# Can you train on AI outputs? What provider terms say

URL: https://infro.io/blog/distillation-terms-of-service-model-outputs
Author: INFRO Editorial (https://infro.io/blog/editorial)
Publisher: INFRO
Published: 2026-10-21
Last modified: 2026-10-21
Last reviewed: 2026-10-21 (next review due 2026-11-20)
Topic: Security and governance
Format: Analysis

> OpenAI, Anthropic and Google all forbid using their models' outputs to develop competing models: OpenAI's business and consumer terms, Anthropic's Commercial Terms and Usage Policy, and Google's Gemini API and Cloud service terms. Each has carve-outs: OpenAI permits internal classifiers and embeddings and fine-tuning its own models, Google permits its own fine-tuning and distillation features, and Anthropic allows it only with prior authorization. Open-weight licences such as Apache 2.0 or Llama 3.1 set their own rules. If someone else uses your key for distillation, the activity is still on your account, which can lead to suspension and charges. This is not legal advice.

Key takeaways:
- All three major providers forbid using outputs to develop competing models, in both business and consumer terms.
- Owning the output does not cancel the restriction: OpenAI assigns you the output and still restricts what you train with it.
- Each provider has carve-outs: OpenAI for internal classifiers and embeddings and its own fine-tuning, Google for its own fine-tuning and distillation features, Anthropic only with prior authorization.
- Open-weight models carry their own licences: gpt-oss is Apache 2.0, while Llama 3.1 requires naming and attribution for models trained on its outputs.
- If someone else distils through your key, the terms make you responsible for activity on your account, and providers may suspend first and talk later.

**Can you train a model on ChatGPT, Claude or Gemini outputs?** Not one that competes with the provider. OpenAI, Anthropic and Google all forbid using their outputs to develop competing models, in their business terms and in their consumer terms. Each makes narrow exceptions, mostly for the provider's own fine-tuning and distillation features, and open-weight models follow their own licences. One consequence matters to every company with an API key: if someone else uses your key to distil a model, the activity is on your account.

> **Warning**
> This is an analysis of published terms, not legal advice. Terms change, enterprise agreements can differ from the published versions, and "compete" is a word lawyers argue about. Ask your counsel before you rely on any reading here.

## Why this question is everywhere in 2026

Distillation, training a smaller model on a larger model's outputs, is an ordinary technique when you own both models. It became a security story when providers described doing it at scale against them. Anthropic said in February 2026 that three labs ran over 16 million exchanges through about 24,000 fraudulent accounts, and that the activity broke its terms of service and regional restrictions[1]. In October, OpenCode described a breach of its Console as related to a large-scale distillation attack in which provider keys saved by its users may have been exposed; some users reported unauthorized charges and Google project suspensions[2]. [What happened in the OpenCode Console breach](https://infro.io/blog/opencode-console-breach-what-happened) keeps to what has been confirmed.

So two questions now arrive together: what may *we* do with model outputs, and what happens to *us* if someone else uses our key to do what we may not.

## What each provider's terms say

| Provider | Document | The restriction | The carve-out |
| --- | --- | --- | --- |
| OpenAI | Services Agreement (API and business) | No using Output to develop AI models that compete with OpenAI's products and services[3] | Internal classifiers, embeddings and similar models not distributed; fine-tuning OpenAI's own models[3] |
| OpenAI | Terms of Use (consumer) | No using Output to develop competing models; no automated extraction of data or Output[4] | None stated |
| Anthropic | Commercial Terms | No accessing the services to build a competing product, including to train competing AI models, and no supporting a third party's attempt[5] | As expressly approved by Anthropic[5] |
| Anthropic | Usage Policy | No training a model on outputs, naming model scraping and distillation[6] | With prior authorization from Anthropic[6] |
| Google | Gemini API Additional Terms | No using the services to develop models that compete with them; no attempts to reverse engineer, extract or replicate components[7] | None stated in the additional terms |
| Google | Cloud Service Specific Terms (AI/ML services) | No using output to create or improve models similar to a Google model, or to substitute for one[8] | Google's own fine-tuning and distillation features, to make modified Google models[8] |

### OpenAI

The Services Agreement, which governs the API, lists among the things a customer may not do, or let end users do, using Output "to develop artificial intelligence models that compete with OpenAI's products and services," except for a Permitted Exception[3]. The exception covers two cases: models "primarily intended to categorize, classify, or organize data," such as embeddings or classifiers, provided they are not distributed or offered to third parties; and fine-tuning or customizing models through OpenAI's own fine-tuning services[3]. The same list forbids extracting data other than as the services permit, and buying, selling or transferring API keys to or from a third party[3].

The consumer Terms of Use say the same thing more briefly: no using Output to develop models that compete with OpenAI, and no automatic or programmatic extraction of data or Output[4]. They also say that, as between you and OpenAI, you own the Output[4]. Ownership and permitted use are separate: you own the text, and you agreed not to train a competing model with it.

### Anthropic

Anthropic's Commercial Terms, which cover API keys, bar accessing the services to build a competing product or service, "including to train competing AI models," except as Anthropic expressly approves, and bar reverse engineering or duplicating the services[5]. They add a clause that matters for the stolen-key case: the customer may not support any third party's attempt at the same conduct[5].

Anthropic's Usage Policy, in the version its site lists as effective 12 November 2026, puts it under not abusing the platform: no using outputs to train an AI model, naming "model scraping" and "model distillation," without prior authorization; no coordinating activity across many accounts to avoid detection; and no providing access to Claude through unauthorized means[6].

### Google

The Gemini API Additional Terms state that you may not use the services "to develop models that compete with the Services," giving the Gemini API and Google AI Studio as examples, and may not try to reverse engineer, extract or replicate any component[7].

The Google Cloud Service Specific Terms, which cover Google's AI/ML services in Cloud, go further. Customers may not use an AI/ML service or its output to develop a similar or competing product, and Google may immediately suspend or terminate use based on any *suspected* violation[8]. A separate model restriction bars using output to substitute for a Google model or to create or improve similar models; where a service offers fine-tuning or model distillation, its output may be used to make modified Google models[8]. The terms also say the competitive-use restriction does not apply to Google's agent platform (formerly Vertex AI Platform) as long as the customer does not use a Google pre-trained model[8]. Read the current section yourself before relying on that nuance.

## What is allowed

The terms are narrower than "never train on outputs." Read together, these uses are expressly permitted or outside the restriction:

- **The provider's own distillation and fine-tuning.** OpenAI's exception covers fine-tuning its models through its services[3]; Google's covers its own fine-tuning and distillation features[8]. You end up with a customized model that runs on the provider, not a portable one.
- **Internal classifiers and embeddings with OpenAI**, if they are not distributed or offered to third parties[3].
- **Anything Anthropic has authorized in advance**[6]. Ask in writing.
- **Distilling your own models from your own models.** If you own or license the teacher, the provider terms above do not apply.
- **Open-weight teachers under their licence.** OpenAI publishes gpt-oss-120b under Apache 2.0[9]. The Llama 3.1 licence lets you use outputs to train or improve another model, but a distributed model built that way must have a name beginning with "Llama" and display "Built with Llama"; licensees above 700 million monthly active users need a separate licence from Meta[10]. Licences differ by model and version: check the exact one.

> **Note**
> A practical test before a training run: write down which model produced each part of the dataset, through which account, under which terms or licence. If any row is a closed provider's API and the result could be called a competing model, stop and ask counsel.

## Your exposure when someone else distils through your key

This is the part most companies have not thought about. The terms put responsibility on the account holder, and they do not wait for intent.

- **You are responsible for activity on your account.** OpenAI's Services Agreement makes the customer responsible for all activities under its account[3]. Anthropic's Commercial Terms say the same, and require the customer to notify Anthropic promptly if it believes the account has been compromised[5].
- **Suspension can come first.** OpenAI may limit or suspend access for a breach of the agreement or a security emergency, making reasonable efforts to notify first but not always[3]. Anthropic may suspend if it reasonably believes the services are being used in breach of its use restrictions[5]. Google may suspend immediately on suspected competitive use[8]. In the OpenCode case, users reported Google suspending projects for "abusive activity consistent with hijacked resources"[2].
- **The charges are on your bill.** A distillation campaign wants volume and the best model, which is the most expensive traffic you can have. Some OpenCode users reported more than $1,700 in unauthorized charges overnight[2].
- **Your account can look like an attacker's.** Anthropic describes detecting distillation through classifiers and behavioral fingerprinting of API traffic[1]. A stolen key running a harvesting workload produces exactly that pattern, under your name.

None of this means a provider will treat a victim as a perpetrator. It means the first contact may be a suspension notice, and the burden of showing what happened is on you. [What to do when a provider suspends your account](https://infro.io/blog/ai-provider-account-suspended-abuse) covers the response, and [How stolen API keys feed distillation campaigns](https://infro.io/blog/stolen-api-keys-distillation-campaigns) the mechanics.

## Reducing the exposure

- [ ] Give every workload its own provider key, project or workspace, so a stolen key's traffic is separable and one suspension does not take everything down.
- [ ] Restrict keys at the provider: models, IP ranges where available, and a spend limit.
- [ ] Watch per-key usage for models you never call and output volumes you never produce. [Signs your AI API key is being abused](https://infro.io/blog/signs-ai-api-key-abused) lists the signals.
- [ ] Write down who may train on model outputs, from which providers, under which terms, and review it before any training run.
- [ ] Know where to report a compromise at each provider before you need to; Anthropic's terms expect prompt notice[5].
- [ ] Keep the provider's own usage records for the window; they are your evidence in an appeal.

## How INFRO handles keys and model access

INFRO is a gateway that runs on provider accounts you own; it does not train models and does not use your traffic for training. For traffic sent through it, model allowlists refuse a model you have not approved with `403 model_not_allowed` ([model allowlists](https://infro.io/docs/org/model-allowlists)), gateway keys can be limited to chosen providers and IP ranges, budgets refuse a request with `402` before the provider is called, and every change to keys and policy is in the [audit log](https://infro.io/docs/org/audit-log). Provider credentials are encrypted with AES-256-GCM under a key held outside the database and are never shown again after saving ([security](https://infro.io/docs/org/security)).

Those controls do not reach a provider key used directly. A provider key stolen from anywhere, including from a platform that stores provider keys as INFRO does, can be used straight against the provider and bypasses INFRO's allowlists and budgets. For that case your backstops are at the provider: scoped keys, provider spend limits, revocation at the provider, and the provider's own usage reports.

## Summary

You may own the output and still not train a competing model on it: OpenAI, Anthropic and Google all say so, each with narrow carve-outs for their own fine-tuning and distillation features or prior authorization. Open-weight licences set different rules. If someone else distils through your key, the terms put the account, the bill and the first conversation on you, so scope keys, watch per-key usage and know how to report a compromise. Take legal advice before relying on any of this.

## Questions

### Can I train my own model on ChatGPT or OpenAI API outputs?

Not a model that competes with OpenAI's products and services: both OpenAI's Terms of Use and its Services Agreement forbid that. The Services Agreement makes two exceptions: internal models that classify or organize data, such as embeddings or classifiers, if not distributed, and fine-tuning OpenAI's own models through its services. What counts as competing is a question for your counsel.

### Do I own the output if I cannot train on it?

OpenAI assigns you its rights in the output, and still restricts using it to develop competing models. Ownership and permitted use are separate questions: the contract binds you as a customer regardless of who owns the text.

### Is distillation allowed with open-weight models?

It depends on the licence. OpenAI's gpt-oss models are published under Apache 2.0. The Llama 3.1 licence allows using outputs to train another model, but a distributed model built that way must have a name beginning with Llama and show Built with Llama. Read the licence of the exact model and version you use.

### Am I liable if someone steals my API key and uses it for distillation?

Provider terms make you responsible for activity under your account, and providers can suspend an account they suspect of a violation. In practice the immediate risks are suspension and the charges. Revoke the key, report the compromise to the provider promptly, and take legal advice on anything beyond that.

## Sources

1. Detecting and preventing distillation attacks — Anthropic. https://www.anthropic.com/news/detecting-and-preventing-distillation-attacks (accessed 2026-10-21)
2. OpenCode hack exposes API keys amid AI distillation attacks — BleepingComputer. https://www.bleepingcomputer.com/news/security/opencode-hack-exposes-api-keys-amid-ai-distillation-attacks/ (accessed 2026-10-21)
3. OpenAI Services Agreement — OpenAI. https://openai.com/policies/services-agreement/ (accessed 2026-10-21)
4. Terms of Use — OpenAI. https://openai.com/policies/row-terms-of-use/ (accessed 2026-10-21)
5. Commercial Terms of Service — Anthropic. https://www.anthropic.com/legal/commercial-terms (accessed 2026-10-21)
6. Usage Policy — Anthropic. https://www.anthropic.com/legal/aup (accessed 2026-10-21)
7. Gemini API Additional Terms of Service — Google. https://ai.google.dev/gemini-api/terms (accessed 2026-10-21)
8. Google Cloud Platform Service Specific Terms — Google Cloud. https://cloud.google.com/terms/service-terms (accessed 2026-10-21)
9. openai/gpt-oss-120b — Hugging Face. https://huggingface.co/openai/gpt-oss-120b (accessed 2026-10-21)
10. Llama 3.1 Community License Agreement — Meta (via Hugging Face). https://huggingface.co/meta-llama/Llama-3.1-8B/blob/main/LICENSE (accessed 2026-10-21)