Can you train a model on ChatGPT, Claude or Gemini outputs? Not one that competes with the provider. OpenAI, Anthropic and Google all forbid using their outputs to develop competing models, in their business terms and in their consumer terms. Each makes narrow exceptions, mostly for the provider's own fine-tuning and distillation features, and open-weight models follow their own licences. One consequence matters to every company with an API key: if someone else uses your key to distil a model, the activity is on your account.
Why this question is everywhere in 2026
Distillation, training a smaller model on a larger model's outputs, is an ordinary technique when you own both models. It became a security story when providers described doing it at scale against them. Anthropic said in February 2026 that three labs ran over 16 million exchanges through about 24,000 fraudulent accounts, and that the activity broke its terms of service and regional restrictions1. In October, OpenCode described a breach of its Console as related to a large-scale distillation attack in which provider keys saved by its users may have been exposed; some users reported unauthorized charges and Google project suspensions2. What happened in the OpenCode Console breach keeps to what has been confirmed.
So two questions now arrive together: what may we do with model outputs, and what happens to us if someone else uses our key to do what we may not.
What each provider's terms say
| Provider | Document | The restriction | The carve-out |
|---|---|---|---|
| OpenAI | Services Agreement (API and business) | No using Output to develop AI models that compete with OpenAI's products and services3 | Internal classifiers, embeddings and similar models not distributed; fine-tuning OpenAI's own models3 |
| OpenAI | Terms of Use (consumer) | No using Output to develop competing models; no automated extraction of data or Output4 | None stated |
| Anthropic | Commercial Terms | No accessing the services to build a competing product, including to train competing AI models, and no supporting a third party's attempt5 | As expressly approved by Anthropic5 |
| Anthropic | Usage Policy | No training a model on outputs, naming model scraping and distillation6 | With prior authorization from Anthropic6 |
| Gemini API Additional Terms | No using the services to develop models that compete with them; no attempts to reverse engineer, extract or replicate components7 | None stated in the additional terms | |
| Cloud Service Specific Terms (AI/ML services) | No using output to create or improve models similar to a Google model, or to substitute for one8 | Google's own fine-tuning and distillation features, to make modified Google models8 |
OpenAI
The Services Agreement, which governs the API, lists among the things a customer may not do, or let end users do, using Output "to develop artificial intelligence models that compete with OpenAI's products and services," except for a Permitted Exception3. The exception covers two cases: models "primarily intended to categorize, classify, or organize data," such as embeddings or classifiers, provided they are not distributed or offered to third parties; and fine-tuning or customizing models through OpenAI's own fine-tuning services3. The same list forbids extracting data other than as the services permit, and buying, selling or transferring API keys to or from a third party3.
The consumer Terms of Use say the same thing more briefly: no using Output to develop models that compete with OpenAI, and no automatic or programmatic extraction of data or Output4. They also say that, as between you and OpenAI, you own the Output4. Ownership and permitted use are separate: you own the text, and you agreed not to train a competing model with it.
Anthropic
Anthropic's Commercial Terms, which cover API keys, bar accessing the services to build a competing product or service, "including to train competing AI models," except as Anthropic expressly approves, and bar reverse engineering or duplicating the services5. They add a clause that matters for the stolen-key case: the customer may not support any third party's attempt at the same conduct5.
Anthropic's Usage Policy, in the version its site lists as effective 12 November 2026, puts it under not abusing the platform: no using outputs to train an AI model, naming "model scraping" and "model distillation," without prior authorization; no coordinating activity across many accounts to avoid detection; and no providing access to Claude through unauthorized means6.
The Gemini API Additional Terms state that you may not use the services "to develop models that compete with the Services," giving the Gemini API and Google AI Studio as examples, and may not try to reverse engineer, extract or replicate any component7.
The Google Cloud Service Specific Terms, which cover Google's AI/ML services in Cloud, go further. Customers may not use an AI/ML service or its output to develop a similar or competing product, and Google may immediately suspend or terminate use based on any suspected violation8. A separate model restriction bars using output to substitute for a Google model or to create or improve similar models; where a service offers fine-tuning or model distillation, its output may be used to make modified Google models8. The terms also say the competitive-use restriction does not apply to Google's agent platform (formerly Vertex AI Platform) as long as the customer does not use a Google pre-trained model8. Read the current section yourself before relying on that nuance.
What is allowed
The terms are narrower than "never train on outputs." Read together, these uses are expressly permitted or outside the restriction:
- The provider's own distillation and fine-tuning. OpenAI's exception covers fine-tuning its models through its services3; Google's covers its own fine-tuning and distillation features8. You end up with a customized model that runs on the provider, not a portable one.
- Internal classifiers and embeddings with OpenAI, if they are not distributed or offered to third parties3.
- Anything Anthropic has authorized in advance6. Ask in writing.
- Distilling your own models from your own models. If you own or license the teacher, the provider terms above do not apply.
- Open-weight teachers under their licence. OpenAI publishes gpt-oss-120b under Apache 2.09. The Llama 3.1 licence lets you use outputs to train or improve another model, but a distributed model built that way must have a name beginning with "Llama" and display "Built with Llama"; licensees above 700 million monthly active users need a separate licence from Meta10. Licences differ by model and version: check the exact one.
Your exposure when someone else distils through your key
This is the part most companies have not thought about. The terms put responsibility on the account holder, and they do not wait for intent.
- You are responsible for activity on your account. OpenAI's Services Agreement makes the customer responsible for all activities under its account3. Anthropic's Commercial Terms say the same, and require the customer to notify Anthropic promptly if it believes the account has been compromised5.
- Suspension can come first. OpenAI may limit or suspend access for a breach of the agreement or a security emergency, making reasonable efforts to notify first but not always3. Anthropic may suspend if it reasonably believes the services are being used in breach of its use restrictions5. Google may suspend immediately on suspected competitive use8. In the OpenCode case, users reported Google suspending projects for "abusive activity consistent with hijacked resources"2.
- The charges are on your bill. A distillation campaign wants volume and the best model, which is the most expensive traffic you can have. Some OpenCode users reported more than $1,700 in unauthorized charges overnight2.
- Your account can look like an attacker's. Anthropic describes detecting distillation through classifiers and behavioral fingerprinting of API traffic1. A stolen key running a harvesting workload produces exactly that pattern, under your name.
None of this means a provider will treat a victim as a perpetrator. It means the first contact may be a suspension notice, and the burden of showing what happened is on you. What to do when a provider suspends your account covers the response, and How stolen API keys feed distillation campaigns the mechanics.
Reducing the exposure
- Give every workload its own provider key, project or workspace, so a stolen key's traffic is separable and one suspension does not take everything down.
- Restrict keys at the provider: models, IP ranges where available, and a spend limit.
- Watch per-key usage for models you never call and output volumes you never produce. Signs your AI API key is being abused lists the signals.
- Write down who may train on model outputs, from which providers, under which terms, and review it before any training run.
- Know where to report a compromise at each provider before you need to; Anthropic's terms expect prompt notice5.
- Keep the provider's own usage records for the window; they are your evidence in an appeal.
How INFRO handles keys and model access
INFRO is a gateway that runs on provider accounts you own; it does not train models and does not use your traffic for training. For traffic sent through it, model allowlists refuse a model you have not approved with 403 model_not_allowed (model allowlists), gateway keys can be limited to chosen providers and IP ranges, budgets refuse a request with 402 before the provider is called, and every change to keys and policy is in the audit log. Provider credentials are encrypted with AES-256-GCM under a key held outside the database and are never shown again after saving (security).
Those controls do not reach a provider key used directly. A provider key stolen from anywhere, including from a platform that stores provider keys as INFRO does, can be used straight against the provider and bypasses INFRO's allowlists and budgets. For that case your backstops are at the provider: scoped keys, provider spend limits, revocation at the provider, and the provider's own usage reports.
Summary
You may own the output and still not train a competing model on it: OpenAI, Anthropic and Google all say so, each with narrow carve-outs for their own fine-tuning and distillation features or prior authorization. Open-weight licences set different rules. If someone else distils through your key, the terms put the account, the bill and the first conversation on you, so scope keys, watch per-key usage and know how to report a compromise. Take legal advice before relying on any of this.