What is a distillation attack? AI model extraction explained

What is a distillation attack? It is the unauthorized, large-scale harvesting of an AI model's outputs, and often its step-by-step reasoning, through the model's API, so that another party can train a "student" model that reproduces its abilities. Distillation itself is an ordinary training method. It becomes an attack when the model's owner has not permitted it, and in 2026 the campaigns that providers described ran on fraudulent accounts, proxy services and stolen API keys.

This guide explains the difference between distillation and a distillation attack, how a campaign works, why model providers treat it as theft and a safety problem, and why it reaches companies that have never trained a model: their keys are part of the supply.

Distillation, and when it becomes an attack

Distillation trains a smaller model on the outputs of a stronger one. Anthropic calls it a widely used and legitimate training method that frontier labs use to produce smaller versions of their own models¹. The White House memorandum NSTM-4 makes the same distinction: distillation used to produce lighter models is part of a healthy ecosystem, while industrial campaigns to extract proprietary capabilities are not².

Google's Threat Intelligence Group uses "model extraction attack" and "distillation attack" as names for the same thing: an adversary uses legitimate access to probe a mature model systematically and collect what it needs to train a new one, which Google calls a form of intellectual property theft³. OpenAI's term is adversarial distillation, the systematic and unauthorized use of one model's outputs or reasoning to train, reproduce or improve another⁴.

How a distillation attack works

The public accounts from Anthropic, Google, OpenAI and the US agencies describe the same pipeline in different detail.

1. Get access at scale, and hide who you are. Anthropic's February post describes proxy services running "hydra cluster" networks of fraudulent accounts; one network managed more than 20,000 accounts at once and mixed distillation traffic with unrelated customer requests¹. The joint advisory describes routing across native APIs, cloud providers, third-party aggregators, relays and account pools, plus a gray market of proxies called "transfer stations" that resell access⁵.
2. Write prompt sets aimed at specific capabilities. Individual prompts look benign. The campaigns Anthropic described targeted agentic reasoning, tool use, coding and computer vision, and the giveaway was the pattern: volume, repetitive structure and concentration on high-value skills¹.
3. Harvest the reasoning, not only the answer. Several campaigns tried to make the model reveal the chain of thought behind an answer. Google describes one campaign of more than 100,000 prompts that tried to coerce Gemini into revealing its full reasoning traces³. OpenAI describes operators copying encrypted reasoning from one conversation and asking the model in another conversation to decrypt and transcribe it⁴.
4. Filter and grade the haul. The advisory describes quality evaluation frameworks that check whether outputs have been degraded by a defender, and automated failover to another pathway when one is blocked⁵.
5. Fine-tune the student. The collected prompts, answers and reasoning become supervised or reinforcement training data for the attacker's own model.
6. Retarget when something new ships. Anthropic reports that one lab redirected nearly half its traffic to a newly released Claude model within 24 hours¹.

Why model providers treat it as an attack

The providers give three reasons, and the US government adds a fourth.

• Capability theft. A frontier model costs a great deal to train. Google states that distillation from Gemini without permission violates its terms of service and that violators may face takedowns and legal action³.
• Safety removal. A distilled copy rarely inherits the safeguards of the original. Anthropic and OpenAI both make this point: extracted capabilities can spread with the protections that prevent misuse stripped out^{1 4}.
• Export controls. Anthropic argues that distillation lets labs close the gap that chip export controls were meant to keep open, and can make it look as if the controls do not work¹.
• National security. The CISA, NSA and FBI advisory of 8 September 2026 names six China-based companies and says they extracted billions of tokens from US frontier models, likely with Chinese government awareness⁵. NSTM-4 commits the administration to sharing information about these campaigns with US AI companies².

Distillation attacks in the OWASP Top 10 for LLMs

If your risk register uses the OWASP Top 10 for LLM applications, distillation sits under LLM10:2025 Unbounded Consumption. Two of its listed vulnerabilities are model extraction via the API, where crafted queries are used to build a shadow model, and functional model replication, where the target model generates synthetic training data for another model⁶. The same entry covers denial of wallet, the cost side of uncontrolled inference, which is the part an ordinary company feels first. The OWASP unbounded consumption guide covers the whole entry.

Distillation is different from stealing weights. Nobody needs to break into the provider; the attacker only needs a lot of permitted-looking requests. That is why access, meaning accounts and keys, is the contested resource.

Why it matters to a company that is not an AI lab

Google's report says model extraction does not usually threaten ordinary users' confidentiality, integrity or availability; the main risk falls on model developers and service providers³. That is true of the model theft itself. It is not true of the access the campaigns consume.

Your keys can be the vehicle. Anthropic's September 2026 threat report says the proxy services behind these campaigns create accounts with false identities, fake or stolen cards and stolen API keys, and often use stolen API credentials belonging to legitimate companies or individuals⁷. The same report lists what stolen AI credentials give an operator: resale value, compute at someone else's expense, and cover, because the activity is attributed to the key's owner⁷. Stolen API keys and distillation campaigns goes into that economy.

Your bill pays for it. Extraction traffic is high-volume and runs around the clock; the advisory lists 24/7 usage with no idle periods among its indicators⁵. On a key with no spend ceiling, that is the shape of a large invoice.

Your account takes the enforcement. Providers ban accounts that show distillation patterns. After the OpenCode Console breach, which OpenCode said was related to a large-scale distillation attack, BleepingComputer reported that some users said Google had suspended their projects for activity consistent with hijacked resources⁸. The cause of that breach has not been published; the OpenCode incident write-up keeps to what is confirmed.

Your own model or AI feature can be the teacher. Google advises organizations that offer AI models as a service to monitor API access for extraction patterns, and gives the example of a specialised model targeted by a competitor³. A product that exposes a fine-tuned model, or a free AI feature anyone can call, is a smaller version of the same target.

What to do about it

You cannot stop a lab from distilling a provider's model; that is the provider's fight. You can make sure your keys are a poor vehicle and that you notice quickly if one is used.

• Give every workload its own key, scoped to one project or workspace and to inference only. See securing LLM API keys.
• Set spend limits at the provider on every project or workspace that holds a key. Provider spend limits compared lists what each provider offers.
• Read the provider's own usage report by key, not only your application's logs. Traffic on your key that your systems did not send is the signal that matters.
• Rotate keys on a schedule and immediately after any tool that stored them reports a breach.
• Buy model access only from the provider or an authorized channel, never from an unknown proxy. AI API proxy risks explains why.
• If you expose a model to the public, rate-limit per user, cap per-account consumption and watch for repetitive, reasoning-seeking prompt patterns.

How INFRO handles distillation risk on your keys

INFRO does not detect or block distillation of a provider's model; the providers' classifiers and account controls do that. What INFRO does is limit what your own keys can be used for and tell you when spending departs from normal.

• Provider keys you connect are encrypted with AES-256-GCM under a key held outside the database, with the organization and connection bound in, and are never shown again after saving.
• Applications call INFRO with gateway keys that can be limited to chosen providers and IP ranges (403 ip_not_allowed before any provider call), belong to one environment, expire, and rotate with an overlap window.
• Budgets at organization, project, member and key level are checked before the provider is called and refuse with 402. Spend alerts flag a day that departs from a scope's usual day.
• On the Business plan, reconciliation compares INFRO's estimates with OpenAI's and Anthropic's own cost reports daily; with usage access on, it reads token counts by provider key and can show traffic on your account that did not go through INFRO. Without usage access it reports "not checked".

The limit is plain. INFRO's controls govern traffic that goes through INFRO. A provider key stolen from anywhere, including from a platform like INFRO that stores customers' keys, can be used directly against the provider and bypasses every INFRO budget and allowlist. The backstops for that case are at the provider: scoped keys, provider spend limits, revocation and the provider's own usage monitoring.

Summary

A distillation attack copies a model by harvesting its outputs and reasoning through the API at scale, without permission. It is the providers' problem first, and they have published how they fight it. It becomes your problem through access: the campaigns need accounts and keys, some of those keys are stolen from ordinary companies, and the owner of a stolen key pays for the traffic. Scope your keys, cap them at the provider, and read usage by key.