OpenAI said on Wednesday that it has identified and dismantled a large-scale coordinated campaign aimed at extracting what it calls "protected reasoning" from its artificial intelligence models, a practice it describes as adversarial distillation.
In a statement reported on October 1, the company said the activity was designed to extract the models' internal reasoning and use it to help train, reproduce or improve another model.
"We recently identified and disrupted a coordinated campaign designed to extract protected reasoning from our models," the company said.
How the campaign unfolded
According to OpenAI, the activity began on July 1 at a low volume before increasing sharply. On July 24 and 25, the company observed 16,000 requests using a relevant extraction pattern from more than 4,000 users. Further investigation identified related prompt-pattern activity across a cluster of more than 15,000 users, which OpenAI said it fully disrupted by July 28.
OpenAI said the operators did not break its encryption, compromise a database or gain direct access to stored user conversations. Instead, they manipulated interactions with the models so that protected reasoning — the internal record of how a model works through a task — could be reproduced in forms visible to the requester.
In one method, operators copied encrypted reasoning from one conversation and asked a model in another conversation to decrypt and transcribe the hidden reasoning.
Attribution and risks
OpenAI said it was unclear whether all the operators originated from a single actor, but a core cluster of the activity was attributed to individuals associated with Moonshot AI, the developer of the Kimi models.
The company said adversarial distillation creates safety and national security risks, because extracted reasoning could be used to train another model without preserving the safeguards applied to the original model's user-facing outputs. It added that as models gain capabilities in dual-use areas, the risks become more significant.
Response
To disrupt the campaign, OpenAI said it used a combination of account enforcement, technical controls and coordination with partners. It banned or restricted fraudulent accounts, strengthened signup and infrastructure controls, and expanded monitoring for related networks.
The company also strengthened protections for hidden reasoning across users, workspaces, organisations and model families, and closed a pathway that could allow someone possessing another user's encrypted reasoning to replay it and recover its contents.
OpenAI expects adversarial-distillation attempts to grow more sophisticated as frontier models improve and actors seek cheaper ways to mimic their capabilities. It said work on technical protections, detection, enforcement and threat-information sharing would continue.
Sources: OpenAI says it disrupted large-scale effort to copy its AI model capabilities
In a statement reported on October 1, the company said the activity was designed to extract the models' internal reasoning and use it to help train, reproduce or improve another model.
"We recently identified and disrupted a coordinated campaign designed to extract protected reasoning from our models," the company said.
How the campaign unfolded
According to OpenAI, the activity began on July 1 at a low volume before increasing sharply. On July 24 and 25, the company observed 16,000 requests using a relevant extraction pattern from more than 4,000 users. Further investigation identified related prompt-pattern activity across a cluster of more than 15,000 users, which OpenAI said it fully disrupted by July 28.
OpenAI said the operators did not break its encryption, compromise a database or gain direct access to stored user conversations. Instead, they manipulated interactions with the models so that protected reasoning — the internal record of how a model works through a task — could be reproduced in forms visible to the requester.
In one method, operators copied encrypted reasoning from one conversation and asked a model in another conversation to decrypt and transcribe the hidden reasoning.
Attribution and risks
OpenAI said it was unclear whether all the operators originated from a single actor, but a core cluster of the activity was attributed to individuals associated with Moonshot AI, the developer of the Kimi models.
The company said adversarial distillation creates safety and national security risks, because extracted reasoning could be used to train another model without preserving the safeguards applied to the original model's user-facing outputs. It added that as models gain capabilities in dual-use areas, the risks become more significant.
Response
To disrupt the campaign, OpenAI said it used a combination of account enforcement, technical controls and coordination with partners. It banned or restricted fraudulent accounts, strengthened signup and infrastructure controls, and expanded monitoring for related networks.
The company also strengthened protections for hidden reasoning across users, workspaces, organisations and model families, and closed a pathway that could allow someone possessing another user's encrypted reasoning to replay it and recover its contents.
OpenAI expects adversarial-distillation attempts to grow more sophisticated as frontier models improve and actors seek cheaper ways to mimic their capabilities. It said work on technical protections, detection, enforcement and threat-information sharing would continue.
Sources: OpenAI says it disrupted large-scale effort to copy its AI model capabilities
