Terms

Model Distillation

Model distillation is a technique in which a student model learns from the outputs of a more extensively trained model. It may be used legitimately to reduce training costs or maliciously to extract a model’s capabilities through APIs.

Model distillation is a method of training a new model using the outputs of another, more extensively trained model, often by sending requests to the target model and analyzing its responses. This can help reduce training costs and accelerate the development of new models, but it raises concerns when done without authorization or by circumventing access restrictions.

How does model distillation work?

The student model learns from the teacher model’s answers and behavior on tasks such as programming, data analysis, and reasoning. Rather than training a competing model from scratch, a developer can collect a large number of outputs and use them to prepare training data.

Legitimate distillation and adversarial extraction

  • Legitimate distillation: Conducted under clear access terms and with the consent of the entity that owns the target model.
  • Adversarial extraction: Uses APIs, accounts, and multimodal methods to access model outputs outside the environment controlled by the owning company.

The sources explain that the dispute is not about allowing any form of model copying, but about access terms, scope of use, proof of authorization, and compliance with contractual and security restrictions.

How can it be used maliciously?

Press sources and security warnings have reported that some alleged campaigns distributed requests across multiple or shared accounts, cloud services, aggregators, and agents in an effort to bypass geographic restrictions, usage limits, and monitoring systems. Attempts have also been reported to extract inferences or information that is not normally shown to users.

Indicators of API misuse

  • New accounts reaching usage limits quickly after being created.
  • Continuous activity lacking the periods of inactivity typical of human users.
  • The same accounts being used from many internet addresses or browser identifiers.
  • Repeated prompts across multiple providers.
  • Coordinated switching between different access paths.

Why does model distillation matter?

Distillation may reduce the time and expense required to develop a new model because it enables the use of outputs from a ready-made model instead of bearing the cost of building a competing model from scratch. At the same time, it may threaten the value of closed models and their ability to control the use of their interfaces and outputs.

The sources also connect this discussion to the balance between closed commercial models and open-weight models. The idea of permitting authorized distillation for laboratories developing open-weight models has been proposed, while impersonation, fraud, and credential theft have been rejected.

What remains unresolved?

The sources include allegations and reports from companies and agencies, and in all cases do not provide independent verification or a judicial ruling. The practical boundaries between legitimate distillation and adversarial extraction, as well as questions concerning intellectual property rights and contractual terms, also remain unresolved in the available evidence.