Anthropic calls on governments to update their policies to keep pace with the acceleration of AI capabilities through two proposals covering advanced-model governance and preparation for the technology’s impact on workers and the economy. The company believes that the speed of model development makes transparency alone insufficient, and that government authorities need greater powers to address deployments that may entail catastrophic risks, while establishing safeguards to prevent the misuse of those powers.
The first proposal consists of the Advanced AI Framework, which focuses on the most powerful models and the risks that may arise from them. The second proposal, the Economic Framework, addresses how to prepare workers and the economy for AI’s impact and ensure that its financial benefits are broadly shared. Anthropic says the two frameworks address two aspects of a single challenge: responsibly directing the technology as it advances and preparing for the changes it will bring to society.
Scope of the Proposed Rules
The advanced framework proposes applying the rules to models trained using more than 1025 floating-point operations and developed by companies that generate more than $500 million in AI-related revenue or spend more than $1 billion on research and development in this field. Anthropic says this scope targets frontier models rather than imposing the same requirements on all AI systems.
The company links the need for this type of regulation to the sharp upward trend in model capabilities. It notes that models a few years ago were barely capable of writing code, while the Claude Mythos Preview model, according to Anthropic, discovered thousands of high-severity software vulnerabilities, including vulnerabilities in every major operating system and browser. The company concludes that if this trend continues, the risks of catastrophic harm may increase.
Four Types of Risk
The framework focuses on four primary areas of risk:
- Biological risks: Unprotected systems could make the development of biological weapons easier and less costly, although the same capabilities could also accelerate drug discovery.
- Cyber risks: Advanced AI models can discover critical vulnerabilities at scale, which could help defend systems but also raise the level of threat to critical infrastructure, such as hospitals and energy networks.
- Loss-of-control risks: Controlling systems that act beyond the scope of their developers may become more difficult as their capabilities improve.
- Automated research and development: Systems could automate AI research and development, potentially multiplying the three previous risks.
Requirements for Advanced-Model Developers
Anthropic proposes that advanced-model developers test their systems and publish a summary of the results, along with a safety framework explaining how catastrophic risks are assessed and system cards describing the models’ capabilities and risks. The framework also calls for regular reports on the overall state of risks and for independent evaluations to be conducted regularly. The company notes that some transparency requirements are already mandated under laws in California and New York, but believes its proposal goes beyond them in some respects.
Under the proposal, every advanced-model developer should engage at least one qualified independent evaluator, who would publish a review of the company’s evaluations and risk reports. Anthropic also calls on governments and the technology sector to develop an ecosystem of independent evaluators by setting standards for them, providing funding, and ensuring sufficient access to models.
The framework also includes security requirements to protect model weights and the infrastructure used for training, as these are valuable targets for attackers, including well-resourced government entities. The company proposes securing the entire development environment against external and internal threats, describing the security program to the public at a high level, providing details to a designated government agency upon request, establishing channels for reporting model-extraction attacks, and testing defenses regularly.
Government Powers with Safeguards
Anthropic believes governments should have the legal ability to prevent or deter the deployment of models that pose a significant risk of causing catastrophic harm, with civil penalties tied to annual global revenue and increasing penalties for repeated violations. At the same time, it warns against broad or burdensome regulatory powers and proposes a specific mechanism for preventing dangerous deployments and concrete safeguards to limit the misuse of that authority.
The framework primarily focuses on the U.S. federal government, but it does not call for eliminating state laws unless a federal law at least as strong as the proposal is enacted. Anthropic says the preemption of state authority should be limited, so states remain able to regulate issues such as child safety and consumer protection, as long as they fall outside the defined safety functions covered by federal law.
Strengthening Society’s Resilience
The second part of the framework proposes measures to increase society’s ability to confront risks. In the biological field, recommendations include screening gene synthesis, monitoring new outbreaks through early-warning systems, and preparing by stockpiling protective equipment and means of limiting airborne transmission.
In the cyber field, Anthropic calls for strengthening the software on which the internet depends, providing technical support to critical-infrastructure operators, and replacing the legacy systems used in that infrastructure. It also proposes creating a dedicated government function to track the cyber capabilities of advanced models, as well as government and technology-sector cooperation to develop safeguards that enable cyber capabilities to be shared broadly.
The company acknowledges that the resilience agenda for loss-of-control risks and automated research and development remains less developed and requires more work across the industry. The directions it identifies include developing capabilities to detect and respond to systems that act beyond the control of their developers, and building infrastructure to contain or shut down those systems.
Anthropic expects its proposals to prompt extensive discussion, but urges policymakers to engage with these issues now as AI capabilities continue to improve over the coming months. It emphasizes that technology governance should keep pace with this development, while the proposals presented remain subject to discussion and evaluation.