Two stories from the AI sector collided this week in ways that should concentrate minds across the political spectrum. First, Anthropic, the AI safety company, announced it had blocked what appeared to be a genuine attempt to use one of its AI models to design biological weapons. The company said its safety systems had detected and blocked the attempt, and it declined to provide details about the would-be attacker. Simultaneously, US legislators pushed forward a package of AI safety bills responding to public warnings from AI researchers — including an Anthropic researcher’s estimate of a greater than 10% probability that AI could pose a human extinction risk. Vermont Senator Bernie Sanders has proposed legislation to ban the development of AI superintelligence altogether. The twin developments — a private company stopping what might have been mass-casualty weapon development, and legislators proposing to constrain a technology they do not fully understand — represent the governance inflection point that AI researchers have been warning about for years.
The received wisdom
The mainstream progressive response to AI risk combines genuine alarm with enthusiasm for regulatory intervention. If AI systems can block bioweapon design attempts, they can also be manipulated to enable them; the same capability that makes a model useful for drug discovery makes it potentially catastrophic in hostile hands. The human extinction warnings from researchers inside frontier AI companies are not fringe positions — they represent a significant professional consensus that something qualitatively new is happening. On this reading, the market alone cannot be trusted to handle existential risks. Only democratic governments with binding legal frameworks, international coordination, and real enforcement power can impose the necessary guardrails. The EU’s AI Act is a model; the US needs something comparable or stronger. The Sanders superintelligence ban is a blunt instrument, but the impulse behind it is reasonable: some technological capabilities may be too dangerous to develop at all.
A different read
The governance instinct is understandable, but it risks being precisely backwards about the mechanism of the risk. Consider what actually happened in the Anthropic case. A private company’s safety systems detected and blocked a bioweapon attempt. The system worked. That is not an argument for complacency — it is an argument for taking seriously what makes safety systems work and what would compromise them. The risk of poorly-designed regulation is not that it fails to constrain dangerous AI development, but that it concentrates that development in jurisdictions — China, Russia, or non-signatory states — where no safety systems operate at all. The history of arms control suggests that unilateral capability restrictions by democracies do not eliminate the capability; they redistribute it to less scrupulous actors.
This is not an argument against all AI regulation. It is an argument against the specific form of regulatory thinking that treats AI governance as analogous to, say, pharmaceutical approval — where a single national regulator can meaningfully control access. AI capabilities exist at the intersection of open-source model weights, freely circulating academic research, and commodity hardware. The knowledge required to build a dangerous AI system is not like the knowledge required to build a nuclear weapon; it does not depend on a small number of controlled inputs that a state can physically interdict. Regulatory frameworks that focus on big Western frontier labs — Anthropic, OpenAI, Google DeepMind — while leaving the global open-source ecosystem unaddressed are addressing roughly 30% of the risk surface.
The bioweapon case also deserves scepticism about what “blocking” actually means. Anthropic’s safety systems intercepted the attempt, which is genuinely good. But if the same technical knowledge could be accessed through a less safety-conscious open-source model, the blocking achieved delay rather than prevention. The arms-control analogy returns: the Non-Proliferation Treaty slowed nuclear diffusion but did not stop it, and its failure modes are instructive. Every new nuclear-armed state acquired capability through a combination of indigenous development and knowledge transfer that the treaty’s signatories were unable to prevent. The biosecurity challenge is harder still because the relevant knowledge base is far more distributed.
The Sanders superintelligence ban proposal is politically coherent but operationally meaningless. His BBC interview showed a politician responding to a genuine public anxiety with a legislative symbol — a bill that declares the intent to ban something that cannot currently be precisely defined, cannot be physically interdicted at a border, and whose development is proceeding simultaneously in seventeen countries. The serious regulatory work is less dramatic: mandatory incident reporting (which would have generated public information from the Anthropic case), liability frameworks that give companies financial incentives to invest in safety rather than route around it, and international agreements — however incomplete — on the most catastrophic near-term uses.
What the week’s events actually demonstrated is that the most consequential AI safety work is happening inside private companies rather than inside regulatory agencies. That is not inherently illegitimate — Anthropic was founded specifically for safety reasons, and its commitment appears genuine — but it means that the governance of potentially civilisation-threatening technology currently rests on the culture and values of a few dozen engineers in San Francisco. That is a foundation worth worrying about regardless of one’s political priors.
What to watch
- Congressional progress on the AI safety bill package: whether substantive liability and incident-reporting provisions survive the lobbying process, or whether the final legislation is largely procedural.
- Any further details Anthropic releases about the blocked bioweapon attempt — the degree of specificity the attempt demonstrated will indicate whether we are in the “curious amateur” phase or the “sophisticated actor” phase of AI-enabled biosecurity threats.
- Whether the EU’s AI Act framework produces any enforceable cross-border mechanism, or whether it functions primarily as a compliance burden on European companies rather than a genuine safety guardrail.
- China’s domestic AI regulation posture: Beijing has announced AI content rules but not safety frameworks comparable to what Western labs operate internally. The gap between Chinese frontier lab capability and safety investment is the least-discussed variable in the global AI governance debate.
— J