Amodei's slow-down call and the AI governance trap

On Saturday, Anthropic CEO Dario Amodei publicly called for AI development to slow down, a remarkable statement from the head of one of the world’s most heavily capitalised AI laboratories. The call comes just days after an Anthropic researcher told the BBC that there is “more than a 10% chance AI could kill all humans” — a statement that, if made by a researcher at a nuclear weapons facility or a biosafety laboratory, would trigger immediate regulatory review. Meanwhile, the UK government this week rejected the idea of a “kill switch” for dangerous AI, while broader expert fears about AI takeover are mounting. The paradox at the heart of this week’s AI news cycle is almost too neat: the man leading one of the firms most responsible for the pace of AI development is calling for the pace to slow, while the government of a major democracy is simultaneously rejecting the most basic proposed safety mechanism. Something is structurally broken.

The received wisdom

The techno-optimist consensus will frame Amodei’s statement as responsible leadership — a sign that the AI industry is maturing, that founders are taking existential risk seriously, and that self-regulation by serious people is preferable to blunt legislative intervention by politicians who do not understand the technology. The UK government’s rejection of a kill switch, on this reading, reflects sensible regulatory pragmatism: a kill switch is a poorly-defined concept that could be misused, that would chill beneficial AI research, and that in any case could not be implemented in a multi-jurisdictional environment where China and other state actors would simply continue development regardless. The answer, progressives and tech-liberals tend to agree, is thoughtful international cooperation, safety standards developed by industry experts, and careful monitoring — not panic-driven legislation. This is not an incoherent position.

A different read

But consider what we are actually being asked to accept. A researcher at a major AI company publicly assigns a greater-than-one-in-ten probability to a scenario in which their company’s product kills every human being on earth. Their CEO responds not by halting the research or placing it under independent oversight, but by calling — from the CEO’s chair of an ongoing, actively-funded research programme — for “slowing down.” The UK government, presented with a proposed mechanism for halting a technology that its own technology sector describes as potentially extinction-level, declines on the grounds that it’s not the right tool.

The closest historical analogy is not nuclear power or genetic engineering, though those are the comparisons most commonly reached for. It is actually closer to the early decades of leaded petrol. For thirty years after the toxicological evidence of lead’s neurological effects on children began to accumulate, the combination of industry lobbying, regulatory capture, and the genuine difficulty of phasing out an economically useful product kept the decision deferred. The structural similarity is this: the people with the most direct knowledge of the harm (or risk) are also the people with the largest financial stake in continued development, and they are operating in a regulatory environment that has not caught up with the pace of their industry.

The nuclear weapons analogy is instructive in a different way. After Hiroshima, the physics community — J. Robert Oppenheimer most famously, but also Niels Bohr, Leo Szilard, and others — launched a sustained, organised campaign for international governance of atomic weapons. They did not simply call for their own institutions to slow down. They called for structural international arrangements that would apply to everyone. The Bulletin of the Atomic Scientists, the Pugwash Conferences, the Treaty on the Non-Proliferation of Nuclear Weapons — all emerged from a recognition that the only way to manage technology with civilisational stakes was through architecture, not through individual corporate virtue.

What is conspicuously absent from the current AI moment is any equivalent push for hard architecture. OpenAI claimed to have cracked a 90-year-old mathematical problem in 88 hours, demonstrating the kind of capability acceleration that, even if the specific result is subsequently contested, illustrates the pace of change. Amodei’s call to slow down is admirable at a personal level and nearly meaningless at a structural level if it is not accompanied by proposals that would actually constrain the competitive dynamics driving the acceleration. In a race to the frontier, a voluntary “slow down” by one participant simply advantages those who don’t slow down — which is why Amodei, to his credit, presumably knows his call alone will change nothing, and is perhaps making a public case for the kind of regulation he privately knows is necessary.

The question of who should hold the kill switch, or its functional equivalent, is genuinely hard. Governments? They are slow and nationally bounded. An international body? The precedent of bodies like the IAEA and the OPCW is that international verification regimes can work when great powers agree — and do not work when they don’t. The current geopolitical environment, with US-China tech competition at its most intense, is not obviously favourable to the kind of great-power agreement that would make a meaningful AI governance treaty possible. But the difficulty of the problem is not an argument against attempting it. It is an argument for treating the problem with the urgency and resource that its difficulty requires.

What to watch

Watch whether Amodei’s statement is followed by any concrete proposal — a funding commitment to safety research, a public call for specific legislation, an engagement with international partners — that would give it structural weight rather than leaving it as a notable but inert CEO comment. Watch whether the UK government’s rejection of the kill switch concept is followed by any alternative mechanism or whether it represents a genuine policy vacuum. Watch how the broader public fear, rising according to multiple surveys, translates into electoral pressure: if AI risk becomes a salient voting issue in the 2026 US midterms or the UK’s next electoral cycle, the political calculus changes rapidly. And watch the Chinese government’s response to any emerging international governance framework — Beijing’s position will determine whether any global architecture is viable or symbolic.

— J