Anthropic co-founder Dario Amodei warned this week that AI is developing too fast without adequate human oversight, calling on the industry to pause and allow safety frameworks to catch up with capability advances. Separately, Al Jazeera reported that Anthropic is urging AI labs broadly to slow down, warning that humans risk losing meaningful control over the systems being built. The calls come in the same week that Trump announced a meeting with AI company leaders to discuss US investment in their companies, and as US stock markets slumped partly on fears about Big Tech overvaluation — a convergence suggesting that AI is simultaneously the centre of political attention, financial speculation, and genuine safety anxiety. SpaceX’s $1.75 trillion valuation target for a secondary share sale adds to the picture of a technology sector in which companies of extraordinary scale are being built faster than any governance structure can track.
The received wisdom
The progressive-technocratic mainstream has two somewhat contradictory responses to Anthropic’s call. The first is enthusiastic endorsement: yes, AI development needs guardrails, the companies should be regulated, and the fact that a leading AI lab is itself calling for a pause shows how serious the risks are. This view was crystallised by the 2023 “pause letter” signed by Elon Musk, Yoshua Bengio, and others, and it has been the dominant framing in European regulatory circles, where the EU AI Act has moved through legislative process.
The second response — increasingly common in American policy circles — is scepticism about whether voluntary pauses or even legislated constraints can work when the competitive dynamic is global. China is not pausing. The logic of mutual restraint requires verification mechanisms that do not exist. On this view, the answer is American investment and dominance, not self-imposed constraints, which is precisely the posture that Trump’s meeting with AI executives seems designed to reinforce. The question of who controls AI is, in this reading, geopolitical before it is technical.
Both readings contain real insights. Both are also, in different ways, self-serving.
A different read
The most important sentence in Anthropic’s warning is the one that has received the least attention: it is Anthropic, the company building some of the world’s most capable AI systems, telling other AI companies to slow down. This is not a disinterested call. It is a company with a specific vision of AI safety — its own approach, embodied in its “Constitutional AI” methodology and its models — making a public argument that happens to cast its competitors’ approaches as reckless and its own as responsible.
This is not necessarily cynical. The people at Anthropic may genuinely believe what they are saying; many of them left OpenAI precisely because they thought safety was being subordinated to capability development. But the structural incentive is clear: a “pause” that is long enough for safety frameworks to be formalised is also a pause that tends to benefit the companies already ahead. Regulatory capture — the phenomenon by which industries shape the regulations ostensibly constraining them — operates in AI as in every other technologically intensive sector.
There is a deeper problem, which is that no one has agreed on what “adequate human oversight” means, and the companies building AI are the only entities with sufficient access to their own systems to propose definitions. When Amodei says humans risk losing control, he is describing a real phenomenon — the alignment problem, which concerns whether AI systems will reliably pursue goals that humans intend — but he is also implicitly positioning Anthropic as the entity that understands and is managing that problem. The public and its elected representatives are being asked to trust a private company’s self-assessment of the risk its own products pose.
This is not unique to AI. The pharmaceutical industry shapes drug approval processes; the financial industry shapes capital requirements; the nuclear industry shaped the regulatory bodies that oversaw it. What is different about AI is the speed: the gap between a technology’s deployment and the institutional capacity to evaluate it is larger than in almost any previous case. Trump’s summit with AI executives this week suggests that the current US government’s answer to this problem is partnership with industry rather than independent oversight — a bet that competitive advantage and safety are aligned, which is precisely the assumption that Anthropic is questioning.
The Wall Street dimension matters too. US stocks slumped this week partly on AI valuation concerns. The financial markets are beginning to ask whether the AI investment cycle is creating value or inflating a bubble — a question that has significant implications for whether development continues at current pace regardless of what any company’s safety team recommends. A sharp financial correction would do more to slow AI development than any voluntary pause, and would do so in a way that is entirely outside the safety discourse. That is either reassuring or alarming depending on your view of whether markets are better or worse at calibrating technological risk than regulatory bodies.
The honest position is that no existing institution — national government, international body, industry consortium, or individual company — has demonstrated the capability to govern transformative AI development in real time. That is a structural problem, not a personnel problem, and it will not be solved by calling summits or signing letters.
What to watch
Whether Trump’s meeting with AI executives produces anything resembling a governance framework or merely a set of investment announcements. The distinction matters enormously for what kind of AI environment the next decade produces.
Congressional movement on AI liability: the question of who is legally responsible when an AI system causes harm has not been resolved in US law. Without liability, there is no market incentive for the safety behaviours Anthropic is advocating.
Whether the EU AI Act’s implementation creates genuine competitive disadvantage for European AI firms — which would strengthen the “we can’t afford to pause” argument in Washington and Beijing simultaneously.
Anthropic’s own next model release: if it comes within months of calling for a pause, that will be a revealing data point about the gap between the company’s public safety posture and its competitive behaviour.
— J