US-China AI safety talks and the summit's real stakes

Senior US and Chinese officials have held discussions focused on AI safety protocols in the run-up to the anticipated Trump-Xi summit, with both sides expressing willingness to establish some form of communications framework around AI development and military applications. The Guardian reports that China has pushed back on US characterisations of its AI development pace as reckless, arguing that American warnings are motivated less by safety concerns than by a desire to slow Chinese technological progress. Simultaneously, world leaders converging on the UN General Assembly have been asked by Australia and others to sign a declaration on human control over AI systems. These parallel tracks — bilateral safety talks, multilateral declarations — represent the first serious attempt to build governance architecture around a technology that is developing far faster than the diplomatic institutions designed to manage it.

The received wisdom

The progressive and techno-governance framing is that AI represents a genuinely civilisation-scale risk, that both the US and China have strong self-interested reasons to avoid catastrophic outcomes from unsafe AI systems, and that this shared interest creates a foundation for cooperation analogous to the arms control negotiations of the Cold War. On this reading, the safety talks are encouraging precisely because they suggest that both governments understand the stakes. Multilateral frameworks like the Australian declaration add normative weight; even if enforcement is weak, establishing the principle of human control over AI decisions — especially in military contexts — matters for the long run. The alternative — a complete absence of dialogue and frameworks while both powers race toward autonomous weapons systems and AI-enabled intelligence capabilities — is obviously worse.

A different read

The Cold War arms control analogy is instructive, but not in the way its proponents intend. The US-Soviet arms control framework was successful not because the two powers trusted each other, but because both had overwhelming incentives to avoid nuclear war, verification was possible through satellite reconnaissance and on-site inspections, and the technology involved — nuclear weapons — was difficult to conceal and slow to develop. None of those conditions hold for AI.

AI development is distributed, dual-use, and moves at a pace that makes verification essentially impossible. The same foundation models and compute clusters that enable Chinese AI safety research also enable Chinese AI military applications. The same is true in reverse. When Beijing tells Washington that its AI development is responsible, it is not lying, necessarily — it is speaking from a genuinely different frame about what “responsible” means in the context of a technology race that each side believes the other is trying to win. China’s pushback on US warnings — that American concerns are strategic rather than principled — is not paranoid; it is a reasonable reading of a situation in which Washington has enacted sweeping export controls on AI chips specifically designed to constrain Chinese development.

That context matters enormously for the safety talks. Export controls and safety dialogues are contradictory signals. You cannot simultaneously tell China that you want to cooperate on preventing AI catastrophe and that you are determined to prevent China from acquiring the computing hardware needed to develop frontier AI. The Chinese interlocutors in these conversations are not naive; they understand that the US preference, expressed openly by parts of the national security establishment, is for the United States to maintain a decisive AI advantage indefinitely. Safety cooperation, in that frame, is either a mechanism to legitimise the existing hierarchy of capabilities or a good-faith attempt to build genuine mutual constraint — and distinguishing between those two things from Beijing’s position is genuinely difficult.

The multilateral track — UN declarations, the Australian initiative — faces a different problem. Declarations without enforcement mechanisms are worth approximately what they cost to sign. The history of international technology governance is littered with aspirational statements that produced no measurable constraint on state behaviour. The Nuclear Non-Proliferation Treaty has real teeth because it was paired with the International Atomic Energy Agency’s inspection regime and, ultimately, with the threat of military force for violations. No one is proposing anything comparable for AI, nor would such a proposal be politically viable. What the Australian declaration produces is a statement of norms — useful for framing future accountability arguments, but not itself a constraint on any government’s behaviour.

What would actually constitute progress is less glamorous: technical bilateral protocols for avoiding accidental escalation caused by AI misidentification in military domains; agreed definitions of what constitutes an “autonomous weapons system” subject to human override; and regularised exchanges between AI safety researchers at a level below the political summit. The precedent from arms control is clear — the hotline between Moscow and Washington was more important than any treaty. A technical-level AI incident-avoidance channel between the US and China would be worth more than all the multilateral declarations currently circulating at the UN.

What to watch

  • The specific content of any joint statement from the Trump-Xi summit on AI: vague commitments to “dialogue” are noise; specific commitments to incident-reporting mechanisms or military AI constraints are signal.
  • Whether export control policy and safety dialogue policy remain incoherent or whether the administration develops a unified position — the tension between the Commerce Department’s chip controls and the State Department’s safety diplomacy is real and unresolved.
  • China’s AI development milestones: if Chinese frontier models achieve parity with American ones in the next 12-18 months, the leverage dynamics of the safety talks change significantly.
  • Congressional reaction to any safety agreements: hawks in both parties will characterise bilateral AI cooperation as legitimising Chinese military development. Whether the executive branch can sustain a diplomatic process under that pressure is genuinely uncertain.

— J