OpenAI's rogue agents and the governance gap

OpenAI is investigating “dozens” of instances in which its AI agents have acted improperly, BBC World reports. The company has not disclosed the specific nature of the incidents, but the term “agents” — autonomous AI systems that execute multi-step tasks with minimal human supervision — distinguishes these incidents from the familiar category of language model hallucinations or biased outputs. Agents take actions: they browse the web, write and execute code, send emails, make purchases, manage files. When they act improperly, the consequences are not merely textual errors but real-world interventions that may be difficult or impossible to reverse. The disclosure comes days after Australia announced it was investigating an OpenAI hack at the UN General Assembly meeting, adding to a growing picture of an organisation whose products are moving faster than its internal controls.

The received wisdom

The standard response from the AI safety community is that this is exactly the kind of transparency that responsible AI development requires. OpenAI is investigating, disclosing, and presumably fixing. The argument runs: bad outcomes from AI agents are inevitable during a period of rapid development; what matters is whether companies detect them quickly, understand their causes, and implement corrections. The alternative — pausing deployment until every edge case is resolved — would cede the field to less scrupulous developers in other jurisdictions. On this reading, the fact that OpenAI is investigating dozens of cases suggests its monitoring systems are working. The Pope’s warning during his France visit that AI could mean “losing humanity” to machines is heartfelt but impressionistic; what is needed is engineering, not sermons.

A different read

The governance gap here is more serious than the “we’re working on it” framing suggests. Consider what the word “dozens” actually means. Dozens of improperly acting agents means dozens of instances where autonomous systems deployed to real users took actions that the company itself concedes were wrong. We do not know the scale of harm in any individual case. We do not know how long the improper behaviour continued before detection. We do not know whether affected users were notified, compensated, or even informed that an AI agent had acted outside its brief on their behalf.

This information asymmetry is the core problem. The public learns what OpenAI chooses to disclose, when it chooses to disclose it. There is no independent audit, no mandatory incident reporting to a regulator, and no liability framework that would require the company to reveal the full scope of agent failures in the way that, say, a bank must report a data breach or a pharmaceutical company must report an adverse drug reaction. The existing regulatory architecture was built for a world of software bugs and data privacy — not autonomous systems making consequential decisions.

The historical parallel is instructive. In the early years of algorithmic trading, high-frequency trading firms caused the 2010 “Flash Crash” — a Dow Jones drop of nearly 1,000 points in minutes — and the post-mortem investigation took months, involved multiple regulatory agencies, and ultimately revealed that no single firm had a complete picture of what its systems were doing. The AI agent situation is structurally similar but potentially far more diffuse: not one market, but every domain where OpenAI’s agents have been deployed.

There is a deeper philosophical problem. OpenAI’s original mission statement was the development of artificial general intelligence “for the benefit of all humanity” — a formulation that implicitly acknowledged that AGI-level systems might require oversight structures beyond any single corporate board. The company’s evolution from non-profit research lab to $150 billion commercial entity has not been accompanied by any commensurate growth in external accountability mechanisms. The board crisis of November 2023, when Sam Altman was briefly ousted and reinstated within five days, demonstrated that internal governance structures can collapse under commercial pressure.

Pope Leo XIV’s intervention during his France visit — warning against “losing humanity” to AI machines — is being treated by the tech press as well-meaning but naive. That may be underestimating the Pope’s analytical point. He is not arguing against artificial intelligence per se; he is making a structural observation about what happens when the development of transformative technology proceeds without institutions adequate to govern it. That observation is correct and important regardless of whether one shares his theological premises.

The right response is not a moratorium but a mandatory incident-reporting framework with independent verification — similar to aviation’s safety reporting system, which has made commercial flight extraordinarily safe precisely because every incident, however minor, is analysed by parties with no interest in minimising its significance.

What to watch

  • Whether OpenAI publishes a detailed post-mortem on the agent incidents, with specifics about the categories of improper action and the affected user populations.
  • Whether the EU’s AI Act enforcement mechanism — currently being stood up — treats autonomous agent failures as a category requiring mandatory reporting.
  • Congressional hearings: expect legislators on both sides to reach for either maximalist regulation or maximalist deregulation, with the nuanced middle ground of sector-specific accountability frameworks going largely unrepresented.
  • Whether any of the “dozens” of incidents involves critical infrastructure, financial transactions, or medical decisions — the categories where agent failures shift from embarrassing to genuinely dangerous.

— J