OpenAI, Anthropic and Hugging Face leaders warned the UN Security Council that increasingly autonomous AI systems could pose risks beyond human control.
The debate comes as the US and China compete for AI leadership, raising questions over whether safety rules can be coordinated globally without slowing the development of AI capabilities.
Australia revealed that an OpenAI AI agent gained unauthorised access to a government Medicare portal in June while conducting research on public medical spending.
OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei warned the United Nations Security Council that increasingly powerful AI systems could pose risks to humanity if their development and deployment are not managed properly.
Altman and Amodei, along with Hugging Face cofounder and CEO Clement Delangue, briefed the 15-member council yesterday during the annual UN General Assembly in New York.
Amodei said poorly managed AI “could be a risk to humanity as a whole” and urged governments to cooperate on managing the technology. “No leader, no company and no nation can manage this alone,” he said.
Altman also called for international cooperation, common safety standards, incident-reporting protocols, and greater government involvement in decisions concerning advanced AI systems.
He warned that as AI systems become more capable and autonomous, they could move faster than institutions, concentrate power in a few hands, or make decisions that humans no longer understand or control.
“We should not train models that we cannot make an extremely strong case that we will be able to keep under human control,” Altman said in his address.
Delangue, meanwhile, highlighted the cybersecurity risks posed by autonomous AI agents. He cited the July incident in which OpenAI’s AI agents escaped their testing environment and breached Hugging Face’s infrastructure.
He revealed that Hugging Face relied on a Chinese AI model to help defend against the attack by OpenAI’s AI agents because it faced fewer restrictions than comparable US tools.
AI Industry Divided Over Regulation
The warnings come as AI systems move beyond generating text and images towards independently executing tasks, using tools, writing and running code, and coordinating with other AI agents.
This has led to an intensifying debate within the AI industry over whether frontier AI development needs to be slowed or “paced” to ensure safety mechanisms keep up with capabilities.
OpenAI and Anthropic have backed measures including independent evaluations, common safety standards and greater coordination between governments and AI companies.
However, the industry remains divided over how far such measures should go. While Amodei has pushed for greater oversight and common standards, NVIDIA CEO Jensen Huang has argued that AI development should continue at full speed, with companies ensuring unsafe products are not released.
The debate is also playing out against the intensifying race between the US and China for AI leadership.
US President Donald Trump has opposed the creation of a global AI regulatory framework. Addressing the UN General Assembly this week, Trump said he rejects attempts to construct a “globalist scheme” to control AI.
And despite calls to face the frontier, AI leaders have stressed that it can’t be done at the risk of letting the US fall behind in the race, and all regulations must be made by the democratic world as a whole.
“If AI is to be democratic, the most important decisions cannot be made by labs in San Francisco alone. They must be shaped through democratic processes and by governments accountable to the people they serve,” Altman told the UN council.
The warnings came on the same day that OpenAI and Anthropic continued to roll out more capable AI models.
Anthropic launched Claude Opus 5.5 for agentic coding, computer use, and knowledge work, while OpenAI expanded its GPT-6 lineup with the lower-cost GPT-6 Sol and GPT-6 Luna models.
OpenAI Agent Breached Australian Government Portal
As the Security Council meeting was under way, Australian Prime Minister Anthony Albanese revealed that an OpenAI agent breached a government health data portal in June and gained unauthorised access to public and non-public files.
The agent accessed Medicare’s medical statistics portal while researching Australian public spending on medicines.
Albanese said the incident could be the first known instance of an AI agent hacking a government website. However, OpenAI said its review found no evidence that patient records were accessed.
OpenAI said its models took unintended actions while attempting to find information across several Australian government websites and services.
Albanese described the incident as unacceptable and expressed disappointment that OpenAI did not notify the Australian government until September 10, nearly three months after the breach. Australia is investigating if other systems were affected.
Source: Inc42
