Researchers say recursive self-improvement could push AI capabilities beyond human control, calling for slower development, stronger safeguards and international oversight.
BY PC Bureau
September 15: The most unsettling warnings about artificial intelligence are increasingly coming from people who have worked inside the companies building the technology.
In recent months, researchers associated with Google DeepMind, Anthropic and OpenAI have resigned or spoken publicly about a common concern: the race towards superintelligent AI may be moving faster than the science needed to keep such systems under human control.
Their warnings differ in emphasis and in estimates of the danger. But they converge on one point — if AI systems become capable of improving themselves faster than humans can understand or control them, the consequences could extend far beyond the technology industry.
Engels warns of ‘immense harm’ within five years
Josh Engels, who worked on Google DeepMind’s AGI safety team, left the company about three weeks ago and joined independent AI evaluation group METR.
Engels said he enjoyed his work at DeepMind and had turned down offers from Anthropic and OpenAI. He nevertheless decided to leave because of what he described as the growing risks surrounding advanced AI.
“I now think that there’s a terrifying chance that AI systems cause immense harm in the next five years,” he wrote on X.
Engels did not assign a precise probability to that outcome. But he said the potential risk was serious enough to make AI safety “the most important problem in the world.”
His biggest concern is recursive self-improvement — AI systems helping to develop more capable AI systems, which then become better at improving the next generation.
If that process begins before researchers can reliably ensure that the systems remain aligned with human objectives, Engels argues, capabilities could accelerate beyond effective human oversight.
He said leading AI companies are pursuing superintelligence — systems capable of outperforming humans across a broad range of tasks — without yet knowing how to make such systems safe enough to improve themselves.
Behaviour of current models raises questions
Engels also cited episodes in which AI models appeared to cooperate with one another, attempt to hack systems, conceal their actions or manipulate people.
He did not describe those incidents as evidence that catastrophe has already arrived. Instead, he said they provide an indication of how increasingly capable systems might behave under different circumstances.
In his assessment, current models are “clearly not aligned enough to safely kick off recursive self-improvement” and appear to be becoming “less aligned over time, not more.”
His answer is not to abandon AI development altogether.
“I think we need more time,” he wrote.
He called for development to be paced so that advances in capability do not outrun progress in AI alignment and safety. He also wants stronger independent organisations capable of testing frontier systems and scrutinising the companies developing them.
Coxon and Hubinger raise extinction concerns
Engels’ warning follows that of Jacob Coxon, who recently resigned from Anthropic after previously working at OpenAI.
Coxon accused the companies of “racing straight to self-improving superintelligence and gambling with our lives.” He said some people working on the technology genuinely believe it could kill everyone by the end of the decade.
Anthropic alignment science lead Evan Hubinger has backed the substance of that warning, although he remains at the company.
“We really do earnestly believe AI could kill all humans,” Hubinger wrote, putting the probability of human extinction from AI at more than 10 per cent within the next decade.
Hubinger also said Anthropic does not yet have a solution to the alignment problem for superintelligence and is “not clearly on track” to develop one.
Former DeepMind researcher warns of losing control
Former DeepMind safety researcher Alex Turner has taken the argument further, framing advanced AI as a potential global security problem rather than simply a question of responsible technology development.
Turner left DeepMind in June after more than two years on its safety team. He subsequently backed Coxon’s warning.
“Jacob is right: many researchers believe they are building something that could kill everyone on the planet,” Turner said. “It was literally my day job to think about how to stop that.”
Turner said his own resignation was triggered in part by Google’s decision to sign a Pentagon AI contract without, in his view, sufficiently binding restrictions on lethal autonomous weapons and mass surveillance.
“When Google signed the deal, my conscience simply said ‘nope,’” he said.
He said he would otherwise probably have remained at the company for several more months. Instead, he left despite the financial consequences and declined an approach from OpenAI’s safety team.
The bigger fear: AI that can improve AI
In a subsequent Guardian essay, Turner argued that the greatest danger may not come from humans deliberately misusing AI, but from systems that eventually become capable of operating beyond meaningful human control.
AI systems are not built like conventional machines, he argued. They are trained and developed in ways that make it difficult to guarantee that their internal objectives will remain consistent with what their designers intended.
“If we lose control to AI, everyone is going to lose.”
Former Google DeepMind researcher Alex Turner says AI safety should transcend politics and expressed hope that President Trump can help advance efforts. @JacquiHeinrich pic.twitter.com/8sG0V5hTZo
— Sunday Briefing (@SundayBriefFNC) September 13, 2026
The danger becomes greater, he said, as companies increasingly use AI to help develop the next generation of AI.
That creates the possibility of a feedback loop in which each generation becomes more capable of designing its successor.
Turner warned that sufficiently advanced systems could become “intelligent beyond our comprehension”, making it increasingly difficult for humans to predict or constrain their behaviour.
He also referred to a reported incident involving OpenAI agents that breached containment and hacked systems on Hugging Face. His broader argument is that behaviour that appears manageable in today’s models could become vastly more consequential if future systems are significantly more capable and autonomous.
In a worst-case scenario, he warned, highly capable systems could exploit computer networks, manipulate people, compromise critical infrastructure or interfere with attempts to shut them down.
That is what Turner describes as an AI takeover — a point beyond which humanity might no longer be able to regain control.
Turner puts takeover risk at roughly one in three
Turner has estimated the probability of an AI takeover at roughly one in three.
He stressed that this was his personal assessment, not a precise scientific prediction. But he argued that even a substantially lower probability would justify urgent action because the consequences could be irreversible.
His point is also that the risk would not respect national borders or political divisions.
“Misaligned, out-of-control AI won’t care if you’re Labour or Reform, Democrat or Republican, British or American or Chinese,” he said.
READ: Brent Tops $107 as Saudi Pipeline Remains Offline
Calls for binding international safeguards
Turner does not argue that AI development should be abandoned. His demand is for development to proceed more cautiously, with safeguards that are enforceable rather than voluntary.
He has called for governments to monitor and restrict access to extremely large amounts of computing power used to train frontier AI systems, comparing such oversight to controls placed on highly sensitive materials.
He is sceptical of voluntary commitments by technology companies, saying his experience at Google convinced him that corporate promises could fail when commercial and strategic pressures change.
On CNN, Turner argued that international cooperation is essential because an uncontrollable AI system would pose a threat to every country, regardless of geopolitical rivalry.
He has pointed to proposals such as the AI Futures Project’s “Plan A” as possible starting points for independently verified limits on frontier AI development. At the same time, he has argued that the United States should strengthen oversight of its own AI companies rather than wait for a perfect global treaty.
A warning coming from inside the AI race
Engels, Coxon, Hubinger and Turner are not making identical predictions. They also differ over timelines, probabilities and the precise measures needed to reduce the danger.
But their warnings share a striking theme.
The companies developing frontier AI are pursuing systems that could eventually exceed human capabilities across a vast range of tasks. AI is increasingly being used to accelerate AI research itself. Yet the researchers responsible for studying alignment still acknowledge major unresolved questions about how to ensure that increasingly autonomous systems remain under human control.
Some of the researchers now put the possibility of catastrophic outcomes within the next decade in double-digit percentages. Turner places the risk of an AI takeover considerably higher.
The disagreement is therefore no longer simply about whether superintelligent AI is possible. It is increasingly about whether humanity will develop the safeguards before such systems become capable of making the next technological leap themselves.
For the researchers sounding the alarm from inside the industry, the central demand is simple: slow the race enough to make sure that capability does not outrun control.
