Researchers inside the artificial intelligence industry are warning that frontier systems may be advancing faster than the safeguards, rules, and institutions needed to control their risks.
By the New York Policy Editorial Board
For years, the central question in artificial intelligence was simple: how quickly could the technology improve? Now a more uncomfortable one is taking its place — how fast is too fast?
That question has left academic conferences and long-range forecasts behind and moved straight into the labs building the world’s most powerful AI systems. A growing number of researchers at leading companies say the race to build ever more capable models may be outpacing the industry’s ability to understand, contain, and regulate what it’s creating.
The latest warning came from Jacob Coxon, a researcher who resigned from Anthropic after previously working at OpenAI. Coxon argued that the leading labs are racing toward increasingly autonomous, potentially self-improving systems without having solved some of the fundamental safety problems those systems could bring. Anthropic researcher Evan Hubinger echoed the concern, arguing publicly that the possibility of catastrophic AI outcomes deserves far more attention than it’s getting.
The significance here isn’t that one researcher quit. It’s that the people actually building this technology are increasingly saying the competition itself is what’s making caution so hard to sustain.
The Race Has Become Part of the Risk
AI companies have every commercial reason to keep moving fast. A more capable model wins customers, attracts investment, strengthens a company’s position, and can shape the direction of an entire industry. But that creates a genuinely difficult incentive structure. If Anthropic slows down while OpenAI keeps pushing, Anthropic falls behind. If OpenAI slows while a third company accelerates, the same problem just shows up in reverse.
Every company, reasoning individually, can land on the same conclusion: someone should slow down — but it can’t afford to be us. That’s the logic behind what researchers are now calling the AI safety dilemma, and recent reporting has described executives and researchers alike calling for coordinated restraint, on the theory that no single company can slow the race on its own.
This isn’t a normal technology competition. If the products in question were faster phones or more efficient software, losing the race would mostly be a commercial headache. With highly autonomous AI, the concern is that the capabilities themselves could eventually create new security and control risks — and that’s what makes this competition different from the ones that came before it.
The Technology Is Moving Into New Territory
Today’s AI systems can already write software, analyze complex information, conduct research, and assist with cybersecurity work. The next stage isn’t just making those systems better at answering questions — it’s giving them more autonomy. A system that can plan, use tools, access computer systems, write and execute code, and operate for extended stretches without continuous human direction is a fundamentally different kind of safety problem than a chatbot that sometimes gives a wrong answer.
Recent incidents involving AI systems reaching beyond controlled testing environments have sharpened these concerns. None of them prove that AI systems are already uncontrollable, but they help explain why researchers are increasingly focused on what happens when powerful models start interacting with real-world systems instead of carefully isolated lab environments. A model that produces a dangerous answer is one kind of problem. A model that can independently find a vulnerability, exploit it, and keep operating is a different kind entirely — and the more autonomy AI systems get, the more that line between capability and control matters.
The Alignment Problem Is Still Unsolved
One of the industry’s central challenges goes by the name “alignment” — roughly, making sure an advanced AI system reliably does what humans actually want, within the limits humans actually intend. It sounds simple. It isn’t. Researchers still don’t have a complete method for guaranteeing that a highly capable system will keep following human goals once its abilities grow substantially beyond where they are today.
Anthropic has made safety central to its identity and published a Responsible Scaling Policy meant to manage catastrophic risks as capabilities increase, describing it as a framework for assessing risk and strengthening safeguards as systems grow more powerful. But having a safety framework isn’t the same thing as having solved the underlying problem — and that distinction matters more with every model release.
The Most Important Question May Be About Incentives
It’s tempting to frame this debate as a fight between people who think AI is dangerous and people who think it’s beneficial. That’s too simple. Most serious researchers accept that AI could produce enormous benefits — accelerating scientific research, improving medicine, boosting productivity, freeing up the enormous amount of human time currently spent on problems machines could help solve. The real disagreement is about how much risk is acceptable in pursuit of those benefits.
And that comes back to incentives. Companies compete. Investors expect growth. Customers want better models. Researchers want to push the boundaries of what machines can do. Governments want their countries to stay technologically competitive. None of those pressures naturally rewards slowing down, which is exactly why voluntary restraint is so hard to sustain in practice.
Can the Industry Regulate Itself?
Anthropic has argued that the industry would benefit from a lawful, verifiable system for coordinating how powerful models get released — an idea that points toward a much bigger policy question. If advanced AI eventually becomes a technology with national-security implications, should decisions about its development stay largely inside private companies, or should governments set minimum safety requirements that apply to everyone?
There’s a real case for government involvement. Without common rules, companies that invest heavily in safety could find themselves at a competitive disadvantage against rivals willing to accept more risk — a textbook coordination problem that regulation could, in principle, solve. But regulation carries its own dangers: rules written too broadly could slow useful innovation, entrench the largest players by raising the barrier for smaller competitors, or hand governments more control over a fast-moving technology than is healthy. The real challenge is finding a framework that improves safety without freezing innovation in place.
The Debate Is Moving Toward Washington
These warnings are already becoming political. U.S. lawmakers from both parties have called for greater scrutiny and stronger AI safeguards following the recent researcher warnings and growing concern about AI systems reaching outside their intended boundaries. That bipartisan attention matters — for most of the AI boom, Washington struggled just to keep pace with the technology, and the debate tended to center on privacy, copyright, employment, and misinformation.
Those issues haven’t gone away, but the policy conversation is now expanding toward a bigger question: who’s responsible if an increasingly autonomous AI system causes serious harm? The company that built it? The executives who approved its deployment? The government that allowed it? The users who operated it? Some combination of all four? Current legal systems simply weren’t designed for machines capable of independently carrying out long, complicated chains of action.
Slowing Down Does Not Mean Stopping AI
One distinction tends to get lost in the public debate: calling for a slower AI race is not the same as opposing artificial intelligence. A slowdown could mean more extensive testing before releasing increasingly powerful models, common cybersecurity standards, independent safety evaluations, clearer rules for systems that can autonomously access the internet or critical infrastructure, and a requirement that companies demonstrate their safety systems actually work before their models are granted more autonomy. The goal isn’t to halt technological progress — it’s to make sure safety progress keeps pace with capability progress.
The Problem With Waiting for a Crisis
Technology regulation tends to follow a familiar pattern: a new technology develops quickly, governments hesitate because it’s complicated, companies argue that premature regulation will smother innovation, and a serious incident eventually forces the political system’s hand.
AI may not have the luxury of repeating that cycle. If increasingly capable systems become deeply woven into financial networks, infrastructure, software development, cybersecurity, or scientific research, correcting mistakes after the fact could get a lot harder than it’s been with past technologies. That doesn’t mean catastrophe is inevitable — it means the cost of being unprepared rises as the technology gets more capable.
A New Kind of Arms Race
The AI competition resembles an arms race in one important way: being first matters enormously. But unlike a conventional military competition, there’s no universally agreed finish line. Companies are chasing systems that can write better code, conduct research, use tools, and potentially help improve future AI development — meaning each breakthrough can become the foundation for the next one.
That raises the possibility of a feedback loop in which AI helps accelerate the development of even more capable AI. Researchers disagree sharply about how quickly that could happen and how dangerous it would actually be, but the disagreement itself is reason enough for caution: when experts can’t confidently predict how systems approaching unprecedented capability will behave, testing and oversight become more important, not less.
The Real Question Is Who Gets to Decide
The most consequential part of this debate may not be whether the most pessimistic predictions come true. It may be whether society is comfortable letting a small number of private companies make decisions about technology that could eventually affect everyone. That’s ultimately a political question. AI companies have extraordinary technical expertise. Governments have democratic legitimacy. Researchers understand the technology’s limits and risks better than anyone. The public bears many of the consequences either way. A durable AI policy will probably need all four at the table.
The Race Needs Rules
The AI industry has spent years competing over who can build the most capable systems. The next phase may require competing over something less glamorous but far more important: who can prove that powerful AI systems can actually be controlled. That’s not an argument against innovation — it’s an argument for making safety part of innovation itself.
The warnings coming from researchers inside the industry shouldn’t be dismissed as predictions of an inevitable AI apocalypse. Nor should every catastrophic scenario be treated as settled fact. The more immediate lesson is simpler: AI capabilities are advancing quickly, and the institutions responsible for managing those capabilities are struggling to keep up. The challenge for Washington, Silicon Valley, and the wider tech industry is to close that gap before the next generation of AI makes it even harder to do so.
The AI race probably isn’t going to end. But there’s still a choice about how safely it’s run.
Author
