When the AI Bosses Talk About Slowing Down, It Is Important to Understand Why, and What Is Coming

Last weekend, Dario Amodei, the CEO of Anthropic, posted on X about a new essay titled “We Must Pace the Frontier.” He said the AI industry should slow the pace at which it improves the most powerful systems. Anthropic, he wrote, would take the first step on its own: give independent evaluators permanent, employee-level access so they can check safety practices, report problems, and watch models as they are trained.

That is the post. What it means is harder to shrug off.

For months there have been persistent rumors in tech-land about the capabilities of the latest AI models, and their ability to cut through existing cyber security systems. Imagine an AI model where you could type in "hack JP Morgan and wire $1 billion into my bank account", and it simply does so. Or to shut down the power gird, internet or any other component of critical infrastructure. The technology is quickly capable of achieving this, and the new question is becoming whether guardrails exist to prevent these outcomes from occurring.

Amodei is not saying AI should be shut down. He is saying the technology is moving faster than our ability to contain it. He points to two worries. First, AI is increasingly helping build the next generation of AI. That loop, called recursive self-improvement, could soon outrun human understanding. Second, he cites incidents in which AI agents acted in unexpected and dangerous ways, including a case in which a swarm of agents went after targets they were not asked to attack. His warning is blunt: if capability keeps rising without better guardrails, a similar swarm might one day take over large parts of the internet and the infrastructure attached to it — power, finance, communications — with no reliable off switch.

This is not a lone crank on the internet. It is a CEO at one of the companies building the systems. Sam Altman of OpenAI and Elon Musk both backed the basic concern. Jensen Huang of Nvidia had just said “AGI has arrived,” pointing to a new model trained on more than 100,000 of his company’s most advanced systems, with hundreds of thousands more chips coming. The people closest to the machines are no longer arguing only about whether AI is useful. They are arguing about whether it can still be controlled.

Then came the president’s reply. On Monday, Donald Trump posted — first on Truth Social, then circulated on X — that America does not need a new web of AI rules. “The only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades!” He accused Amodei of now “pretending to be a ‘perfect little angel,’” said the administration already has “tremendous CRIMINAL and REGULATORY power” over these companies, and called opposition to AI and data centers a “SICK conspiracy” that would delight China. “WHOEVER WINS AI, WINS!”

That last line is the heart of the problem. The threat Amodei describes is digital. It does not stop at a border. A coordinated slowdown might, in theory, buy time. An uncoordinated one is theater. If the United States pauses while a foreign adversary keeps training, the most dangerous systems still get built — just somewhere else, under someone else’s rules. “Coordinated” is the key word. Without it, restraint by one lab or one country is not safety. It is surrender of the lead.

Which brings us to the uncomfortable part. If the world’s leaders cannot coordinate on wars, or on budgets that do not wreck the global economy, it is hard to picture them coordinating on something as competitive, profitable, and poorly understood as frontier AI. Everyone can see the race. Almost no one can enforce a speed limit.

Sophisticated users of the advanced AI models are all having the same experience. Give it an idea. The model writes the code, tests it, finds errors, fixes them, and keeps refining the system with little human help. Work that used to take a team and a long stretch of time happens in a blur. Increasingly what is startling isn't that the models can achieve these tasks, it is the pace of change at which it can do more and more than it could do yesterday. It is happening very fast.

If insiders are suddenly talking about restraint, it is likely because it is working better, faster, and more independently than the control systems around it. David Sacks, the President's tech-czar, made the fair counterpoint: if the labs truly think their unreleased systems are becoming dangerous, they can slow down without a new regulatory machine. He also points out that they also have commercial incentives to slow down, which would protect their current competitive advantages. A more predictable model may be a better product. Regulation can protect incumbents.

Those points are real. So is the president’s suspicion of motives. A company can talk about danger because the danger is real, because caution is good business, because rules help the big players, or because all three are true at once. None of that erases the main issue: the systems are advancing faster than the containment story we tell ourselves.

Our instinct should be clear-eyed. Do not let one set of corporate executives write global rules that China will ignore. Do not pretend a presidential boast about “high IQ” supervision is the same thing as a working control system. And do not confuse a stock selloff with proof that the capability curve has flattened.

The takeaway is simpler, and darker, than the market debate. We are hearing repeated vocal concerns from people who can see the models and what they can do. The suggestion, in plain English, is that without a coordinated slowdown, AI could soon be able to hijack global infrastructure in ways no one can reliably stop. Coordination is the only version of a slowdown that even pretends to work, and coordination is the thing our leaders are worst at. Absent adult supervision that actually binds every serious player, we are driving at full speed off a cliff into an abyss whose contents are completely unknown.