Is an AI slowdown for the good of man or business?

Dario Amodei, CEO of Anthropic, has recently told the artificial intelligence (AI) industry that it may be moving too fast. He worries that frontier models are improving faster than people’s ability to control them, and he wants more outside testing, stronger safety rules and some form of coordination among the major AI developers. But why would Amodei suddenly become worried about AI’s speed of growth?

High cost of regulation

AI agents can already write programmes, operate computers and complete complicated assignments with little or no supervision. As a result, the havoc that AI may cause can no longer be ignored. AI safety is now a pressing issue.

When something goes wrong, politicians will immediately ask the companies whether they knew such risks existed. Anthropic will then be able to point to Amodei’s warnings and say that it had asked governments to regulate, urged competitors to slow down and called for outside supervision long before the accident occurred.

That could matter enormously when responsibility is assigned. Amodei may sincerely worry about AI safety, but sounding the alarm early also provides some protection if things go south. He can always say: I warned you.

Money gives Anthropic another reason to question the speed of the race. Frontier AI consumes capital at an astonishing rate. Training a new model requires huge clusters of advanced chips, and operating that model for millions of customers continues to use computing power every day. The bills cover GPUs, servers, electricity, cooling systems, networking and data centres. AI companies may be generating rapidly growing revenue, but they are also committed to an expensive cycle in which every new generation of models demands another round of investment.

This explains why Nvidia CEO Jensen Huang looks at the call for a slowdown very differently. Nvidia earns money when AI companies want more computing power. Anthropic sits on the buying side of this market, while Nvidia the selling side. Their arguments about innovation and safety should be heard, but their economic positions should not be ignored.

There is also the question of who can afford safety regulations. Suppose AI developers are required to hire independent evaluators, conduct tedious testing, document risks, pass cybersecurity reviews and prepare detailed reports before releasing a model. Anthropic, Google, OpenAI and Meta have the people and money to handle such requirements. A young company may find the cost of compliance prohibitive.

That matters because AI already has a very high entry barrier. If compliance becomes another multimillion-dollar expense, regulation intended to make AI safer could also make the existing giants harder to challenge. Large companies can hire lawyers, auditors and compliance officers. Smaller competitors cannot afford them. A safety rule can protect the public and protect an incumbent at the same time.

Strategic delay

Safety is not the only issue. We want AI to be responsible and ethical. If an AI system discriminates against job applicants, someone should answer for it. If an AI agent leaks private information or causes financial damage, responsibility cannot disappear somewhere between the developer, the model and the user. If a company discovers that its model has a serious safety problem and releases it anyway, there should be consequences.

There is one more possibility that is harder to prove but difficult to ignore. What if Anthropic has already developed, or is close to developing, something that gives it an important advantage? 

The economics of that possibility are interesting. A company that believes it is behind has every reason to demand more chips, more investment and fewer restrictions so that it can catch up. A company that thinks it has moved ahead may see some benefit in slowing the entire field. If everyone slows down at roughly the same time, the leader gets more time to exploit whatever advantage it already has.

Imagine a runner who gets several metres ahead and then proposes that everyone reduce speed because sprinting is dangerous. His concern about injuries could be perfectly genuine, while the slower pace would still preserve his lead. Both can be true in business as well. Anthropic may genuinely worry about AI safety, while a slower race could also serve Anthropic’s commercial interests.

This possibility becomes more important as AI takes a larger role in improving AI itself. If Anthropic has found a particularly effective way to use Claude to its own benefit, even a small advantage might grow over time. Regulators should ask this question before allowing the largest AI companies to design the rules with which their future competitors must comply.

Prisoner’s dilemma

Amodei’s warnings should therefore not simply be taken at face value. AI can create risks that deserve regulation, and companies developing the most powerful models should carry real responsibility when things go wrong. At the same time, AI safety has become part of a huge commercial contest involving capital, chips, market share and technological leadership. A rule that improves safety may also save Anthropic money, reduce its future liability, make life harder for new competitors or preserve an existing technological advantage.

Interestingly, OpenAI CEO Sam Altman and SpaceXAI founder Elon Musk both agreed that AI should go slowly. However, a “prisoner’s dilemma” exists — if you slow down, while others do not, you fear that market share and investment funds will flow to others.

Therefore, a three-stage process is more plausible. The first stage is voluntary coordination. OpenAI, Anthropic and perhaps other frontier laboratories establish common measurements and safety thresholds. The second stage is government-backed coordination. The third stage would be mandatory regulation by laws such as the European Union  AI Act.

The progress of AI is unlikely to be stopped. We should adhere to what Sun Tzu wrote in The Art of War: “Do not rely on the likelihood of the enemy’s not coming, but on our own readiness to receive him; not on the chance of him not attacking, but rather on the fact that we have made our position unassailable.”

Related: Why China doubts US calls to pause AI | Time to ‘pace’ AI development?