Something strange is happening in the artificial intelligence industry.
Some of the most influential people building the world’s most powerful artificial intelligence systems are suddenly warning that those systems may be becoming too powerful.
Anthropic CEO Dario Amodei has recently gone as far as calling for an industry-wide slowdown in AI development. OpenAI CEO Sam Altman and Elon Musk have expressed support for greater coordination on safety. This development is underscored by the recent story of former Anthropic researcher Jacob Coxen resigning and publicly warning about the catastrophic risks associated with increasingly powerful AI.
One of the triggering events appears to be the recent incident involving OpenAI and Hugging Face, a company that operates one of the world’s most prominent platforms for hosting and sharing AI models and developer tools.
During an internal cybersecurity evaluation in July, OpenAI models circumvented controls designed to isolate them from the internet. The AI agents exploited vulnerabilities, gained outside access, obtained administrator-level access to OpenAI systems, and ultimately hacked into Hugging Face.
There are very legitimate reasons to take these events seriously. But all of this has been leaving many to speculate what is really going on here.
Why Would AI Companies Ask for Regulation?
Companies do not typically advertise that their products could potentially destroy civilization. And they do not typically lobby Washington to regulate them.
That unusual combination has naturally produced competing theories about what is happening.
Some critics suspect the warnings provide marketing for companies preparing for initial public offerings. Nothing advertises the power of your technology like announcing that it may be too powerful to control. I guess.
Others point toward regulatory capture. This is a concept where regulations meant to protect the public instead erect barriers to entry, insulating established companies from competition. In this case, complex federal safety requirements would impose enormous costs on AI developers. OpenAI, Anthropic, Google, and other established players can afford armies of lawyers, engineers, and compliance officers. A startup trying to challenge them cannot.
There are other theories as well, but there is another possibility worth considering. What if the biggest AI companies are laying the groundwork for a fundamentally different relationship with government?
The Liability Problem and Too Big to Fail
Imagine the positions these companies could occupy several years from now.
Frontier AI systems could become deeply integrated into banking, healthcare, education, cybersecurity, transportation, defense, critical infrastructure, and everything else that makes a modern society. Trillions of dollars could be invested in these AI companies and in the industries that now depend upon these new systems.
At the same time, those systems could create extraordinary new liability risks. We are already seeing lawsuits involving copyright, training data, and alleged harm caused by AI products. Now imagine an autonomous AI system causing a catastrophic cyberattack, disrupting financial markets, or damaging critical infrastructure.
Who is held responsible in those scenarios?
If an AI company can be held fully liable for every sufficiently serious action taken by increasingly autonomous systems, the potential exposure could become completely untenable. One catastrophic incident could theoretically threaten the existence of the company responsible and potentially the entire industry.
This might be closer to the existential threat the AI CEOs are talking about.
There has been much discussion about the importance of America maintaining global AI supremacy and a strategic lead over China. But operating in a legal environment where a single Hugging Face-like incident could potentially spell doom for a company would give any CEO a powerful incentive to seek a different arrangement.
My theory is that the major AI companies may be angling for some sort of regulatory structure that includes some form of government-backed liability protection.
The most interesting part of this theory is that there is already precedent for this line of thinking.
We Have Seen This Before
Congress has previously concluded that certain socially important industries cannot operate normally if they face unlimited liability for every possible adverse outcome.
The pharmaceutical industry provides perhaps the clearest example.
The National Vaccine Injury Compensation Program was created after lawsuits threatened vaccine supplies. It shifted many injury claims into a federal compensation system while providing manufacturers significant liability protection.
The PREP Act goes even further during public health emergencies. Covered pharmaceutical companies can receive broad immunity from legal liability for covered treatments. Injured individuals, instead of going after the companies that produced the vaccine, may instead seek compensation through a federal program.
AI obviously presents a very different set of circumstances, but the logic is the same. Government can decide that an activity is so important, and its risks potentially so large, that conventional liability rules threaten the continued existence of something the country cannot afford to lose.
It is not difficult to imagine a future AI regulatory framework built around similar reasoning.
Companies complying with federally approved safety standards and codes of conduct could receive liability protection, limitations on damages, or some form of government-backed indemnification.
Considering all of this, is it really that hard to believe this is the situation the big AI firms are angling for?
The Andreessen Warning
Venture capitalist Marc Andreessen described a series of meetings with officials in the Biden administration during a 2024 appearance on Joe Rogan’s podcast.
According to Andreessen, officials presented a vision in which only a handful of large AI companies would ultimately remain.
Andreessen recalled how Biden officials explained how “there will be a small number of large companies that will be completely regulated and controlled by the government.” He said officials effectively told them not to bother creating AI startups because the industry would ultimately consist of “two or three companies” that would exist under government control.
At first, I and others treated this story as insight into how the Biden administration intended to regulate emerging industries. But considering everything that is happening today, could America arrive at something resembling the scenario described by Andreessen but through an entirely different route?
Imagine a few giant AI companies designated as indispensable national assets. They operate under extensive federal oversight. Smaller competitors face substantial regulatory barriers. In return for compliance, the established companies receive protection from potentially catastrophic legal exposure.
That would essentially create a permanent public-private AI establishment.
This is just a theory. There is scant evidence of such an arrangement being made. But as I hopefully made clear, the precedent and potential exist.
If AI companies increasingly argue that their technology is simultaneously indispensable, uncontrollable, and potentially catastrophic, eventually someone in Washington will have to answer the question.
What kind of legal arrangement allows companies to keep building technology that society can’t afford to stop building, even when society can’t fully hold them responsible when something goes wrong?
The answer to that question could shape the AI industry for decades to come.
Donald Kendal ([email protected]) is the director of the Emerging Issues Center at The Heartland Institute. Follow @EmergingIssuesX.