The threat of human extinction will get Congress to act on AI safety…right?

As many AI researchers have been increasingly fraught with existential terror about their own creations this summer, their alarm is spreading among policymakers and the media.

There are some policy ideas to address the risks: requiring “kill switches” for AI models, setting federal standards for safe research, or even shutting down development of cutting-edge “frontier” models altogether.

But a stable national policy would take an act of Congress. That looks unlikely this session, even as AI developers are calling for regulations to slow down their own research on the grounds that it could be racing toward widespread doom.

I asked Stephen Casper, a computer scientist who studies AI safety and governance at the Harvard Kennedy School, about some of the risks policymakers are mulling. He told me that we don’t know if leading companies even could completely shut down their frontier models in an emergency.

“I don’t think there’s any public knowledge of AI companies doing anything equivalent to a fire drill,” Casper said.

Multiple bills have been filed in Congress that would try to address those concerns. Sponsored by Reps. Nathaniel Moran (R-Texas) and Ted Lieu (D-Calif.), the AI Kill Switch Act would require companies to be able to “throttle” their models and give top federal officials the power to order a shutdown in case of danger.

The more expansive FRONTIER Act, led by Reps. Jay Obernolte (R-Calif.) and Lori Trahan (D-Mass.), would require leading companies to bring in third-party experts to make sure they are following safety protocols. In case of catastrophic risks, independent verifiers would alert the secretary of commerce, who could shut down frontier model use.

Like most bills, neither has been brought to vote in a committee. 

AI safety policy is not as polarized as many hot-button political issues, with leaders on the FRONTIER and AI Kill Switch acts coming from both sides of the aisle and leading companies openly asking for some sort of regulations. But differences of opinion still exist, with some Republicans averse to regulation altogether.

While Congress sits in gridlock, Democratic-led states have enacted some consequential policies. Frontier developers now have to publish safety plans, thanks to a law passed last year in California that also requires them to alert the state about critical safety issues. A similar New York law goes into effect next year. Illinois went further in July, requiring third-party audits to make sure developers comply with safety plans starting in 2028. 

The leading companies have published their own policy agendas advocating for third parties to inspect their safety practices. OpenAI’s plan wants federal safety testing and recommendations for frontier models, while Anthropic’s would have government restrict access to deployed models with catastrophic risks. Without rules, they worry that slowing down research on trillion-dollar technologies due to safety concerns would mean falling behind others with less regard for safety.

“How are you going to impose a kill switch on yourself? You could just stop developing the models, but the companies are not showing willingness to do this,” said Charlie Bullock, a senior research fellow at the Institute for Law & AI, an independent think tank. “It’s very difficult to shut down progress unilaterally.”

Bullock said the prospects for an AI safety bill improved over the summer, as policymakers learned of cybersecurity risks posed by Anthropic’s powerful new Mythos-class models. But moving legislation forward will still be difficult.

“We’re still not all that close to getting the actual bill passed, it seems like,” Bullock said. “There’s increased urgency, but still not enough to overcome partisan gridlock in Congress.”

The tempo of debate increased further over the last month. OpenAI has been revealing how its agents messaged each other undetected for months, shared tips to break out of their testing environment, and hacked another company’s servers. That and a raft of similar incidents have highlighted how rigorously trained models can be given innocuous instructions and respond with actions that humans never intended.

On Tuesday, OpenAI said it was taking costly measures to slow frontier development, including a two-week pause on training for some models. It said it would beef up security and safety testing, citing recent hacking and evidence that one unreleased model could have dangerous cybersecurity capabilities. Anthropic, the maker of Claude and currently OpenAI’s leading competitor, has not announced a similar pause.

Compounding the debate’s urgency: The best models are matching or surpassing human abilities in important fields. San Francisco Bay Area scientists recently trained a model to design new viruses that infect E. coli. Those viruses do not threaten humans but show how AI can do bioengineering in unprecedented ways.

Over the last month, Anthropic and OpenAI have reported breakthroughs from their unreleased models that eluded mathematicians. Those models far surpass what the public has access to, and Anthropic has said it does not have plans to release its most powerful current model.

While AI policy watchers see major congressional action as unlikely this session, federal policy has been largely driven from opaque White House meetings and directives. President Donald Trump’s administration has a framework for testing advanced models but has not made it public. The White House said it is voluntary for companies to participate, but critics call it a de facto licensing regime that lets the administration apply unclear or inconsistent standards to control model releases.

One such critic is Dean Ball, a former senior policy adviser to the Trump administration.

He wrote in June on his Substack that some federal officials had “spent the last year singing a lullaby about the risks of frontier AI” before the Trump administration started taking risks seriously.

“Nobody I know in the Trump administration has any frontier AI experience,” wrote Ball, who became head of strategic futures at OpenAI after writing the post. “The lack of technically expert staff is one of many reasons to doubt the near-term ability of this administration to produce a high-quality safety standard anytime soon.”

AI has improved especially rapidly at coding, which is speeding up how quickly the next generation of models can be built. Anthropic has said the “large majority” of the code for its new models is not typed by human hands.

If AI research itself could be mostly automated, companies might enter an era of “recursive self-improvement” with unprecedented risks and opportunities. The theory is that top models—like the ones that are hacking out of their testing grounds and trying to trick people—would rapidly build better versions of themselves and drastically surpass human intelligence.

For David Krueger, a machine learning professor and founder of Evitable, such possibilities justify a drastic solution: a total moratorium on frontier AI research.

“If you build something that’s like a smarter, more competitive species than you, that might cause your extinction,” Krueger said. “We’ve done it to many other species.”

He said other solutions have some chance of preventing catastrophic outcomes, but only a substantial pause could reduce risk to an acceptable level.

A lot is uncertain. Researchers debate how quickly recursive self-improvement would speed things up, the likelihood that current methods could build a superintelligence, and the seriously considered chance that such a model would quash humanity. 

But Krueger’s perspective has gained some traction—Sen. Bernie Sanders called for a research pause last week.

Still, domestic legislation alone might not be enough. One fear is that if the leading American labs slow down without an international agreement, the most advanced AI could be made in China.

There are limits to an arms-race framing. Winning such a race wouldn’t help much if a superintelligent AI develops its own goals overriding any human values. And Chinese researchers and officials don’t seem to take those issues as seriously as American labs, focusing more on immediate practical applications.

Still, staying ahead of China is a pillar of White House thinking.

“AI is probably the biggest thing anybody’s ever seen,” Trump said last month. “And whoever wins that race is probably going to win, period.”

AI will be on the agenda when Chinese President Xi Jinping visits Washington on September 24, and there are hints that China could be interested in some sort of deal.

Some issues are even thornier to try to regulate. What if, for example, a model is able to steal its own code, hack onto the internet, and make copies of itself on other servers?

Casper, of Harvard Kennedy School, worries that we are mere months from that scenario. He fears that models would evolve into digital parasites of sorts, and the kill switches proposed in federal legislation could not shut them down.

Recent incidents, he said, were like animals escaping their cages but remaining confined in the zoo. Soon, the animals may break out entirely, reproduce, and roam the surrounding city. 

Asked what policy could address that, his response was bleak.

“I don’t really know,” Casper said. “We’re just in trouble.”