Does it really matter if there’s an intelligence explosion?

Vernor Vinge is possibly my favorite science fiction author of all time. The Singularity, which he famously predicted would happen somewhere between 2005 and 2030, features heavily in most of his stories. It usually happens the same way: an Intelligence Explosion, of the type envisioned by I.J. Good — the smarter AI gets, the faster it’s able to make itself even smarter, meaning that AI becomes godlike nearly instantaneously once it reaches a certain threshold of capability.

Almost every discussion about AI safety that I hear in San Francisco or read in the tech blogosphere centers around this sort of event. The discussions are all about whether recursive self-improvement (RSI) is near, and whether we should “pace the frontier” — i.e., deliberately slow down or pause AI progress — to stop RSI from creating an intelligence explosion.

As I’ve listened to these arguments, I’ve grown increasingly skeptical that this is the right discussion we should be having. But before I explain why, let me quickly summarize the debate itself, to give an idea of what the arguments look like.

In fact, I’ve published some pieces of that discussion on this blog. Back in September, Ramez Naam wrote a guest post arguing that an intelligence explosion is not imminent:

Where’s the “intelligence explosion”?

Where’s the “intelligence explosion”?

His basic arguments were:

  • OpenAI data show that AI is still only able to automate short chunks of the AI research process.

  • AI research is subject to diminishing returns, so that each increase in AI capability should buy less of a speedup in the pace of automated research.

  • Diminishing returns also mean that accelerating AI progress will require exponentially more of other resources, such as compute and money.

  • There’s no sign yet of an acceleration in the rate of AI progress.

In a follow-up post, Ramez did a deeper dive on the topic of agent swarms, arguing that they’re not as effective as many people believe.

Theo Jaffee echoed these arguments, noting that OpenAI’s data shows that AI hasn’t even been able to fully automate smaller research tasks yet:

Luis Garicano and Alex Imas also wrote a post downplaying the likelihood of an intelligence explosion:

Are we approaching an intelligence explosion?

(Dear reader: this is an unusual post; it gets a bit technical. Alex and I are trying to do something hard here: look at the data and try to understand what they tell us about the intelligence explosion. There is math, Greek letters, and two models. So please bear with us, and if you are bored, apologies. We think it is worth making an exception here…

Read more

7 days ago · 65 likes · 13 comments · Luis Garicano and Alex Imas

Their arguments are more economic in nature. They show that although AI progress has accelerated, it also shows diminishing returns in terms of research spending, meaning that every unit of progress has come with a higher price tag:

They also estimate an economic model of RSI, and conclude that ideas in AI research are probably getting harder to find, reducing the likelihood of an intelligence explosion.

On the other side of the debate, we have the recent METR report about Anthropic’s internal progress in automating AI research. The report uses Anthropic’s own internal measure of AI capability, AECI, rather than the better-known public ECI. It also relies on data that aren’t public, meaning that deciding how much of the report to believe requires a lot of guesswork and assumption-making. But the basic story METR tells is that Claude is already speeding up Anthropic’s research process by a lot:

As we describe below, we were able to confidently determine that the acceleration [of AI progress in late 2025] was not due to an acceleration in the rate of growth of training compute…We guess that a majority (perhaps ≈80%) of the current…acceleration is due to AI.

Dwarkesh Patel, meanwhile, argues that AI research isn’t that different from math, and that the same factors that made AI suddenly go from useless to godlike at math will happen soon with AI’s own self-improvement:

The avalanche that hit math is coming for AI development

Read more

7 days ago · 547 likes · 103 comments · Dwarkesh Patel

He writes:

It’s crazy how fast we went from, in 2023, “This model can’t reliably add four-digit numbers,” to…about a month ago, “Okay, it fucking solved a Millennium Prize problem in 88 hours.”…

AI research is actually surprisingly similar to the type of mathematical progress that current AIs are disproportionately good at: unlocking a ton of low-hanging fruit by applying a combination of incredible persistence and a superhuman capacity to connect ideas from lots of adjacent domains…In every domain where we can check answers and give AIs thousands of rounds of practice—for example, competitive programming contests, or well-scoped open math problems—they’ve gone from worse than most humans to better than the best humans within a few years…AI research has those same verifiable properties…

Once AI is as good as the best humans at AI research, algorithmic progress will get radically faster…Today’s AIs already do work at about 10 times human speed. And automated researchers would also be working around the clock, nights and weekends included. As a result, each individual automated researcher will be cramming in roughly 50x more work every calendar week than a human researcher can do…During the intelligence explosion, automated engineers will build tens of thousands of times more [reinforcement learning] training environments than we have currently.

So that’s the debate. Basically, it depends on whether ideas get harder to find faster than AI gets better at finding ideas, and only time will tell which one wins.

But as I said, while this debate is intellectually interesting, I’m not sure how much it matters.

Whether an intelligence explosion happens is obviously very important. A sudden unfathomable leap in AI capabilities to godlike levels would be, to put it mildly, a very big deal. But that doesn’t mean an intelligence explosion is the thing we should be thinking and arguing about.

First of all, if an intelligence explosion is possible, it’s going to happen, and there’s probably just not much we can do to affect how it all shakes out.

Although people disagree about AI’s contribution to its own uplift, everyone agrees that AI capabilities are improving at a steady, rapid rate. Here’s Epoch AI’s ECI number, the most common measure of AI capabilities:

Most discussions about AI safety policy revolve around the idea of “pacing the frontier” — i.e., using government policy and AI labs’ own decisions to bend that trendline of progress downward. The usual reason given is that this will allow alignment research time to catch up, so that more powerful AI is safe by the time it arrives.

But this approach simply can’t deal with an intelligence explosion. If there’s some threshold at which AI gets good enough to start an intelligence explosion, that threshold will be reached. Even if we “paced the frontier” and slowed the rate of AI progress down by half, that would mean we’d hit the intelligence explosion in 2037 instead of 2032, or 2030 instead of 2028, or something like that. It would only buy us a few extra years.

I guess two years or five years isn’t nothing — we could use it to hang out with our loved ones and try some cool new restaurants while we waited for the arrival of the computer gods. But ultimately, it seems like small potatoes, especially if you’re an AI safety type who thinks an intelligence explosion would be a catastrophic event for humanity.

If an intelligence explosion is possible, the only way to prevent it is to halt AI progress completely, rather than just slow it down. In 1993, Vernor Vinge made a powerful argument as to why that’s impossible:

But if the technological Singularity can happen, it will. Even if all the governments of the world were to understand the “threat” and be in deadly fear of it, progress toward the goal would continue. In fiction, there have been stories of laws passed forbidding the construction of “a machine in the form of the mind of man”. In fact, the competitive advantage — economic, military, even artistic of every advance in automation is so compelling that passing laws, or having customs, that forbid such things merely assures that someone else will get them first.

I’m a little more optimistic than Vinge here — I think we could engineer an AI slowdown if we got sufficiently scared — but I don’t for one minute believe we could engineer a complete halt.

Nor do I believe that alignment research will be able to make an intelligence explosion safe. “Pacing the frontier” so that alignment research can catch up might be effective in a world of steady linear AI progress. But in a world of sudden superexponential progress, AI alignment research will immediately be left behind. Even if there is a Singularity for understanding alignment, in a world with an intelligence explosion it will be the superintelligent AI, not us, that carries out that Singularity and decides on its alignment for itself.

A second point is that it’s not obvious why an intelligence explosion would be more dangerous than — or even distinguishable from — the type of rapid, non-explosive progress we’re already getting. Remember, AI going from “can’t add two numbers together” to “solves problems too hard for all human mathematicians combined” in three years is not counted as an intelligence explosion!

It’s clear that at the current rate of steady progress, AI will become superhuman in most domains in a fairly small number of years. Even robotics and other physical tasks look conquerable with the current type of AI that powers Claude and GPT. This means that AI will basically look like a god within our lifetimes, whether or not it looks like the kind of infinitely self-improving gods that populate Vernor Vinge novels.

This is why I’m not sure what an intelligence explosion means for our economy. It’s easy to write down a model in which smoothly increasing AI capabilities drive a smooth increase in GDP. But why should we believe a model like that? Other technologies haven’t really worked that way.

Usually, technologies reach some threshold of usefulness where adopting them makes economic sense. When they hit that threshold, their local contribution to productivity growth becomes very rapid, but also finishes fairly quickly. Fiszbein et al. (2024) shows this for electrification, Bartel (2005) shows it for IT-enabled machines. Each new invention creates a whole bunch of little local Singularities. It’s only when you add all these up across the economy that you get something that looks like a big smooth wave of growth.

A suddenly superintelligent AI isn’t really limited by locality — it could, essentially by definition, take over all or most human tasks if it wanted to. Whereas if an intelligence explosion doesn’t occur, these takeovers are likely to be staggered and irregular. But they will still happen, and the economic effect after the first few years probably won’t be that different.

In other words, I think:

  • The AI we’ll get from our current trend of improvement will look about as godlike — and as economically important — as an AI that suddenly bootstraps itself to godhood.

  • It’s not possible to stop AI improvement completely, and so if an intelligence explosion is possible, it’s going to happen to us soon.

  • Slowing down the rate of AI progress will not help us deal with post-Singularity superintelligence.

This is why I think this whole debate is misplaced. We need to be thinking about how to deal with the AI we already have, and the one we’ll almost certainly get soon — an AI that’s smarter than all humans at most economically important tasks. We need to think about how to regulate that actually existing superintelligence, how to prevent AI-enabled bioterror, how to make sure the economic benefits of AI are widely shared, and so on. We need to think about competition policy, to make sure the benefits of AI don’t exclusively go to one company.

I think we need to accept that nobody is “pacing the frontier” enough to prevent the emergence of superintelligence — and that nobody is likely to do so. If the machine god awakes, then it awakes, and we’ll simply deal with it. In the meantime, we need to focus on dealing with the AI we have now.

Share

Source link

Tagged:

Leave a Reply

Your email address will not be published. Required fields are marked *