Is democracy indirectly being fundamentally threatened by the rollout of artificial intelligence?

I think the honest answer is yes.

Hello and welcome to GZERO World.

I'm Ian Bremmer.

And today, what does it mean to win the race to create the most powerful technology the world has ever seen?

It has been an exciting and terrifying few months in AI development.

Advanced models are transforming medicine and mathematics.

OpenAI's latest model, GPT-6 Astra, took nearly an octillion calculations to train.

That's a thousand times the number of stars in the known universe.

Granted, maybe still less than the universes that we don't know about.

But all this unrelenting progress, hard to comprehend progress, has also led to growing fears of an AI apocalypse.

There have been a string of worrying incidents over the summer.

Open AI agents escaping a controlled testing environment, Anthropic, says it had to block several accounts of users who were trying to use Claude models to create bioweapons.

And an Anthropic whistleblower claimed the people building AI believe it could kill us all by 2030.

Industry leaders started calling for a slowdown.

Even Bill Gates, one of technology's biggest champions, began sounding the alarm.

Well, there's never been a product that's less understood in terms of what its capabilities are than AI.

This makes nuclear weapons look like nothing.

By the end of September, CEOs from the biggest AI firms had signed a, quote, "morally binding agreement on safety standards," whatever that means.

But it was voluntary, with no penalties for breaking them.

Now, I'm not an AI doomer.

I'm not wired that way, if you will.

But I do think the risks are real.

And without meaningful guardrails, the consequences of reckless development will be enormous.

Americans agree with me.

Nearly three quarters of US adults want the government to slow AI development.

And with that much popular bipartisan support, you would think regulation would be a rare political no-brainer.

So why has Congress shrugged its shoulders?

Three reasons.

Number one, the technology is moving so fast that lawmakers are struggling to understand what it is they're even regulating.

Number two, the sheer amount of money pouring into AI is creating huge pressure to keep building, keep deploying, keep generating returns.

And then, number three, China.

Washington is afraid that slowing down would give Beijing the strategic advantage in a race that will define global power and influence for generations.

As President Trump put it, "Whoever wins AI wins."

But what does it mean to win?

If dangerous, uncontrollable AI is the result, it doesn't really matter who gets there first.

Victory should be about deploying AI safely, efficiently, and in ways that benefit society.

If the United States wants to win, it needs to show the world that powerful AI can be built responsibly, which means getting governance right.

Tech leaders, most tech leaders, do understand this, which is why they've been practically begging for responsible AI rules.

Innovation and safety shouldn't be mutually exclusive.

In a democracy, figuring out how to strike that balance shouldn't be left to the people building AI alone.

Joining me to talk about the stakes of the global AI race, Ben Buchanan.

He's former White House AI advisor and co-author of "The Bitter Struggle-- Superintelligence, Superpowers, and the Fate of the World."

Don't worry, I've also got your puppet regime.

Hello.

- Uh, yes, may I please speak with Lame Donchik Duck?

But first, a word from the folks who help us keep the lights on.

Funding for GZERO World is provided by our lead sponsor, Prologis.

Every day, all over the world, Prologis helps businesses of all sizes lower their carbon footprint and scale their supply chains.

With a portfolio of logistics and real estate and an end-to-end solutions platform addressing the critical initiatives of global logistics today.

Learn more at Prologis.com.

And by Cox Enterprises is proud to support GZERO.

Our people are working to build a brighter future in automotive, greenhouse agriculture, clean tech, environmental conservation and beyond.

Cox Enterprises in the business of what's next since 1898.

Additional funding provided by the Andrew Carnegie Foundation, Koo and Patricia Yuen, committed to bridging cultural differences in our communities, and... - Ben Buchanan, good to have you on the show.

Thanks for having me.

So, you know, you and I have talked a little bit about this whole P-Doom versus P-Utopia, and you are actually closer to the "we're going to have a crisis soon" scale of the spectrum.

Is that just because having spent a lot of time in the White House, that's the way you're wired or do you think, no, actually, this is baked in?

Maybe the White House does make you a little bit doomy, but I think you're right to set the bar at crisis, as opposed to existential apocalypse.

And I think there's pathways in the reasonably near term, in which we could see AI systems start to cause some real or enable some real havoc in our society, probably the most immediate one is cyber.

We've talked a little bit about this in the past.

And the way in which AI can enable offensive cyber operations, I fear, is starting to come online pretty quickly.

I think in the medium run, in the long run on cyber, AI will help us with cyber defense and help us in the first instance, ship more secure code.

But we're going to have, I think, a bumpy transition period that we're just starting to see of AI enabled cyber operations, letting actors do things they wouldn't be able to do otherwise.

How long do you think, I mean, let's first get to the positive side.

How long will it take and what kind of resources will it take to use AI to actually no longer have all of these cyber vulnerabilities?

I think when you're talking about developing new code, that doesn't take that long because we have a development process that's quickly integrating AI, huge amounts of code at these companies is written by AI, probably more secure.

So new digitally native companies will be more secure going forward.

- Than a few years ago?

Yeah, I think so.

And especially once they start unlocking something called formal methods, which is a mathematical way of proving that software doesn't have vulnerabilities in it.

And historically, it's been too cumbersome, too finicky to really work at any kind of scale.

With AI, I think it's possible that we can unlock a lot of that technology in the interim and in the near term.

Where I am worried is all the stuff we've already shipped, and especially things like critical infrastructure and the like, where it takes a very long time to update that code, if it's possible at all.

That's going to be the transition period.

That's going to take billions of dollars and much longer.

When I first saw, I mean, certainly the OpenAI Hugging Face incident, where it escaped the sandbox and training and breached this external source, and the way that it was trying to cover up for its track, sacrifice agents, knew that it was doing things that was not being directed in the programming.

That to me, kind of felt like, let's ring the bell, this is a crisis.

Did you have the same view?

- Yes, and I would break that case into animating two areas of concern.

The first is it shows the cyber capabilities of AI systems, it was breaking into, the agents, were breaking into a reasonably well defended company, having good effects, all that.

The second is the agents were doing it on their own.

And this will be called misalignment.

And this is another path to worry.

So cyber is one path to worry.

misalignment is another the AI systems themselves can not because the human is intentionally misusing them, but because they have misunderstood their training objective or any number of other factors can take actions we do not want them to take.

And Hugging Face is the best exemplar of misalignment risk.

As AI systems get more powerful, we should expect that the dangers of misalignment, the dangers of the system going rogue, if you'd like, are much higher than they are today.

So there's more risk embedded in that.

So when open AI, and certainly not only open AI, lots of all of the American hyperscalers and the Chinese as well.

When they see that their own products, right, the things that they are programming, they are responsible for, have these capabilities that they did not intend, why are they continuing to ship them?

Are they not aware?

Do they underprice the legal risk that they're taking on?

It's hard to know the thinking inside.

I don't know the legal risk of who's liable for what.

I think the courts haven't worked it out.

But I think the underlying question is a really good one, which is why continue to do this?

What the companies have said, and OpenAI did say this over the summer, is they're trying to slow down relative to where they were and so forth.

I do think there's a role for the government in making sure that these companies are held, whether it's through liability or through regulation, to a very high standard of operational excellence.

And to say, if you're going to be in this business of training very powerful AI systems, we as a society need to make sure you're doing it in a way that's safe and secure and trustworthy.

And certainly my posture in the government and my posture today is that that is not something that can be left to the companies.

We need actual regulation in the same way we have regulations for many other high stakes fields from aviation to drugs to power.

Doesn't need to be the same type of regulation, but the idea that we would have nothing strikes me as foolish, given the stakes here.

We saw all the CEOs last collect in the White House and they signed something that President Trump described as a moral responsibility.

That doesn't strike me as having binding or legal force.

Is it fair to say that as of right now, those regulations, that governance does not exist?

There are no binding regulations on the books on frontier AI development.

Trump AI Accord, as he calls it, or superintelligence accord, as he calls it, is morally binding, but I'm not sure morals bind markets.

And I think we need to have more from the government.

And certainly my posture in the government was to have regulations in place, the beginnings of a more expansive regulatory structure that would constrain what the companies are doing and try to manage this risk a little bit better.

Ultimately, part of the reason why I think liability is not enough is if these incidents happen again, if they happen in a worse way, the consequences will be felt by all of us.

And I think we all have a stake in making sure we get this right.

- But let's talk about what specifically these near-term risks around cyber are, because I mean, Hugging Face is pretty abstract for most people.

Having a vulnerability found, which 80% of the known internet actually functions on, that appears like everyone might not have internet access for a while.

Is that what we're talking about?

Like, what are the actual, when you think about the kind of real cyber crisis that could be felt literally tomorrow with what's already shipped, the present vulnerabilities and the capabilities of just the models that presently exist, what's the risk there?

- Let's take the one that you're alluding to here.

So, PHP is a well-known internet scripting language that's been around for several decades.

OpenAI using their systems appears to have found a vulnerability in the underlying language itself, meaning that if you as a web developer are using PHP, you can have the most secure practices in the world, but the mere fact that you are using this now broken language seems to make you vulnerable.

And PHP is used, according to the National Security Agency, 70%, 80% hardship over here, something like that of sites on the internet.

So a really big percentage of the internet was relying on this technology, is relying on this technology that OpenAI finds is vulnerable.

In response to this, the National Security Agency, usually a pretty quiet organization, kind of puts out a DEF CON 1 call saying, fix your stuff, take your sites offline, some government sites go offline.

This is not hypothetical.

These are the kinds of things that were hypothetical in 2022, when we were debating them in the White House, but they're not hypothetical now.

And that's an example, I think, of where we are right now with AI finding vulnerabilities that are really widespread across the internet.

The next level of the game is can AI find vulnerabilities that are not just used in websites, but are also used in critical infrastructure, and whether that's power or water or the financial system.

And it does seem like there's an increasing level of concern from sophisticated organizations like big banks saying, "We're really not sure what these systems are going to turn up, and we've got to get ahead of this."

When I see the alignment problem and the CEOs talking about the fact that within a year, most of them seem to truly believe that at the frontier, it will no longer be humans.

Humans will be out of the loop.

At the frontier, the research will be led by artificial intelligence.

And that drives a very different kind of risk, right?

That's not just alignment.

That's what they call recursive self-improvement of the models, where the people that own them don't know what they're doing anymore.

First of all, do you have an ability, do you have a sense of whether or not that this is being hyped or this is accurate?

- I think this is accurate.

And certainly when I say this is accurate, the capacity for AI systems to help generate or help create more powerful successors, I think it's already demonstrated.

If you look at the writing of the code for the AI systems, three or four years ago, essentially all the code AI companies was written by humans.

Right now, essentially none of it is, maybe 10% of it is.

So the code itself is being written by AI.

Now you might say, well, humans are still directing that.

The real question is, can AI do research?

And in February of 2026, the answer emphatically was no.

Anthropic tracks these things very carefully.

Anthropic showed that less than 1% of the research tasks at Anthropic were led by Claude.

August of 2026, just eight months later, 26%.

So from 1% to 26% in eight months.

As with most things in AI, that trend line is not going to stop.

It may even accelerate.

So what you are moving towards, what we are all moving towards as a society is a world in which these AI companies are using AI to create more powerful systems, which is an additional accelerant in the overall capability of AI systems that are coming online.

And we should expect that to show up in terms of raw capability in 27 and 28.

Now, to what degree is that risky is an interesting question.

I think it compounds the risk in two ways.

The first is it brings more capabilities online faster than would otherwise be the case, meaning that we as a society have to confront more good and ill AI capability than would be the case without this loop.

And then the second, as you were alluding to, is it does create more difficulty in looking inside the AI systems themselves.

Can the companies understand their creations, which are already fairly impenetrable to begin with?

And there's real risk there.

So if you are, again, the companies are not going to be able or willing to regulate this themselves.

It's not what they're created to do, and they're not incentivized because they've got a collective action problem.

And they don't trust each other.

They don't like each other.

The leaders of these companies are like much less aligned than even the leaders of, say, the G7 nations, right?

So if we accept that, what does the government have to, because the cyber issue at least feels like you throw resources at it.

You make it urgent.

You understand what the nature of the risk and the fix is.

This one is harder, right?

So if you were sitting inside the White House right now and in control of technology policy, how would you start to address the humans outside of the loop problem?

- A couple of points.

I think the first is there needs to be some degree of mandatory testing and evaluation before systems are released to the public.

And we were setting up for this when I was in the Biden administration with the Biden executive order required the company to disclose to the government their safety tests.

The second is, there needs to be some regulation of the company's internal deployment and development of models.

What's so striking about the Open AI Hugging Face case is that was not an externally deployed system.

That accident happened during training.

So it's almost like the airplane had its incident on the ground, but it was still significant because of the way AI works.

So regulation before release, regulation of the development process.

And then the one that I think is increasingly more and more important is this question of what we call pacing the frontier.

Is there some way... Which means slowing down.

Slowing down.

And I fully believe that the posture for the United States should be to build a big lead over China, so that we are not in an arms race with China on AI.

And then the phrase we used to use in the White House was spend that lead on safety.

Slow down more than you otherwise would to make sure you're doing the technological development in a way that's safe and secure and trustworthy.

But that slowdown for safety is something that functionally needs to happen now.

This can't wait for a next administration.

- That's right.

Okay, but when I see the nature of both the CEOs who are all over the map in terms of their position on this issue, and the administration, which is saying both feet on the accelerator because this is the one thing driving the economy, it appears that is not going to happen.

I don't think it's likely.

Again, you do see a range of companies saying different things.

The only mechanism that would cause this to happen is the politics of AI are changing.

In the government, it was very clear in the Biden administration, the public did not really engage with AI relative to other issues.

The salience of AI politically has vaulted up amongst other issues.

That is certainly true.

But it's mostly being articulated in things that we aren't talking about.

It's data centers, it's jobs, it's not pacing the front.

I think that is true.

I think as these systems continue to get better, that will change.

Now, when you talk to the Chinese, number one, they don't seem to have much interest in getting involved with the United States in a pacing conversation because their view is you guys want to slow down, fine, we'll just catch up.

But secondly, they believe that they have less of a security and a vulnerability challenge because at the end of the day, the state's in charge, they know what the companies are actually doing, and they have a closed system.

They're in charge of their internet.

They know what their people do and don't do.

How convinced are you by that argument?

And how much does that change the nature of arms race dynamics between the two countries?

I'm convinced they believe it.

And certainly that was my experience with the Chinese was when we pushed the Chinese very hard.

President Biden raised us with President Xi.

We had a secret meeting in Geneva.

I was there.

We talked at deep level with the Chinese government about these risks, and they did not take them seriously.

And their posture was this is American imperialism.

Any discussion of slowing down is locking in American lead, all the rhetoric.

We made a little progress after that, but not very much.

So I think it is genuinely at least the position that their political apparatus holds.

Whether or not it's right, I'm not sure.

And in fact, I think they may in some ways be, because of their experience with the Great Firewall, overconfident in their ability to clamp down on AI systems.

It is worth noting they have started just this fall to at least suggest they want to do more regulation of AI.

What that looks like is a very open question.

Of course, their top priority is regime survival, so it will have a lot more on censorship than the equivalent American regulation would.

And a lot of the models that are being used for cyber operations are Chinese models without terribly strong safeguards.

How aligned do you think the development of AI is or is not with an open versus a closed system?

Is democracy indirectly being fundamentally threatened by the rollout of artificial intelligence?

I think the honest answer is yes.

But the good news is democracy has more cards to play, and in particular, the United States has more cards to play than anyone else.

Why is the answer yes first?

So the answer is yes, because this gives autocracies the opportunity to leapfrog ahead of democracies.

In some ways, there's a mechanism through which AI could enable tyranny, surveillance, crackdowns at home.

That's what autocracies can do to solve some of their problems.

Democracies could slide into that.

And we have to make sure we develop these systems and deploy systems in a way that align with our values and our police-- Because AI easily facilitates that.

AI absolutely facilitates that.

So I think it poses, maybe not a threat, but it certainly poses a series of challenges and questions, as many technologies do and have throughout history, to our democratic way of governance.

And that's a capability that leaders might like to engage with.

That's right.

But, and one dimension as well that makes this particularly challenging for the US government is that AI, I argue, is the first revolutionary technology, at least for the last hundred or so years, that is not funded by the US government.

And you think about nukes and space, the early days of the internet, early days of jet aviation, radar, GPS, all of that, at its core, kind of had its roots in Department of Defense money.

And what that meant is the US government had the capacity to understand what was going on, because some program manager was cutting the checks, and also had some capacity to shape it.

Now, AI actually is that way, if you go back to the 1980s.

But the current paradigm of AI since 2012, scaling of deep learning is really a private sector thing.

And it was acutely obvious to me as the president's AI advisor that democracies had fewer cards to play in understanding and shaping AI than we did in the government in almost any other area.

So that is, in some sense, I think, the core through line for every AI policy decision is, what is the relationship between the public and private sector?

And embedded in that relationship question are both challenges and opportunities.

What are the muscles that the United States and other democracies need to exercise in order to best ensure the resilience of democratic institutions?

My co-author and I have called this the bitter struggle because we think it has three parts.

And the first is the United States and its allies and partners need to have an advantage in invention.

This making the AI and gaining a geopolitical advantage through making AI.

The good news here is the lifeblood of AI invention is computing power.

That we have what's called the scaling law.

The more computing power it's used to train an AI system, the more powerful the system that results.

- And the chip advantage matters for that.

- And the chip advantage.

And that's why our big action in the Biden administration was to deny China access to advanced chips and chip making equipment.

And that has built the lead for the United States in AI.

And the more effective we are at enforcing those rules, Trump has backslid a lot, the more effective we'll be at building a lead.

So the first part of the bitter struggle, as we call it, is get the advantage for the United States.

The second part, where we're less good, is take that advantage and apply it to our military and our intelligence community and saying, "Can we take this technology and use it for things like American cyber operations on offense and defense?

Can we use it for intelligence analysis with guardrails, but also really taking this and using it to compete in the dog-eat-dog world of geopolitics?"

And then the third and maybe the most challenging one, where I'm less confident we have an advantage, is lining up with our values and saying can we adopt AI, can we develop AI in a way that doesn't concentrate power too much in corporations or doesn't concentrate power too much in the state that guards against misuse by individuals but also protects our civil liberties and this poses a real set of challenges I think to the democratic way of life.

There's ways to manage it and we think it's possible but that is I think the third and most vexing part of the challenge facing American policymakers.

Ben Buchanan, thanks for joining us.

Thanks so much for having me.

Now let's hear from our friends who are just as artificial but far less intelligent.

It's time for Puppet Regime.

Hello.

- Yes, may I please speak with lame, donchik duck?

Yeah, yeah, yeah, very funny Vladimir.

No, but seriously, those midterms were a disaster.

Yeah, maybe.

Oh, come on, I haven't seen crushing defeat like that since Yaroslav the Wise lost Battle of Lisztven in year 1020... Ah, shut up.

What do you even know about elections anyway?

Ah, rule number one.

Don't hold them unless you will win them.

Easy for you to say.

Yes, actually it is.

But anyway, look on bright side, Donchik.

Life as a lame duck can be great.

Oh, yeah?

Yeah, now you don't have to worry anymore about voters, about elections.

You can just live your best life, manifest.

Yeah, I guess I could just let Steve Miller run America and come hang out with you and Xi and the fellas.

You know where to find us, big boy.

I could become Generalissimo of Cuba.

No one is stopping you.

I could... You could bomb Iran with me again.

I sure could.

How did he get on the call?

Anyway, look, Vladimir, while we're in this very strong new phase, you want to end the war in Ukraine?

Maybe.

- Really?

Really?

- Of course not.

Don't be ridiculous.

I'm just being supportive.

Alright, you want to help me pick some curtains for the ballroom?

Ooh.

That's our show this week.

Come back next week if you like what you've seen, or even if you don't.

You yourself are a super intelligent robot ready to take over the world.

I hope you'll be a merciful overlord and come check us out at GZEROmedia.com.

[music] Funding for GZERO World is provided by our lead sponsor, Prologis.

Every day, all over the world, Prologis helps businesses of all sizes lower their carbon footprint and scale their supply chains.

With a portfolio of logistics and real estate and an end-to-end solutions platform addressing the critical initiatives of global logistics today.

Learn more at Prologis.com.

- And by Cox Enterprises is proud to support GZERO.

Our people are working to build a brighter future in automotive, greenhouse agriculture, clean tech, environmental conservation and beyond.

Cox Enterprises, in the business of what's next since 1898.

Additional funding provided by the Andrew Carnegie Foundation, Koo and Patricia Yuen, committed to bridging cultural differences in our communities, and... [music playing]