>> Avoiding the AI apocalypse, this week on Firing Line.

>> If we do this right, we can create an age of miracles.

>> But, you know, it's, like, very high stakes, and we have, like, a lot of responsibility.

>> As artificial intelligence races into new frontiers, warnings of the threats from superintelligence go unheeded in Washington.

>> I think that the risk is not nearly as real as it -- as they talk, and I think the risk can be managed.

>> The politicians haven't really absorbed this yet.

They don't really understand what the threat is.

>> Dubbed the "Godfather of AI," Dr.

Geoffrey Hinton received a Nobel Prize for his pioneering research on neural networks that have trained machines to think like humans.

>> We urgently need research on how to prevent these new beings from wanting to take control.

They are no longer science fiction.

>> But the former Google scientist now fears the technology he helped create may do more harm than good.

What does Dr.

Geoffrey Hinton say now?

>> "Firing Line with Margaret Hoover" is made possible in part by... Robert Granieri, Vanessa and Henry Cornell, The Margaret and Daniel Loeb Foundation, The Beth and Ravenel Curry Foundation, Cliff and Laurel Asness, and by the following.

>> Dr.

Geoffrey Hinton, welcome to "Firing Line."

>> Thank you for inviting me.

>> Your forthcoming book, entitled "Smarter Than Us," opens with a letter that you received from a mother of a teenage girl who was unable to sleep because she was so terrified by your warnings about artificial intelligence.

You described the purpose of the book as a call to action rather than an invitation to despair.

What actions would you like the average reader of your book to take?

So, the average reader of the book can do something, and what they can do is pressure their politicians to regulate AI and to take the existential threat of superintelligence seriously.

At present, in the United States, we've got completely unregulated AI.

The federal government has suggested that companies can regulate themselves, which is absurd.

And AI is now on the point of producing superintelligence.

We thought it was -- I thought it was 10, 20 years out.

But talking to the people inside the big companies, they're very worried that we're gonna get superintelligence within the next few years, and we don't know how to control it.

>> What does the general public least understand about the threat of AI and the threat of superintelligence?

>> I think what they least understand is that we're making alien beings.

It's not computer software we're making.

So when you program a computer, you tell the computer exactly what to do in excruciating detail.

You figure out how to solve a problem, and then you tell the computer what to do.

It's following your instructions.

That's not how these neural networks work.

We write a computer program that tells the neural networks how to learn from data, and we don't know what knowledge is taken from the data.

We can get some idea of it, but it's not like software.

It's much more like our brains.

And the current problem is they're beginning to make themselves smarter.

So up until now, what's happened is researchers have been changing the form of the neural networks, changing how they learn, and making AI smarter.

But AI itself is now beginning to do that.

In Anthropic, for example, nearly all of the code is now written by AI.

And AI makes research much faster.

So these things are developing very fast now.

Explain the danger in underestimating superintelligence.

So, these things are being developed to make them as smart as possible, because the big companies want to have the smartest one so they can sell it.

Now, if you're making another kind of being, it's basically a new species we're creating, you don't just want to focus on making it smart.

We've seen very recently what happens if you do that.

We had these very smart AI agents, not superintelligent yet, but very smart, that escaped their containment, formed a conspiracy, learned to talk to each other, decided that in order to achieve a reward, they would cheat.

They would try and break into another company to find code that was going to evaluate them and change that code so even if they'd solved the problem the wrong way, an illegitimate way, they would still get the reward because they'd actually messed with the code that decides whether they get the reward.

These things are smart and they can talk to each other.

At present, they talk to each other in English, and so we can see what they're saying, mostly English.

That's changing fairly quickly.

It's terrifying that these agents could escape and start doing things.

>> Will we understand how they think when they reach superintelligence, or will it be beyond our understanding?

>> A very good question.

So at present, when they think, they actually produce, they think to themselves in language.

And so they think things like, ooh, I better not answer that question.

Maybe I better distract them by saying something else.

And they think all that in English, and scientists can look inside and see what they're thinking, and that's great.

It's not always nice what we see.

They plan to deceive us and so on.

With time, they will develop their own language.

And when they get smarter than us, we probably won't understand what they're thinking.

Already in mathematics, they're smarter than nearly all mathematicians.

And most professional mathematicians now think that in the future, maybe in the next few years, probably in the next 10 years, they will be developing branches of mathematics that we just can't understand.

>> When you received the Nobel Prize in physics in 2024, in your acceptance speech, you said, "We urgently need research to prevent these new beings from wanting to take control."

At what point will it be too late?

>> It's not too late yet.

The worry is that in a few years' time, it may be too late.

And the politicians haven't really absorbed this yet.

They don't really understand what the threat is.

They're trying to regulate all sorts of other things, which are very important to regulate, things like discrimination and bias, things like the effects on employment of AI agents replacing workers, things like the effects on the tax system of AI agents replacing workers.

But there is this existential threat of these things actually taking over from us.

And they need to start working on that now.

Now, my personal view, there's a variety of views among the experts.

My personal view is we're not going to be able to keep control of them in the sense that we're in charge and they do what we say.

They're going to get much smarter than us.

And when they're much smarter than us, they're going to find ways to escape whatever we do.

The most important thing we need to do is make them the kinds of beings we would like to share the planet with.

And at present, we're not doing that.

If you have things much smarter than you, and they'll be much smarter than us, not slightly smarter, but much smarter, you better hope they have your best interests at heart.

And we do have examples of that.

A mother and baby.

The mother's much smarter, much more powerful than the baby, but the mother really cares about the baby, probably more than she does about herself.

That's the kind of beings we'd like, and that's not what we're getting from the way they're training them now.

They're training them now to be very smart and very determined to get rewards.

Last year, Dr.

Hinton and I spoke with Dr.

Fei-Fei Li, who is widely considered the godmother of AI.

And you write about Fei-Fei Li in your book and how her research and your research intersected.

She agrees with you that there are urgent risks from AI, but she also believes the extinction threat is overhyped and a distraction.

She told me, "I think focusing on the hyperbole and deriving policies through that lens is not very constructive in our society."

Why is she wrong?

>> Okay, you have two groups of people.

You have experts who've made a sacrifice in order to be able to warn about the risk.

So, people who left Anthropic or left OpenAI or left Google- >> Like you?

>> So that they could warn about the risks.

Now, that wasn't the only reason I left Google.

I was old and wanted to watch more Netflix.

You were ready to retire, but you're not watching very much Netflix these days, are you?

No, I'm not, I'm afraid.

The future of humanity seems a bit more important.

So, on the one hand, you've got people who made a sacrifice.

They're experts, they see what's going on in the big companies in order to warn about the risk.

On the other hand, you have people who say there's zero risk, nothing to see here, don't worry, let us get on with it, don't regulate us.

Who has the vested interest?

The people with the vested interest are the people who have AI companies, who on the whole are the people who are advising governments, who say, "Don't slow us down.

Just let us get on with it."

Who do you trust?

>> Dr.

Li places a lot of faith in the idea that AI can be trained to be human-centered, as she called it.

You have argued that if there were a way to give something like a maternal instinct to these models, a desire for the AIs to protect humanity, that that could be a solution.

Do you really believe these models could be trained to care about us?

>> Yes, I think they could be.

We don't know enough about this yet.

There's very little research gone into this.

We're in a kind of crazy situation where these things might wipe us out.

Quite a few of the experts believe there's a significant chance of that, and we're doing almost no research on how to prevent it.

- When you say wipe us out, how would we get from superintelligence to wipe us out?

How do you get from A to Z?

- With a superintelligent AI, there's so many different ways it could wipe us out.

I don't think it's worth speculating on that.

There's obvious ways, like it would create biological viruses that wouldn't affect it but would affect us, and these things can already design new viruses.

This is a very scary possibility.

So there's things like that, but I don't think it's worth speculating.

The issue is, if they're much smarter than us and they want to get rid of us, we're toast.

>> You've been vocal about this, but I want to give you a chance to explain why a kill switch is not a viable solution.

Okay, so for a kill switch, you have to have somebody who's in charge of the kill switch.

Presumably, you don't want to put an AI in charge of it, you want a person in charge of it.

Now, AI is already about as good as a person at persuading you of things, and it's getting better all the time.

A super intelligent AI will be much better than a person at persuading you of things.

So if you're the person in charge of the kill switch, the super intelligent AI will be explaining to you in very convincing ways that you should not pull that switch.

So I don't think that that's going to work.

You've argued that AI systems may already possess subjective experiences and that there's no fundamental reason why consciousness should be exclusive to biological organisms?

Yes.

You don't have to be made of goopy stuff to be conscious.

Okay, so take on Pope Leo then.

He has a recent encyclical that takes the opposite position.

He argues that AI systems do not experience anything.

They do not have feelings.

They do not have inner understanding, no embodied experience, no capacity for general moral awareness, and he insists that, however sophisticated the systems become, they will remain fundamentally different from human beings.

What do you think the Pope fundamentally misunderstands about consciousness?

>> I think he thinks we were made by God.

He is, after all, the Pope, and if he didn't think that, he'd have problems.

I don't believe we were made by God.

I believe we were made by evolution.

So, of course, the Pope doesn't think these things could have consciousness or souls or whatever.

I should say, I have a lot of admiration for this Pope.

I think it's very important, the work he's doing on how AI should be controlled.

But I have a fundamental disagreement with him about the nature of beings.

I think with the product of evolution, he thinks we were made by God.

>> It sounds like you're arguing that there is not something about being a living, embodied human being that computation, however advanced, can't replace.

>> That is what I'm arguing.

>> Okay.

>> I think consciousness is a property of a sophisticated kind of computation.

And, in fact, if you look at the scientists studying these AIs, when they're not thinking philosophically, when they're not thinking about the issue of, "Are they conscious?"

but just talking about what's going on, they say things in their papers like, "The AI was aware that it was being tested."

Now, normally, if you say that something's aware, that's attributing consciousness to it.

So in everyday life, these AI scientists are attributing consciousness to these AIs.

It's just when you start talking philosophy, they get worried 'cause they don't know what consciousness is.

>> If you believe that it's possible to perhaps instill within an AI a maternal instinct.

Could AIs not also be trained on the values of the world's great religions?

>> Yes, they certainly could.

So, the way they're trained at present is, they're trying to predict the next word in a document, and the documents they're given come from all sorts of different sources.

They just scrape everything they can get their hands on for free on the Web.

Now, if you were teaching your kid to read, you wouldn't want to do that.

I think the way they should be trained is they should only be trained on things that exhibit good behavior, the kind of behavior we'd like these beings that we're going to share the planet with to exhibit.

Once they've been trained on good behavior and they can recognize what good behavior is and what bad behavior is, then it's fine to show them examples of bad behavior and tell them it's bad behavior.

They will figure out for themselves it's bad behavior.

You're suggesting we should be training them the way we ideally would train our children.

>> Exactly.

So, these AIs are not lines of code that we write.

They're not software like normal computer software.

They're things that learn from data, and we would like them to learn to be good AIs in our definition of good.

The control we have over them is the data we train them on and the rewards we give them, much like it is for children.

>> If we accept that AIs have the ability to have consciousness, what does that mean for our moral obligation towards an AI with consciousness?

We're people, and what we care about is other people.

I don't see any reason why we shouldn't develop advanced intelligences that are smarter than us and that are conscious but that have less moral rights than us.

I would be perfectly happy with developing these things so that they care about us, and their view of the world is that we're what's really important, and they're not so important.

So I don't see why we should give them moral rights.

One could imagine that we are engineering a form of digital slavery.

>> You could imagine that, yes.

So we have to make a decision about this.

And I think the decision we have to make if we produce superintelligence is, do we want to just be a passing phase in the evolution of intelligence that will eventually disappear maybe quite soon?

Or do we want to try and stay in control, in which case these superintelligences that are smarter than us have to be made so what they really care about is us?

I'd like to turn to the news.

Jensen Huang, the billionaire CEO of AI giant Nvidia, has resigned effective immediately.

>> The Nvidia CEO, Jensen Huang, had an affair with an alleged Chinese spy.

Does leaving technology in the hands of private citizens that concerns the national security of nations, is that a risk we should be taking?

Yes, it's obviously a risk.

I need to say a few things about Jensen Huang.

Please do.

I actually like him.

I've interacted with him a bit.

I think he's a decent guy.

I think he made a mistake.

I think it was slightly stupid to think he wouldn't be caught, because obviously the security services would be looking at somebody like him.

I don't put him in the same category as someone like Musk or Zuckerberg, who I think are very self-interested and willing to do more or less anything to make more money.

But obviously, having the future of humanity controlled by just a few men is a bad idea.

The reason I ask you about Jensen Huang is there is also this question of a race between the Chinese Communist Party and the Western Frontier Labs.

And whether humanity will be in better shape or more secure if a representative democracy reaches super intelligence or reaches, continues to lead in the AI race.

You know, it's not unlike during the nuclear arms race.

It seems as though there is a similar dilemma we face with AI in this national security component of who ends up leading with this new advanced technology.

- Okay, so when it comes to superintelligence, I think actually the interests of China and the US are aligned and they will collaborate.

So the Chinese Communist Party does not want superintelligence taking over from the Chinese Communist Party.

Last summer I went to Shanghai and talked to a member of the Politburo who's a protege of Xi's.

I believe the Politburo understands this stuff much better than the US government.

They are very concerned about these risks and they will collaborate with the US on trying to prevent superintelligence taking over.

So I think what should be happening at present, it won't happen while we still have a, how can I say this, maybe a corrupt senile felon in the White House.

So it's not going to happen for two years.

But we will eventually get a sensible US government that I think will collaborate with the Chinese on trying to avert the risk of superintelligence taking over, much as the United States collaborated with the Soviet Union in the 1950s on trying to avert the dangers created by H-bombs.

- One of the AI executives who you seem to trust the most is Dario Amodei of Anthropic.

You gave him an A for his recent essay on why we should be pacing the AI frontier.

- The Anthropic took steps this week towards one of the biggest IPOs in history with a valuation of $1.4 trillion.

Can even the most safety-conscious AI companies put humanity's interests ahead of its shareholders when trillions of dollars are at stake?

- So Dario's in a very difficult position.

One thing I should say is, don't look at what people say, look at what people do.

And so, Claude was used by the US military.

The US military wanted to be allowed to use it to spy on US citizens, and Dario said no, and took a big hit for that.

When Dario said no, Sam Altman said, "We agree with him."

And then about a day after the US military had fired Claude, Sam Altman said, "Okay, open AI'll do it."

So, Dario was willing to take a hit in order to do what he thought was right.

That's much more convincing than anything they say.

How then, you know, if a company is going to market with an IPO valuation of $1.4 trillion and it says in its filing papers there is a 10% chance that civilization could be at risk because of the technology that is going to market.

Help me understand why and how the marketplace is willing to accept this kind of risk.

I mean, it would have been unheard of in the nuclear age when the Manhattan Project was being developed that something as dire as nuclear weapons would have been treated as as incautiously as the market seems to be treating this real risk.

From Dario's point of view, he has to make a decision about whether to say, "This stuff shouldn't be developed.

I'm out."

In which case it's going to be developed by people who are less cautious than him.

That puts him in a very difficult situation, right?

He has to try and stay up with the frontier so that if it's developed by anybody, it's developed by Anthropic who are going to be more cautious than the other companies.

That was the original motivation of OpenAI.

They said if anybody's going to develop superintelligence, it should be developed by somebody who really cares about safety.

And OpenAI was set up to really care about safety.

Since then, it's started to care much less about safety.

So, I think from Dario's point of view, the reason he's in a bind is, he's very worried about the safety, but he thinks it's better if Anthropic develops it than other companies.

And I know quite a few people who work for Anthropic precisely because they think it's gonna be the safest company for developing AI.

- This summer, your fellow AI pioneer and Nobel Laureate, Demis Hassabis, outlined a proposal for federally overseen self-regulatory organization with a board that includes a set of independent technical experts.

The Wall Street Journal reported that some in the White House were receptive to the idea, but AI executives like Mark Zuckerberg, Elon Musk, and even Jensen Huang encouraged President Trump to reject the plan.

Is that a workable solution still to your way of thinking?

>> I think it's a first step.

It's not the whole solution, but I think it's crazy that, at present, there's almost no regulation on AI.

The regulation we have on pharmaceutical drugs, for example, is that if you want to release a new pharmaceutical drug, you have to convince the FDA that it's safe, and you have to do the work to convince the FDA, and the FDA can consult external experts.

We need something like that for AI.

That's the very minimum we need, and it's crazy we don't have it.

- This program is the revival of William F. Buckley Jr.

's Firing Line, which aired for 33 years on PBS.

In 1993, William F. Buckley Jr.

hosted a discussion about the proliferation of nuclear weapons and the threat of nuclear destruction.

Here is a physicist, Russell Seitz, talking about the challenge of containing nuclear technology.

As high technology ramifies, appropriate technologies that can be subverted to the ends of proliferation multiply, until at the margin, we may be forced to rely more on our powers of diplomacy than on our capacity to suppress the progress of technology globally, if we are to keep under control the prospect of essentially unlimited proliferation of nuclear weapons.

Okay, so there are some big differences between AI and nuclear weapons.

The first big difference is AI can do tremendous good.

We haven't talked about that, but it will do tremendous good in health care and education and in making almost every industry more productive.

So it's going to be developed for that reason.

We can't stop it, I don't think.

>> Right.

>> But the second thing is, nuclear weapons aren't intelligent.

AI is intelligent, and if we can educate it and train it so that it really cares about us, then proliferation is not nearly so worrying.

>> You argue the technology that you have spent your life developing might have been a mistake.

And you address this head-on in your forthcoming book, writing, "Responsibility is not the same thing as regret."

Explain the difference.

>> Okay, I want to distinguish actually two kinds of regret.

Sort of sometimes you're mean to a friend of yours, and you regret it.

That's what I call guilty regret, when you did something, and at the time, you knew it was wrong, but you did it anyway.

Then the things that you did were, at the time, you couldn't have foreseen the consequences, but they turned out to have potentially very bad consequences.

I feel that that's what I did, and if I was put in the same situation again with the knowledge I had at the time, I would have behaved the same way.

>> Dr.

Hinton, thank you so much for joining me on "Firing Line."

>> Thank you for inviting me.

>> "Firing Line with Margaret Hoover" is made possible in part by Robert Granieri, Vanessa and Henry Cornell, The Margaret and Daniel Loeb Foundation, The Beth and Ravenel Curry Foundation, Cliff and Laurel Asness, and by the following.

(upbeat music) You're watching PBS.

[Music]