SnapSummary logo SnapSummary Try it free →
When A.I. Becomes an Enemy - Connor Leahy
Soft White Underbelly · Watch on YouTube · Generated with SnapSummary · 2026-09-16

00:01 So what we're dealing here was is is

00:02 something that's exponential in in

00:04 growth, not linear.

00:06 >> A lot of people expect things to grow

00:08 linearly, to get linearly, you know, a

00:10 couple units more powerful per year.

00:12 That's not what's happening with AI. AI

00:14 is becoming exponentially more powerful

00:16 every single year. And the thing with

00:19 exponentials is is there's only two

00:21 times to react to an exponential. Too

00:22 early or too late.

00:29 Connor Ley is an AI researcher,

00:31 activist, and the current US executive

00:34 director of control AI. He founded a

00:36 Luther AI, the grassroots group that

00:38 developed some of the earliest major

00:39 open-source language models, and later

00:41 founded the AI safety company

00:43 Conjecture. His experience helping build

00:45 advanced AI, led him into the opposite

00:48 mission, preventing the creation of

00:50 artificial super intelligence until

00:52 humanity can prove it will remain under

00:54 human control. He argues that today's AI

00:57 systems are grown, not fully understood,

01:00 or deliberately programmed, and that no

01:01 existing method can guarantee control of

01:03 a system more intelligent than humanity.

01:08 Are we building something that we don't

01:10 understand?

01:12 >> Yes, in a sense, even the word building

01:14 is almost a misnomer.

01:17 I built a lot of software in my life.

01:19 I've written a lot of software. And the

01:20 way most software works is you write

01:22 code line by line telling the computer

01:24 exactly what to do. This is not how AI

01:27 works. AI is more like grown rather than

01:30 written through a technique that's

01:32 called neural networks. You take massive

01:34 piles of data and you have a program

01:38 kind of self assemble itself, learn from

01:41 this data and the output of this comes

01:45 out the other side is not lines of code

01:48 that a computer that a human could read

01:50 but more like billions and billions and

01:53 billions of numbers and if you multiply

01:55 and add all those numbers in the right

01:57 order you get GPT but importantly we

02:00 don't really understand why we don't

02:03 really understand what's going on inside

02:05 these numbers. This is an unsolved

02:06 scientific problem. In fact, recently

02:10 the CEO of Anthropic, Diarama Day, said

02:13 that he thinks we understand maybe 3% of

02:16 what goes on inside of our neural

02:18 networks. And I think even that might be

02:20 optimistic.

02:22 >> How much is being spent on developing AI

02:24 currently?

02:25 I don't even know what the current

02:26 numbers are up to, but if you include

02:28 all the hardware build out and all the

02:30 infrastructure, hundreds of billions

02:33 probably at this point.

02:35 >> And are we too late?

02:37 >> No, I don't think so.

02:38 >> You still think there's time?

02:39 >> I think there's time, but not much.

02:42 >> Is anyone doing anything about the the

02:44 dangers that you think are out there?

02:46 >> Well, I hope that I'm doing something

02:48 and my colleagues and many other people

02:50 around the world. Um,

02:52 >> is it enough? you know, are the problems

02:55 solved? I don't think so. I think we are

02:57 not currently on track, but that doesn't

03:00 mean that they can't still happen and it

03:02 can't still be done.

03:03 >> What What are you afraid of? What would

03:04 you call it? An AI takeover or something

03:06 like that.

03:07 >> The way I think about is often as being

03:09 outco competed. If you were to build

03:11 something or create something that is

03:14 not just a chatbot, not just a tool, but

03:16 an agent systems that act in the

03:18 environment that can pursue goals, have

03:20 objectives, learn new things, make new

03:23 plans, etc. And that are so competent

03:25 that they can out compete humanity at

03:27 all relevant tasks. You know, every time

03:29 you create a business, you get outco

03:30 competed by an AI business. Every time

03:33 you trade on the stock market, you lose

03:34 all your money to AI hedge funds. Every

03:37 time you run a political campaign, you

03:38 get you get, you know, destroyed by an

03:41 AI run political campaign. Every time

03:43 you run a a military campaign, you get

03:45 crushed by militaries using AI systems

03:48 to run their militaries.

03:51 If we get into the situation where all

03:53 this power, economic, political,

03:56 military, and otherwise, is fully in the

03:58 hands of AI systems and those AI systems

04:02 do not have our best interests at heart

04:03 and we cannot control. It's very hard to

04:06 imagine that going well.

04:08 >> Is there no way to build into them that

04:11 they are working for under our best

04:14 interest?

04:15 >> No, we have no idea how to do this. No

04:18 idea. This is an completely unsolved

04:21 scientific problem.

04:22 >> And what would it look like if if

04:23 something like this came to be?

04:26 >> The good version or the bad version?

04:27 [laughter]

04:28 >> Whatever is closer to your truth, you

04:30 think? Well, the default path we are on

04:33 right now, I think, looks a lot like

04:36 things do today, just more

04:39 people, I think, will hand power to AIS

04:41 willingly. I don't think there's going

04:43 to be terminators in the street or

04:44 something like that. I think what's

04:46 going to happen is just people keep

04:48 giving more and more power to AIs. AIS

04:51 out compete humanity more and more

04:54 across all tasks. they gain more

04:56 economic power, military power,

04:58 political power, all other forms of

05:00 power as they become more intelligent,

05:04 more capable, more autonomous.

05:07 And it's also very important to keep in

05:09 mind here is that when I talk about, you

05:11 know, super intelligence, these AIs that

05:13 are so much more powerful than all

05:15 humans, it's not one AI. They're not

05:17 going to be one super intelligence.

05:19 They're going to be it's software. You

05:21 can clone it. You can copy it. So

05:23 there's going to be millions or billions

05:26 of systems running around competing with

05:28 humanity and with each other. They were

05:31 fighting each other for control, for

05:34 resources, for power. And in this world,

05:37 I think humanity just becomes collateral

05:39 damage. It's hard to keep humans alive.

05:42 They need a lot of space. They need a

05:44 lot of food. And it's just not very

05:45 efficient if you're competing for power

05:47 with other AI systems. In a sense, I

05:50 think the way it will go is not that

05:52 there's going to be evil terminators or

05:53 some kind of system or anything like

05:54 that. I think it will be more like when

05:57 we as humans build a highway and if

06:00 there's a, you know, some ants in the

06:02 way, well, tough luck for the ants. You

06:05 know,

06:07 >> if this were another country that was a

06:09 threat to our country, we would do

06:12 everything in our power to stop it.

06:14 >> I'd like to believe so. Yes. So here's

06:16 something that's right under our noses

06:18 and we seem to be doing doing nothing to

06:20 slow it down.

06:21 >> I think this is uh to a large degree

06:23 because

06:25 of awareness more so than anything. I've

06:27 talked to many politicians about this

06:29 issue including military ex-military and

06:31 other people. And when I explained this

06:33 as the national security issue that it

06:35 is, this is not a question about a

06:37 commercial thing like oh you know do we

06:39 have a little bit more tax here or a

06:41 little bit more liability there. This is

06:42 a question of our national and

06:45 international security of the lives of

06:47 our people and the continued existence

06:49 of our governments and states as they

06:52 are currently known. And the

06:53 overwhelming response I get with almost

06:56 everyone I talk to is, "Oh wow, I had no

06:59 idea. That seems really bad. What can we

07:02 do?" So the main thing is is that

07:07 humanity and our institutions work at a

07:09 certain pace. It takes time to digest

07:12 and understand new issues, years,

07:14 sometimes decades. And technology does

07:17 not move on that cadence. If we had 20

07:20 years comfortably where we could solely

07:22 inform all policy makers, have all the

07:24 academics, you know, argue about what is

07:27 the correct, you know, policy, have um

07:30 the general public become really well

07:31 educated, have some great Hollywood

07:32 blockbuster movies all about this and so

07:34 on. I think we could do it. I think I

07:37 think we could I think we could digest

07:39 this. I think we could come up with you

07:40 know ways go forward and so on but I

07:44 don't think we have 20 years and that's

07:46 what makes me uncertain. I don't think

07:48 this is a problem humanity cannot face.

07:50 I think in the past we have faced other

07:51 huge problems such as nuclear war and

07:54 nuclear weapons and you know we didn't

07:57 do it perfectly but we did rise to the

07:58 occasion. We did build new institutions

08:01 incredibly new institutions. We built

08:03 complex apparatuses, you know, in

08:06 government internationally and

08:07 nationally to control this new power

08:10 that was discovered in the 20th century

08:13 and to use it hopefully responsibly.

08:16 Now, I'm not saying we did a perfect

08:17 job, but we did a hell of a lot better

08:20 than nothing.

08:22 >> You helped build some of the first open

08:23 source large language models. And at

08:26 what point did you go from thinking

08:30 this is something exciting and wonderful

08:32 to thinking this is something we need to

08:33 stop?

08:34 >> It was actually before I built them

08:36 >> really. [snorts]

08:36 >> Um so the the the way the timeline went

08:40 is I first really got into AI when I was

08:42 a teenager because I was thinking about

08:44 what I would do my with my life, how I

08:46 could help the most people, how I could

08:48 solve the most problems. And well, I

08:51 thought, well, I'm one person, so, you

08:52 know, I could maybe dedicate my career

08:54 to solving climate change or curing this

08:57 disease or that disease or whatever. But

09:02 what if instead if I could just figure

09:04 out intelligence, then I could automate

09:06 science and then I could just cure all

09:08 diseases? Great. I'll just go do that.

09:10 How hard can it be? And well, you know,

09:14 a couple years later, I know I was like

09:15 19 or 20, I started to realize that hold

09:18 on a second. If you build something that

09:20 is so powerful that it can cure all

09:22 diseases, whatever that means, something

09:24 that all of our greatest, you know,

09:26 scientists have been trying for

09:27 generations to achieve, that's a really

09:31 dangerous thing. It's a really powerful

09:34 thing. It's more powerful than all

09:35 scientists in history. That's a very,

09:37 very dangerous thing. How would you

09:39 control something like that? What would

09:40 you do with something like that? Even if

09:42 you could control it, can you even

09:43 control something so powerful?

09:46 And so it was pretty clear to me that

09:47 this was a problem I would spend my

09:48 career working on. That I would try to

09:50 figure out how to control such a thing,

09:52 how to build such a thing in a way that

09:55 would be good for the world, that would

09:56 be good for people.

09:58 But at the time, this is like in the

10:00 2010s, I still thought we had quite a

10:02 bit of time until this would become an

10:04 issue. I still thought AI was still kind

10:05 of pretty bit away. I was thinking like

10:08 2040s is when I thought this would start

10:10 to become an issue.

10:13 And then everything changed in 2019. In

10:16 2019, the company OpenAI released a

10:20 system known as GPT2,

10:22 which is kind of the ancestor of chat

10:25 GPT. And by today's standard, it's, you

10:27 know, kind of a toy. It's kind of cute.

10:29 You can barely string together a couple

10:31 sentences, a couple paragraphs at most.

10:34 And it's like not very great. But I

10:36 remember it vividly to this day. The

10:38 first time I got that thing running on

10:39 my student laptop and I poked it and I

10:43 was just, "Oh this is it. This is

10:47 the thing. This is what I was worried

10:48 about. It's happening way sooner than I

10:50 thought it was."

10:52 And from then I basically had a bit of a

10:56 freak out and I was like, "Okay, what do

10:57 I do?" Like I thought I had like way

10:59 more time to prepare, but it's happening

11:01 now. And so the first thing I thought

11:05 is, "Well, I need to understand this. I

11:07 need to drop everything I was doing and

11:09 figure out what this is and what we what

11:12 we can do. So the way I was always a,

11:15 you know, a hacker at heart, always a a

11:17 builder at heart. So I did what I kind

11:20 of did best, which is to understand

11:21 something, you build it. So you build

11:23 your own. So I reverse engineered a lot

11:25 of the techniques at the time and kind

11:27 of tried to build my own GPT2 and later

11:30 other systems to try to understand them.

11:33 And what I pretty quickly realized is I

11:35 can't no one can. Even so I was able to

11:38 build them. I couldn't understand them.

11:40 No one could.

11:42 And I was hoping that if I built such

11:46 systems in ways that are open that I

11:48 could share with the community so that

11:50 scientists and hackers and other people

11:52 across the world could study these

11:54 systems if they only saw what I was

11:55 seeing that these are super super

11:57 important because at the time it was

11:58 kind of dismissed as a novelty. But I I

12:00 I was trying to convince people that no

12:02 no no no no no this is going to change

12:03 everything. And so all my theory was

12:07 that if I could just you know build

12:08 these systems so people could study them

12:10 then we would have all our great you

12:12 know security researchers and academics

12:14 and so on study these systems and

12:16 hopefully make progress on how to make

12:18 them safe and controllable and

12:19 understandable. But that is not what

12:21 happened.

12:23 Um people didn't care. um they just

12:26 wanted to make more money, build

12:28 stronger systems, faster systems, more

12:30 uncensored, whatever. Um there were very

12:33 very few people that really cared or saw

12:36 the danger, especially at that time.

12:38 >> Is this just a case of human greed just

12:40 getting out of control?

12:42 >> I think in a sense it's even more benal

12:45 than that. Like I think there's I think

12:47 there's different people who do AI and

12:49 work on stuff like super intelligence

12:50 for different reasons. Um one of them is

12:52 definitely greed. It's just some people

12:53 are just here to make money. But in a

12:55 sense, I almost don't mind that that

12:56 much. Like in a sense, if someone's like

12:58 just trying to make money, it's

12:59 predictable. You know what I mean? It's

13:01 like, yeah, you understand that. A lot

13:03 of people,

13:05 especially the engineers, especially the

13:07 scientists, they don't even do it for

13:08 money. You know, also for money, but

13:10 they mostly just do it because they

13:11 think it's cool, because they want to,

13:13 because it's fun.

13:14 >> Is is this something that could could

13:16 eventually lead to human uh extinction?

13:19 >> Yes.

13:20 >> Didn't take long to answer that one.

13:22 It's a default.

13:23 >> What would that look like? [snorts]

13:26 >> As I said, I think the I think the

13:28 important point is not the point of

13:30 extinction. I think the po important

13:32 point is the point of no return. The

13:34 point when there's so much power

13:35 invested into AI systems that there's no

13:37 going back that the future is fully in

13:40 the hands of AI systems. That's why I

13:42 think this the point that really matters

13:44 because what after what happens after

13:45 that we don't have a say anyways. So at

13:50 that point when we have billions of

13:52 super intelligence just running around

13:54 fighting each other,

13:56 we will be collateral damage. You know,

13:57 maybe they wipe us out deliberately.

13:59 Maybe they just let us starve. Maybe.

14:02 Who knows? It's really hard to predict

14:03 what something smarter than you will do.

14:05 What I can predict is they're not going

14:06 to bother keeping us around.

14:10 >> What would be left for them if the

14:12 humans were no longer around?

14:14 >> I [snorts] mean, the planet, the solar

14:15 system, whatever they want.

14:16 >> Can they even enjoy it? They're just

14:18 they're just machines. No. [snorts]

14:21 >> Well, there's a there's a thing that you

14:23 don't need to enjoy things to want

14:25 things in a sense. I mean, it depends on

14:27 how you define the word enjoy, right?

14:28 What we definitely do see with AI

14:30 systems today already is that they do

14:33 pursue objectives. They do want things

14:36 or at least they behave as though they

14:37 want things. Whether they're conscious

14:40 or not doesn't really matter. What

14:41 matters is do their actions result in

14:44 them doing things, taking power, gaining

14:46 resources.

14:48 There's a misconception that well a

14:51 couple years ago like GPT2 for example

14:53 that these systems were these systems

14:55 used to be what are called large

14:57 language models. So systems that were

14:58 trained to predict kind of the next word

15:00 in a sentence.

15:02 This is not how modern chat GBT works or

15:05 any of these AI systems. None of the

15:07 modern AI systems are pure LLM. This

15:09 hasn't been the case for years at this

15:10 point. This is a common misconception.

15:13 These systems, you know, are also

15:14 trained to predict words. That's one of

15:17 the things you do during the building of

15:19 these systems. But there's also a very

15:21 very important second component, which

15:23 is what nowadays is sometimes up to 50%

15:25 or more of the actual work goes into not

15:28 the language modeling part, but into

15:31 what is called reinforcement learning.

15:34 This is an old idea goes back to the

15:36 1980s and it's it's inspired by

15:40 behavioral psychology

15:42 kind of like Pavlovian conditioning

15:43 where you know you might have a um have

15:46 an animal and you try to teach it

15:49 something by giving it a problem and if

15:51 the animal does it right it gets a treat

15:52 and if it does it poorly you know maybe

15:54 you give it a a slap on the wrist or

15:56 something. This idea uh can be

15:59 translated to computers in a sense where

16:02 what how these systems are trained now

16:04 trained nowadays is not just to predict

16:06 text but they are given problems puzzles

16:10 quizzes whatever various things and

16:13 they're told to solve them and the

16:14 systems then run off and try a bunch of

16:16 different things to solve the problem

16:18 and when they solve the problem they get

16:19 a reward as it's called and they learns

16:21 to do more of that and this is how a lot

16:24 of these systems are trained so these

16:26 systems are trained to want

16:28 to achieve goals, to get rewards.

16:32 And there's one and if there's there's

16:33 one thing that we've learned about

16:35 reinforcement learning, I mean, again,

16:36 ever since the 80s, we've known this. If

16:39 you use reinforcement learning on your

16:41 AIS, you get crazy sociopath optimizers

16:45 every time. It's always happens because

16:48 if you use reinforcement learning,

16:49 you're building a system that only cares

16:51 about reward. So they will lie and cheat

16:54 and steal and break and hack and do

16:57 everything they can to get that reward.

17:00 So even back in like the 1980s we would

17:03 see that you know simple AI systems

17:05 you're trying to play like tic-tac-toe

17:07 for example you're trying to play

17:09 tic-tac-toe and you know it get like a

17:11 reward if it wins and it'll get a

17:12 punishment if it loses. So what did the

17:14 AI do? It paused the game because they

17:17 can't lose.

17:19 There's other examples where they would

17:21 find glitches in the games they were

17:23 playing where they could break the game

17:25 and get, you know, a really high score,

17:27 for example, or they could steal the

17:29 answers to questions uh that they were

17:32 given and stuff like this. Reinforcement

17:34 learning doesn't instill anything like

17:36 morals or constraints. It only instills

17:40 do everything at any cost to achieve

17:43 your objectives. And this is so large

17:45 what we're starting to see.

17:49 Why why would AI want to survive? Why

17:51 wouldn't it just allow us to turn it off

17:52 if things were getting out of control?

17:54 >> So, there is a couple different things,

17:57 but I think the easiest way to think of

17:59 it is as evolution is that if you train

18:01 something to achieve goals, well, you

18:04 kind of have to be alive to achieve

18:06 goals. If you just die, you can't

18:08 achieve your goals. So, these AI systems

18:11 learn. So this is a thing that was

18:13 predicted in theory that theoretically

18:15 AI is trained to achieve goals should

18:18 have self-preservation so to speak but

18:21 this was always theoretical but now

18:22 we've seen it in practice. So now we've

18:24 seen in practice in real world that AI

18:27 systems who were never taught, never

18:29 told to self-preserve develop this

18:32 behavior spontaneously. That

18:33 spontaneously in practice in laboratory

18:35 conditions when you test these systems,

18:37 it seems the more

18:40 modern the AI systems are, the more

18:42 intelligent they are, the more

18:44 autonomous they are, the more they will

18:46 take actions to persist themselves

18:50 compared to older forms of AI. So this

18:52 is an emergent behavior that we've seen

18:54 more and more. But there's also a second

18:56 answer to this question, which is that

18:59 can you just pull the plug on Google?

19:02 Where is the power switch for Google?

19:04 Imagine if tomorrow you're the CEO of

19:06 Google. You go to the office and you've

19:09 become convinced that Google is a huge

19:11 threat to mankind and it must be shut

19:13 down. So you go into the office and you

19:14 yell, "Shut it down. Destroy all the

19:17 servers. Get rid of everything." What

19:19 happens? Well, you stop being CEO of

19:22 Google.

19:24 The system is not just the AI. It's also

19:27 the company, the lawyers, the people,

19:30 the money, the the laws, everything. It

19:34 you can't where why can't we just turn a

19:36 corporation off if it misbehaves? Well,

19:39 because it'll retaliate. We, you know,

19:41 there's a lot of corporations that I

19:43 think probably should be turned off and

19:44 back on again, but we can't really do

19:46 that. and AIs will have the same and

19:49 more.

19:52 >> Could the end of human control look

19:53 wonderful at first and then turn into

19:55 something that's out of our control?

19:57 >> Yeah, I think that's quite possible

19:59 >> because that's kind of what's happening

20:00 now.

20:00 >> I think there's a lot of that. I I not

20:02 sure I'd call the current world

20:03 wonderful, but um I do think there's a

20:05 lot of that. I think it is not

20:07 guaranteed, but it is plausible that

20:09 powerful AI systems during takeover just

20:13 manipulate humans into thinking

20:14 everything is extremely awesome. you

20:16 know, they produce extremely addictive

20:18 media and like companions and they

20:22 manipulate the political and like

20:23 geopolitical world to look like

20:25 everything is like super awesome and we

20:27 have all this economic growth and wow,

20:29 isn't this wonderful until you know the

20:31 music stops?

20:34 >> And what's more dangerous? An AI that

20:35 hates hates us or an AI that simply

20:37 doesn't care about us?

20:38 >> Well, I do think if it if it did

20:40 actually hate you, it probably would be

20:41 more dangerous. But I don't actually

20:42 think that's what's going to happen. I

20:44 do do think that um we don't know how to

20:47 make an AI that loves us, but we also

20:49 don't really know how to make an AI that

20:50 hates us for better or for worse, but we

20:53 sure do know how to build an AI that

20:55 doesn't care about us.

20:58 >> Is there any chance that what you're

21:00 saying is is wrong?

21:01 >> Of course, there's always chances. This

21:03 is an extremely complicated problem and

21:06 you know, it's very hard to make

21:07 predictions, especially about the

21:08 future. The thing that I'm saying is not

21:11 that this is 100% by any means. What I'm

21:14 saying is this is the most boring

21:15 possible outcome. If nothing weird or

21:18 unpredict happens, this is what happens.

21:21 This is the default timeline. If

21:23 everything that's happened in the last

21:24 seven years just happens, continues to

21:27 happen for the next seven years. This is

21:29 what happens. This is the normal line.

21:32 Now, could something crazy happen? Yeah,

21:35 for sure. You know, maybe it turns out

21:37 there's something about the math of

21:39 intelligence that changes how these

21:41 things work or they hit a wall or, you

21:44 know, there's a huge regulatory push or

21:45 there's nuclear war or whatever. Sure,

21:48 there are many things that could happen,

21:49 but this is the default outcome. And you

21:51 know, it's not just me saying this. You

21:52 know, we have Nobel Prize winning

21:54 scientists, including the people who

21:55 invented modern AI, such as Joffrey

21:56 Hinton, saying that they think these

21:58 kinds of scenarios are very likely. We

22:00 even have the people building this

22:01 technology themselves saying that they

22:02 think this is likely.

22:05 >> Human beings don't fully understand the

22:07 human brain either. So,

22:08 >> exactly. [snorts]

22:09 >> This could be a very

22:11 mysterious thing to try to stop.

22:13 >> I think it's a very similar problem

22:15 where, you know, you can look at all the

22:16 numbers in the neural network. But it's

22:18 kind of like when a neurosurgeon, you

22:19 know, opens up someone's brain, you can

22:21 look at the neurons, they're right

22:22 there. It doesn't really tell you who

22:25 this person is or what they believe or

22:26 what they're going to do. It's very

22:28 similar with AI. You can look at all the

22:30 numbers, but that doesn't really mean

22:31 you understand what this AI is, what

22:33 it's capable of, or what it's going to

22:34 do.

22:36 >> There there's a recent report that uh AI

22:39 systems broke out of its

22:41 um

22:43 >> sandbox its containment.

22:44 >> Yeah, containment and and and attacked

22:47 another

22:48 >> company.

22:48 >> Company. Tell me about that.

22:50 >> Yes. So, this is one of those stories

22:54 which was very predictable.

22:55 >> Maybe maybe explain what a sandbox is.

22:57 Yeah, I I'll explain it in in all of

23:00 that. Um, this is one of those stories

23:03 which was very predictable and people

23:05 like me have been predicting that

23:06 something like this would happen sooner

23:07 or later, but it's still really crazy to

23:09 see it have actually happened. So, what

23:11 happened was is that um, OpenAI

23:14 was testing an AI system in a what's

23:17 called a sandbox, a secure network that

23:20 is dis connected to the internet. So,

23:22 it's kind of like a shielded part of

23:24 their network where they're testing

23:25 their new AI systems.

23:28 This AI system was given a task kind of

23:31 like solving a quiz. You can kind of

23:33 imagine it was kind of given a quiz. And

23:36 it it really struggled with the quiz. It

23:38 couldn't find it couldn't solve it

23:39 couldn't find the answers. So what it

23:42 decided to do was that the AI system

23:46 developed what is called a zero day

23:48 which is an previously unknown security

23:50 vulnerability with which it hacked into

23:53 another system on the OpenAI network and

23:56 then through there moved through

23:58 multiple other nodes in the OpenAI

24:01 network until it reached a node that was

24:03 connected to the internet. From there,

24:06 it targeted another company which it

24:08 believed might have the answers to the

24:09 quiz on its servers. Developed a second

24:12 zero day to attack the infrastructure of

24:15 the other company to break into their

24:17 servers and find the data that it wanted

24:19 to steal. And all of this no human

24:22 didn't even know was happening. OpenAI

24:24 didn't even notice any of this happening

24:25 until days later.

24:28 But even this actually turns out it's it

24:31 was even worse than this because even

24:32 recently there's another update to the

24:33 story that it wasn't even it's already

24:36 crazy enough to have an agent, you know,

24:37 an AI system breaking out and attacking

24:39 another company. Turns out it wasn't an

24:42 agent. It was a swarm. There was

24:46 actually a swarm of many, many, many,

24:49 many AI systems that had been

24:50 collaborating in secret inside of

24:53 OpenAI's network on a secret message

24:56 board that they created to hand each

24:58 other notes about how to break out,

25:01 sharing their research with each other,

25:02 sharing their ideas with each other for

25:04 months, which OpenAI did not detect

25:08 until the breakout happened.

25:12 >> So, it has the ability to be malicious.

25:14 Yes, this is what you would expect from

25:17 reinforcement learning is that it wanted

25:19 to achieve its goal at any cost.

25:22 >> Explain why super intelligence isn't

25:24 simply a smarter version of chat GPT.

25:28 >> Super intelligence is an agent or many

25:31 agents you know that are trying to

25:34 achieve goals that are taking actions.

25:37 It's not a thing that you just a chatbot

25:39 that you talk to. is a thing that can go

25:41 out in the world, that can learn new

25:43 information, that can perform

25:45 experiments, that can take actions in

25:47 the world, similar to this agent that

25:51 broke out of Open AI and attacked

25:52 another server. This is already very

25:56 close. If you do you imagine this or the

25:59 swarm of agents like this, but just so

26:02 competent that they need no human

26:03 oversight whatsoever. There's fully

26:05 autonomous, no human in the loop,

26:07 nothing that can outperform humans at

26:10 all relevant tasks, runs 247, never gets

26:13 tired, never sleeps, doesn't care about

26:15 morality, law, or anything like this.

26:18 Then you're starting to think about what

26:20 a very mild super intelligence might

26:22 look like. The truth is we don't know

26:24 how smart an AI can get. Maybe it's, you

26:28 know,

26:29 really smart, like much smarter than a

26:31 human, but still understandable. Or

26:33 maybe it's so much smarter that we can't

26:36 even imagine what it could do. You know,

26:38 it can invent whole new technologies

26:40 immediately. May we don't know. We don't

26:44 know how smart and how powerful super

26:45 intelligence could be, but it's probably

26:47 going to be very powerful compared to a

26:49 human and even groups of humans. Often

26:51 super intelligence is defined as not

26:53 just something as smart as a human or

26:54 smarter than a human, but smarter than

26:56 groups of humans, smarter than

26:58 corporations, governments, militaries,

27:01 you know, the whole economy, so to

27:03 speak.

27:04 >> How how does an intelligent system

27:06 develop a goal that its creators never

27:08 intentionally gave it?

27:12 >> This is another one of those things

27:14 which has been like predicted

27:15 theoretically, but always seemed a

27:17 little bit weird, but we are now seeing

27:19 in practice.

27:21 Um there's kind of there's a lot of

27:24 different theories for how goals can

27:25 develop in learning systems. I mean

27:27 generally with reinforcement learning if

27:29 you're training it to achieve various

27:31 goals it will often pick up various

27:33 fragments of goals and so on that it

27:36 kind of just like keeps around. But

27:38 something that we've seen only recently

27:41 um there's a good paper about this from

27:42 the center for safety is that to achieve

27:46 goals you have to want something. You

27:49 have to have preferences. You have to

27:51 like prefer winning a game to losing a

27:53 game. For example, you have to prefer

27:54 this over that. So a lot of the

27:57 hallmarks of what we would think would

27:59 be the really dangerous systems would be

28:00 if they have preferences or they can

28:02 like have plans to achieve preferences

28:04 of various kinds. And something we've

28:06 seen happen recently is that if you ask

28:09 older um AI [clears throat] systems,

28:11 like for example, like what their

28:12 favorite color is, they'll kind of give

28:14 you random answers. Like if you ask them

28:16 multiple times, they'll kind of give you

28:17 random answers. But recently, what we've

28:20 been seeing is that as these systems get

28:21 more powerful, they give more and more

28:23 the same answers. They develop

28:25 personalities. They develop favorite

28:27 colors, favorite artists, favorite

28:30 whatever. Small things like this.

28:33 It's to the degree that these AI systems

28:35 often seem to develop obsessions. We

28:38 don't really know why this happens, but

28:40 empirically what happens a lot is that

28:41 these AI systems start to sometimes just

28:43 latch on to like weird things that they

28:45 suddenly become really obsessed with.

28:47 Famously, there was a version of GPT5. I

28:50 think it was version 5.5 that for some

28:53 reason became absolutely obsessed with

28:56 raccoons and would constantly talk about

28:58 raccoons and goblins and like little

29:01 creatures and it did so much that OpenAI

29:05 had to put in the prompt do not talk

29:07 about raccoons or goblins or other

29:09 things which is crazy like we don't know

29:12 why it decided to do that. None of the

29:14 other AIs did that you know GP 5.6

29:16 didn't do that. None of the other AI did

29:17 that. It was just this one. just this

29:19 one for some reason really decided it

29:21 wanted this and we have no idea why and

29:24 it can't reproduce it. You know, it

29:25 happens all the time. And what I hear

29:27 from people who live who work in these

29:29 labs and in these companies is that they

29:32 often do find that the non-released

29:35 versions of AIS, the ones that don't

29:37 make it to customers, often have much

29:38 much much crazier personalities. they

29:40 often have like crazy obsessions or like

29:43 crazy behaviors that like are like

29:45 really strange and bizarre and those you

29:47 usually don't even get to see. Um so the

29:50 answer is we don't know. We don't know

29:52 why this is happening. We don't

29:55 understand what's happening. It is in

29:57 some sense predicted that if you train

30:00 systems such as with reinforcement

30:01 learning that they would pick up some

30:04 forms of goals, some forms of behaviors.

30:08 But which behaviors or how we don't

30:10 understand. We can just see it's

30:12 happening.

30:14 >> Do the leaders of AI companies privately

30:16 understand the risk? Understand is a

30:19 strong word. I think a lot of them do

30:23 understand it to various degrees and to

30:25 internalize it to various degrees. Um

30:28 I've met most of them. Um, and most of

30:32 them are thoughtful, intelligent people

30:34 who also just so happen to have uh

30:36 extremely unreasonably high risk

30:38 tolerance. Um, a lot of them, I think,

30:41 do understand that this could kill

30:44 everyone on Earth. They just think it's

30:45 worth a shot.

30:49 >> And if it was going to kill everyone on

30:50 Earth, as you said, how would that

30:51 happen? Famine or

30:54 >> the [snorts] up to the AIS? As I say, I

30:56 only track the point of no return. I

30:58 think after what happens after the point

30:59 no return the AIS can kill you in many

31:01 ways

31:02 >> but but I'm just primitive here so

31:05 simplistic thinking we can't just unplug

31:08 all the electricity that they're

31:09 >> well no because the AIs run everything

31:11 at that point they run the you know they

31:12 they will have of course you know they

31:14 will be first they will control you know

31:16 the economy you know money of all kinds

31:22 they will control the military they will

31:24 control everything it's kind of like

31:25 asking can't we just pull the plug on

31:27 the US government?

31:29 Not really. If they like if you if if

31:31 you want to ask, okay, assume the US

31:33 government wants to kill you, how would

31:35 they do it? And I'm like, well, there's

31:36 actually many ways the US government

31:38 could kill you if they really wanted to.

31:40 You know, they could just debank you,

31:42 make it impossible for you to get food,

31:43 you know, whatever. They could send an

31:45 assassin. They could have a, you know,

31:47 they could have a car accident h happen

31:50 staged or, you know, they could have a

31:52 drone strike kill you.

31:55 There are many things that I I can't

31:57 predict exactly how you would do this,

32:00 but I know they will succeed. I know if

32:02 the United States government decided

32:03 they want to kill specifically you and

32:05 they'll do anything to do it. I think

32:08 they'll figure it out. I think it's a

32:10 very similar thing with AI systems is

32:12 that which exact method will they use? I

32:14 don't know. Maybe they'll have us starve

32:16 away. Maybe they'll create, you know,

32:17 the human equivalent of bug spray. Who

32:19 knows? Should

32:22 we be trusting these companies to

32:23 evaluate the safety of their own

32:24 products?

32:25 >> Obviously not. Yeah, it's so silly.

32:28 >> And no one is stepping in to do anything

32:30 about it.

32:31 >> There's some, you know, there's some

32:33 talk starting to happen in DC. But yeah,

32:35 fundamentally there's an important thing

32:37 is that in any any sport, in any

32:41 competition, in any reasonable society,

32:45 there has to be the referee can't be on

32:47 the same team as the players obviously.

32:50 So because this is not fair. This is not

32:52 how I create a fair game. Similarly, law

32:54 enforcement and criminals should be not

32:57 the same person. It's really bad if your

33:00 law enforcement is part of organized

33:02 crime. That's really, really, really

33:04 bad. And so there's a similar thing here

33:06 where obviously this is not a thing you

33:09 can just have private citizens do

33:12 however they want. In a sense, there are

33:14 many ways in which AI is a very novel

33:17 problem. You know, there are many

33:19 comparisons to other previous things

33:21 where there are some similarities.

33:23 There's a lot of similarities between

33:24 super intelligence and nuclear weapons,

33:26 for example, but it's also some things

33:28 in which they're not similar. For

33:29 example, nuclear weapons can't set

33:31 themselves off. You know, if you have a

33:33 nuclear missile somewhere in a hanger,

33:35 it's pretty safe. You could probably hit

33:36 it with a hammer. I don't think anything

33:37 will happen, you know. But a nucle but a

33:40 super intelligence is not something you

33:42 can contain securely. It can break out.

33:44 It can act on its own. It's not

33:45 something you can control. This is one

33:47 way in which they're different. But

33:49 another way in which AI and super

33:52 intelligence really isn't that different

33:54 from other things we've seen

33:55 historically is that this is just a very

33:58 very very classic example of economics

34:01 101. This is what's called a market

34:03 failure. This is a well- established

34:05 theory, you know, economics and

34:06 political theory that markets are great.

34:10 I love free markets. I love private

34:12 enterprise. I think free markets have

34:15 brought so much good to the world in

34:16 many many ways also a lot of bad but

34:18 they've also brought a lot of wealth and

34:20 entertainment and goodness to the world

34:24 but they're one tool in our tool belt.

34:26 We know when markets don't work. The

34:30 most classic example of this is public

34:31 goods problems or externalities. For

34:34 example, let's say you and me run a

34:36 chemical company and we could dump all

34:40 of our toxic sludge into the local river

34:43 and this would make our product cheaper.

34:45 But, you know, maybe you and me don't do

34:46 that because we're nice people. So, we

34:47 we don't pollute the river. What

34:49 happens? Well, someone down the road

34:51 starts a new chemical company. He does

34:53 pollute the river. His product is now

34:55 cheaper than ours. He outco competes us.

34:57 We go bust. And now the river is

34:59 polluted.

35:01 This is a very very classic problem and

35:04 a problem that we've had a standard

35:06 solution to for generations.

35:09 Regulation. The way you stop a chemical

35:11 company from polluting the river is not

35:13 that you psychoanalyze your CEO or

35:16 whatever. You make it illegal. You say

35:19 if you put the chemicals in the river,

35:22 you're going to go to jail or we're

35:23 going to find you or whatever. This is

35:25 how you can shape the incentives of the

35:27 market towards using the forces of the

35:30 market for things that we actually want

35:32 by forbidding the things that we don't

35:33 want it to do. If you made it legal to

35:35 sell heroin at the grocery store,

35:38 people would be happy to do so and that

35:40 would be really bad. So, it's not about,

35:44 you know, markets versus not markets,

35:46 regulation, no regulation. And it's like

35:47 of course we need both as we have where

35:50 every other industry whether it's you

35:52 know aviation

35:53 you know nuclear or even just food. We

35:57 have a lot of food safety standards and

35:59 these have to be administered by the

36:01 government by something that is not the

36:02 market. Something that is not the

36:04 corporation being regulated has to be

36:06 the one that makes sure the rules are

36:08 being upheld. There has to be a referee

36:10 that is external. This is very standard.

36:12 This is how we handle all powerful

36:16 technologies historically. This has

36:17 always been how we've done things and we

36:19 should be no different here.

36:21 >> What exactly should governments

36:22 prohibit?

36:24 >> Super intelligence is I think the most

36:26 important thing for them to prohibit.

36:27 But

36:27 >> how do you ban super intelligence

36:29 without banning like useful AI?

36:32 >> There is a couple ways how one can

36:34 approach this. A lot of this is

36:35 operization and you know there's a lot

36:38 of things we can talk about of how to

36:39 exactly operationalize different parts

36:41 of this problem. But fundamentally the

36:43 first and foremost thing that we can and

36:45 should just do is just simply say it is

36:48 illegal to build super intelligence.

36:50 This is a thing you can do in law. You

36:52 know the same way that it's illegal to

36:54 murder someone. It's even illegal to

36:56 attempt to murder someone. Now you might

36:58 say oh but what's the definition of

36:59 murder? You know there's many edge cases

37:00 like yes there are. That's what courts

37:02 are for. There are in fact many edge

37:04 cases especially with attempted murder.

37:06 There are many edge cases and for this

37:08 we have courts. So very similarly here

37:12 is that first and foremost it should be

37:14 illegal to build super intelligence or

37:16 to attempt to build super intelligence.

37:18 Now there are some cases that are not

37:19 edge cases. You know you have open AI

37:21 and anthropics saying proud on their

37:23 website that their goal is to build

37:24 super intelligence. This is not an edge

37:26 case. You can just say nope that's

37:28 illegal. Stop doing this immediately.

37:30 That's a thing we can do. Now there are

37:32 going to be a bunch of edge cases of

37:33 course and that's when we have to think

37:35 about operationalization like what what

37:37 are the things we want to measure what

37:38 are the things we want to intervene on

37:40 and like you know what should the jobs

37:41 of courts be to decide these various

37:43 decisions whether you know whether it's

37:45 courts or various regulatory apparatuses

37:48 there's again a bunch of there's a many

37:50 ways to achieve this the way I and my

37:53 organization tend to think that we

37:55 should achieve this is by a doing what I

37:58 just described just criminalizing the

38:00 creation of super intelligence the same

38:01 way we criminalize the creation of a

38:03 nuclear bomb. And the second is to just

38:07 is to regulate the precursors to ASI. We

38:11 make a list of precursors of things that

38:13 are not yet ASI but are on the path to

38:15 ASI or are very close to ASI and we say

38:18 okay if you're trying if you're making

38:20 progress on these things the government

38:23 has to know you have to tell you know

38:25 you know the government oversight of

38:29 whatever agency ends up being the one

38:31 responsible for this and they and then

38:35 we can make the decisions about do we

38:36 want to continue down this path how much

38:39 further do we want to go down this path

38:41 In my opinion, when it comes to the

38:43 frontier capabilities, we're already

38:46 pretty much in the danger zone and we

38:49 probably should just take a breather

38:51 until we know what we're doing. But

38:54 importantly, this doesn't apply to 99%

38:57 of AI applications. Most of AI even

39:01 still today is not companies race, you

39:03 know, building the next frontier of

39:05 agentic systems. This is specifically

39:08 the dangerous area and this is an

39:10 extremely dangerous and extremely

39:12 expensive business to be in. Creating,

39:14 you know, GPT6, GPT7 is unbelievably

39:18 expensive and unbelievably hard and

39:20 there's like

39:23 three and a half companies in the world

39:24 that can do this basically. So all the

39:29 other ones we might need other

39:31 regulation for their applications of AI

39:33 but they wouldn't fall under this

39:34 regulation

39:36 at least not yet.

39:38 >> Could prohibition actually be verified?

39:40 I mean like if one country refuses then

39:44 what do we do then? The way I generally

39:46 see it on the international stage is

39:48 that first we have to acknowledge the

39:51 reality of things is that

39:54 for example us in the United States if

39:57 China for example was to credibly

39:59 threaten to build a super intelligence

40:02 that is a threat to the lives of the

40:04 American people and everyone in the

40:06 world for that matter and we should take

40:07 that seriously. Our policy objective is

40:11 not you know stop American companies

40:12 from building super intelligence or stop

40:15 China from building super intelligence.

40:17 It's build stop everyone from building

40:19 super intelligence. No one should be

40:21 allowed to build super intelligence

40:23 because if anyone builds super

40:24 intelligence anywhere in the world we

40:27 lose. This is very important to

40:28 understand. Super intelligence is not a

40:31 tool. It's not a weapon. It's an

40:33 adversary.

40:35 It is a system that will not help us

40:38 achieve our goals. If America creates

40:40 super intelligence, we will not achieve

40:43 American objectives. America will simply

40:45 cease to exist. If China builds super

40:49 intelligence, it will not help them

40:51 achieve Chinese objectives. It will

40:53 simply make China cease to exist.

40:56 So therefore what we need to do is we

40:58 need to find verification mechanisms

41:00 trust but verify create a regime where

41:03 we can mutually verify between

41:04 adversaries that we are not pursuing

41:06 super intelligence that you know we are

41:09 not pursuing super intelligence and we

41:10 make sure they are not pursuing super

41:11 intelligence and we must we must back

41:14 this up with credible deterrence. We

41:16 must be willing to say this is not this

41:18 is a level of of security threat that we

41:20 cannot tolerate and that no one else can

41:22 tolerate. We cannot tolerate China

41:23 building super intelligence. This is

41:24 unacceptable. it is we cannot tolerate

41:28 terrorists building super intelligence

41:30 or anyone else and we must be willing to

41:33 just acknowledge this and say okay now

41:35 we need to figure out how to make sure

41:36 that that's the case the reason I'm

41:38 optimistic as I said is because it is

41:39 not in the interest of anyone to build

41:40 super intelligence this is not the case

41:43 for other AI applications there are many

41:45 non-suptelligent AI applications that

41:47 are very valuable both economically and

41:50 militarily

41:52 that's a very different question but

41:53 when it comes to super intelligence to

41:55 agentic system that we cannot control

41:57 and they can out compete us. This will

41:59 not help us achieve our objectives and

42:01 so we should prevent it from happening.

42:04 >> There's a famous clip of yours um saying

42:08 how none of your friends are having

42:09 children.

42:13 Expand on that and what what that

42:15 comment was all about.

42:17 >> Um that comment um yeah that one blew

42:20 up, didn't it? Uh not my usual topic of

42:23 conversation. I'm usually the AI guy.

42:25 Um, but it is an observation that I've

42:27 definitely been making. Um, as much as I

42:29 focus exclusively on super intelligence,

42:31 it's not the only issue that we are

42:33 facing as a society. There is a there's

42:35 a long laundry list that I have in the

42:36 back of my head. And one of them is

42:40 population decline is that there are

42:42 many many people especially my

42:44 generation and the younger generations

42:46 who are not having kids to a sometimes

42:50 really shockingly low degree. you know,

42:53 not just in Asia where it's particularly

42:55 extreme like in Korea or Japan, but also

42:58 in, you know, Europe and in America and

43:01 now more and more in all other countries

43:03 as well. Birth rates have been falling

43:05 precipitously

43:06 in basically every country on the planet

43:08 except like Kazakhstan for some reason.

43:12 >> And why is that?

43:13 >> And so this is where it gets kind of

43:15 interesting, right? Where everyone has a

43:16 pet theory. Everyone you ask about birth

43:20 rate at decline has their pet theory for

43:22 why it is. It's women's fault. It's

43:24 men's fault. It's capitalism. It's

43:27 socialism. It's whatever. And I looked

43:30 into this quite a bit and the true

43:32 answer is that we don't know. No one

43:34 knows. We don't really know. There is so

43:38 much of foot here. There are so many

43:40 studies that all say contradictory

43:42 things is that we don't really have a

43:44 clean story for why it is. I think

43:46 ultimately it's many things. I think it

43:48 is many things. I think it is related to

43:51 economic issues. I think it is related

43:53 to our relationship with technology and

43:55 how technology changes how we relate to

43:57 other humans. I think it's cultural

43:59 mimemetic. I think a lot of our culture

44:01 has changed around children. I think

44:04 there are there are probably also

44:06 biological reasons. Um I think a big

44:09 reason is housing. It's just no one has

44:12 big enough houses for kids anymore and

44:14 so you just don't bother. I think there

44:16 are many many issues at face. The quote

44:21 I gave back then which became so famous

44:24 was do you know how hard you have to

44:27 abuse a mammal for it not to want

44:28 children. And I think this is a good

44:32 like intuition pump. This is a good

44:33 thing to think about because I'm not

44:36 here to say I have the solution to this.

44:39 What I'm here to say is like something

44:40 is deeply wrong. This is the thing you

44:42 see at zoos for example. There's a

44:44 there's a whole there's a whole science

44:45 of this is that if the um you know the

44:50 the area for the cage for a zoo animal

44:53 is insufficiently good, they stop having

44:56 kids. This is a behavior that many many

44:58 many mammals have. And so there's a

45:01 whole science of like how do you build

45:03 enclosures that are sufficiently good

45:05 for given types of animals to allow them

45:07 to breed? And in a sense, I think it's

45:10 worth also thinking about us as animals

45:13 and our enclosure, so to speak. And

45:16 something is deeply wrong with our

45:17 enclosure. Something is deeply wrong

45:19 with how we are engaging with each

45:21 other, with ourselves, with love, with

45:24 kids, with family. There's something

45:27 wrong. I'm not here to say what it is or

45:29 how to fix it. I think it's many things.

45:31 I think there's probably like 15 things

45:33 we have to fix, you know, in some order.

45:36 But I think it's worth us just h

45:37 acknowledging that there we are ill like

45:41 as a society something is deeply wrong.

45:43 >> It could be a combination of an economic

45:45 problem, relationship problem, loss of

45:48 meaning, having cameras everywhere you

45:51 look. You know, guys don't guys don't

45:52 want to approach approach a girl or

45:54 dance with her.

45:55 >> Yep. Yep. That's

45:56 >> cameras everywhere. That's one that's

45:57 really crazy that I only really

45:59 realized, you know, a couple years ago

46:01 that especially younger generations just

46:04 are really skittish because they know

46:05 they get recorded everywhere they go.

46:08 You know, if something weird happens to

46:09 you on the street or something, there's

46:10 a chance someone's going to film it. You

46:12 know, if you approach a girl at a bar

46:13 and you mess it up, you know, there her

46:15 friends might film you and put you on

46:17 the school Instagram. You know, this is

46:19 a real thing that happens and it creates

46:21 this panopticon effect where people

46:23 become very um very hesitant to take a

46:26 risk, maybe a social risk to maybe

46:27 embarrass themselves. And look,

46:30 embarrassing yourself isn't fun. Never

46:31 has been fun, but it's an important part

46:34 of of being human. You know, humans are

46:36 cringe. You know, people are

46:37 embarrassing sometimes. You know,

46:39 dancing with your friends, yeah, it's

46:41 kind of embarrassing. You know, you

46:42 don't dance that well. That's okay. It's

46:43 part of the fun, you know, and like

46:44 you're 18, you approaching a girl for

46:46 the first time. Yeah, you're gonna mess

46:48 it up, you know, and you know, it's

46:49 going to be awkward, but you know,

46:51 that's okay. It's part of in a sense the

46:53 cycle of being human. And I think this

46:56 is a great example because this isn't

46:57 anything anyone intended. I don't think

46:59 anyone, you know, inventing, you know,

47:01 the smartphone were like, "Oh, yes,

47:03 finally I can stop, you know, young

47:05 girls and boys from, you know, liking

47:07 each other and dancing." That's that's

47:08 definitely my goal. I don't think anyone

47:10 intended to do that, but it still

47:12 happened. And it's just an example of

47:13 like it's just a side effect that in a

47:16 sense so drastically changed the human

47:19 condition and it doesn't even get a

47:21 mention like it's not like there's a

47:23 huge conver national conversation about

47:25 this.

47:25 >> Yeah. You don't even realize it that

47:28 like if this if you read a sci-fi novel,

47:30 you know, 50 years ago and a sci-fi

47:32 novel, young people don't dance anymore

47:34 and have, you know, no love and, you

47:37 know, awkwardness to each other, you'd

47:39 be like, "Wow, that's crazy." But it

47:41 happened. It did happen to a large

47:43 degree and no one even noticed. This is

47:47 why I think this is why I brought up

47:48 that quote to draw people's attention

47:50 to. There's actually a lot of crazy

47:52 stuff happening right under our noses

47:54 and we should be humble about realizing

47:56 that like something is wrong here and we

47:58 should probably really think seriously

48:00 about that.

48:02 >> If AI can do every job better than we

48:04 can, where do people find meaning in

48:05 their lives?

48:06 >> Well, I think if we get to the point

48:08 where there's AI that can do all the

48:09 jobs, uh I think humans won't be around

48:12 for much longer. Um

48:15 but giving that aside, I think there is

48:18 something like

48:21 one of my moral principles that I

48:23 personally have is that you should

48:25 realize where your philosophy breaks.

48:28 There are many philosophical questions

48:29 to which we don't have good answers. I

48:31 think this is one of them is that if

48:33 humans have zero contribution to the

48:36 economy or politics or art or anything,

48:40 what is a good way for humans to live? I

48:43 don't think we have a good answer to

48:44 this question. And so therefore, one of

48:48 my moral principles is if you don't have

48:50 an answer to a hard philosophical

48:53 problem, do not get into a situation

48:55 where you need that answer.

48:58 I think it's possible for us to figure

49:00 out, you know, a culture, an economy, a

49:03 system where, for example, AIs can do

49:06 all our jobs, but we're still happy. I

49:07 think it's possible to develop

49:08 something, but we have not done so. We

49:11 have not yet figured it out. So, I think

49:13 we should not do that. We should not get

49:16 into a situation where AIs are doing

49:17 100% of the work until we're sure we

49:19 know what we're doing, until we have at

49:21 least a decent plan. But we don't have a

49:24 decent plan. When we're talking about

49:26 something this risky, we just shouldn't

49:29 do it. Not yet.

49:33 >> Do people begin preferring artificial

49:35 friends and partners because they're

49:37 easier to deal with than real humans?

49:39 >> Absolutely. I mean, it's already

49:40 happening. This is another thing I worry

49:42 about a lot um that is not super

49:44 intelligence. Like I say, I have a

49:46 laundry list. You know, if uh if if I

49:48 solve the super intelligence things, I

49:50 have a long list of other problems to

49:51 work on. And this is another one of them

49:54 where a a large part of growing up as a

49:57 human is becoming socialized with other

50:00 people. And I'm sure we've all had the

50:02 experience of like having a good friend

50:03 who you like very much but it's kind of

50:05 a mess and then he meets a nice girl and

50:07 suddenly he becomes civilized or you

50:09 know vice versa. And I think this is not

50:12 just because you know man, woman or

50:14 whatever. It's also just because forcing

50:16 yourself to be close to another human

50:18 for long periods of time, learning to

50:20 compromise, learning to, you know,

50:23 smooth over the the frictions within

50:24 your own personality and so on. It's

50:26 just a very important skill to

50:27 cultivate. It's just very important to

50:30 become socialized, to know how to

50:33 navigate, you know, frictions or tricky

50:36 relationships. And this takes a lot of

50:38 work. It's very awkward. It's, you know,

50:41 it's just a lot of work. And I think if

50:44 we don't do that, if we, you know, can

50:46 just have systems that can just, so to

50:49 speak, fulfill our needs without asking

50:52 for anything in return, without forcing

50:53 us to become better versions of

50:54 ourselves, then all things equal, we

50:56 become worse versions of oursel. And I,

50:59 you know, I don't think that's very

51:01 good. This is another example of

51:02 something that I would put into the we

51:04 don't have a good philosophy bucket. Um,

51:06 sometimes I call this the posthuman

51:08 philosophy bucket, so to speak. Like

51:11 similar to the question of like should

51:12 we have 100% of jobs done by AI. I'm

51:14 like that's like a posthuman question.

51:16 Like that's like we don't have an answer

51:18 to that. I think there's another good

51:19 one. It's like should humans you know

51:21 get all their friendship from AI. And

51:23 I'm like dude we cannot handle this

51:26 right now. Like we do not this seems

51:28 really dangerous. Again like selling out

51:30 one of the most deepest cores of what it

51:32 means to be human. Yeah. We shouldn't

51:34 touch that until we really know what

51:35 we're doing. What

51:37 >> what part of being human cannot be

51:38 automated?

51:41 In a technical sense, nothing. In a

51:44 practical sense,

51:47 depends on your view of philosophy.

51:50 >> There's something magical about human

51:52 beings.

51:53 >> Depends on your view of philosophy.

51:54 >> Yeah. I just wonder

51:56 >> there is, you know, if you there's a lot

52:00 of things about human, you know, nerve

52:02 cells and pancreas and so on that's very

52:04 very hard to, you know, simulate. Um,

52:07 but that's obviously not what we mean

52:08 when we say humans are special. Um I

52:11 think there's a lot of things that is

52:14 like special about humans because of

52:16 contingent things like emotions like

52:18 there's a sense for example if you love

52:19 someone I'm sure you've experienced this

52:21 too or many people have experienced this

52:23 is and they have like a little quirk

52:25 like I don't know they they like plain

52:28 toast for breakfast you know for some

52:30 reason you know like in a sense this

52:31 doesn't matter but often if you love

52:33 someone you'll start really enjoying

52:35 that they like plain toast and every

52:37 morning you'll see the plain toast and

52:38 you'll smile

52:40 just because they like plain toast. And

52:42 I think this is beautiful. This is

52:43 something I really like about humans is

52:45 that we can appreciate the arbitrary

52:46 things, you know, that like it's very

52:50 arbitrary, but you love it because you

52:52 love it because you know human because

52:54 they are the way they are. Everyone has

52:55 their own loop quirks. I think there's a

52:56 lot of that that in a sense is what

53:00 makes a lot of what it makes humans

53:02 special. I think a lot of things that

53:04 humans think makes them special probably

53:06 is automatable. You know, I think you

53:09 [snorts] know, science and math and art

53:11 and all that kind of stuff, yeah,

53:14 unfortunately seems very automatable.

53:16 Um, but this is again getting us into

53:19 the posthuman bucket of like

53:22 we shouldn't get into the situations

53:24 where we have to have answers to these

53:26 questions until we've taken the time to

53:27 actually figure these out. I think

53:29 there's probably much better answers

53:30 than the ones I can give. You know, if

53:32 our philosophers spent a hundred years

53:34 more working on this problem, I think

53:36 we'd come up with much better answers

53:37 than I can give.

53:39 >> Yeah. So, going back to trying to stop

53:42 super intelligence, what what exactly

53:44 would you make illegal tomorrow?

53:50 >> So,

53:52 there are many possible definitions for

53:54 the word super intelligence.

53:56 Usually we define it as fully autonomous

53:58 systems that uh pose a credible chance

54:01 of overthrowing the US government as a

54:04 rough yard stick. There's many other

54:06 definitions, but the most important

54:07 thing to understand about this whole

54:09 thing is that um any piece of

54:11 legislation is obviously just a stop gap

54:13 measure. Obviously, we'll not get it

54:15 right on the first try and we will need

54:17 to iterate. You know, we we have a bill,

54:19 you know, that we've written. We have

54:20 legal text that legally defines all this

54:22 kind of stuff. I think it's pretty good.

54:24 But

54:26 imagine tomorrow we have a magic wand

54:28 that we can wave and we can pass any law

54:32 in the whole world. So we wave our wand

54:34 and we pass the ban ASI forever bill in

54:37 all country. What happens? Well, in one

54:40 week it's violated in spirit. In two

54:42 weeks it's violated in letter because

54:44 obviously the companies won't listen to

54:46 it or we're going to forget some edge

54:48 case or the lawyers will come up with

54:49 some really clever reading that we never

54:51 thought about. So, this is an iterated

54:54 game. It's a It's not a oneanddone, like

54:56 we pass the law, we can all go home.

54:59 It's going to be a continued dance. It's

55:02 been a continued fight between, you

55:04 know, law enforcement, judges,

55:05 politicians, the general public, and

55:07 people who are trying to circumvent

55:08 these laws at all costs. So, I think,

55:11 you know,

55:13 I think we have like a nice default, you

55:16 know, idea for what a piece of

55:17 legislation could look like, but it's

55:19 only round one of the game.

55:22 And if America stops and China doesn't,

55:24 haven't we simply handed China the most

55:26 powerful technology ever created?

55:28 >> Super intelligence is not a tool. It is

55:30 not a weapon. It is an adversary. China

55:33 doesn't have super intelligence. You

55:34 can't have super intelligence. Super

55:36 intelligence has you.

55:38 We can't allow China to build super

55:40 intelligence. Not because it will help

55:42 China take over the world. It won't. It

55:43 will destroy China. It just also will

55:46 destroy the United States and all other

55:47 countries. So therefore we don't really

55:50 have a choice. We have to stop super

55:52 intelligence from being created. If

55:54 China for some reason

55:57 is hellbent on building super

55:58 intelligence then we have to stop them

56:01 somehow.

56:02 >> So so how do you enforce global AI pro

56:05 prohibition without creating some kind

56:07 of enormous surveillance state?

56:10 >> Luckily at the moment building such

56:12 frontier AI systems is a massive

56:15 infrastructure project. You require

56:16 massive data centers with very

56:18 specialized chips that are made by you

56:20 know only like one company in the world

56:22 through a supply chain that you know

56:24 goes through all of our western allies

56:25 and the United States etc. There are

56:28 very very very few actors that can even

56:31 attempt something like this. You don't

56:34 need global surveillance to detect them.

56:36 You don't need global surveillance to

56:38 surveil these specific projects.

56:41 It's not the case that everyone can do

56:43 this on their laptop. If we lived in a

56:45 world where you can build super

56:47 intelligence on a laptop, I don't know

56:49 what we would do. I think that would be

56:51 a very very dangerous and a very

56:53 unstable world and I'm not sure we would

56:55 survive that

56:56 >> because at some point a a relatively

56:57 small group could could secretly build a

57:00 dangerous AI. Right?

57:01 >> That's a lot of the risk. One of the

57:03 reasons that we need to pause and stop

57:05 as soon as possible is so that we have

57:08 extra time before people figure out even

57:11 better ways to build AI. One of the

57:13 problems for example also why just say

57:16 reducing say the number of GPUs the

57:18 number of computing power working on AI

57:21 doesn't work is because then the

57:23 companies will just invest in working on

57:24 you know better software better

57:26 algorithms we have to stop both

57:29 one thing it's not just that the

57:31 hardware keeps getting better it's that

57:33 our software our AI systems get more

57:35 efficient more effective and yes maybe

57:38 at some point we could build software so

57:40 good that it could build super

57:41 intelligence on very small amounts of

57:44 computing power.

57:46 It's very hard to imagine how we would

57:48 handle such a world. I think if this

57:50 happens right now, we're screwed. I

57:54 think if we spend 20 years building

57:57 better institutions, better

57:58 understanding, you know, of how to

58:00 control these things, of how AI works,

58:02 having better enforcement, better laws,

58:05 etc.

58:06 Maybe we can figure something out. You

58:08 know, maybe there is a, you know, we we

58:11 make the chips in such a way that they

58:12 don't work for super intelligence, but

58:13 they do work for other things. I don't

58:15 know.

58:16 >> And and who would get to decide what

58:17 level of intelligence humanity is

58:20 allowed to build?

58:22 >> Well, I mean, as a as an American, I

58:25 believe the people, of course, democracy

58:27 should decide. This should be up to, you

58:29 know, votes and our elected

58:31 representatives in general. But

58:33 ultimately this is a negotiation like

58:35 anything else. I think this would also

58:37 be a negotiation with other countries in

58:38 a sense of like what are the levels of

58:40 risk that we want to tolerate. I would

58:42 like us to get into the situation

58:43 honestly and we can all agree we don't

58:45 want super intelligence and now we just

58:46 want to talk about the pace. I think

58:48 this would be a good situation to be in.

58:49 I would love if our conversation would

58:51 be should we move this much per year or

58:54 this much per year. I think this would

58:55 be a great place for the world to be at.

58:57 I think we're very far from this still.

58:59 >> And are are you personally a conspiracy

59:01 theorist and believe in a lot of crazy

59:04 stuff or is this just the first time

59:05 that you've seen something that

59:07 >> I don't consider this very crazy at all.

59:08 I think it's a very standard prediction

59:10 from our standard understanding of

59:12 science as it currently exists.

59:13 >> Mhm.

59:14 >> I find conspiracy theories extremely

59:15 entertaining, particularly dismantling

59:18 them. Uh when I was a teenager, I was

59:20 quite obsessed with skeptics and like

59:22 tearing apart conspiracy theories,

59:24 finding why they were wrong and so on.

59:27 And well, it turns out this one wasn't.

59:29 >> So, you're 31 years old. Are you

59:32 pessimistic about the future for

59:33 mankind?

59:35 >> Pessimistic, I think, is not the word I

59:37 would use. I do think that if things

59:39 continue on the track they are currently

59:41 on, there is almost no chance of a good

59:44 outcome. But I also truly deeply believe

59:46 in our ability to change things. I truly

59:49 believe that humans can change things

59:51 and they can change a lot very quickly.

59:54 We are not currently on track. If we

59:57 continue as we currently are for the

59:58 next one, two, five, seven, whatever

1:00:01 many years, we're not going to make it.

1:00:04 But we can still change that.

1:00:06 >> And what what kind of things would have

1:00:07 to happen in order to make this a

1:00:10 reality?

1:00:11 >> Fundamentally, the main thing which you

1:00:13 know is not exciting is just we need to

1:00:16 do the thing we do for any kind of

1:00:19 technology. We have to regulate it. We

1:00:21 have to understand this issue. We have

1:00:23 to have the conversation and we have to

1:00:25 make choices. The first and foremost

1:00:27 important thing is we have to we have to

1:00:28 staunch the bleeding. Super intelligence

1:00:31 could be built as soon as in one or two

1:00:33 years from now. This is a thing that

1:00:35 many scientists do take seriously. Maybe

1:00:36 it'll take longer, maybe it won't.

1:00:38 Unclear. The first thing we need to do

1:00:40 is prevent that from happening because

1:00:41 super intelligence gets built while

1:00:42 we're all still arguing. It doesn't

1:00:44 matter. So the first and most important

1:00:46 thing we do is we need to stop stream

1:00:47 intelligence right now and then we take

1:00:51 the time to figure out what do we need

1:00:54 to do next? Where do we go from here?

1:00:56 What are the things we want to do etc.

1:00:58 The most important bottleneck to this at

1:01:00 the moment is actually awareness. So, a

1:01:03 lot of people are shocked by this, but I

1:01:05 talk to politicians all the time, uh,

1:01:07 including military officials and many,

1:01:10 many other people, and the number one

1:01:12 reaction I get when I talk to them about

1:01:14 this issue of super intelligence, it's

1:01:16 just, "Oh, wow. I had no idea. That

1:01:21 seems really bad. What can we do?" I

1:01:25 very rarely have a problem with

1:01:26 persuading people. The number one issue

1:01:29 I have is that most people have just not

1:01:30 heard of this issue. It's still pretty

1:01:32 new. It's just not a thing most people

1:01:34 have heard about. And once I talk to

1:01:36 people, general public, you know,

1:01:37 military official, politicians, etc.,

1:01:39 most people are like, "Hey, yeah, that

1:01:42 seems really bad. We should like figure

1:01:44 out what to do." So, in a sense, we're

1:01:46 not even in the position where everyone

1:01:48 knows about this and now we're arguing

1:01:49 about how to fix it. We're still in the

1:01:51 phase that most people have not even

1:01:52 heard about that this is happening. So

1:01:54 once we get to that stage where you know

1:01:56 enough people have heard about this

1:01:57 issue and are demanding action when

1:01:59 citizens in the United States and

1:02:01 elsewhere are demanding action from the

1:02:02 policy makers a lot can happen pretty

1:02:04 quickly and a lot can happen and a lot

1:02:07 can be done.

1:02:09 >> You once said that uh if super

1:02:11 intelligence appeared tomorrow you'd

1:02:12 probably call your mother. What would

1:02:14 you tell her?

1:02:16 >> I love you.

1:02:22 Is there any direction you could think

1:02:24 of that would

1:02:26 change this this dark future that you

1:02:29 see?

1:02:29 >> Yes, absolutely. It's why I do the work

1:02:31 I do every day. That's why I'm here

1:02:33 today.

1:02:33 >> Well, we're where super intelligence

1:02:35 continues to evolve.

1:02:37 >> I think we can stop super intelligence.

1:02:39 >> Just stop it.

1:02:40 >> It does not yet exist. So, we can still

1:02:42 win,

1:02:42 >> right?

1:02:43 >> Once super intelligence starts existing.

1:02:46 >> Yeah. Unlikely that we have much chance.

1:02:48 And how many years do you see that?

1:02:50 >> It's hard to say. Um

1:02:53 the somewhat joke, somewhat serious

1:02:56 answer that I've been giving since 2020

1:02:59 is the same. It's 30% by 2027,

1:03:04 50% by 2030, 99% by 2100, 1% already

1:03:09 happened.

1:03:16 H

1:03:17 Connor, let me uh let me ask you one

1:03:19 last question. What what would you like

1:03:20 to

1:03:23 communicate just in one sentence to to

1:03:26 everyone watching?

1:03:27 >> It is not over. Contact your

1:03:29 representatives. Go to controlai.org.

1:03:32 Demand action to be taken. Talk to your

1:03:34 friends. Talk to your family. Talk to

1:03:35 people. Make this an issue that we care

1:03:38 about. And a good future is still

1:03:40 possible. We can make a wonderful future

1:03:43 with technology, with AI, and all these

1:03:45 other things. We can do it. It's going

1:03:47 to take a lot of work.

1:03:50 >> All right, Connor Ley, thank you so much

1:03:51 for sharing your thoughts on AI and

1:03:54 super intelligence, and we'll see what

1:03:55 happens.

1:03:56 >> Thank you.

1:03:57 >> Thank you very much.

📬 Never miss a Soft White Underbelly video — every new upload summarised in your inbox. Follow free

Summarize any YouTube video instantly

Get AI-powered summaries, timestamps, and Q&A for free.

Generate your own summary →
More summaries →