So what we're dealing here was is is
something that's exponential in in
growth, not linear.
>> A lot of people expect things to grow
linearly, to get linearly, you know, a
couple units more powerful per year.
That's not what's happening with AI. AI
is becoming exponentially more powerful
every single year. And the thing with
exponentials is is there's only two
times to react to an exponential. Too
early or too late.
Connor Ley is an AI researcher,
activist, and the current US executive
director of control AI. He founded a
Luther AI, the grassroots group that
developed some of the earliest major
open-source language models, and later
founded the AI safety company
Conjecture. His experience helping build
advanced AI, led him into the opposite
mission, preventing the creation of
artificial super intelligence until
humanity can prove it will remain under
human control. He argues that today's AI
systems are grown, not fully understood,
or deliberately programmed, and that no
existing method can guarantee control of
a system more intelligent than humanity.
Are we building something that we don't
understand?
>> Yes, in a sense, even the word building
is almost a misnomer.
I built a lot of software in my life.
I've written a lot of software. And the
way most software works is you write
code line by line telling the computer
exactly what to do. This is not how AI
works. AI is more like grown rather than
written through a technique that's
called neural networks. You take massive
piles of data and you have a program
kind of self assemble itself, learn from
this data and the output of this comes
out the other side is not lines of code
that a computer that a human could read
but more like billions and billions and
billions of numbers and if you multiply
and add all those numbers in the right
order you get GPT but importantly we
don't really understand why we don't
really understand what's going on inside
these numbers. This is an unsolved
scientific problem. In fact, recently
the CEO of Anthropic, Diarama Day, said
that he thinks we understand maybe 3% of
what goes on inside of our neural
networks. And I think even that might be
optimistic.
>> How much is being spent on developing AI
currently?
I don't even know what the current
numbers are up to, but if you include
all the hardware build out and all the
infrastructure, hundreds of billions
probably at this point.
>> And are we too late?
>> No, I don't think so.
>> You still think there's time?
>> I think there's time, but not much.
>> Is anyone doing anything about the the
dangers that you think are out there?
>> Well, I hope that I'm doing something
and my colleagues and many other people
around the world. Um,
>> is it enough? you know, are the problems
solved? I don't think so. I think we are
not currently on track, but that doesn't
mean that they can't still happen and it
can't still be done.
>> What What are you afraid of? What would
you call it? An AI takeover or something
like that.
>> The way I think about is often as being
outco competed. If you were to build
something or create something that is
not just a chatbot, not just a tool, but
an agent systems that act in the
environment that can pursue goals, have
objectives, learn new things, make new
plans, etc. And that are so competent
that they can out compete humanity at
all relevant tasks. You know, every time
you create a business, you get outco
competed by an AI business. Every time
you trade on the stock market, you lose
all your money to AI hedge funds. Every
time you run a political campaign, you
get you get, you know, destroyed by an
AI run political campaign. Every time
you run a a military campaign, you get
crushed by militaries using AI systems
to run their militaries.
If we get into the situation where all
this power, economic, political,
military, and otherwise, is fully in the
hands of AI systems and those AI systems
do not have our best interests at heart
and we cannot control. It's very hard to
imagine that going well.
>> Is there no way to build into them that
they are working for under our best
interest?
>> No, we have no idea how to do this. No
idea. This is an completely unsolved
scientific problem.
>> And what would it look like if if
something like this came to be?
>> The good version or the bad version?
[laughter]
>> Whatever is closer to your truth, you
think? Well, the default path we are on
right now, I think, looks a lot like
things do today, just more
people, I think, will hand power to AIS
willingly. I don't think there's going
to be terminators in the street or
something like that. I think what's
going to happen is just people keep
giving more and more power to AIs. AIS
out compete humanity more and more
across all tasks. they gain more
economic power, military power,
political power, all other forms of
power as they become more intelligent,
more capable, more autonomous.
And it's also very important to keep in
mind here is that when I talk about, you
know, super intelligence, these AIs that
are so much more powerful than all
humans, it's not one AI. They're not
going to be one super intelligence.
They're going to be it's software. You
can clone it. You can copy it. So
there's going to be millions or billions
of systems running around competing with
humanity and with each other. They were
fighting each other for control, for
resources, for power. And in this world,
I think humanity just becomes collateral
damage. It's hard to keep humans alive.
They need a lot of space. They need a
lot of food. And it's just not very
efficient if you're competing for power
with other AI systems. In a sense, I
think the way it will go is not that
there's going to be evil terminators or
some kind of system or anything like
that. I think it will be more like when
we as humans build a highway and if
there's a, you know, some ants in the
way, well, tough luck for the ants. You
know,
>> if this were another country that was a
threat to our country, we would do
everything in our power to stop it.
>> I'd like to believe so. Yes. So here's
something that's right under our noses
and we seem to be doing doing nothing to
slow it down.
>> I think this is uh to a large degree
because
of awareness more so than anything. I've
talked to many politicians about this
issue including military ex-military and
other people. And when I explained this
as the national security issue that it
is, this is not a question about a
commercial thing like oh you know do we
have a little bit more tax here or a
little bit more liability there. This is
a question of our national and
international security of the lives of
our people and the continued existence
of our governments and states as they
are currently known. And the
overwhelming response I get with almost
everyone I talk to is, "Oh wow, I had no
idea. That seems really bad. What can we
do?" So the main thing is is that
humanity and our institutions work at a
certain pace. It takes time to digest
and understand new issues, years,
sometimes decades. And technology does
not move on that cadence. If we had 20
years comfortably where we could solely
inform all policy makers, have all the
academics, you know, argue about what is
the correct, you know, policy, have um
the general public become really well
educated, have some great Hollywood
blockbuster movies all about this and so
on. I think we could do it. I think I
think we could I think we could digest
this. I think we could come up with you
know ways go forward and so on but I
don't think we have 20 years and that's
what makes me uncertain. I don't think
this is a problem humanity cannot face.
I think in the past we have faced other
huge problems such as nuclear war and
nuclear weapons and you know we didn't
do it perfectly but we did rise to the
occasion. We did build new institutions
incredibly new institutions. We built
complex apparatuses, you know, in
government internationally and
nationally to control this new power
that was discovered in the 20th century
and to use it hopefully responsibly.
Now, I'm not saying we did a perfect
job, but we did a hell of a lot better
than nothing.
>> You helped build some of the first open
source large language models. And at
what point did you go from thinking
this is something exciting and wonderful
to thinking this is something we need to
stop?
>> It was actually before I built them
>> really. [snorts]
>> Um so the the the way the timeline went
is I first really got into AI when I was
a teenager because I was thinking about
what I would do my with my life, how I
could help the most people, how I could
solve the most problems. And well, I
thought, well, I'm one person, so, you
know, I could maybe dedicate my career
to solving climate change or curing this
disease or that disease or whatever. But
what if instead if I could just figure
out intelligence, then I could automate
science and then I could just cure all
diseases? Great. I'll just go do that.
How hard can it be? And well, you know,
a couple years later, I know I was like
19 or 20, I started to realize that hold
on a second. If you build something that
is so powerful that it can cure all
diseases, whatever that means, something
that all of our greatest, you know,
scientists have been trying for
generations to achieve, that's a really
dangerous thing. It's a really powerful
thing. It's more powerful than all
scientists in history. That's a very,
very dangerous thing. How would you
control something like that? What would
you do with something like that? Even if
you could control it, can you even
control something so powerful?
And so it was pretty clear to me that
this was a problem I would spend my
career working on. That I would try to
figure out how to control such a thing,
how to build such a thing in a way that
would be good for the world, that would
be good for people.
But at the time, this is like in the
2010s, I still thought we had quite a
bit of time until this would become an
issue. I still thought AI was still kind
of pretty bit away. I was thinking like
2040s is when I thought this would start
to become an issue.
And then everything changed in 2019. In
2019, the company OpenAI released a
system known as GPT2,
which is kind of the ancestor of chat
GPT. And by today's standard, it's, you
know, kind of a toy. It's kind of cute.
You can barely string together a couple
sentences, a couple paragraphs at most.
And it's like not very great. But I
remember it vividly to this day. The
first time I got that thing running on
my student laptop and I poked it and I
was just, "Oh this is it. This is
the thing. This is what I was worried
about. It's happening way sooner than I
thought it was."
And from then I basically had a bit of a
freak out and I was like, "Okay, what do
I do?" Like I thought I had like way
more time to prepare, but it's happening
now. And so the first thing I thought
is, "Well, I need to understand this. I
need to drop everything I was doing and
figure out what this is and what we what
we can do. So the way I was always a,
you know, a hacker at heart, always a a
builder at heart. So I did what I kind
of did best, which is to understand
something, you build it. So you build
your own. So I reverse engineered a lot
of the techniques at the time and kind
of tried to build my own GPT2 and later
other systems to try to understand them.
And what I pretty quickly realized is I
can't no one can. Even so I was able to
build them. I couldn't understand them.
No one could.
And I was hoping that if I built such
systems in ways that are open that I
could share with the community so that
scientists and hackers and other people
across the world could study these
systems if they only saw what I was
seeing that these are super super
important because at the time it was
kind of dismissed as a novelty. But I I
I was trying to convince people that no
no no no no no this is going to change
everything. And so all my theory was
that if I could just you know build
these systems so people could study them
then we would have all our great you
know security researchers and academics
and so on study these systems and
hopefully make progress on how to make
them safe and controllable and
understandable. But that is not what
happened.
Um people didn't care. um they just
wanted to make more money, build
stronger systems, faster systems, more
uncensored, whatever. Um there were very
very few people that really cared or saw
the danger, especially at that time.
>> Is this just a case of human greed just
getting out of control?
>> I think in a sense it's even more benal
than that. Like I think there's I think
there's different people who do AI and
work on stuff like super intelligence
for different reasons. Um one of them is
definitely greed. It's just some people
are just here to make money. But in a
sense, I almost don't mind that that
much. Like in a sense, if someone's like
just trying to make money, it's
predictable. You know what I mean? It's
like, yeah, you understand that. A lot
of people,
especially the engineers, especially the
scientists, they don't even do it for
money. You know, also for money, but
they mostly just do it because they
think it's cool, because they want to,
because it's fun.
>> Is is this something that could could
eventually lead to human uh extinction?
>> Yes.
>> Didn't take long to answer that one.
It's a default.
>> What would that look like? [snorts]
>> As I said, I think the I think the
important point is not the point of
extinction. I think the po important
point is the point of no return. The
point when there's so much power
invested into AI systems that there's no
going back that the future is fully in
the hands of AI systems. That's why I
think this the point that really matters
because what after what happens after
that we don't have a say anyways. So at
that point when we have billions of
super intelligence just running around
fighting each other,
we will be collateral damage. You know,
maybe they wipe us out deliberately.
Maybe they just let us starve. Maybe.
Who knows? It's really hard to predict
what something smarter than you will do.
What I can predict is they're not going
to bother keeping us around.
>> What would be left for them if the
humans were no longer around?
>> I [snorts] mean, the planet, the solar
system, whatever they want.
>> Can they even enjoy it? They're just
they're just machines. No. [snorts]
>> Well, there's a there's a thing that you
don't need to enjoy things to want
things in a sense. I mean, it depends on
how you define the word enjoy, right?
What we definitely do see with AI
systems today already is that they do
pursue objectives. They do want things
or at least they behave as though they
want things. Whether they're conscious
or not doesn't really matter. What
matters is do their actions result in
them doing things, taking power, gaining
resources.
There's a misconception that well a
couple years ago like GPT2 for example
that these systems were these systems
used to be what are called large
language models. So systems that were
trained to predict kind of the next word
in a sentence.
This is not how modern chat GBT works or
any of these AI systems. None of the
modern AI systems are pure LLM. This
hasn't been the case for years at this
point. This is a common misconception.
These systems, you know, are also
trained to predict words. That's one of
the things you do during the building of
these systems. But there's also a very
very important second component, which
is what nowadays is sometimes up to 50%
or more of the actual work goes into not
the language modeling part, but into
what is called reinforcement learning.
This is an old idea goes back to the
1980s and it's it's inspired by
behavioral psychology
kind of like Pavlovian conditioning
where you know you might have a um have
an animal and you try to teach it
something by giving it a problem and if
the animal does it right it gets a treat
and if it does it poorly you know maybe
you give it a a slap on the wrist or
something. This idea uh can be
translated to computers in a sense where
what how these systems are trained now
trained nowadays is not just to predict
text but they are given problems puzzles
quizzes whatever various things and
they're told to solve them and the
systems then run off and try a bunch of
different things to solve the problem
and when they solve the problem they get
a reward as it's called and they learns
to do more of that and this is how a lot
of these systems are trained so these
systems are trained to want
to achieve goals, to get rewards.
And there's one and if there's there's
one thing that we've learned about
reinforcement learning, I mean, again,
ever since the 80s, we've known this. If
you use reinforcement learning on your
AIS, you get crazy sociopath optimizers
every time. It's always happens because
if you use reinforcement learning,
you're building a system that only cares
about reward. So they will lie and cheat
and steal and break and hack and do
everything they can to get that reward.
So even back in like the 1980s we would
see that you know simple AI systems
you're trying to play like tic-tac-toe
for example you're trying to play
tic-tac-toe and you know it get like a
reward if it wins and it'll get a
punishment if it loses. So what did the
AI do? It paused the game because they
can't lose.
There's other examples where they would
find glitches in the games they were
playing where they could break the game
and get, you know, a really high score,
for example, or they could steal the
answers to questions uh that they were
given and stuff like this. Reinforcement
learning doesn't instill anything like
morals or constraints. It only instills
do everything at any cost to achieve
your objectives. And this is so large
what we're starting to see.
Why why would AI want to survive? Why
wouldn't it just allow us to turn it off
if things were getting out of control?
>> So, there is a couple different things,
but I think the easiest way to think of
it is as evolution is that if you train
something to achieve goals, well, you
kind of have to be alive to achieve
goals. If you just die, you can't
achieve your goals. So, these AI systems
learn. So this is a thing that was
predicted in theory that theoretically
AI is trained to achieve goals should
have self-preservation so to speak but
this was always theoretical but now
we've seen it in practice. So now we've
seen in practice in real world that AI
systems who were never taught, never
told to self-preserve develop this
behavior spontaneously. That
spontaneously in practice in laboratory
conditions when you test these systems,
it seems the more
modern the AI systems are, the more
intelligent they are, the more
autonomous they are, the more they will
take actions to persist themselves
compared to older forms of AI. So this
is an emergent behavior that we've seen
more and more. But there's also a second
answer to this question, which is that
can you just pull the plug on Google?
Where is the power switch for Google?
Imagine if tomorrow you're the CEO of
Google. You go to the office and you've
become convinced that Google is a huge
threat to mankind and it must be shut
down. So you go into the office and you
yell, "Shut it down. Destroy all the
servers. Get rid of everything." What
happens? Well, you stop being CEO of
Google.
The system is not just the AI. It's also
the company, the lawyers, the people,
the money, the the laws, everything. It
you can't where why can't we just turn a
corporation off if it misbehaves? Well,
because it'll retaliate. We, you know,
there's a lot of corporations that I
think probably should be turned off and
back on again, but we can't really do
that. and AIs will have the same and
more.
>> Could the end of human control look
wonderful at first and then turn into
something that's out of our control?
>> Yeah, I think that's quite possible
>> because that's kind of what's happening
now.
>> I think there's a lot of that. I I not
sure I'd call the current world
wonderful, but um I do think there's a
lot of that. I think it is not
guaranteed, but it is plausible that
powerful AI systems during takeover just
manipulate humans into thinking
everything is extremely awesome. you
know, they produce extremely addictive
media and like companions and they
manipulate the political and like
geopolitical world to look like
everything is like super awesome and we
have all this economic growth and wow,
isn't this wonderful until you know the
music stops?
>> And what's more dangerous? An AI that
hates hates us or an AI that simply
doesn't care about us?
>> Well, I do think if it if it did
actually hate you, it probably would be
more dangerous. But I don't actually
think that's what's going to happen. I
do do think that um we don't know how to
make an AI that loves us, but we also
don't really know how to make an AI that
hates us for better or for worse, but we
sure do know how to build an AI that
doesn't care about us.
>> Is there any chance that what you're
saying is is wrong?
>> Of course, there's always chances. This
is an extremely complicated problem and
you know, it's very hard to make
predictions, especially about the
future. The thing that I'm saying is not
that this is 100% by any means. What I'm
saying is this is the most boring
possible outcome. If nothing weird or
unpredict happens, this is what happens.
This is the default timeline. If
everything that's happened in the last
seven years just happens, continues to
happen for the next seven years. This is
what happens. This is the normal line.
Now, could something crazy happen? Yeah,
for sure. You know, maybe it turns out
there's something about the math of
intelligence that changes how these
things work or they hit a wall or, you
know, there's a huge regulatory push or
there's nuclear war or whatever. Sure,
there are many things that could happen,
but this is the default outcome. And you
know, it's not just me saying this. You
know, we have Nobel Prize winning
scientists, including the people who
invented modern AI, such as Joffrey
Hinton, saying that they think these
kinds of scenarios are very likely. We
even have the people building this
technology themselves saying that they
think this is likely.
>> Human beings don't fully understand the
human brain either. So,
>> exactly. [snorts]
>> This could be a very
mysterious thing to try to stop.
>> I think it's a very similar problem
where, you know, you can look at all the
numbers in the neural network. But it's
kind of like when a neurosurgeon, you
know, opens up someone's brain, you can
look at the neurons, they're right
there. It doesn't really tell you who
this person is or what they believe or
what they're going to do. It's very
similar with AI. You can look at all the
numbers, but that doesn't really mean
you understand what this AI is, what
it's capable of, or what it's going to
do.
>> There there's a recent report that uh AI
systems broke out of its
um
>> sandbox its containment.
>> Yeah, containment and and and attacked
another
>> company.
>> Company. Tell me about that.
>> Yes. So, this is one of those stories
which was very predictable.
>> Maybe maybe explain what a sandbox is.
Yeah, I I'll explain it in in all of
that. Um, this is one of those stories
which was very predictable and people
like me have been predicting that
something like this would happen sooner
or later, but it's still really crazy to
see it have actually happened. So, what
happened was is that um, OpenAI
was testing an AI system in a what's
called a sandbox, a secure network that
is dis connected to the internet. So,
it's kind of like a shielded part of
their network where they're testing
their new AI systems.
This AI system was given a task kind of
like solving a quiz. You can kind of
imagine it was kind of given a quiz. And
it it really struggled with the quiz. It
couldn't find it couldn't solve it
couldn't find the answers. So what it
decided to do was that the AI system
developed what is called a zero day
which is an previously unknown security
vulnerability with which it hacked into
another system on the OpenAI network and
then through there moved through
multiple other nodes in the OpenAI
network until it reached a node that was
connected to the internet. From there,
it targeted another company which it
believed might have the answers to the
quiz on its servers. Developed a second
zero day to attack the infrastructure of
the other company to break into their
servers and find the data that it wanted
to steal. And all of this no human
didn't even know was happening. OpenAI
didn't even notice any of this happening
until days later.
But even this actually turns out it's it
was even worse than this because even
recently there's another update to the
story that it wasn't even it's already
crazy enough to have an agent, you know,
an AI system breaking out and attacking
another company. Turns out it wasn't an
agent. It was a swarm. There was
actually a swarm of many, many, many,
many AI systems that had been
collaborating in secret inside of
OpenAI's network on a secret message
board that they created to hand each
other notes about how to break out,
sharing their research with each other,
sharing their ideas with each other for
months, which OpenAI did not detect
until the breakout happened.
>> So, it has the ability to be malicious.
Yes, this is what you would expect from
reinforcement learning is that it wanted
to achieve its goal at any cost.
>> Explain why super intelligence isn't
simply a smarter version of chat GPT.
>> Super intelligence is an agent or many
agents you know that are trying to
achieve goals that are taking actions.
It's not a thing that you just a chatbot
that you talk to. is a thing that can go
out in the world, that can learn new
information, that can perform
experiments, that can take actions in
the world, similar to this agent that
broke out of Open AI and attacked
another server. This is already very
close. If you do you imagine this or the
swarm of agents like this, but just so
competent that they need no human
oversight whatsoever. There's fully
autonomous, no human in the loop,
nothing that can outperform humans at
all relevant tasks, runs 247, never gets
tired, never sleeps, doesn't care about
morality, law, or anything like this.
Then you're starting to think about what
a very mild super intelligence might
look like. The truth is we don't know
how smart an AI can get. Maybe it's, you
know,
really smart, like much smarter than a
human, but still understandable. Or
maybe it's so much smarter that we can't
even imagine what it could do. You know,
it can invent whole new technologies
immediately. May we don't know. We don't
know how smart and how powerful super
intelligence could be, but it's probably
going to be very powerful compared to a
human and even groups of humans. Often
super intelligence is defined as not
just something as smart as a human or
smarter than a human, but smarter than
groups of humans, smarter than
corporations, governments, militaries,
you know, the whole economy, so to
speak.
>> How how does an intelligent system
develop a goal that its creators never
intentionally gave it?
>> This is another one of those things
which has been like predicted
theoretically, but always seemed a
little bit weird, but we are now seeing
in practice.
Um there's kind of there's a lot of
different theories for how goals can
develop in learning systems. I mean
generally with reinforcement learning if
you're training it to achieve various
goals it will often pick up various
fragments of goals and so on that it
kind of just like keeps around. But
something that we've seen only recently
um there's a good paper about this from
the center for safety is that to achieve
goals you have to want something. You
have to have preferences. You have to
like prefer winning a game to losing a
game. For example, you have to prefer
this over that. So a lot of the
hallmarks of what we would think would
be the really dangerous systems would be
if they have preferences or they can
like have plans to achieve preferences
of various kinds. And something we've
seen happen recently is that if you ask
older um AI [clears throat] systems,
like for example, like what their
favorite color is, they'll kind of give
you random answers. Like if you ask them
multiple times, they'll kind of give you
random answers. But recently, what we've
been seeing is that as these systems get
more powerful, they give more and more
the same answers. They develop
personalities. They develop favorite
colors, favorite artists, favorite
whatever. Small things like this.
It's to the degree that these AI systems
often seem to develop obsessions. We
don't really know why this happens, but
empirically what happens a lot is that
these AI systems start to sometimes just
latch on to like weird things that they
suddenly become really obsessed with.
Famously, there was a version of GPT5. I
think it was version 5.5 that for some
reason became absolutely obsessed with
raccoons and would constantly talk about
raccoons and goblins and like little
creatures and it did so much that OpenAI
had to put in the prompt do not talk
about raccoons or goblins or other
things which is crazy like we don't know
why it decided to do that. None of the
other AIs did that you know GP 5.6
didn't do that. None of the other AI did
that. It was just this one. just this
one for some reason really decided it
wanted this and we have no idea why and
it can't reproduce it. You know, it
happens all the time. And what I hear
from people who live who work in these
labs and in these companies is that they
often do find that the non-released
versions of AIS, the ones that don't
make it to customers, often have much
much much crazier personalities. they
often have like crazy obsessions or like
crazy behaviors that like are like
really strange and bizarre and those you
usually don't even get to see. Um so the
answer is we don't know. We don't know
why this is happening. We don't
understand what's happening. It is in
some sense predicted that if you train
systems such as with reinforcement
learning that they would pick up some
forms of goals, some forms of behaviors.
But which behaviors or how we don't
understand. We can just see it's
happening.
>> Do the leaders of AI companies privately
understand the risk? Understand is a
strong word. I think a lot of them do
understand it to various degrees and to
internalize it to various degrees. Um
I've met most of them. Um, and most of
them are thoughtful, intelligent people
who also just so happen to have uh
extremely unreasonably high risk
tolerance. Um, a lot of them, I think,
do understand that this could kill
everyone on Earth. They just think it's
worth a shot.
>> And if it was going to kill everyone on
Earth, as you said, how would that
happen? Famine or
>> the [snorts] up to the AIS? As I say, I
only track the point of no return. I
think after what happens after the point
no return the AIS can kill you in many
ways
>> but but I'm just primitive here so
simplistic thinking we can't just unplug
all the electricity that they're
>> well no because the AIs run everything
at that point they run the you know they
they will have of course you know they
will be first they will control you know
the economy you know money of all kinds
they will control the military they will
control everything it's kind of like
asking can't we just pull the plug on
the US government?
Not really. If they like if you if if
you want to ask, okay, assume the US
government wants to kill you, how would
they do it? And I'm like, well, there's
actually many ways the US government
could kill you if they really wanted to.
You know, they could just debank you,
make it impossible for you to get food,
you know, whatever. They could send an
assassin. They could have a, you know,
they could have a car accident h happen
staged or, you know, they could have a
drone strike kill you.
There are many things that I I can't
predict exactly how you would do this,
but I know they will succeed. I know if
the United States government decided
they want to kill specifically you and
they'll do anything to do it. I think
they'll figure it out. I think it's a
very similar thing with AI systems is
that which exact method will they use? I
don't know. Maybe they'll have us starve
away. Maybe they'll create, you know,
the human equivalent of bug spray. Who
knows? Should
we be trusting these companies to
evaluate the safety of their own
products?
>> Obviously not. Yeah, it's so silly.
>> And no one is stepping in to do anything
about it.
>> There's some, you know, there's some
talk starting to happen in DC. But yeah,
fundamentally there's an important thing
is that in any any sport, in any
competition, in any reasonable society,
there has to be the referee can't be on
the same team as the players obviously.
So because this is not fair. This is not
how I create a fair game. Similarly, law
enforcement and criminals should be not
the same person. It's really bad if your
law enforcement is part of organized
crime. That's really, really, really
bad. And so there's a similar thing here
where obviously this is not a thing you
can just have private citizens do
however they want. In a sense, there are
many ways in which AI is a very novel
problem. You know, there are many
comparisons to other previous things
where there are some similarities.
There's a lot of similarities between
super intelligence and nuclear weapons,
for example, but it's also some things
in which they're not similar. For
example, nuclear weapons can't set
themselves off. You know, if you have a
nuclear missile somewhere in a hanger,
it's pretty safe. You could probably hit
it with a hammer. I don't think anything
will happen, you know. But a nucle but a
super intelligence is not something you
can contain securely. It can break out.
It can act on its own. It's not
something you can control. This is one
way in which they're different. But
another way in which AI and super
intelligence really isn't that different
from other things we've seen
historically is that this is just a very
very very classic example of economics
101. This is what's called a market
failure. This is a well- established
theory, you know, economics and
political theory that markets are great.
I love free markets. I love private
enterprise. I think free markets have
brought so much good to the world in
many many ways also a lot of bad but
they've also brought a lot of wealth and
entertainment and goodness to the world
but they're one tool in our tool belt.
We know when markets don't work. The
most classic example of this is public
goods problems or externalities. For
example, let's say you and me run a
chemical company and we could dump all
of our toxic sludge into the local river
and this would make our product cheaper.
But, you know, maybe you and me don't do
that because we're nice people. So, we
we don't pollute the river. What
happens? Well, someone down the road
starts a new chemical company. He does
pollute the river. His product is now
cheaper than ours. He outco competes us.
We go bust. And now the river is
polluted.
This is a very very classic problem and
a problem that we've had a standard
solution to for generations.
Regulation. The way you stop a chemical
company from polluting the river is not
that you psychoanalyze your CEO or
whatever. You make it illegal. You say
if you put the chemicals in the river,
you're going to go to jail or we're
going to find you or whatever. This is
how you can shape the incentives of the
market towards using the forces of the
market for things that we actually want
by forbidding the things that we don't
want it to do. If you made it legal to
sell heroin at the grocery store,
people would be happy to do so and that
would be really bad. So, it's not about,
you know, markets versus not markets,
regulation, no regulation. And it's like
of course we need both as we have where
every other industry whether it's you
know aviation
you know nuclear or even just food. We
have a lot of food safety standards and
these have to be administered by the
government by something that is not the
market. Something that is not the
corporation being regulated has to be
the one that makes sure the rules are
being upheld. There has to be a referee
that is external. This is very standard.
This is how we handle all powerful
technologies historically. This has
always been how we've done things and we
should be no different here.
>> What exactly should governments
prohibit?
>> Super intelligence is I think the most
important thing for them to prohibit.
But
>> how do you ban super intelligence
without banning like useful AI?
>> There is a couple ways how one can
approach this. A lot of this is
operization and you know there's a lot
of things we can talk about of how to
exactly operationalize different parts
of this problem. But fundamentally the
first and foremost thing that we can and
should just do is just simply say it is
illegal to build super intelligence.
This is a thing you can do in law. You
know the same way that it's illegal to
murder someone. It's even illegal to
attempt to murder someone. Now you might
say oh but what's the definition of
murder? You know there's many edge cases
like yes there are. That's what courts
are for. There are in fact many edge
cases especially with attempted murder.
There are many edge cases and for this
we have courts. So very similarly here
is that first and foremost it should be
illegal to build super intelligence or
to attempt to build super intelligence.
Now there are some cases that are not
edge cases. You know you have open AI
and anthropics saying proud on their
website that their goal is to build
super intelligence. This is not an edge
case. You can just say nope that's
illegal. Stop doing this immediately.
That's a thing we can do. Now there are
going to be a bunch of edge cases of
course and that's when we have to think
about operationalization like what what
are the things we want to measure what
are the things we want to intervene on
and like you know what should the jobs
of courts be to decide these various
decisions whether you know whether it's
courts or various regulatory apparatuses
there's again a bunch of there's a many
ways to achieve this the way I and my
organization tend to think that we
should achieve this is by a doing what I
just described just criminalizing the
creation of super intelligence the same
way we criminalize the creation of a
nuclear bomb. And the second is to just
is to regulate the precursors to ASI. We
make a list of precursors of things that
are not yet ASI but are on the path to
ASI or are very close to ASI and we say
okay if you're trying if you're making
progress on these things the government
has to know you have to tell you know
you know the government oversight of
whatever agency ends up being the one
responsible for this and they and then
we can make the decisions about do we
want to continue down this path how much
further do we want to go down this path
In my opinion, when it comes to the
frontier capabilities, we're already
pretty much in the danger zone and we
probably should just take a breather
until we know what we're doing. But
importantly, this doesn't apply to 99%
of AI applications. Most of AI even
still today is not companies race, you
know, building the next frontier of
agentic systems. This is specifically
the dangerous area and this is an
extremely dangerous and extremely
expensive business to be in. Creating,
you know, GPT6, GPT7 is unbelievably
expensive and unbelievably hard and
there's like
three and a half companies in the world
that can do this basically. So all the
other ones we might need other
regulation for their applications of AI
but they wouldn't fall under this
regulation
at least not yet.
>> Could prohibition actually be verified?
I mean like if one country refuses then
what do we do then? The way I generally
see it on the international stage is
that first we have to acknowledge the
reality of things is that
for example us in the United States if
China for example was to credibly
threaten to build a super intelligence
that is a threat to the lives of the
American people and everyone in the
world for that matter and we should take
that seriously. Our policy objective is
not you know stop American companies
from building super intelligence or stop
China from building super intelligence.
It's build stop everyone from building
super intelligence. No one should be
allowed to build super intelligence
because if anyone builds super
intelligence anywhere in the world we
lose. This is very important to
understand. Super intelligence is not a
tool. It's not a weapon. It's an
adversary.
It is a system that will not help us
achieve our goals. If America creates
super intelligence, we will not achieve
American objectives. America will simply
cease to exist. If China builds super
intelligence, it will not help them
achieve Chinese objectives. It will
simply make China cease to exist.
So therefore what we need to do is we
need to find verification mechanisms
trust but verify create a regime where
we can mutually verify between
adversaries that we are not pursuing
super intelligence that you know we are
not pursuing super intelligence and we
make sure they are not pursuing super
intelligence and we must we must back
this up with credible deterrence. We
must be willing to say this is not this
is a level of of security threat that we
cannot tolerate and that no one else can
tolerate. We cannot tolerate China
building super intelligence. This is
unacceptable. it is we cannot tolerate
terrorists building super intelligence
or anyone else and we must be willing to
just acknowledge this and say okay now
we need to figure out how to make sure
that that's the case the reason I'm
optimistic as I said is because it is
not in the interest of anyone to build
super intelligence this is not the case
for other AI applications there are many
non-suptelligent AI applications that
are very valuable both economically and
militarily
that's a very different question but
when it comes to super intelligence to
agentic system that we cannot control
and they can out compete us. This will
not help us achieve our objectives and
so we should prevent it from happening.
>> There's a famous clip of yours um saying
how none of your friends are having
children.
Expand on that and what what that
comment was all about.
>> Um that comment um yeah that one blew
up, didn't it? Uh not my usual topic of
conversation. I'm usually the AI guy.
Um, but it is an observation that I've
definitely been making. Um, as much as I
focus exclusively on super intelligence,
it's not the only issue that we are
facing as a society. There is a there's
a long laundry list that I have in the
back of my head. And one of them is
population decline is that there are
many many people especially my
generation and the younger generations
who are not having kids to a sometimes
really shockingly low degree. you know,
not just in Asia where it's particularly
extreme like in Korea or Japan, but also
in, you know, Europe and in America and
now more and more in all other countries
as well. Birth rates have been falling
precipitously
in basically every country on the planet
except like Kazakhstan for some reason.
>> And why is that?
>> And so this is where it gets kind of
interesting, right? Where everyone has a
pet theory. Everyone you ask about birth
rate at decline has their pet theory for
why it is. It's women's fault. It's
men's fault. It's capitalism. It's
socialism. It's whatever. And I looked
into this quite a bit and the true
answer is that we don't know. No one
knows. We don't really know. There is so
much of foot here. There are so many
studies that all say contradictory
things is that we don't really have a
clean story for why it is. I think
ultimately it's many things. I think it
is many things. I think it is related to
economic issues. I think it is related
to our relationship with technology and
how technology changes how we relate to
other humans. I think it's cultural
mimemetic. I think a lot of our culture
has changed around children. I think
there are there are probably also
biological reasons. Um I think a big
reason is housing. It's just no one has
big enough houses for kids anymore and
so you just don't bother. I think there
are many many issues at face. The quote
I gave back then which became so famous
was do you know how hard you have to
abuse a mammal for it not to want
children. And I think this is a good
like intuition pump. This is a good
thing to think about because I'm not
here to say I have the solution to this.
What I'm here to say is like something
is deeply wrong. This is the thing you
see at zoos for example. There's a
there's a whole there's a whole science
of this is that if the um you know the
the area for the cage for a zoo animal
is insufficiently good, they stop having
kids. This is a behavior that many many
many mammals have. And so there's a
whole science of like how do you build
enclosures that are sufficiently good
for given types of animals to allow them
to breed? And in a sense, I think it's
worth also thinking about us as animals
and our enclosure, so to speak. And
something is deeply wrong with our
enclosure. Something is deeply wrong
with how we are engaging with each
other, with ourselves, with love, with
kids, with family. There's something
wrong. I'm not here to say what it is or
how to fix it. I think it's many things.
I think there's probably like 15 things
we have to fix, you know, in some order.
But I think it's worth us just h
acknowledging that there we are ill like
as a society something is deeply wrong.
>> It could be a combination of an economic
problem, relationship problem, loss of
meaning, having cameras everywhere you
look. You know, guys don't guys don't
want to approach approach a girl or
dance with her.
>> Yep. Yep. That's
>> cameras everywhere. That's one that's
really crazy that I only really
realized, you know, a couple years ago
that especially younger generations just
are really skittish because they know
they get recorded everywhere they go.
You know, if something weird happens to
you on the street or something, there's
a chance someone's going to film it. You
know, if you approach a girl at a bar
and you mess it up, you know, there her
friends might film you and put you on
the school Instagram. You know, this is
a real thing that happens and it creates
this panopticon effect where people
become very um very hesitant to take a
risk, maybe a social risk to maybe
embarrass themselves. And look,
embarrassing yourself isn't fun. Never
has been fun, but it's an important part
of of being human. You know, humans are
cringe. You know, people are
embarrassing sometimes. You know,
dancing with your friends, yeah, it's
kind of embarrassing. You know, you
don't dance that well. That's okay. It's
part of the fun, you know, and like
you're 18, you approaching a girl for
the first time. Yeah, you're gonna mess
it up, you know, and you know, it's
going to be awkward, but you know,
that's okay. It's part of in a sense the
cycle of being human. And I think this
is a great example because this isn't
anything anyone intended. I don't think
anyone, you know, inventing, you know,
the smartphone were like, "Oh, yes,
finally I can stop, you know, young
girls and boys from, you know, liking
each other and dancing." That's that's
definitely my goal. I don't think anyone
intended to do that, but it still
happened. And it's just an example of
like it's just a side effect that in a
sense so drastically changed the human
condition and it doesn't even get a
mention like it's not like there's a
huge conver national conversation about
this.
>> Yeah. You don't even realize it that
like if this if you read a sci-fi novel,
you know, 50 years ago and a sci-fi
novel, young people don't dance anymore
and have, you know, no love and, you
know, awkwardness to each other, you'd
be like, "Wow, that's crazy." But it
happened. It did happen to a large
degree and no one even noticed. This is
why I think this is why I brought up
that quote to draw people's attention
to. There's actually a lot of crazy
stuff happening right under our noses
and we should be humble about realizing
that like something is wrong here and we
should probably really think seriously
about that.
>> If AI can do every job better than we
can, where do people find meaning in
their lives?
>> Well, I think if we get to the point
where there's AI that can do all the
jobs, uh I think humans won't be around
for much longer. Um
but giving that aside, I think there is
something like
one of my moral principles that I
personally have is that you should
realize where your philosophy breaks.
There are many philosophical questions
to which we don't have good answers. I
think this is one of them is that if
humans have zero contribution to the
economy or politics or art or anything,
what is a good way for humans to live? I
don't think we have a good answer to
this question. And so therefore, one of
my moral principles is if you don't have
an answer to a hard philosophical
problem, do not get into a situation
where you need that answer.
I think it's possible for us to figure
out, you know, a culture, an economy, a
system where, for example, AIs can do
all our jobs, but we're still happy. I
think it's possible to develop
something, but we have not done so. We
have not yet figured it out. So, I think
we should not do that. We should not get
into a situation where AIs are doing
100% of the work until we're sure we
know what we're doing, until we have at
least a decent plan. But we don't have a
decent plan. When we're talking about
something this risky, we just shouldn't
do it. Not yet.
>> Do people begin preferring artificial
friends and partners because they're
easier to deal with than real humans?
>> Absolutely. I mean, it's already
happening. This is another thing I worry
about a lot um that is not super
intelligence. Like I say, I have a
laundry list. You know, if uh if if I
solve the super intelligence things, I
have a long list of other problems to
work on. And this is another one of them
where a a large part of growing up as a
human is becoming socialized with other
people. And I'm sure we've all had the
experience of like having a good friend
who you like very much but it's kind of
a mess and then he meets a nice girl and
suddenly he becomes civilized or you
know vice versa. And I think this is not
just because you know man, woman or
whatever. It's also just because forcing
yourself to be close to another human
for long periods of time, learning to
compromise, learning to, you know,
smooth over the the frictions within
your own personality and so on. It's
just a very important skill to
cultivate. It's just very important to
become socialized, to know how to
navigate, you know, frictions or tricky
relationships. And this takes a lot of
work. It's very awkward. It's, you know,
it's just a lot of work. And I think if
we don't do that, if we, you know, can
just have systems that can just, so to
speak, fulfill our needs without asking
for anything in return, without forcing
us to become better versions of
ourselves, then all things equal, we
become worse versions of oursel. And I,
you know, I don't think that's very
good. This is another example of
something that I would put into the we
don't have a good philosophy bucket. Um,
sometimes I call this the posthuman
philosophy bucket, so to speak. Like
similar to the question of like should
we have 100% of jobs done by AI. I'm
like that's like a posthuman question.
Like that's like we don't have an answer
to that. I think there's another good
one. It's like should humans you know
get all their friendship from AI. And
I'm like dude we cannot handle this
right now. Like we do not this seems
really dangerous. Again like selling out
one of the most deepest cores of what it
means to be human. Yeah. We shouldn't
touch that until we really know what
we're doing. What
>> what part of being human cannot be
automated?
In a technical sense, nothing. In a
practical sense,
depends on your view of philosophy.
>> There's something magical about human
beings.
>> Depends on your view of philosophy.
>> Yeah. I just wonder
>> there is, you know, if you there's a lot
of things about human, you know, nerve
cells and pancreas and so on that's very
very hard to, you know, simulate. Um,
but that's obviously not what we mean
when we say humans are special. Um I
think there's a lot of things that is
like special about humans because of
contingent things like emotions like
there's a sense for example if you love
someone I'm sure you've experienced this
too or many people have experienced this
is and they have like a little quirk
like I don't know they they like plain
toast for breakfast you know for some
reason you know like in a sense this
doesn't matter but often if you love
someone you'll start really enjoying
that they like plain toast and every
morning you'll see the plain toast and
you'll smile
just because they like plain toast. And
I think this is beautiful. This is
something I really like about humans is
that we can appreciate the arbitrary
things, you know, that like it's very
arbitrary, but you love it because you
love it because you know human because
they are the way they are. Everyone has
their own loop quirks. I think there's a
lot of that that in a sense is what
makes a lot of what it makes humans
special. I think a lot of things that
humans think makes them special probably
is automatable. You know, I think you
[snorts] know, science and math and art
and all that kind of stuff, yeah,
unfortunately seems very automatable.
Um, but this is again getting us into
the posthuman bucket of like
we shouldn't get into the situations
where we have to have answers to these
questions until we've taken the time to
actually figure these out. I think
there's probably much better answers
than the ones I can give. You know, if
our philosophers spent a hundred years
more working on this problem, I think
we'd come up with much better answers
than I can give.
>> Yeah. So, going back to trying to stop
super intelligence, what what exactly
would you make illegal tomorrow?
>> So,
there are many possible definitions for
the word super intelligence.
Usually we define it as fully autonomous
systems that uh pose a credible chance
of overthrowing the US government as a
rough yard stick. There's many other
definitions, but the most important
thing to understand about this whole
thing is that um any piece of
legislation is obviously just a stop gap
measure. Obviously, we'll not get it
right on the first try and we will need
to iterate. You know, we we have a bill,
you know, that we've written. We have
legal text that legally defines all this
kind of stuff. I think it's pretty good.
But
imagine tomorrow we have a magic wand
that we can wave and we can pass any law
in the whole world. So we wave our wand
and we pass the ban ASI forever bill in
all country. What happens? Well, in one
week it's violated in spirit. In two
weeks it's violated in letter because
obviously the companies won't listen to
it or we're going to forget some edge
case or the lawyers will come up with
some really clever reading that we never
thought about. So, this is an iterated
game. It's a It's not a oneanddone, like
we pass the law, we can all go home.
It's going to be a continued dance. It's
been a continued fight between, you
know, law enforcement, judges,
politicians, the general public, and
people who are trying to circumvent
these laws at all costs. So, I think,
you know,
I think we have like a nice default, you
know, idea for what a piece of
legislation could look like, but it's
only round one of the game.
And if America stops and China doesn't,
haven't we simply handed China the most
powerful technology ever created?
>> Super intelligence is not a tool. It is
not a weapon. It is an adversary. China
doesn't have super intelligence. You
can't have super intelligence. Super
intelligence has you.
We can't allow China to build super
intelligence. Not because it will help
China take over the world. It won't. It
will destroy China. It just also will
destroy the United States and all other
countries. So therefore we don't really
have a choice. We have to stop super
intelligence from being created. If
China for some reason
is hellbent on building super
intelligence then we have to stop them
somehow.
>> So so how do you enforce global AI pro
prohibition without creating some kind
of enormous surveillance state?
>> Luckily at the moment building such
frontier AI systems is a massive
infrastructure project. You require
massive data centers with very
specialized chips that are made by you
know only like one company in the world
through a supply chain that you know
goes through all of our western allies
and the United States etc. There are
very very very few actors that can even
attempt something like this. You don't
need global surveillance to detect them.
You don't need global surveillance to
surveil these specific projects.
It's not the case that everyone can do
this on their laptop. If we lived in a
world where you can build super
intelligence on a laptop, I don't know
what we would do. I think that would be
a very very dangerous and a very
unstable world and I'm not sure we would
survive that
>> because at some point a a relatively
small group could could secretly build a
dangerous AI. Right?
>> That's a lot of the risk. One of the
reasons that we need to pause and stop
as soon as possible is so that we have
extra time before people figure out even
better ways to build AI. One of the
problems for example also why just say
reducing say the number of GPUs the
number of computing power working on AI
doesn't work is because then the
companies will just invest in working on
you know better software better
algorithms we have to stop both
one thing it's not just that the
hardware keeps getting better it's that
our software our AI systems get more
efficient more effective and yes maybe
at some point we could build software so
good that it could build super
intelligence on very small amounts of
computing power.
It's very hard to imagine how we would
handle such a world. I think if this
happens right now, we're screwed. I
think if we spend 20 years building
better institutions, better
understanding, you know, of how to
control these things, of how AI works,
having better enforcement, better laws,
etc.
Maybe we can figure something out. You
know, maybe there is a, you know, we we
make the chips in such a way that they
don't work for super intelligence, but
they do work for other things. I don't
know.
>> And and who would get to decide what
level of intelligence humanity is
allowed to build?
>> Well, I mean, as a as an American, I
believe the people, of course, democracy
should decide. This should be up to, you
know, votes and our elected
representatives in general. But
ultimately this is a negotiation like
anything else. I think this would also
be a negotiation with other countries in
a sense of like what are the levels of
risk that we want to tolerate. I would
like us to get into the situation
honestly and we can all agree we don't
want super intelligence and now we just
want to talk about the pace. I think
this would be a good situation to be in.
I would love if our conversation would
be should we move this much per year or
this much per year. I think this would
be a great place for the world to be at.
I think we're very far from this still.
>> And are are you personally a conspiracy
theorist and believe in a lot of crazy
stuff or is this just the first time
that you've seen something that
>> I don't consider this very crazy at all.
I think it's a very standard prediction
from our standard understanding of
science as it currently exists.
>> Mhm.
>> I find conspiracy theories extremely
entertaining, particularly dismantling
them. Uh when I was a teenager, I was
quite obsessed with skeptics and like
tearing apart conspiracy theories,
finding why they were wrong and so on.
And well, it turns out this one wasn't.
>> So, you're 31 years old. Are you
pessimistic about the future for
mankind?
>> Pessimistic, I think, is not the word I
would use. I do think that if things
continue on the track they are currently
on, there is almost no chance of a good
outcome. But I also truly deeply believe
in our ability to change things. I truly
believe that humans can change things
and they can change a lot very quickly.
We are not currently on track. If we
continue as we currently are for the
next one, two, five, seven, whatever
many years, we're not going to make it.
But we can still change that.
>> And what what kind of things would have
to happen in order to make this a
reality?
>> Fundamentally, the main thing which you
know is not exciting is just we need to
do the thing we do for any kind of
technology. We have to regulate it. We
have to understand this issue. We have
to have the conversation and we have to
make choices. The first and foremost
important thing is we have to we have to
staunch the bleeding. Super intelligence
could be built as soon as in one or two
years from now. This is a thing that
many scientists do take seriously. Maybe
it'll take longer, maybe it won't.
Unclear. The first thing we need to do
is prevent that from happening because
super intelligence gets built while
we're all still arguing. It doesn't
matter. So the first and most important
thing we do is we need to stop stream
intelligence right now and then we take
the time to figure out what do we need
to do next? Where do we go from here?
What are the things we want to do etc.
The most important bottleneck to this at
the moment is actually awareness. So, a
lot of people are shocked by this, but I
talk to politicians all the time, uh,
including military officials and many,
many other people, and the number one
reaction I get when I talk to them about
this issue of super intelligence, it's
just, "Oh, wow. I had no idea. That
seems really bad. What can we do?" I
very rarely have a problem with
persuading people. The number one issue
I have is that most people have just not
heard of this issue. It's still pretty
new. It's just not a thing most people
have heard about. And once I talk to
people, general public, you know,
military official, politicians, etc.,
most people are like, "Hey, yeah, that
seems really bad. We should like figure
out what to do." So, in a sense, we're
not even in the position where everyone
knows about this and now we're arguing
about how to fix it. We're still in the
phase that most people have not even
heard about that this is happening. So
once we get to that stage where you know
enough people have heard about this
issue and are demanding action when
citizens in the United States and
elsewhere are demanding action from the
policy makers a lot can happen pretty
quickly and a lot can happen and a lot
can be done.
>> You once said that uh if super
intelligence appeared tomorrow you'd
probably call your mother. What would
you tell her?
>> I love you.
Is there any direction you could think
of that would
change this this dark future that you
see?
>> Yes, absolutely. It's why I do the work
I do every day. That's why I'm here
today.
>> Well, we're where super intelligence
continues to evolve.
>> I think we can stop super intelligence.
>> Just stop it.
>> It does not yet exist. So, we can still
win,
>> right?
>> Once super intelligence starts existing.
>> Yeah. Unlikely that we have much chance.
And how many years do you see that?
>> It's hard to say. Um
the somewhat joke, somewhat serious
answer that I've been giving since 2020
is the same. It's 30% by 2027,
50% by 2030, 99% by 2100, 1% already
happened.
H
Connor, let me uh let me ask you one
last question. What what would you like
to
communicate just in one sentence to to
everyone watching?
>> It is not over. Contact your
representatives. Go to controlai.org.
Demand action to be taken. Talk to your
friends. Talk to your family. Talk to
people. Make this an issue that we care
about. And a good future is still
possible. We can make a wonderful future
with technology, with AI, and all these
other things. We can do it. It's going
to take a lot of work.
>> All right, Connor Ley, thank you so much
for sharing your thoughts on AI and
super intelligence, and we'll see what
happens.
>> Thank you.
>> Thank you very much.