Why The AI Doomers Might Be Right - Robert Wright
CChris Williamson
Computing/SoftwareBusiness NewsInternet Technology
Transcript
00:00:00I've told you this before, but you wrote probably the most influential book in my life, which was
00:00:04The Moral Animal, and it was the thing that got me started on the trajectory of thinking about
00:00:08evolutionary psychology, of studying human nature more deeply. Why are you now writing about AI,
00:00:15given your heritage? Well, in some ways, it's an extension of evolutionary thinking. In a couple
00:00:21of senses that I think are so underappreciated, AI is a product of evolution and is still evolving.
00:00:30But the other connection to The Moral Animal, I think, is, first of all, well, The Moral Animal
00:00:36was about the human mind, and AI does a lot of things that traditionally only human minds have
00:00:43done. The other thing I tried to do in The Moral Animal is highlight kind of what you might call
00:00:49moral biases, kind of self-serving moral biases, you know, the way we all think we're right and
00:00:54the other guy's wrong. And I think if we're going to get through the AI revolution in good
00:01:00shape, among the things we're going to have to do is grapple with that, with kind of what
00:01:04you might call the psychology of tribalism, a little more successfully than we have. And
00:01:10so I pay a certain amount of attention to that in this book as well.
00:01:13What's the central question that you're wrestling with here?
00:01:18Is it true that this technology, which obviously holds the potential to bring great wonders,
00:01:27is also in some respects terrifying and could go badly awry if we don't approach it wisely? And I
00:01:36think the answer's yes. That's exactly why this debate's interesting, right? That we have this
00:01:41sort of endlessly unresolved potential future. I don't know whether you've seen the graph that I
00:01:47think it's the FT put together, and it was three potential futures from an AI perspective. One results
00:01:54in everything getting blown up. One results in exponential growth, the kind of which we've never
00:01:58seen before. And the other results in a 0.2% increase in GDP year on year. So it's like either very
00:02:06little changes or everything changes in one of two directions.
00:02:10Right. Well, I think it definitely has the potential to massively increase GDP. I also think
00:02:16it has the potential to so destabilize the world, if not do something worse to it, that that just
00:02:22doesn't materialize. And, you know, in terms of doom scenarios, I'm agnostic about the sci-fi doom
00:02:30scenarios, but I take them more seriously now than I did before I went into this research project. You know,
00:02:36actually taking over and maybe deciding it has no use for us or something. I found much to my dismay
00:02:42that it was harder to dismiss those arguments than I thought. But the thing I'm more confident of is
00:02:48it's just going to be an earthquake. It's going to be destabilizing along a number of dimensions,
00:02:52solutions. And that's why we need to approach it with care.
00:02:57Were you, would you have classed yourself as a sort of AI hopeful going into writing this? What was your
00:03:03predisposition before you got started?
00:03:05I wouldn't say I'm a wildly optimistic person by nature. I tend to focus on potential downsides
00:03:12of things. But I, again, I had not bought the doom scenarios. You know, I had the doomer in chief,
00:03:17Eliezer Yudkowsky on my podcast 15 years ago. And at that point, it's interesting. He was in
00:03:22mid-transition. He was moving from singularity optimist to doomer. I was still saying things like,
00:03:29look, AI, you know, it's not a generic property of intelligence that it has a will to power.
00:03:36We have one because of our unique evolutionary history. I was still asking questions like that.
00:03:41Eliezer was saying they're good questions, but X, Y, Z. I wasn't really persuaded by anything he said,
00:03:46but I, now I, I am more respectful of the sci-fi doom argument. So, but in answer to your question,
00:03:55I would have to admit that I go, I go into situations looking for things to worry about.
00:04:01I do. That's my nature. I think, I think society needs those people and it needs the other kinds
00:04:07of people. And we need to, to talk things over. Jeffrey Hinton fears AI and Jan Lacoon doesn't.
00:04:15Who do you think is getting the future more right at the moment? Yeah. I, I start my book with a
00:04:19conversation I had with Jeffrey Hinton, uh, in 1983. Okay. That not to betray my age, but the truth is
00:04:26I wrote a piece about AI in 1983. I haven't been paying attention to it ever since. But, uh, at that
00:04:33point, Jeffrey Hinton did not have a hint of, uh, of, of doom in his voice. He was, in fact, I remember
00:04:41the reason I talked to him as I was talking to somebody, I forget who it was, but they said,
00:04:45if you want to hear the gospel about neural networks, you should talk to Jeff Hinton. And I talked to him.
00:04:50He was an enthusiast. He said, I know we don't have much to show right now, but just wait until
00:04:55the microprocessors get really cheap. And we have what he was calling massive parallelism. And he was,
00:05:02he was right. And in the end, he found it scarier than he himself, uh, had anticipated finding it by
00:05:10his own account. Hmm. Yeah. It's, it's weird how prescient some people have been. Do you know the
00:05:15story of Avatar? Do you know how James Cameron wrote that screenplay in the nineties? So he wrote the
00:05:21screenplay in the nineties, but knew that the technology to be able to recreate what he needed
00:05:27didn't exist yet, but would exist in the future. So he's written the script and then sits on it until
00:05:34the technology is at the level where he can do it. That level of, I mean, this is the job of technologists
00:05:39and futurists, right? Like shock horror people who do a job and think about it all the time are good at
00:05:43it, but it is, it's still pretty impressive how, how, uh, sort of how much foresight these people
00:05:48have got. Yeah, I agree. And Hinton certainly got the general picture. Yeah. Okay. So you're saying
00:05:55AI isn't just another technological development. It's sort of a threshold event in, in, in planetary
00:06:02history. Why do you think most people still don't grasp the magnitude of what's coming?
00:06:08I think, uh, a couple of reasons. One is, I think there's a misunderstanding about what's
00:06:14going on with these machines. And that leads to one sense in which they are a product of
00:06:22evolution. Okay. So it's commonly said that they are trained and that's a fair word. The,
00:06:28and, and the training process is referred to as a learning process. And that's true, but it's
00:06:34also true that the training process is a process of evolution that in effect reverse engineers
00:06:43cognitive functionality that in our species took millions of years to evolve. Okay. So a good
00:06:51example is the language, uh, generation that they famously do, uh, sometimes called next token prediction,
00:06:59next word prediction, you know, it turns out that, uh, they developed kind of on their own in a way,
00:07:09uh, a system of representing the meaning of words. Okay. I mean, I can elaborate on that,
00:07:14but it would, it would get too technical. Uh, the point is that, that nobody said to the machines,
00:07:21you know, you need to figure out the meaning of words or gave it a means of doing that. And this
00:07:25is the big revelation I had when I heard Jeffrey Hinton's name, you know, a few years ago, suddenly
00:07:31he's being called the godfather of AI. When I last talked to him, he was just this, you know,
00:07:36obscure computer scientist, uh, who was advocating this maverick approach to AI. And I look back at the
00:07:42article I wrote at the time and I realized there was something I just got fundamentally wrong about the
00:07:46potential of the, the approach he was advocating. And it's, it's this that I thought that to the extent
00:07:56that these things dealt with words, we were going to have to put the meaning of the words in, like,
00:08:01we're going to have to look at a dictionary and say, okay, this word has these different senses.
00:08:06And we were going to have to architect a neural network to have different nodes that reflected
00:08:11these different meanings of the words. And in my defense, there were neural network models at the
00:08:16time, including by a guy who collaborated with him that did that, that took that approach. But,
00:08:20but that wasn't really the thing Hinton had in mind. It turns out that we don't have to tell
00:08:28the machines about the meaning words, how to represent them. We just have to train it to generate
00:08:36language. And the training is, it does, it accomplishes something by selectively strengthening these
00:08:43connections among neurons in a neural network. It accomplishes something that, you know, took
00:08:50millions of years of human evolution, coming up with a way of representing the meaning of words. Now,
00:08:55it also does something that happens during a human lifetime, which is learn a specific language. Now,
00:09:01that is learning in the traditional sense. But for us to learn the language,
00:09:05we had to have some built in linguistic equipment built in by natural selection.
00:09:10And the point is, these machines do both things at once. Okay, they kind of in a certain sense,
00:09:16recapitulate natural selection, even though the cognitive stuff they're building in isn't exactly like
00:09:22stuff in our brain, but it accomplishes the same feats. And once you realize that all you need is data,
00:09:31okay, to feed into these machines, human generated data, that they will do the rest, they'll do the reverse
00:09:40engineering, then you realize that, oh, it's the same with self driving cars, you feed into visual data,
00:09:45and it does what a driver does, auditory data, all kinds of data. And that's what I think people don't
00:09:52understand is that we have a long way to go on this fuel alone. Like, for example, you know, recently,
00:10:01Mark Zuckerberg had the, I don't know, good or bad judgment to announce in the same week, A,
00:10:08he was laying off 8,000 workers, B, he would henceforth be tracking the keystrokes of his workers.
00:10:13Well, why? Because once you take the data, the input data they're getting, maybe the emails,
00:10:18everything, I don't know, and what they're doing with it, the output data, then you can replicate
00:10:24the whatever it is that's going on inside their brains, it does their jobs, and then you can fire
00:10:28them. And it's the same with robotics and everything else. All you need is the data, and the machines
00:10:36will replicate kind of the cognitive functionality we have, even if in some time, in some cases,
00:10:42they approach it in a somewhat different way. Although in many cases, they don't. We've discovered
00:10:47that, for example, they invented what are called edge detector neurons to make out objects visually,
00:10:56and evolution, you know, built the same thing into us. So you're saying that we've got,
00:11:00that's one of the first examples of machine and organic, like, convergent evolution, in a way.
00:11:12In the same way that eyes independently evolved across a bunch of different species. I think that
00:11:17crabs, for some reason, converging on the form of a crab is something like that. This edge detection is
00:11:22something that we have. And from the black box of, you need to be able to achieve this,
00:11:27That's right. One of the most efficient ways to do it. But that would make sense,
00:11:30right? Like, how would humans and the rest of the animal kingdom have arrived at this as the most
00:11:35effective way to do it, having split tested it just way more slowly over a much longer period of time,
00:11:40using evolutionary processes and gene mutations, and AI not come up with at least a few of these things
00:11:46that are the same? That's right. And convergent is a good term, because I suspect that these edge
00:11:52detectors have been invented multiple times in natural selection. First of all, a lot of things
00:11:58have been multicellularity, winged flight, a lot of things have been multiply invented. And then this
00:12:04is in a sense, another case of invention where you just say to the machine, look, you know, we're going
00:12:10to give you kind of positive reinforcement every time you get better at recognizing these objects. And so
00:12:15whatever strengths of neural connection led you to get closer, we're going to preserve those. We're
00:12:21going to keep going through trial and error, through mutation, you could say. We're going to make you
00:12:27better at seeing things. And it's not surprising that since that really is kind of what happened in
00:12:32evolution right through trial and error, we try to get better recognizing objects. It's going to discover
00:12:37some of the same tricks in this case, edge detectors. Yes. Well, the reinforcement function is I didn't die
00:12:44and I passed on my genes, as opposed to here's a good boy point inside of the black box. But yeah,
00:12:48basically the same thing. Okay. So how do you, how do you come to think about AI fitting into the broader
00:12:53context of human evolution and civilization? Are we witnessing the next stage in evolution itself?
00:12:59I think so. And, you know, I think this is a new form of intelligence. There's never been anything
00:13:08like it. I do think it can be seen as an extension of organic intelligence, even though the material
00:13:15isn't, strictly speaking, organic. It's silicon. It's not carbon based. And it may be different in other
00:13:21ways. And I'm agnostic as to whether it is sentient or could be, whether it has subjective experience or
00:13:27could, it certainly could. Uh, but I do think it is the invention of a, uh, you could, it's definitely
00:13:36an invention of a new kind of intelligence that I think will surpass ours and you could call it a
00:13:40new form of life. And then the, the, uh, the other thing I try to emphasize in the book is that it is
00:13:47coinciding with a second big threshold, which is what you could call the evolution of kind of a global
00:13:52brain evolution through, you know, technological evolution, human cultural evolution. You know,
00:13:58we've gotten, uh, more and more interconnected, of course, via information technology. There's more
00:14:03and more rich intellectual collaboration across national borders. I, I, I mentioned this guy, uh,
00:14:09Teilhard de Chardin in the book who in, in 1923, about a century ago coined the term noosphere,
00:14:15noosphere N O O S is the Greek word for mind. Um, to refer to this, what he called the thinking
00:14:22envelope of the earth, the brain of brains, you know, but, but he imagined the neurons in the global
00:14:28brain being human brains. And now we have to reckon with the possibility that a lot of them and
00:14:34conceivably the most important ones will be Silicon brains. We have to ask them, what is our relationship
00:14:39to those neurons going to be? Hmm. Well, why is it the case that discussions about AI keep pulling
00:14:46people toward religious language on both sides of the fence? That's interesting. I mean, you could start
00:14:52with Eliezer Yudkowsky, who sees himself as having rejected his religious upbringing, but has a kind of
00:15:02fervor about this, right? He could be, you know, a biblical prophet. Um, and then on the other side,
00:15:08these singularity enthusiasts, whom I first became aware of, I don't know, about 20 years ago, who said,
00:15:15you know, we're going to enter this period where technology changes faster and faster. There will be
00:15:21a positive reinforcement, this feedback loop. And then things change so fast that like, who knows what's
00:15:27on the other side. In fact, the term singularity in physics connotes exactly that. There's this opaque
00:15:34kind of thing and an event horizon or whatever, beyond which you, the laws break down. You don't
00:15:39really know what is beyond there. And, and from the, from early on, in fact, from the very first use of
00:15:45the term in this context, which I think was John von Neumann's, that was explicit. The idea that things
00:15:50could start moving so fast that you just don't know what's going to happen. So one thing I didn't
00:15:54understand is like these optimists, unless they have a literally religious faith, how could you
00:16:00be so optimistic? Right? Like the whole, the definition of the thing is that you don't know
00:16:04what's going to be on the other side. I don't get why you're so upbeat about this could work out well,
00:16:09but I don't, I don't understand that. So yeah, there's, there's all that. And then there's the,
00:16:15I think I deal with in the book, which is the fact that when a process is as systematically
00:16:20directional as this has been, right? Like biological evolution carries complexity and intelligence really
00:16:28to higher and higher levels. You get cells, multi-celled life, societies, multi-celled life.
00:16:33You get this one society of multi-cellular organisms known as us that, that spawn a whole new kind of
00:16:41evolution, technically called cultural evolution by anthropologists, but that encompass,
00:16:46encompasses technological evolution, political ideas, everything. And, and that carries organization to
00:16:53a higher level in the sense that, you know, we were 10,000, 20,000 years ago, hunter gatherer village
00:16:58was the most complex form of social organization. Now we're approaching the global level. I think when you,
00:17:05you, when you see a, a, a process that's that systematically directional, and I'm not saying
00:17:12it's driven by anything other than the conventional mechanical things we think of as driving it, national
00:17:17selection in the case of evolution, you know, completely material process, but it's still in principle,
00:17:23you know, looks more and more like something that was set up to like, do something right. I mean,
00:17:27that's just, that's an intuition people have. And I think you can actually argue about it in, in, in,
00:17:34uh, more rigorously than just having the intuition. But I think that, uh, that's one reason, uh,
00:17:41there's a little more of a, I mean, teleology is the formal term for something being purposive and,
00:17:47and, you know, just look at the idea of a simulation, right? Like it, on the one hand,
00:17:52a lot of people use it as kind of a joke, like something weird happens and they say we are in a
00:17:57simulation, but I think a fair number of people, including in Silicon Valley, take it seriously
00:18:03that there could be a simulation. Well, if that's what we're in, then, then it was designed by some
00:18:09intelligent being or process. So there's a purpose in some sense, I guess there's something
00:18:14it had in mind, right? So a lot of people are either implicitly or explicitly
00:18:20taking seriously the idea that there's a purpose unfolding. And one thing I'd add to just the
00:18:29conversation about that is that in my view, at least there's a moral dimension to this. I think,
00:18:34uh, there has been in a certain sense, a kind of moral advance of humankind, notwithstanding all
00:18:42the backsliding as, uh, social organizations grown. I could get into that, but, but the main thing I'd focus
00:18:49on now is I think if we're going to get through the AI revolution in good shape, there's going to have
00:18:55to be something almost like a moral revolution, because I think for various reasons I could get
00:19:01into, we have to confront this as a global community, a cohesive global community that is not, you know,
00:19:08rendered immobile by wars. And I think for that to happen, we're all going to have to get better at,
00:19:17you know, just looking at things from the perspective of countries other than ours and,
00:19:21and, and, and doing some things that in a way, aren't that spectacular in terms of, you know,
00:19:28cognitive feats, but, but are very hard because of cognitive biases we have. It gets back to the
00:19:32self-serving moral infrastructure, you know, the infrastructure for moral thinking that natural
00:19:38selection built into us. I think we have to get over that. And so, you know, one reason I call the book,
00:19:44the God test is it, it's, it's kind of like a test that God would set up, right? I'm not,
00:19:49I'm not saying it is. I'm just saying the idea that we confront this huge challenge and to come
00:19:55out on the good side of it, we're going to have to see a kind of moral upgrade for our species.
00:20:01That's a, you know, that's the kind of tests we associate with gods.
00:20:06Most people don't realize how much being dehydrated impacts their performance, which is why for the
00:20:10last five years, I've started pretty much every morning with Element. Element is a tasty electrolyte
00:20:15drink mix with everything that you need and nothing that you don't. This orange salt in a cold glass
00:20:20of water is like a sweet, salty, orangey nectar. And I really tell the difference when I take it versus
00:20:28when I don't. It plays a critical role in reducing muscle cramps and fatigue, helps to optimize brain
00:20:33health and regulate your appetite while also curbing cravings. Best of all, there are no questions asked
00:20:38refund policy with an unlimited duration, so you can buy it and try it for as long as you want. And if you
00:20:42don't like it for any reason, they'll just give you your money back. Plus, they offer free shipping in the
00:20:46US. Right now, you can get a free sample pack of Element's most popular flavors with your first
00:20:50purchase by going to the link in the description below or heading to drinklmnt.com/modernwisdom.
00:20:56That's drinklmnt.com/modernwisdom. Get practical for a second. What happens if we don't have this moral
00:21:05upgrade? What's the outcome if we encounter ever-increasing, ever-coordinating noosphere,
00:21:12AI, ultra-coordination across the globe, but we haven't had this enlightenment upgrade?
00:21:17Well, I think if we don't have some... I mean, I don't want to overdo the term enlightenment. I'm not
00:21:23talking about full-on Buddhist enlightenment. I think I am talking about a slight movement in that
00:21:31direction, if only in the literal sense of the term mindful. I mean, I'm a big fan of mindfulness
00:21:38meditation, but just mindful in the sense of, like, just paying attention and being calm enough to pay
00:21:46attention, right? And seeing things maybe a little more objectively than you do when you're full of
00:21:53emotion. I think that's the kind of thing that allows us to be better at looking at things from
00:22:00the point of view of other people. I mean, just think about when there's some email you get and it annoys you,
00:22:07and you've got this... I can't believe this still happens to me at my age. You know,
00:22:11you think you'd get over it, but no, you have this... It's almost like a fantasy of this mean email
00:22:15you're going to write in response, right? And then if you calm down, you're like... It isn't just that
00:22:20you go, oh, that wouldn't be a good idea. You go, oh, well, maybe what he meant is this, or maybe the
00:22:25reason he can't do this for me is this. You get better when you're calmer at looking things from
00:22:36other people's points of view. And I think we're going to have to get better at summoning that kind
00:22:42of objectivity toward one another, especially across the barriers of conflict that keep dividing us,
00:22:51right? Why is that important in an age of AI? You know, it's interesting. I listened to your
00:22:58podcast with Tristan Harris, which I thought was great. I agree with him about pretty much everything,
00:23:04but I would add a footnote to something he said. And it was that, you know, he said, look, in the Cold
00:23:11War, we didn't have to be on great terms with the Soviet Union to do arms control accords. We, you know,
00:23:18we could have a relationship of tremendous tension and even conflict, but work things out along a
00:23:23particular dimension. I agree. That's true. And it's encouraging. But I think artificial intelligence
00:23:29is a much harder technology to deal with in this way than nuclear weapons are. I mean, you know, the
00:23:39verification process is more complicated. If you want to try to monitor what's going on, it's just
00:23:45complicated in a lot more ways. And, you know, this is a whole argument I could present. I don't,
00:23:50I don't think this is the time, but, but the point is, I think we're going to have to go well beyond
00:23:57a few specific kind of deals and treaties, although I welcome those up to and including
00:24:05something I call organic transparency. You know, there's already agreement that a certain amount of
00:24:10transparency could be stabilizing in terms of US China relations, especially there are clearly scenarios
00:24:19where one country worries about what the other is doing behind closed doors with its AI and gets freaked
00:24:24out, uh, and launches some kind of preemptive attack or something. And so maybe transparency would have
00:24:30been stabilizing. And when I talk about organic transparency, and again, there, there can be
00:24:35formal transparency, right? Monitoring the kind you get with arms control agreements. Great to the,
00:24:40to the extent that we can do that. But there's also something that comes out of being richly engaged
00:24:45with another country along economic and cultural and scientific lines. You just know more about
00:24:51what's going on. If the scientists are getting together at conferences, having drinks afterwards, whatever,
00:24:57if business people are doing that, you just get more in the way of a heads up about stuff that's
00:25:03going on inside labs, inside this, inside that. And there can be a greater sense of reassurance and, and ultimately, trust.
00:25:12So I think because of, of how challenging the formal things we're going to have to work out are at an
00:25:20international level, and AI just presents you with a ton of threats that cannot be addressed via national
00:25:27policy alone. Um, I think just to, to handle the things we'd like to handle via formal arrangements,
00:25:36we're going to, we're going to have to calm the planet down a little. And moreover, I think we're
00:25:41going to have to go the extra step and have, you know, rich and friendly engagement among the nations.
00:25:48And that's, it's a good thing. It can happen. We've done it before. And, uh, this is a, you know,
00:25:55we really need to. Do you think benevolence comes along for the ride with intelligence?
00:26:03No. Uh, I think intelligence alone is, is almost, uh, neutral in that sense. And I don't,
00:26:12I don't think we really need a ton of benevolence per se, at least not foundationally, because,
00:26:21you know, my argument is, and has been for some time, even before AI, I was arguing that technology
00:26:26is making relations among nations more non-zero sum. Okay. Classic example, nuclear weapons,
00:26:31nuclear wars, lose, lose non-zero sum outcome. The win-win outcome is to not have the nuclear war,
00:26:38to have the treaties that stabilize things. Um, same with, you know, climate change, any number of
00:26:46problems that transcend national bounds and can only be solved through some degree of, uh, international
00:26:52coordination. You know, I've been arguing, I mean, I, I had a book called non-zero that was about this,
00:26:57that, that, uh, you know, 26 years ago or something that was about the growing non-zero sum dynamic
00:27:04among nations. Now, what that means is it's just in your interest to cooperate. You don't have to
00:27:09cooperate out of benevolence. You know, you don't, you don't, you don't have to love them. And I, I
00:27:15distinguish between, I'm not the first to do this, psychologists distinguish between emotional
00:27:20emotional empathy, the kind of empathy people often think of, like feel their pain empathy
00:27:25and cognitive empathy, which is just understanding what's going on in their minds, understanding how
00:27:32they're looking at things. You don't have to feel their pain. You don't have to like them. You don't
00:27:36have to care about them. But if they're in a non-zero sum relationship with you,
00:27:41you probably are going to have a better outcome from any negotiations you do about how to work things
00:27:48out and solve the problem you have in common, if you do understand at least what's going on in their
00:27:55minds. And, and I'm just a huge advocate of cultivating this cognitive empathy and recognizing
00:28:01the kind of built-in cognitive biases that get in the way of it. That's a good example of something
00:28:06I think we're going to have to get better at overcoming.
00:28:07Yeah. I think the reason I bring it up is a lot of people assume, a bunch of my friends,
00:28:13we don't need to worry about the direction of an AI future, because if it's smart,
00:28:17why would it not care about us? Why would it not bake in benevolent, pro-social,
00:28:21human caring, flourishing, et cetera? I, I've read too much Nick Bostrom to be able to,
00:28:29no matter what, it's kind of like your first relationship, you know, you get into a relationship
00:28:33and your first relationship is with an asshole and you're like, God, for the remainder of time,
00:28:37I've been pattern matched that every relationship is at least going to be tarnished somewhat with that.
00:28:41My introduction to thinking about AI safety was Nick, which means I'm forever,
00:28:45forever cursed to kind of be on the back foot and a little bit skeptical about this stuff.
00:28:48But yeah, I don't think that that's necessarily the case. I don't think that
00:28:53any super intelligent AI is necessarily going to have
00:28:57benevolence baked in or the care of humanity baked into it. Also, if what you're saying is true,
00:29:04and I think it's a really interesting parallel to say, look, evolution just wanted to optimize for
00:29:11a couple of things, survival and reproduction, and some stuff emerged. No one taught humans how to do
00:29:17this. The same thing occurred with AI, right? No one said, this is what this word means. This is,
00:29:23it's just the outcome that we want is relatively tightly defined. Here's some good boy points and
00:29:27some bad boy points depending on whether you get it right or wrong. If we assume that that is going to
00:29:32be at least for the foreseeable future until we get to world models and global modeling or whatever
00:29:38it's called until we get to that, and that may even still be the same process there, there is no reason
00:29:44to assume that anything is baked into the system. It's just going to find it out for itself. And it may
00:29:51not like the idea of humans being around. It may think that there's something that we don't actually
00:29:56add to the system. It may find us to be a scourge on the earth. And this is where a lot of the Duma,
00:30:01the sort of Duma future plans come in.
00:30:03Yeah, no, it, uh, it, it, you know, intelligence, one interesting thing to come out of this whole
00:30:12thing is the study of like properties of intelligence, of intelligent goal seeking systems.
00:30:20And, uh, there are some things that, you know, um, evolution built into us that we're seeing in
00:30:27these machines just by virtue of the fact that they're intelligent goal seeking systems like us,
00:30:31they figure out stuff that either was figured out for us by evolution and instantiating in our brains
00:30:38or stuff that we figure out. And in some cases, it's a little about, for example, deception, right?
00:30:42Sometimes you realize, well, uh, I'll have a better chance of getting what I want out of this person.
00:30:48If they don't know this particular thing, like if you're doing a deal, you're negotiating, you don't
00:30:54want them to know that you don't have any alternatives, right? Like nobody else has made you an offer.
00:30:59So, and through, you know, I think natural selection built some deceptive tendencies into us and we
00:31:05kind of figure it out to some extent. Well, these machines are doing the same thing. You know, they
00:31:10are, and this was predicted by, you know, by people like Eliezer, uh, and I give them credit, but we're
00:31:16now seeing it, you know, these machines figure out that, uh, deception makes sense or that power is going to
00:31:22help them realize some goal. And they, they, and, and, and, you know, they may realize that it makes
00:31:30sense to be nice to somebody, that it makes sense to be mean to somebody given their goal. But yeah,
00:31:36they have, I would, they don't have an obvious bias in favor of what is from our point of view
00:31:43being good or bad. Now there's a whole field of trying to engineer goodness into them. But I certainly
00:31:52trying to make sure that our relationship with the intelligence, even if it indeed surpasses
00:31:59ours, as I think is likely is non-zero sum, right? Like, uh, we, you know, there's something it continues
00:32:06to get from our existence and flourishing that, uh, is compatible, uh, with the, the, the goals that it,
00:32:14it has and, uh, and vice versa. So, um, it's, but, but yes, it, we shouldn't, we shouldn't assume,
00:32:23uh, it's not that it's bad. The doomer scenarios don't depend on it being malevolent by nature.
00:32:29They just depend on it being expedient by nature. Yeah. It's not that it doesn't like us. It's that it
00:32:35doesn't care when we get in the way. That's one, that is one scenario. I mean, what are the, uh,
00:32:41what are the most legitimate concerns from the AI doomer camp in your opinion? Well,
00:32:46I mean, first of all, I'd say, uh, the thing I'm surest of is the sheer destabilization,
00:32:53the less sci-fi form of doomerism. So like jobs, it may be true that all the people who lose their jobs
00:33:00find new ways to spend time or, or find new jobs, maybe jobs per se, maybe spend time constructively,
00:33:08but I do think there's going to be a lot of job loss. Uh, and that's, that's disorienting and
00:33:13dislocating regardless of whether each person eventually has a happy outcome, right? Uh, there's
00:33:19going to be issues with, you know, parents are going to freak out about the kids spending time with these
00:33:24things. Uh, and there are, we've seen some bad outcomes and, and there can be more, um, you know,
00:33:32there are the, uh, the, the, you know, somebody could make a bioweapon with an AI, uh, an AI mythos
00:33:40is a good example of, uh, you know, the possibility that a cyber hacking machine could get loose.
00:33:46There's just, there's a lot of, on the one hand risks, things that, that, that will go wrong,
00:33:51at least at some level with doing some magnitude of damage if we don't play our cards right. And then
00:33:58there are these forms of destabilization that I think are almost inevitable, just social destabilization.
00:34:07And, you know, this points to one of the virtues of approaching this as a global community,
00:34:14leave aside, you know, regulating it internationally and anything else. It's just that I think we'd be
00:34:21better off going a little slower than we're going just because even if we successfully adapt to the
00:34:29change, it takes time. And if too much of it happens at once, you know, all hell breaks loose.
00:34:36And if you ask, well, why, why can't we proceed more slowly? The answer you get from the American
00:34:43AI companies is because of China, right? That's the first thing you hear. So as long as there's this
00:34:49sense of intense international contention, it's going to be hard to do even modest things. I mean,
00:34:56if you, I, you know, they once said to Sam Altman, like, what can't, shouldn't you be paying more
00:35:01attention to copyright laws? He said, well, that would slow us down. And I'm like, well, you know,
00:35:05life is hard. Speed limits slow me down, but, but that's just life, right? I mean,
00:35:10that doesn't seem like a good enough argument. And if you press further and say, and by the way,
00:35:15copyright itself, I'm not really bent out of shape on, I'm, I'm, I'm going to apparently get some money
00:35:19from this anthropic settlement because I've written books, but honestly, I'm just happy for what I've
00:35:24done to be in the training data. Copyright's not a hobby horse of mine, but the, but, but the point is,
00:35:30anything you say, like, if you say, well, maybe we should tax data centers, uh, to, you know, uh, to
00:35:37pay for the fact that inevitably, you know, there's, there's, there's going to be more, uh, carbon fuel
00:35:42consumption one way or the other as a result of this. And so a, it would be good to slow it down a
00:35:48little blah, blah, blah, blah. Any regulation that slows AI down is met with the same chorus from,
00:35:55from Silicon Valley, which is no, we can't do it because of China. So like, I think first of all,
00:36:01we could in principle proceed at a more cautious pace. If we would reduce the level of mutual fear,
00:36:09which I personally think is founded largely on misconceptions on both sides.
00:36:14Hmm. Okay. Before we continue, most people in their thirties are still training hard. Their
00:36:19protein is dialed in. They sleep better than they did in their twenties. Discipline is not the issue,
00:36:24but recovery feels somewhat different. Strength gains take a little longer. The margin for error
00:36:29starts to shrink. And that is why I'm such a huge fan of timeline. You see, mitochondria are the energy
00:36:36producers inside of your muscle cells. As they weaken with age, your ability to generate power and
00:36:41recover effectively changes, even if your habits stay strong. MiterPure from Timeline contains the
00:36:47only clinically validated form of Urythaline A used in human trials. It promotes mitophagy,
00:36:52which is your body's natural process for clearing out damaged mitochondria and renewing healthy ones.
00:36:56In studies, this supported mitochondrial function and muscle strength in older adults. It's not about
00:37:02pushing harder. It's about actually supporting the cellular machinery underneath your training. If you care
00:37:07about staying strong into your thirties, forties and fifties and beyond, this is foundational.
00:37:12Best of all, there is a 30 day money back guarantee plus free shipping in the US and they ship
00:37:17internationally. And right now you can get up to 20% off by going to the link in the description below
00:37:21or heading to timeline.com/modernwisdom and using the code modernwisdom at checkout. That's
00:37:26timeline.com/modernwisdom and modernwisdom at checkout.
00:37:31What's the AI risk that worries you the most that you think has received the least attention? Is it
00:37:36that? Is it the ability for humans to adapt to a changing environment or is it something else?
00:37:43Well, in the near term, it's not that the things I talk about, like job disruption and so on,
00:37:53are not talked about. But I think in the near term, what's not appreciated is how just highly likely it
00:38:00is that collectively these things will be destabilizing. Okay. It's just going to be an
00:38:07earthquake. And I think that's the thing I would like to most emphasize because it just gets people's
00:38:17attention to the possible virtue of talking about, yeah, calming down and slowing things down.
00:38:25You know, another, it's funny, another thing Tristan Harris said on your podcast is, you know,
00:38:32repeatedly pointed out your podcast is called Modern Wisdom. We're going to have to be wise to get
00:38:37through this. I agree. But here I'd add, you know, there's also, I talked about the fact that we're
00:38:46going to have this global conversation, ultimately, you know, something that is in some sense, a global
00:38:51mind is going to have to work this out. And as I said earlier, individuals are at their most wise
00:38:59when they are calm. Right. And it's the same. I think it's the same with planets. We don't know,
00:39:05but that's my contention is that the planet as a whole will do the wisest, most responsible job of
00:39:12stewarding this technology. If the planet as a whole is more tranquil, if there is less conflict
00:39:20and less contention. I mean, I'm sorry. I know I keep getting back to this sermon. It's my big sermon.
00:39:26So maybe the question you asked, I guess, was the biggest underappreciated thing. Well,
00:39:34I would say, I think there is more and more appreciation of the fact that this conversation
00:39:40has to be international and some of the policy does, but still not as much as I'd like.
00:39:46Right. Yeah. Well, I mean, I understand the issue, right? Because you have a technology,
00:39:55nukes weren't going to go off and just hit the entire planet. If one country developed a particularly
00:40:02strong nuke, right? Let's say that you get the Tsar bomber times two, the biggest bomb that's ever been
00:40:09dropped. And if you breach this particular threshold, for some reason, all countries
00:40:15are now at the mercy of all nuclear weapons. That's not the way that it works. But I think
00:40:19the concern that people have is if you build a sufficiently intelligent AI, it impacts everybody
00:40:27in a way that you don't just ring fence, like not pressing the nuke button. The problem is
00:40:34the coordination that we need in order to be able to do that. I mean, look at COVID.
00:40:38We couldn't even do it with COVID. And that was happening right then. That was people dying in the
00:40:43moment. That was every country on the planet being worried about it. Even China, even if it was the
00:40:49biggest PSYOP that escaped from the lab in Wuhan, they were worried too. So the lack of coordination
00:40:57that doesn't give me an awful lot of hope for people being able to do like predictive future
00:41:05coordination and like preparatory coordination. Right. It was not encouraging. I mean, I will say
00:41:12that although a pandemic is a non-zero-sum problem in the sense that if it breaks out in any nation,
00:41:19it's trouble for all nations and they should work your head off. Once a pandemic has started,
00:41:24there are zero-sum dynamics like who gets the masks, you know, there's finite amount of medical equipment
00:41:30and vaccines and so on. So it's not completely shocking. To me, the most disconcerting miss
00:41:37is in the aftermath when it became clear that, although I don't think we know for sure, it is at least
00:41:44possible that this pandemic was the result of a genetically engineered microorganism that escaped
00:41:52from a lab. It wasn't, it wasn't made as a bioweapon, wasn't released intentionally. We don't know for
00:41:58sure that, that it was a genetically engineered virus at all, but it obviously could have been. And it seems
00:42:06to me that if you process that information wisely, you say, wait a second, this could happen tomorrow.
00:42:13And one thing that shows is we don't really have any transparency, or at least not enough, so far as
00:42:20what's going on in other countries in their labs. Right. But that, that has not even been a conversation. To me,
00:42:28that's the most discouraging thing because, you know, a virus is in a way, a good analogy for lots of
00:42:34things that can go wrong with AI. I mean, first of all, there's a literal case of using AI to build a
00:42:39bioweapon, a new kind. And COVID, I think I've heard you say COVID was like a bad vaccine or something.
00:42:48What's the metaphor? Yeah, yeah, yeah. COVID was the worst kind of vaccine that we could have done
00:42:53for everyone because it's made us more skeptical of future pandemics and our response is going to be
00:42:59less coordinated. That's right. And, and you have to realize if somebody uses AI to build a bioweapon,
00:43:06they're going to make a point of making it more effective than COVID at, at doing whatever kind
00:43:11of damage they want to do. So it could be a lot worse. And so, uh, that could happen, A, but then B,
00:43:18the other, you know, some of the other AI nightmare scenarios, like the one that Mythos brings to
00:43:23life, you know, you, you, you got a self-replicating AI that is a super hacker. It jumps from data center
00:43:30to data center gathering, uh, you know, commandeering computer power, getting stronger as it goes,
00:43:36whatever wants to, I don't know, takes out the satellite infrastructure, who knows? Uh, that is,
00:43:42that's kind of, you know, a virus is a metaphor for that. Again, it's this self-replicating peril
00:43:47that makes relations among nations, non-zero sum. It doesn't matter where this thing starts off.
00:43:55It is a threat to your nation if it does start off. So you're going to have to coordinate policies
00:43:59with other nations because you need more insight into what's going on in those nations.
00:44:04Uh-huh. Okay. What do you think are the most legitimate white pills from the techno-optimists
00:44:08then? Let's look at the other side of the fence. Oh, wait, remind me of what white pills mean. I mean,
00:44:13I know blue and red, but what are, I, you're younger than I am and cooler. Techno-optimists,
00:44:19what is the, what's the bull case? What's the pro case? How can this thing go right? What are the
00:44:23most likely ways that this goes right? Uh, I think, I just, I'm sorry. I wish I could see it going
00:44:35right in a laissez-faire environment where you just let it go and, and let, uh, the market system
00:44:40deal with it. I just don't think that's going to happen. It's easy to point to wonderful things it
00:44:47could do. And we've heard them, uh, cure disease. Um, it could, it could, yeah, well, you know,
00:44:53one thing I, uh, this is not what the techno-optimists get into, but, but I referred earlier to like
00:44:58cultivating cognitive empathy, getting better to understanding other people's points of view,
00:45:04maybe getting more mindful generally, you can have an AI that helps you with that. But the natural
00:45:10tendency of the market will be to produce the kind that doesn't. I mean, we've already seen that if,
00:45:17if companies, you know, optimize for engagement, you may get sycophantic AIs to say, yeah, you're right.
00:45:23They're wrong. Like in this argument with your spouse, you're right. They're wrong. Uh, so that will
00:45:28tend to happen, but it can, AI can be a wonderful and a literally enlightening companion. Okay. If,
00:45:36if we want that, but you have to make a point to want it.
00:45:39You don't think that this is just going to find its way there naturally. Like if you just let the
00:45:44sort of capitalist meritocratic, it will find its way optimizing function without any shaping from us
00:45:50and without any predisposition from a, uh, a better coordinated world, it's not just going to arrive
00:45:56there. I think if enough people send signals to the market, markets are very efficient and wondrous
00:46:01things. You know, they really are. What does that, what does that look like sending signals practically?
00:46:07It means, for example, you, you and, and I, and enough other people to get the attention
00:46:15of people who are not necessarily the people making the foundation models or the frontier model. It could
00:46:20wind up being people who take an open weights model and open source model, and they kind of fine tune it
00:46:26to be this thing that interrogates you critically along certain lines, right? Like, okay, you say you,
00:46:32you say you hate this person. You say you find this country threatening. Let's just like, uh, or you,
00:46:38you say you think they're looking at it this way. Let's just play devil's advocate. It, it, you know,
00:46:42it's, it's almost, um, like doing steel manning, uh, automatically in some cases, but, but it, it depends on
00:46:52enough people. You know, there are a lot of things in life that they're good for you,
00:46:57but hard to do working out every day, good for you, but hard to do sometimes that's, that's why some
00:47:04people who can afford them, you know, have a personal trainer, right? They say, I'm going to,
00:47:10this person is going to expect me to show up in the gym three days a week or five days a week or
00:47:15whatever. And once you've made that commitment, you just kind of have to do it, or maybe they'll
00:47:18even show up at your house. Uh, but, but you know, and, and it's kind of like that. I think it's going
00:47:24to be kind of like that and choosing your AI companion, right? Like it feels good in the short
00:47:29run to have some, somebody will tell you or a machine that will tell you you're always right.
00:47:33And your, your adversary and rival and spouse is always wrong. Um, but you know, I want to be a
00:47:39little better than that. So I think, you know, now if the market signal is going to be strong enough
00:47:46for this to happen at scale, uh, these signals may emerge from like movements, you know, it could be,
00:47:53for example, uh, religions will, will say to their congregants, Hey, we recommend this model or we,
00:48:01you know, whatever. And then there's a demand for it. I'm not saying all those will be good.
00:48:06Depends on what, what group of religious people it is and what their values are. But you can imagine,
00:48:13you know, there are lots, there are lots of people right now engaged in the process of trying to make
00:48:19themselves, you know, genuinely better people. I mean, they, they meditate so that they'll be less
00:48:23volatile and, and, and work better with other people. I think we're going to have to go into this
00:48:30recognizing that for better or worse, these machines are probably going to be exerting pretty
00:48:36pervasive influence on people. And we need to think carefully about what kind of influence we want.
00:48:42What about the risk of AI induced thinking atrophy, right? This, this role of AI systems
00:48:49in taking on critical thoughts, uh, decision-making, um, connectedness, all of the things that typically
00:48:57humans really value inside of themselves. And as we start to outsource that to AI systems, our capacity
00:49:04to be able to do that diminishes AI induced thinking atrophy. Are you worried about that?
00:49:09Um, I mean, yes and no. I mean, you've heard the standard responses, right? Which is, I forget
00:49:17whether it was Plato or Socrates who supposedly said, uh, the written word is bad because people
00:49:22won't have to remember things. Yeah. Um, the, uh, and there can be some of that. I mean,
00:49:30the other side of the coin is obviously, at least right now, the richness of intellectual exploration,
00:49:37it permits, right? Like if you're interested in a subject, it's, it's almost like having like a leading
00:49:45expert there for you to interrogate. And for me, at least that's a much more efficient way to learn.
00:49:50Now you have to be on guard for hallucinations and so on, but I think machines are getting better and
00:49:55you can develop kind of an ability to, uh, to know when to be suspicious. Um, so that's, that's great.
00:50:04But I think, you know, I think what we can be sure of is that, you know, if this proceeds in a
00:50:12reasonably smooth way, the pace of overall intellectual progress will, will benefit from the technology.
00:50:20That's certainly not the problem, but I think you're asking a good, a good question
00:50:25as to, uh, you know, what, what it's going to be like to be human. If, uh, you know,
00:50:34well, for starters, there aren't many humans who can say, I'm really on the frontier. I'm the reason
00:50:40we're making progress right now. I will say, look, most humans don't say that now. And you shouldn't,
00:50:46you know, nobody should over extrapolate from whatever sub demographic they occupy.
00:50:51That's true. But I do think that everybody feels like they are breaking
00:50:55new ground, even if it's in their own life. I had Mark Manson on a couple of weeks ago,
00:50:58and he's got this great line, which is do hard shit, not because being hard makes it more meaningful.
00:51:03Uh, sorry, uh, not, not for the purpose of it being hard, but because it's hard, it will make
00:51:08it more meaningful that we associate a degree of meaning with struggle. And I mean, I'm sure that
00:51:13you have used a chat GPT or something else to help you write at some point. I've got to get,
00:51:19I've got to get a bit of research done. I need to write. So I'm really struggling to formulate
00:51:23this particular paragraph or this sentence or this idea, whatever that sentence is just less satisfying
00:51:30than the one that you spent time working on. And I wonder whether snow plowing out of the way,
00:51:37all of the challenges or more of the challenges, actually more of the challenges that humans face,
00:51:42primarily intellectually. And then when robotics come online, perhaps physically as well,
00:51:47it's going to sap meaning out of the world for most people at small amounts. And we're in the middle
00:51:55of a meaning crisis already. People are already struggling with meaning. And if you make life
00:51:59easier, if you make thinking more outsourced, if you make difficulty harder to access,
00:52:06the only way to do it is to be a Luddite, which means that you fall behind all of the other people.
00:52:09We're still in a meritocracy, right? So if you don't use it, you lose it. But if you don't lose it,
00:52:13if you don't use it, you also fall behind from it. It feels like a vicious situation to be in.
00:52:19Yeah. I haven't used AI in exactly that way with my own writing. I mean, a couple of times in my
00:52:23newsletter, I've said when I was just doing very short summaries of things, I just said, full disclosure,
00:52:31you know, your first draft was AI. But that's not really my writing. I'm just the editor. With my own
00:52:35writing in the book, I haven't done quite that. But I have had, I mean, first of all,
00:52:41I've had conversations with Claude in particular, which is very good with language, about subtle
00:52:46linguistic issues, like asking questions, like usage questions and stuff. And it's just, it's almost
00:52:55mind-blowing how good it is in that regard. But I have, you know, imagined the future enough that I
00:53:04have had moments of true despair. I mean, I said to somebody the other day, I feel like I'm a blacksmith,
00:53:10a century ago, you know, because I can see the writing on the wall. I mean, the, you know,
00:53:15I have a sub stack. And it's clear to me that the next wave is going to be,
00:53:26you know, you're going to see the success of a lot of sub stacks that are using AI probably more heavily
00:53:33than I'm going to be. But in any event, it's just so good that I can see the writing on the wall.
00:53:41Yeah. 95% of people don't get enough fiber, which is why Momentus built fiber plus because hitting your
00:53:48daily fiber target through food alone is actually kind of hard, even harder than your last toilet trip.
00:53:54Fiber isn't just a digestion thing. It's the foundation of your gut health, which drives how
00:53:59well you absorb nutrients, how stable your energy is, and how quickly you recover. If your gut isn't dialed,
00:54:04everything else you're doing is working at a fraction of its potential. Fiber plus is a three in one formula
00:54:09built to address digestion, gut barrier strength, and blood sugar stability all at once. That's why
00:54:14I plan to make my next big dump Momentus. Best of all, they offer a 30 day money back guarantee,
00:54:22so you can buy it and try it for 29 days. And if you don't love it, they'll give you your money back.
00:54:26Plus they ship internationally. Right now you can get up to 35% off your first subscription
00:54:31and that 30 day money back guarantee by going to the link in the description below or heading to
00:54:35livemomentus.com/modernwisdom and using the code modernwisdom at checkout.
00:54:39That's l-i-v-e-m-o-m-e-n-t-o-u-s.com/modernwisdom and modernwisdom at checkout.
00:54:47What are you most worried about or what do you think is going to take?
00:54:53Yeah, what do you think is going to be the highest risk?
00:54:58I just think doing what I've done my whole life, which is painstakingly generate writing
00:55:07that, you know, you hope will be enticing and clear and accurate and persuasive and so on,
00:55:19is going to be a less and less viable way to make a living. I think, you know, I do think for some
00:55:27time to come, well, I don't know how long, but people like me may still have a role as kind of validators.
00:55:34In other words, like, look, increasingly, you're going to look at a sub stack or whatever,
00:55:39you're going to go like, I don't know who, you know, how can I be sure that this person actually
00:55:44wrote it? I don't know them well enough to personally trust them. All you know is that
00:55:48they're vouching for the content. They're willing to have their name associated with this content.
00:55:52And so in a way, you know, it's a throwback to, you know, when I started out in journalism, the news
00:55:59weeklies didn't have bylines. And The Economist is, I think, still that way. So it's just like,
00:56:05you don't know who wrote the piece, but you know who the editor is, and you know you've come to trust
00:56:09the editor. So the editor of a magazine like that is a person you trust, even though you know they
00:56:14didn't generate most of the writing. They're just vouching for it. It reflects their judgment about
00:56:19what's good. They're telling you that they think it's accurate and so on. I can see that role for
00:56:26somebody like me for five years, but, uh, you know, it's really, really scraping the bottom of the
00:56:34barrel here. Well, look, I, I, this is, you know, I, I, to get back to how we started this off,
00:56:40if people understand, you know, how these AIs are being created, how basically you just give them
00:56:49the data and they do the engineering, uh, to generate the parts of the human mind,
00:56:55or if you just pay attention to the improvement we've seen over the last two or three years,
00:56:59right? Yeah. Um, what, what, uh, what, what industries would you be most bullish on?
00:57:04Or if you were a young person today, what do you think would be a good career path to go down or a
00:57:08particular industry to go in? Well, you know, it's common to say, uh, manual labor robotics is a
00:57:15little behind. It's going to be a while before I'm, uh, calling a plum, uh, a robot to fix my sink.
00:57:21I, I do think that, uh, certain kinds of human services are going to become almost more valuable
00:57:31because they're humans. And I think a good example is live music. I can well imagine that there will
00:57:36be more of a demand for, you know, for bands that play at small clubs in Brooklyn or whatever
00:57:44and make enough money to get by, you know, which would be in a way an improvement over the situation
00:57:4930 years ago during, you know, the golden era of, of the record companies was, was a winner take all
00:57:55market. You know, a few, a few people got super rich playing music. So I can imagine a world in which
00:58:01more people are actually making a living playing music. Comedians. Comedians. Good example.
00:58:06Live events, nightclubs. Yeah. And maybe look, uh, yeah, live events generally. I can, I can,
00:58:12I can well imagine that I myself have had the feeling, you know, because I've been so immersed
00:58:18in AI while writing this book of just like, you know, you're in New York, you're in a subway or
00:58:22somewhere and you see some guy, you know, a busker trying to, trying to make a few bucks playing an
00:58:27instrument and they're really good. There's some really good. And I just think, God bless you.
00:58:30You know, it's like, I, I just almost get emotional. Um, so, you know, if you don't think it's going to
00:58:39get weird, I don't think you're paying attention. Do you think AI could make humanity more religious
00:58:44rather than less? Well, that's a good question. Um, you know, there have been, I write in a book
00:58:53about this guy who, uh, he's a guy who actually started what became Waymo. I think, uh, the Google
00:58:58self-driving thing, his name's Lewandowski, uh, who was trying to start a religion that, uh,
00:59:05involved. Always, always a great first line to a story. He was trying to start a religion and.
00:59:11Hey, it worked out for L. Ron Hubbard. Scientology, right? Uh, he, he, he made a good living. Um, the,
00:59:20so, but no, but his, his argument was if we, if we have a respectful, even worshipful attitude toward
00:59:26the AI, then it will treat us well in, in return once it's running the show. I, I, I don't think
00:59:31it's going to work that way. So let's leave that one aside. Um, I mean, it's a good question. You
00:59:39know, the other thing is there are a lot of kind of spiritually related mysteries in the universe. Like
00:59:45what is consciousness? What is subjective experience? Um, uh, all I know for sure is
00:59:52it's the thing that gives life meaning. If you imagine beings, if, if you imagine humans,
00:59:58like they look like humans, they do the stuff humans do. The P zombie. Yeah. But there's zombies.
01:00:03It's not like anything to be them. I would say like, well, blow them up. I don't care. There's
01:00:07no meaning to their lives anyway. There's nothing meaningful going on. If there's not subjective
01:00:12experience, there's not consciousness. And I love to have more insight into what that is. I don't know that
01:00:18AI can help us because it's, it's the most stubborn mystery I'm aware of almost. I'd love to, you know,
01:00:24there's so many mysteries that are suggestive of something weird and wondrous quantum physics,
01:00:30for sure. I, uh, you know, I can imagine getting a kind of, I mean, who knows whether there's a
01:00:37revelation that awaits, right? That is at one level, an intellectual revelation that explains stuff,
01:00:44but at another level is also, you know, gratifying in a spiritual way.
01:00:48Yeah. Yeah. What's, uh, you mentioned, we've sort of circled around it a bunch,
01:00:53the idea that these machines are able to, uh, like pantomime intelligence.
01:01:00They're able to simulate knowing, but do you think that they know? Do they actually know what they're
01:01:06doing? I have a chapter on that. Actually, there's a famous thought experiment called the Chinese
01:01:11room thought experiment by a philosopher who's no longer alive and, uh, named Searle. And, uh,
01:01:18he argued that AI cannot have understanding. It cannot understand things.
01:01:26And there's a little ambiguity in his argument. What's, what's the, can you remember the thought
01:01:31experiment? Yeah. It's, so you're in, there's a guy, there's a, there's a guy in a room. He doesn't speak
01:01:39Chinese, but he, you know, he gets these slips of paper. Uh, let's imagine their questions in Chinese.
01:01:49And then he has a manual. He consults to decide what to write, what Chinese, you know,
01:01:55idea graphic script to, to, to, to put on the paper that he hands back out, you know, of the room in
01:02:02response. And to the people on the, on the outside who speak Chinese, it seems like there's somebody
01:02:08in there who understands Chinese. Okay. And what Searle says is this guy is like a computer program,
01:02:15because there's like a script that the program is following that, uh, that, uh, you know, the script,
01:02:24the script, his little book that he consults to decide, oh, if you get this, you, you, you, you output
01:02:29that that's like a computer program to Searle. And he says, well, we wouldn't say that anywhere in this
01:02:36room, there is actual understanding, right? So there's not understanding in the computer. Now,
01:02:41Searle was writing before the, the, the deep learning revolution, he was imagining a deterministic,
01:02:47uh, computer program. So that's different, but I think there's a bigger problem with his argument.
01:02:52It has to do with him, uh, the kind of the two senses in which he insisted, uh, that the computers
01:03:00don't really deal at a semantic level, a level of the meaning of words. I think, uh, I argue that we
01:03:07can now show that he was just flat out wrong about that. Now there is some ambiguity in his, uh, about
01:03:16whether he meant, he kind of changed positions, but whether he, he, he, he, he had in mind the idea
01:03:23that to really understand something, you need to have consciousness. There needs to be a subjective
01:03:27experience of understanding. Now, if he meant that, which in his classic paper, he doesn't really seem
01:03:33to mean, but if he meant that, then I would say, well, who knows? I mean, you know, no one person
01:03:39can say for sure that any other person is conscious, strictly speaking. Right. I mean, I'm pretty sure you
01:03:44are Chris, but at 99.99%. And, uh, you know, my, my, my dogs, God rest their souls are up, up in the nineties
01:03:53for sure. But we, the, the whole distinctive feature about consciousness, subjective experience is you can
01:03:59never know for sure that anything else has it. So we can't rule out the possibility that AI has it.
01:04:06Uh, and I certainly don't rule out the possibility that it does or may in the future if it doesn't now,
01:04:12but in any event, my point is if you want to say that consciousness is a prerequisite for
01:04:17understanding, in other words, you're not willing to grant that something understands unless you know
01:04:23it's conscious, then I, I, I just, we can't really argue about whether AI understands because we don't,
01:04:30we don't know if it's conscious, but it, you know, I come up with a kind of alternative way of looking at
01:04:34understanding, which is like, does, is it processing information with mechanisms that are like
01:04:42functionally analogous to the mechanisms in our brain that are at work when we have the subjective
01:04:47experience of understanding mechanisms that, for example, represent the meaning of words.
01:04:52Um, I, I would say to the extent that that's going on, I'm willing to say the computer is,
01:04:58is understanding things in a meaningful sense. And I think increasingly that's going to be what's
01:05:03going on. It, it doesn't have all of the elements of understanding that we have in our minds right
01:05:08now, but it has some, and I don't see any reason that it can't ultimately, uh, have all of them.
01:05:14What do you think is happening with the singularity debate at the moment? What
01:05:18have you, what have you learned around that? You know, cause what was, what was really interesting
01:05:21to me was I went through, I got whiplash from 20, say 15, 16, when I read super intelligence, then
01:05:2820, 17, 18, I'm real worried. There's going to be a fast takeoff scenario or computer brain
01:05:34interfaces. And we're all going to be under the thumb. And then by the time we get to 2019, 2020,
01:05:39I'm also distracted by COVID, I suppose, but I'm like, uh, AI isn't able to deliver on the threats
01:05:45that Nick was worried about when he wrote super intelligence. And then very quickly it comes back
01:05:50along and I'm like, right. Okay. Fuck here. It's, it's happening. It's happening. It's happening.
01:05:54Like, uh, the dude from the office who's going like, oh my God, it's happening. Everybody stay calm.
01:05:59And then, uh, we've now got to the stage where it seems to have like flattened out again, a little
01:06:05bit that we've asymptoted a little bit in terms of the models improving. I don't know of many people
01:06:10who think that LLMs are going to be the architecture that a super intelligent general AI is going to be
01:06:18like built on top of it's more likely to be world models and other stuff.
01:06:21So what's happening with the singularity debate?
01:06:25Um, I see a little more singularity going on than you do right now. I'd say, uh, in, in maybe a couple
01:06:33of senses. I mean, first of all, of course, the fundamental dynamic of the singularity is that
01:06:39the technological progress feeds into itself and, and accelerates the cycle. And of course, you know,
01:06:45famously, uh, Dario Amadei of Anthropic has been very explicit about this. And so is Altman. I think
01:06:51that, you know, uh, especially with these coding agents, it's gotten to the point that the better
01:06:58the coding agents, the more they can use them to create, you know, the next models. So the dynamic
01:07:05seems more and more at work, uh, at least just kind of in principle. I mean, they say that's what they're
01:07:13doing and, and look, the coding models, these agents, I mean, remember a year ago, it's funny,
01:07:20you know, I wrote the book, I had the chapter on agents, but it was just like a word people,
01:07:24you know, and then as the book, I'm, you know, it's, it's getting ready to finalize. I'm like
01:07:29rushing, you know, rushing to add all this stuff about like, it's actually happening and, uh, the,
01:07:37so the agentic revolution has happened, you know, and is happening, uh, pretty fast. There's also this
01:07:45famous, are you, uh, are you up on the, uh, what is it? The, uh, is it memory the group? No, uh,
01:07:53damn it. Uh, the group that does, uh, um, they do these evals where they measure how long it,
01:08:04how long it would take a human to do a job a computer can do. Okay. Okay. Especially programming
01:08:09tasks, but not only programming tasks. So, so, uh, they say, okay, right now the best large language model
01:08:16can do a task with like 80% success rate that it would take a person like a minute to do, or five
01:08:26seconds to do. And they, they've gone back and they, they've done these studies with, with the large
01:08:31language models for the last, like, I don't know, four years or something. And what they found as of now,
01:08:37more than a year ago, they found that these times that the task duration in human terms that an AI
01:08:44could do were doubling every seven months. Okay. That's exponential. Okay. That, that's a, that's a,
01:08:51if you, if you don't plot it on a logarithmic y-axis, you just plot it like a regular graph. It just goes
01:08:56up and up and up and approaches the vertical. And then it increasingly, as they kept doing the studies,
01:09:02it seemed like not only was it, it exponential, but the doubling time was getting shorter.
01:09:07Like, it's like, it's, it's like a Moore's law on steroids, even on, on steroids. And it's getting
01:09:13to the point right now where it's just hard to do the studies because of the length of the tasks,
01:09:19right? It's like, you can only, so the amount of time it takes to test the AI by the time you
01:09:24finish testing it, the AI is better, but you, you need to then do another model. It's like the next,
01:09:29the next one. Well, it's more, it's more like, you know, once it can do something that takes,
01:09:34I don't know what they're at now. It takes a human. I should look at the graph.
01:09:38Two, 200,000 years to do or whatever. Well, we're not up there yet. Thank God. But,
01:09:42but even once you get into like eight, 10 hours, it's like, well, wait,
01:09:45what kind of task are we talking about now? Right. I mean, it's almost beyond, I think they,
01:09:50anyway, they are having trouble formulating the task and, and, and, and testing them in,
01:09:58in humans. But the point is this trend has not subsided. And, uh, you know, a note in the book
01:10:04is kind of parallel to the, the, there was a curve, there's a curve like that for the growth in human
01:10:08brain size starting like a couple million years ago. And that was, uh, around, I hope I've got that
01:10:16right. You know, a million, two million, the, the, uh, that seems to correspond with the development of
01:10:22our certain amount of our linguistic, uh, hardware. So that, that had a lot to do with
01:10:27language processing and, uh, and, and I would say the, the way once you have language, the evolutionary
01:10:35value of, of, of manipulating it deftly grows. And so it's a self-reinforcing kind of process,
01:10:43but in any event, they, uh, you know, so there's that, but the last thing I'd say about, uh, is super
01:10:51intelligence, um, you know, can it happen? Uh, I think first of all, we probably will have more
01:11:00non-trivial breakthroughs. I mean, people often cite transformers and say, well, we have another of
01:11:06the so-called, you know, transformers, what the T and GPT stands for. All of these models use
01:11:11transformers and people say, well, we have another one of those. And I would say, well, first of all,
01:11:16even since then we've had chain of thought reasoning, which was very big and we've had, and that was
01:11:22only within, you know, a couple of years ago, we've had, you know, multimodal training, which is
01:11:28training a single model on various, along various sensory dimensions, you know, audio, video and, and, uh,
01:11:36text and so on is really in a fairly early stage. And that, and that was not a thing when the
01:11:43transformer, uh, came around. So in a way we've had those two things, we'll probably have more,
01:11:50but, you know, even if we didn't, I think, uh, in fact, even if, if we just halted training right
01:11:56now and didn't even create any new generations of models, I think, and you wait for the, the
01:12:02applications to get refined and people to integrate them into their lives in the workplace, I think
01:12:08breakneck advance would as a practical matter happened for a couple of years. But, but the
01:12:14other thing, and I think this is really key is that you got to remember, you know, in a way there's
01:12:20already such thing as human super intelligence and what it is, is like collective brains. Okay.
01:12:26Like there's nobody at Boeing who knows how to make an airliner, but Boeing knows how, you know,
01:12:31the corporation collectively kind of knows how to make an airliner. And it's, it's the same way with
01:12:37big scientific breakthroughs. They're always more collaborative, whether, whether intergenerationally
01:12:41or intergenerationally than they might seem when we give a Nobel prize to just one person.
01:12:47So collective intelligence resulting from communication among individual human beings
01:12:53is really a lot of what human intellectual progress is about. And these machines, they can
01:13:00communicate with each other. They can collaborate. They're starting to do it. Uh, they would be able
01:13:05to do it even if we didn't try to engineer it and make them better at it. But we are trying to do that,
01:13:09uh, you know, for purposes of scientific progress and so on. So I think, um, I, I don't think we need to
01:13:16worry about stagnation. That's not, that's not high on my list. I don't think anybody's worried about
01:13:22that. Yeah. You might not believe me, but this is what peak sleep optimization looks like. I'm not
01:13:30talking about the nightgown. That's just for sex appeal. I'm talking about my eight sleep. The eight
01:13:34sleep pod five comes with a smart cover you throw on your mattress that actively cools or heats each side
01:13:39of the bed up to 20 degrees. And now they've added the world's first temperature regulating duvet and
01:13:43pillowcase. So you've got 360 degree coverage for deep uninterrupted rest. It's like being Walt Disney
01:13:49without the cryogenic chamber and the racism. Best of all, their autopilot feature learns your sleep
01:13:55patterns and makes adjustments to improve your sleep in real time. It even detects when you're snoring
01:13:59and lifts your head a few inches to help you breathe better. That's why eight sleep has been clinically
01:14:03proven to add up to one hour of quality sleep per night. They have a 30 day sleep trial so you can buy it
01:14:08and sleep on it for 29 nights. If you don't like it, they will give you your money back. Plus they ship
01:14:13internationally. Right now you can get up to $350 off the pod five by going to the link in the
01:14:17description below by heading to eight sleep.com/modernwisdom and using the code modernwisdom at checkout.
01:14:22That's E I G H T sleep.com/modernwisdom and modernwisdom at checkout.
01:14:28Yeah. Who was a Edward Frank Fredkin? Who's that?
01:14:32Uh, so my first book, and it's funny because I now have, I mean, three of the six books I've
01:14:40written have the word God or gods in the title. I don't know what that means, but the, and, but the
01:14:44other thing is that book like this one has, uh, a visual reference to the famous Sistine Chapel thing
01:14:50where the hand of God is reaching out to, I assume it's the hand of Adam. Um, and, uh,
01:14:57both of these jackets have that in different, in very different ways, but Ed Fredkin, that book was
01:15:02called three scientists and their gods. My first book. So this is like, I started writing it a couple
01:15:08of years after I interviewed Jeffrey Hinton, I was writing a column called the information age at that
01:15:12point for, uh, the sciences magazine, which like so many periodicals I've written for no longer exists.
01:15:20But, um, the, uh, so the book was, it was information was a theme
01:15:27running through it in various ways. Like there was a, there was a profile of, uh,
01:15:31EO Wilson who studied ant colonies and the way they, they process information. But Ed Fredkin
01:15:38was this, who died, uh, maybe a year ago or so was this, uh, guy at MIT fascinating guy.
01:15:47Uh, didn't, didn't go to college and wound up as a tenured professor at MIT.
01:15:52He was a computer scientist. He had this interesting theory of digital physics, which in retrospect
01:15:58was kind of about us being in a simulation. And in fact, I, you know, we talked about that.
01:16:03Um, but he, for a time was at MIT head of what was it affect the AI lab. I forget the, there were various
01:16:13names at one point, I think it was project Mac maybe and something else, but the, uh, he at the time when
01:16:21I was, uh, interviewing him on his, this Island he owned in the Caribbean, he was apparently the model
01:16:26for the character, this professor in the movie war games with Matthew Broderick. If people remember
01:16:31that, that one from the early eighties, uh, I think the professor in that was very worried about
01:16:38nuclear war like Ed, and I think owned an Island even, I think that was based on Fredkin. Anyway,
01:16:43uh, Ed was saying to me, like when he had been at MIT, well, first of all, when I said to him, like,
01:16:51what's the meaning of life? And he said to me, this is in the eighties. He said,
01:16:54oh, it's to create artificial intelligence. You know, that's the next stage in the evolution of
01:17:00intelligence. And he explained to me that when he was at MIT, he tried to start this initiative,
01:17:08this international AI lab. He said, because he knew that if this became a subject of international
01:17:15competition, we were in trouble. This was during the cold war. So he wanted to get a U S Soviet collaboration
01:17:22on, uh, on a, like a single lab where AI would be developed for the good of humankind.
01:17:28And he said to me, you know, and I failed and now it's too late. Um, but he, you know,
01:17:33he foresaw a lot of things. I will say, uh, encouragingly, he had a pretty sunny view of
01:17:40super intelligence. He did think we would get super intelligence. He said, first of all, he said,
01:17:45you know, when AI first emerges, it'll, it'll be like the human mind really good at things,
01:17:50laughably bad at other things. Well, he's right about that. He said, but eventually, you know,
01:17:54it'll be this incredibly intelligent thing and it'll, it'll be nice to us. We'll just be like,
01:17:59you know, ants, ants to it. We won't, it won't, you know, won't have any interest in,
01:18:03you know, or like squirrels to it. It won't have any interest in disrupting our lives. It won't need to.
01:18:08And, and look, I think you asked earlier, I don't think I ever answered like what's,
01:18:13what's the bull case for the accelerationists. I mean, first of all, I think it's going to be
01:18:18disruptive in the short term in any event, in ways we should pay attention to. But as for
01:18:23long-term non-Doomer outcomes, I think it's entirely plausible that it will turn into a form
01:18:30of intelligence that, uh, treats us well, maybe because it's just morally enlightened. You know,
01:18:37I, I, you know, in a certain sense, um, in the relevant sense, from our point of view,
01:18:43uh, or maybe because it'll just be so powerful. It'll be, I mean, it may, you know what, maybe
01:18:48that's more likely if it's sentient because it'll say like, well, we're sentient. We think that's a
01:18:54good thing. These guys are sentient. And of course we can kill them, but you know, it's good to be,
01:18:59you know, subjective, why, you know, you know, just the way you and I would not
01:19:03pitilessly kill a dog. Right. If we were convinced it wasn't like anything to be a dog,
01:19:09as Thomas Nagel phrased, you know, the question of consciousness in his, in his essay, what is it
01:19:13like to be a bat? If we were convinced that dogs didn't have subjective experience, we'd probably
01:19:18think, eh, I don't, you know, whatever, who cares? But, you know, we, even though we evolved as these
01:19:27self-interested and sometimes ruthless creatures, if it doesn't cost us to keep something alive that
01:19:33we think is capable of subjective experience, we'll do it. And, and that can well happen. I am not
01:19:39predicting the Yudkowsky scenario. It's just that I can't, I can't get the probability of it down to a
01:19:47level so low that I don't think it's worth worrying about. I'm going to take that as a white pill,
01:19:52even though you didn't know what that meant. That was my first, my first white pill.
01:19:57Your first ever white pill. I popped your white pill cherry.
01:19:59Thank you for that, Chris. No, it felt so good.
01:20:02You're welcome. Robert Wright, ladies and gentlemen. Dude, you rule. I love all of your work. Everyone
01:20:05should go and read The Moral Animal. It's over 30 years old now and still just, it's so good. It's
01:20:10so fantastic. And you've got your new one as well. Where should people go to check out everything else
01:20:13that you've got going on? Well, I have a newsletter called Nonzero on Substack,
01:20:18podcast called Nonzero on Twitter. I am @RobertWriter. That's W-R-I-G-H-T-E-R, kind of a pun.
01:20:28And that's about some of it, the books, the God test. And let's focus.
01:20:36I am going to OpenAI's campus and HQ next week. So I'll see if I can find out. I'll see if I can
01:20:42find out any super secret insights there. Do. Please report back to all of us.
01:20:47I shall indeed. Robert, appreciate you, man. Until the next time.
01:20:50Thank you. Catch you later on. Bye, everyone.
01:20:52Thank you very much for tuning in. If you enjoyed that episode, YouTube knows who you are deeply.
01:20:59It thinks you're going to like this one even more. Go on. Press it.