The stamp collecting machine we talked about last time is a physical impossibility effectivly. Because it has a perfect model of reality, or an extremely good model or realtity, that we just gave it by specifying it. It looks through every possible sequence of output data within a year, which is far too large a search space to search exhaustively. And it is able to evaluate for each of this extremely huge search space a detailed, year-long simulation of reality to figure out how many stamps it gets. So clearly an actual computer to run this algorithm is significantly larger than the universe. So why is it even worth thinking about? Right? Umm.. And the reason it’s worth thinking about is… Self Improvement It is reasonable to expect that a general intelligence which is not this powerful would improve itself over time. and eventually become very powerful. Not as powerful as this, but closer to this than our intuitive understandings of intelligence. Umm And the reason for that is… that intelligence, in this context, is an instrumental value. Whatever you’re trying to achieve, it’s valuable to be more intelligent. If you sense that you’re not intelligent enough to come up with the best possible stamp-collecting plan, the first part of your plan might be spent designing improvements to yourself to allow you to come up with a better plan. [Interviewer] Is the machine realizing it might need to modify itself a bit like somebody saying “I wanna be a doctor” so they decide to go off and get a medical degree? In a sense, yeah, although people are not really able to change our intelligence, our core intelligence. We can acquire more knowledge, Umm and we can acquire more skills, which is in a sense increasing your effective intelligence but… if you could have a… if you could take an action which would actually increase your ability to think, in all senses, uh, that action would be worth taking. even at fairly significant cost. And this is something which is true if you’re trying to collect stamps, but also if you’re trying to do almost anything else. Being better at modeling reality, being better at picking options, better at making plans, better at enacting those plans, Whatever your plan is, that’s worth going for. And we may also have reason to believe that a general intelligence might… …be quite successful at improving itself. Human beings designed it, but it’s quite easy, it’s quite common for humans to design things that are better at what they do than humans are. Right? I am a very weak chess player. But I could, given some time and effort, write a computer that would beat me at chess. So it’s perfectly plausible that we might write, we might design an AI that is actually better than us at AI design. So that it’s able to make improvements to itself they that we haven’t thought of. Machine self-improvement… We don’t really have a sense of the time scale. And it could be extremely short. Right? If it’s just a software change, you write the piece of software, you run it, It might think faster than a human being, I mean, computer clock speeds are much faster than human… brain speeds, Um It might discover improvements that it could make to itself very quickly, rewrite its own code, that’s all in software, It could do that very quickly. I mean it could do that within a day, it could do that within the first second of being turned on. Um And then it becomes very interesting because… There’s a possibility that this process could be recursive. That is to say, that once it’s redesigned itself, it’s now more intelligent, It’s possible that at this new level of intelligence, It is better at AI design than it was. And it’s able to come up with more improvements that it can make to itself. Umm And so on and so on… And so the thing continually increases in its intelligence. Umm… And this could be quite rapid. And then the question becomes: Is this subcritical or supercritical? So there’s kind of an analogy here with a nuclear material, right? If you’ve got some fissile material, say it’s uranium or something, every time the atom, uh, decays, the nucleus splits, releases some energy, it releases three fast neutrons I think. And those three neutrons go off into the material, and may hit other nuclei and cause them to fuse, [probably a mistake, they decay rather than fuse] each of which gives off three of their own. So the question is: For each decay of a nucleus, how many further decays happen as a consequence of that? If the number is less than one, then you’ll get a bit of a chain reaction that then fizzles out, because on average, for everyone that splits, it’s creating less than one new one. If it’s exactly one you’ll get a self-sustaining reaction, where the thing is like a nuclear power plant, where each thing sets off one other one on average, and so the thing just keeps going steadily. If it’s more than one, you’ll get a runaway reaction in which the level of activity increases exponentially. If each fission event results in two more, then you’ve got 2,4,8,16,32… and you’ve got at best a melt-down, and at worst a nuclear bomb. Umm… And so, we don’t know… what the shape… of the landscape is around any general intelligence that we might design.