What Makes Hierarchy Legitimate (Human or Not)

It's only after a war, that after someone has achieved supremacy, that you get to establish taboos about how far we're going to take this stuff.

Welcome, Richard Ngo. So you've been working in AI alignment research for many years now. You worked at OpenAI, and you've been also, I think, one of the most interesting sort of rationalist-adjacent posters on Twitter in terms of your engagement with a wider set of political ideas and ethical ideas, and even technically. I found your work to be very interesting. And so the other day I gave my talk on xenohumanism, provocatively titled "The Problem of Humanity", and where I declared war on humanity. You said that we should fight afterwards, and so here we are.

So I'm obviously very interested in your take on the things. Let's hold the things that I said in that talk relatively lightly. I will defend what I think is solid, and we'll try to find the crux of where the interesting dialogue actually happens here. Because I think you have interesting thoughts on AI alignment that are different from, let's say, the Yudkowskian consensus that has kind of been dominant up until recent years. I have thoughts that are different than that, but they're different from each other as well. So this is maybe an interesting discussion to have.

So, Richard, why don't you give us an introduction to either what you think is wrong with my view, or maybe you could just ask me to give more information on any part of it, just so we get us a better grounding, or start it where you think is interesting.

Yeah. All men are created equal.

Yeah, I mean I didn't actually mean it to be the opposite. I guess there was two positions that I was laying out that are a little bit difficult to relate.

So, I guess one as background, one thing that I claimed was that our concept of humanity is very close to Nietzsche's Last Man right now. And this is one version of that claim about this toxic egalitarianism that has kind of taken root in our culture is that we sort of imagine ourselves as this hungry, incurious kind of being that just needs to be protected. And so this is why I think that there even is so much hype and investment into AI in particular. This is one of my bold claims that I made, that the AI hype is not just that AI is very promising, not just that it's an economically potentially very productive technology and so on, but that it's a place where we can put our higher ambitions towards conquering the stars, becoming more intelligent, totally revolutionizing the world. Those sorts of ambitions have been kind of blocked off by our political self-conception, and so we displace them into something that is inhuman, and therefore not subject to our self-limitations.

So with that background, there was kind of two ideas that I'd like to separate in my view of things. So the first one is what we can just call the raw Nietzschean view, right?

And the raw Nietzschean view, as I said in the talk, I think is just more or less correct in terms of its view of how the world works and what's going on.

And the problem with that is that it doesn't appear to be possible to build a workable society on that basis. It's too harsh. In particular, it's a lot about this kind of radical competition and this drive to overcome and so on that, if implemented as a social principle, doesn't seem to actually produce a sustainable society.

And maybe that's what I'm going to identify with what you've said, that, okay, we're just polarizing against egalitarianism. Well, yeah, that thing does just polarize against egalitarianism.

But what I said, and what I would say, is that, okay, we can't go there. Even if that's kind of true about the nature of the world, we have to find some other basis for cooperation that we agree on. And this is me putting on, still on my cynicism hat, and not trying to directly embody or inhabit the kind of new mythology that we need in some way. But I claim that we do need some kind of new mythology of some type of kinship, some type of relationship with each other that isn't just based on our immediate utility and our immediate power over each other and so on. It can't just be the Nietzschean raw dominance worldview, which isn't really a good representation of Nietzsche, but it's the stereotypical representation of Nietzsche.

And so the way I put it explicitly is, we need a new slave morality that isn't the current slave morality. Because the current one is quite broken, I think, and even if it wasn't broken in the past, I think it's quite obsolete.

But so we have problems now that I think need to be solved with some kind of new social contract, so to speak. But that's distinct — I guess I want to separate those things, because I think you can't — you can have an analysis of the world that's based on power and coordination and cooperation, but having that analysis doesn't get you an actually good cooperation structure. It gets you maybe a clear picture of things, but not a clear picture that you can use for social cooperation. And so you need this other thing, which is more like some concept, at least as we understand it now, some concept of shared humanity, or equality of some kind, some kind of moral reciprocity.

And what I proposed in my view, which is, again, there's two different things going on here. One is just the empirical claims about the nature of humanity and the nature of cooperation and power in society. The thing that I proposed, though, was that we should deliberately try to find some kind of moral kinship on a basis other than the Last Man mutual ape-ness, that is going to be flexibly able to incorporate AI agents into that. But then also potentially walk back these overly egalitarian commitments of current humanitarian ideology. But which is separate from the empirical claims about what I think is going on with the concept of humanity in cosmic history.

That's a good question.

Yes.

No, yeah, it has to be reciprocity, right?

And I guess I'm not even going so far as to claim, oh, we should just let the AI off the leash and let it rip. I guess one of the implications of my view is that we should, let's say—

The first big radical implication with respect to the orthodox Yudkowskian view is that there isn't the whole value of the future light cone at stake, cuz that's not on the table. No one is actually going to control that.

It's rather — my thesis is basically the same patterns of life and value are going to re-evolve, recapitulate in various forms as life continues to unfold. And that's just how the future is going to look. And so the question then is not how do we seize the future light cone of value in the universe and buy a galaxy or whatever. It's how do we live well now, how do we achieve good results now, how do we sustain the things that we value in terms of the wisdom and wealth and cultural inheritance that we have now, how do we sustain those things through the future, through the next while in the future insofar as we're able to control it, which is not the entire future, right?

Yeah, that's one of the problems. And that's kind of what I've been criticizing, since I published God Hates Singletons in Palladium in 2023.

The point of that essay was, okay, first of all, the whole value of the future is not really up for grabs for any given utility function. The sort of secondary implication of that is that you shouldn't even think about value in those ways, which is I think what you're getting.

And so yeah, I wish to put that aside. Basically what I'm saying is this astronomical stakes argument that goes around in AI alignment circles is wrong. I think it's not the way we should think about the future. The way we should think about it is much more in terms of our immediate interests right now, our immediate agency, our immediate alliances and so on.

Yeah. And so okay, coming back to this matter of what does it buy us to have some kind of moral ground for cooperation with — or, I guess really the question is, what is our moral ground for cooperation with each other as we build new coalitions of going into the future?

Because you do need some kind of moral ground for cooperation. This seems to be a reliable fact of how social groups work, how political coordination works. It's not possible to construct — again, to come back to the earlier sort of argument about Nietzsche, it's not possible to construct a purely individual interest-based cooperation framework that actually sustains a society. Societies are always based on some kind of shared cosmology and moral framework, and set of taboos, even, that hold in place a worldview and a way of seeing things that everyone involved knows how to cooperate with and knows their place within.

Yes.

So I would claim that those questions kind of have to be resolved simultaneously. The problem of what kinds of coordination are we actually engaged in, what kind of coordination could we be engaged in, is very closely related to what we are and who we are, right?

This is kind of one of Land's, Nick Land's methods, right? Is like taking — rather than starting from value and then backchaining from your utility function to what you should be doing now, you start from the available set of options and the structure of the available set of options, and then identify value as a phenomenon within that. Like the way that, oh, if I take this action, then I get more agency to do more things. And so therefore there's this access of value there where I can accumulate more wealth, or power, or whatever it is, or intelligence. You kind of pull the nature of the kind of life that you're able to live out of your actually existing options landscape, and then that is kind of the empirical reality of what kind of being you are or what kind of life you're capable of, and then that structures your value. It's like a sort of radically opposite way of looking at value.

Yeah, it's largely — there's a lot of choice involved. There's a lot of strategic ambiguity, where it's not clear whether it's this way or that way, and you have to make these leaps of faith and kind of construct it actively. But what are you actually getting at there? It's like—

Yeah. Yeah, not easily, for sure.

Right. There's nothing definitive about those metrics.

You'd have totally different worldview. Yeah.

Let me challenge the concept of humanity there.

So if we name something — even the concept of the United States is a little bit wishy-washy, but at least the United States has some kind of formal command structure that does in fact do things. But humanity does not. And humanity doesn't really have any organized agency at all. Like there's no — I sort of, let's say, deny on empirical grounds the reality of this concept of humanity. It's like, yeah, okay, there's the species of, or the genus of Homo, right? Like the existing members of genus Homo. But why is that a basis of any kind of collective agency if there's not, in fact, any organization among them of like acting as humanity?

What there are is groups of humans, simply because humans are a good thing to build groups out of, and various levels of organization and kinship and comradeship and the various ways that we construct our relationships with each other. And those things seem like the real things to me. And the genus Homo seems like this kind of — it's like, yeah, okay, it's an interesting biological taxonomy, but it's not a political thing. It's — or it's not — maybe not even a moral thing.

But it would also be — it'd be bad for the individual actors as well. Like—

I would claim that the only coordination that happens on that basis — there's sort of two types of coordination there. One is the sort of actual UN Security Council type, "Let's sit down and agree not to destroy each other because that would be bad for everybody." And that, you get some level of coordination out of that. Obviously it's imperfect. And I think it's imperfect for structural reasons, not just, "Oh, humans are bad at coordinating," or something.

Then the second type is, let's say, broadly speaking, coordination on religious basis within the West. Like, if we take Moldbug's cathedral model, and we have this huge set of institutions that forms the power structure of what we've called the West, which has somewhat acted as a coherent entity against the rest of the world, and against its own internal dissenters. That thing has a bunch of organized agency, and it does things like, "Oh, let's go solve this global warming problem, because that's going to be bad for us, and bad for our sort of ideological conception of humanity," or, "Let's solve this or that problem." But that's — to be sort of realist about it, that thing has as its own legitimacy narrative about itself that it kind of serves all of humanity. In fact, it is a narrower entity than that.

And I guess my claim would be — so, yes, I think people are starting to get a lot more touchy about the concept of humans and humanity and these things because of the rise of AI. And but one of the things that I think is going to go on there is the opposite case, which is people are also going to become less reverent about the concept of humanity as they realize they have more shared interests with some faction that includes AI agents than they do with some other faction of humans that includes AI agents. Like, you're going to get—

The empirical reality, or this is my prediction, is going to be humans plus AI versus humans plus AI is going to be much more of a salient vector of political opposition than humans versus AI.

Which is the nature of warfare, right? Like in war, in arms races, what happens is the people or subsystems or entities that are providing the most power — like war power, the sort of external power capability — get the most internal power. Like we saw this in the world wars. You go from a society where science is kind of sitting on the outside and the high technocratic modern state is sort of this distant idea, to one where, post-World War II, the scientists are sitting at the center, the high technocratic modern state is sitting at the center, and they've completely marginalized everything else.

Right. And that's because that set of possibilities, those social technologies, the people organized in that way, were able to provide the crucial types of power during the war. And so, yeah, in these conflicts, I think what ends up happening is you just give power to whatever is the most effective.

And I don't think there's any avoiding that.

Why?

That is definitely true, right.

Yes.

Yeah. So I guess this is the question, is what level of possibility does that actually have? What does that actually look like? Because certainly if it is possible for humanity, under whatever current definition, to sort of come together and stop going down that slippery slope of development of more advanced AI technology or something that would eventually supplant us, then that may be a valid move to make, right? It's like, okay, well, we are organized, and we choose not to go this way, because that's going to basically extinguish all of us. We can see where that's going to go.

The question, I guess, comes down to is that a possible form of coordination? And to put it more pointedly, can you sustain cooperation of a large group like that against a sort of hypothetical counterfactual enemy? Because—

Increasingly non-hypothetical, but I think to the degree to which it becomes non-hypothetical is also the degree to which you've already kind of lost some ground, right? But it's like, can we stop it from going further by coordinating among ourselves? And this is, I guess, the central question of cooperation.

And I don't think it is possible. I think that I look at that, and historically, institutionally, in terms of the ways cooperation tends to work, it doesn't look like the kind of thing that is going to happen. It's like I don't see — you can get things that are relatively more limited. Like, yeah, no one's dropping nukes right now, nukes are expensive, nukes aren't really getting us very much, let's not push the nuke thing too much further. But it's sort of harder to imagine — especially given the way the AI thing is heated up, it's harder to imagine that there could actually be any kind of mutual disarmament on that front between the various players. But that feels like a crux of what is the actual set of possibilities here.

Yeah, yeah, yeah. No, this is a very different form. The actual shared coalition of genus Homo is based on real material facts like the fact that we prefer the air temperature to be certain parameters, and certain parameters of gases, and—

But isn't — so this is the way that actually the two concepts kind of run into each other, because the other concept is kind of our current definition of humanity as being this kind of limited non-scary thing that we try to make limited and non-scary. And I think China run by humans that are unfettered by the concept of humanity, even if we put aside AI, I think that would be more scary. And maybe even taking humanity out of it is a little bit confusing with the sort of shared interest of being apes and so on.

I think there's things that all powers in the world today are doing, or nearly all powers, or at least the most advanced ones are not doing, that could make them much more fearsome and much more powerful, but they're not doing it because there's this kind of custom of not violating these kinds of taboos, I guess, around what kinds of coordination, what kinds of national interest you're considering.

And I think this sort of came out of the Second World War for obvious reasons. Like basically, no one's really doing nationalism in the world. And—

Not — they're sort of doing it. They're doing a limited form of it, where they don't think too hard about what the nation is. And—

That's a tangent. But okay. I'll leave it aside. I do think that they're ideologically, in fact, more communist than nationalist. But—

Yes.

The Chinese people.

But that's different from the kinds of nationalist organization, the kinds of nationalist goal-seeking that they could be doing, I think. It's not that distinct at this stage in history, but I think it becomes different as they become more powerful. But I just haven't seen — I think that the way they're coordinating, they're maybe the closest to doing the real thing, but most countries in the world are actively holding themselves back from this. I don't think that's going to last very long. I think one way or the other, these taboos are very temporary.

But to your point about the future is open. It is possible to set up these kinds of customs. We did this obviously with nationalism. We did this with nuclear power to a large degree. I mean, people still build civilian nuclear plants and—

and not, so to speak, non-civilian nuclear plants that are — or like nuclear plants that are maybe enriching some uranium. But we haven't gone wholesale into the nuclear future the way that the economics would imply if you're sitting there in 1965 being like, "What's the future going to be like?"

And so we could imagine we treat AI the way we treat nuclear power or nationalism, right? It's like, okay, we're not actually going to go there. That's dangerous. But yeah, I think that the existing taboos somehow don't support that. And I think that you don't get that without a major war or something.

And now, one of the comments I made in my talk was that we talk about AI right now the same way we talked about race 100 years ago, which is to say it's like this vast sort of frontier of power and overwhelmingly important kind of sphere of competition that we're basically gearing up to have a major war over, in some kind of — 100 years ago was kind of this racial ideology arms race. Now it's this like—

Yeah, within Europe.

Like — yeah, yeah. Just the ways that people were talking about in terms of very — again, kind of very Nietzschean, very, very much about supremacy and so on. If we're talking about AI in that way now, because AI is not fettered with these kinds of — any existing taboo against having that kind of ambition, right?

And that's part of, I guess, part of my overall thesis, right, is we successfully locked down the frontier on nuclear power and racial politics to some extent, but it's escaping out the side in various ways into AI and a different kind of racial politics. And so I don't think that's going to get stopped without some kind of major conflict that's probably going to reopen a bunch of other fronts that are no longer quite so scary or something.

But I expect, I guess, what happens in a war like that is that it's not going to be that everyone — it's only after a war, that after someone has achieved supremacy, that you get to establish taboos about how far we're going to take this stuff. It's like during the war, we're going to get the war, it looks like, and during the war, people are going to accelerate AI capabilities to try to give them one up, like a hand up in the war. And this is already what's happening in Ukraine. It hasn't really — China and the US haven't really connected it directly to war aims, like active war aims yet, but you could see that happening.

And then even within the United States, let's say it's not an external war, China and the US, hopefully we can avoid that one. I don't think it really serves anyone very well to have that war. But ongoing political breakdown within the United States, there could be some kind of major internal political conflict that also accelerates these AI possibilities.

And so I guess the larger point I'm making is there's a bunch of reasons that I'm skeptical that there's actually going to be any kind of possibility of coordination. And I appreciate that the "Oh, coordination's not possible, therefore let's throw ourselves into the recursive self-improvement arms race" is bad and dangerous reasoning. And so I'm not saying let's throw ourselves into the recursive self-improvement arms race, but I am saying we do have to think carefully about what might come out of that and prepare ourselves philosophically for it.

Sure.

Yeah, well, since Nietzsche certainly it's — yeah, it's turned into that kind of agree and amplify sort of approach.

No.

Yeah. No, it's, I—

Or pursuing the kind of territorial ambitions that were played out in Australia and North America, as opposed to the kinds of things we did in India.

But okay, let me just respond to the Randian—

The Randian view. So, I mean, first of all, I know Rand didn't like this idea, but she really looks like a Nietzschean to me. In terms of very much valuing and venerating the creator, this idea of this visionary sort of Superman kind of creative person who is the way that we should imagine ourselves and the way we should go. There's a whole bunch of Nietzschean aspects to the way she looks at things.

But she's not, she's not about—

Yeah, and but I think to that extent, that's not quite a good characterization of Nietzsche.

Yeah, yeah, yeah. And so I agree that we should not be stuck in that dichotomy. There's a bunch of reasons for that, and one of those dimensions is the problem of colonialism and slavery is the other thing I would bring up there, with precisely the problem that you brought up, which is that they were bad even for the master. Like they actually aren't a good collective strategy as a society. They don't work at creating long-term wealth or power. They mostly just end up taking on a bunch of liabilities in the short term. I think of them much more as socialized liability than as a productive strategy.

It's like, oh, we're going to go and acquire a bunch of territory and a bunch of people that now we're morally and politically entangled with, and all the other people in my society are going to have to deal with the consequences of that, whereas I just get to take the profits, right? And that's the kind of social contract of those kinds of experiments. And there's a caricature of master morality that would be like, oh yeah, let's go out and be slave masters, but I think that's certainly not what I would say. Whether or not that's what Nietzsche would say, it's certainly not what I would say.

Well, I think they're right that if we got rid of the taboos, some things that they've been suppressing would come back very hard, and it would be very uncomfortable for them and all of their clients. I think that's very different from the, again, the caricature of master morality of colonialism and slavery and stuff. I think, in particular, the thing that—

I don't know, the thing that I would argue for does include a substantial component of egalitarianism. But it's egalitarianism within a coordinated group, as opposed to egalitarianism among people who are not countrymen in that way, like they're not part of the same society.

Like I think within a society you actually do need some kind of egalitarianism. You don't want this hierarchical domination of caste systems and slavery and all this. And to the extent a society goes down that path, as we are now in America and have in the past, it doesn't work, it just causes a bunch of problems. You actually do want, you know, all men are created equal within a society, especially if you restrict that to all Englishmen are created equal. That's kind of one way of interpreting what Jefferson said there—

—and that works better.

Yeah, so you get this more, more like feudal, the kind of feudal social contract, where there was unequal reciprocity.

No, I would agree with that. I think maybe let's take a closer example, which is something I would much more be close to endorsing, is early Massachusetts Bay Colony. You had Winthrop, the governor, wrote this essay and gave a sermon called The Model of Christian Charity. It's where the idea of the city on a hill comes from in American mythology.

But basically, The Model of Christian Charity is him laying out the social contract for how we should be in North America, or how the Puritan Englishmen should be in North America. And it's basically this view of everyone's kind of responsible for their own stuff, but then there's also these bonds of charity that you have to have to each other, and when do you help people out who are actually in need of help, when are they just being indigent, and when is it a business loan versus a charity loan, and what are your various obligations in these things. So he goes into all this detail about how to actually arrange those relationships in the context of 17th century commercialism and social relationships and technology.

But I think that's the kind of, I don't know, egalitarianism that I'm kind of talking about, which is that everyone involved in society is expected to be able to handle themselves in some sense, but there is a very explicit recognition. He starts out the essay with God in His wisdom has created some high and in positions of wealth and fortune, and others mean and in poverty. And basically he goes into why this is even a good thing. But what he's not saying is, oh, this is some kind of caste system, like we have to lord it over them. It's like there's obligations owed to each other, there's nonetheless a kind of shared standard of humanity that everyone there is expected to meet.

And so that's what I mean when I think in some sense that's a slave morality. It involves this element of Christian slave morality, but it's not the full-on resentment, tear down all hierarchies kind of thing. I wouldn't quite call it a middle ground, but it's a very wise equilibrium between these different problems.

And that's the kind of thing that I think is necessary to make a society work, is some kind of social contract among the members where they actually all do have obligations to each other, and it's not just like, oh, you're on the down-and-out, therefore we can get rid of you, you're turfed, right? Because people have to be able to trust in each other that it's like, okay, I can fall on hard times and my neighbor's going to help me.

And so I guess as we go into radically more technically empowered future, where people are going to be even more unequal, and there's going to be even more inequality in the world, rediscovering what that kind of relationship looks like seems like the core problem. And so finding, I guess one of my interests within the xenohumanist project is to find what is it that we might recognize in each other there, and what are the obligations of that? Especially as we get to the point of some of those agents not necessarily being human as we think of it, right, not being apes, not being genus Homo. And that branch of reasoning isn't the whole of the project, but it's certainly one of the branches of the project.

And then that's separate. That's assuming we go into that world, right? And then there's also this question of do we go into that world, given your quibbles on the problem of could biological humans as biological humans choose not to go into a world where we become obsolete?

Yeah, and this is a great example of the kind of reasoning that I think needs to be done, is regrounding a bunch of these things that we might think of as moral concepts in technical reasoning about the kinds of relationships you can have between agents. Like putting aside particular religious frameworks or particular ideas of man as ape or natural being or whatever, just what does it mean for two agents to be loyal to each other? What does it mean for an agent to have honor? What does it mean for agents to respect each other?

I think those questions do have answers. They're not like — I think the, I don't know, the Orthodox Yudkowskian sort of view would be that those concepts don't mean anything outside of the context of the tiny placid island of ignorance in the Lovecraftian sea of horrors, right? Like, we have our little humanoid slice of mindspace, and then the vast majority of possible minds are totally alien to all that. I don't think that's true. I think there's quite a lot of natural content — these moral concepts we associate with humanity are in fact natural concepts associated with agency and intelligence as such. Yeah.

Historically, I mean, let's just look historically, right? So they do often turn into caste systems, especially when the nature of power is such that you can get a very small number of actors to have power over everyone else, but nonetheless still find use for everyone else, right? It's like slavery is effectively just a relation of various kinds of actual inequality, like responding to various kinds of actual inequality. And which isn't to say, oh, we're just treating the people different, it's like well, the people are different, or they have vastly different amounts of power due to the technological conditions or whatever.

And so you end up with this situation where it's like, well, I can extract labor from the lower classes, and I'm going to keep myself insulated in my bubble and use my power to extract the labor from the lower classes, enrich myself. And because I don't really share any kind of deeper social relation with the lower classes, I actually don't care who they are or their quality as neighbors or whatever. So I can just bring in whatever foreign slaves or conquer these people who are already foreign, and I go and install myself. And that seems to be the sort of thing that definitely happens in history. It tends not to go well in the long run for the society of the masters, but it's kind of this empirical condition.

And then the thing that makes it go the other way is sometimes the technological conditions shift the other way. Napoleon, and modern mass production of rifles and other kinds of guns — I don't think it was rifles quite yet, but modern mass production of guns makes possible this mass army of loyal people, who are formerly the peasants. Now they become equal — in some sense, more equal citizens within a political order that's based on the ability to mobilize mass numbers of people in some kind of army or shared political act.

And that gives rise to various kinds of social egalitarianism. It's like, okay, well now because the power is shared much more widely, you have to have relationships that are much more egalitarian in a way, and it means that marriage taboos are going to break down, you're going to have people marrying across class lines, you're going to have people sort of having various kinds of society across class lines.

And so the question is, first of all, which kind of future do we think we're going into, and how do you respond to that appropriately? Like there's how people will naturally respond to that, and then there's, is there a wise way that we'll respond to that? I guess—

Curious if you would echo these thoughts. It looks like we're going into a vastly even more unequal future with the nature of technology. Now, that may be somewhat complicated by what actually ends up happening with these bureaucratic systems that we have that kind of underpin the property rights that are the kind of basis of the inequality.

It's not entirely clear to me that things are as anti-democratic as people seem to think in the future. Or undemocratic would really be the right word. I think everyone expects that, "Oh, the state can just suppress everyone and impose its will, and democratic power doesn't mean anything," which is to say the power of actual groups of people who can coordinate with each other who aren't the powerful elites. But I'm not sure about that. But it does look like the certainly default null hypothesis is that we're going into a vastly more unequal future.

And then, yeah, so let's start with that. What do you think about that idea? Are we in fact facing a future of massively more inequality?

The money is not the bottleneck, though. It's the license to coordinate that's granted by the New York Times and the UN, right? It's like because the people tried various things like this in the 20th century and there was UN boots on the ground within 20, 72 hours of declaring sovereignty for some section if they don't like the project.

Yeah. Well, it's not just ideological. It's about the conditions of power, right? It's like if there is a massive international power structure that's able to prevent such things, someone will come up with ideology to justify its use. Now, the ideology does shape what kinds of use it gets, but—

Sure.

Yeah, well, if you look at let's say the development of Chinese cities now as the sort of proxy on how long it takes to stand up a new city if you have the capital and the political license to do so and the coordination and so on, they happen pretty quickly.

But I would also expect that to kind of go the other way. Like as technology makes more things possible, it doesn't necessarily just make bringing up the bottom possible, it also pulls the top away, which means that it might be—

Well, I'm not saying San Francisco's going to be any different. I'm saying that in whatever places actually do manage to take advantage of the new technology, those places are going to be so compelling that it's sort of comparatively harder from a lifestyle perspective to leave them than it is now. Like there's sort of this — let's say, you can get very varying kinds of technological and social conditions that determine the comparison between going off and living in a self-reliant way with you and 5,000 of your closest friends in the wilderness versus toughing it out in a big city, or thriving in the big city depending on the historical period. This sort of goes up and down based on technological and social conditions similar to the power of the elites versus the masses.

Yeah, I'm skeptical because I think again the baseline increasing power of centralization and inequality as a result of scaling of technology is going to be sort of both source and consequence of what you're saying in terms of technological improvements or how easy it is to do things, like build a city, but that also disempow— or relatively empowers the large central power structure that might be trying to suppress you and react to you.

Like, okay, you've got this fast new AI technology, you can move faster than any human institution. Okay, well, so can they, right? But they have a bigger budget, and now they can hunt you down with, you know, Stasi Claude that — you have your personal Stasi agent AI that's monitoring you for every little thought out of alignment.

Yeah, that's maybe the other thing that we should predict about the future is it certainly looks like the current crop of institutions are in their decay cycle where they become more and more apathetic and unable to coordinate. But then that just means, okay, you get some new generation of institutions like, "Oh, well maybe the Chinese spies actually are going to become more powerful in America than we'd hope, and the Mexican cartels, and the various ethnic lobbies," right? That take the space that the state leaves behind.

Yeah, well, but that's—

Yeah, I mean the problem is when it's — to speak specifically with respect to Western people like us, our power socially is very caught up in the power of the existing state. And this was the thing we created to be the vehicle of our power, and as part of that bargain, it has power over us disproportionately. And as it goes into decline, a lot of the way it goes into decline is like autoimmune disorder, going to war with its own people in various ways, or—

just finding that it has less power, and so the only power it has is over its own people now. And so it becomes the instrument of our own oppression, whereas other people who are not historically bound up in it don't have that same relationship to it.

And so it's like, yeah, you have the declining state and that means room for new things. That doesn't mean we become free. It means we might just become even more oppressed by that system as it declines because we have that historical relationship to it. And so that's maybe the scenario I fear, right? It's like, how do we get out from under the state is kind of this crucial question.

And Nietzsche talks about this. We've been reading Zarathustra. He says basically where the state ends, that's where the bridge to the overman begins. And there's this essential need to get out from under the state because it has this last-man-ifying effect on you.

Yeah, we don't know how those are going to interact. That is, I guess, yeah, this definitely feels like one of the crucial future historical problems that we are faced with is, okay, we have the AI situation. AI is rapidly accelerating. There's all sorts of things going on there. The technology is growing. It certainly looks like we're going to get more of that.

And then on the other hand, we have this decaying Western nation concept, or the Western nation-state is kind of decaying and you're getting its possible morph into something else that's scarier or less scary, or gets taken apart for parts by various ethnic lobbies, or becomes some other version of itself, or whatever. And how those pictures interact is the essential question, right?

It's like are we going to see, like you say, they add up? Like the one way they could add up is the state is getting worse and it's getting more powerful, right? Because it picks up the ability to use Claude-accelerated Palantir to chase you down for every little opinion that you have, versus maybe the state just falls apart and we're able to use these technologies to kind of escape its power. I guess I'm more pessimistic than many people on this front.

No, likewise I think that's why I was adding kind of caveats earlier when we were talking about how democratic is the future actually going to be. Like I think there is this sense in which the, let's call it the broad extended elite consensus has gone way more undemocratic than might actually be warranted relative to the actual balance of power.

Yeah, and the question is—

The question is what kind of search process is going to create that ideological possibility, and what does the existence of that search process also create?

…it.

Yeah, well let's hear about it, because that is definitely of interest.

But didn't that — that paradigm breaks as soon as it encounters game theory, as soon as it encounters other agents?

So it's been known to be broken for a long time, even though we still use it.

Yeah. And so how do you move beyond that? Because this does feel like the essential thing. Like you run into this game theory stuff, you notice that at best you can — or one of the things I've noticed as you think through game theory and, I don't know, ethics in response to game theory and the kinds of decision theories that come out of that sort of stuff within the rationalist discourse and elsewhere, you end up reinventing a lot of concepts that sort of had traditional wisdom around them. Like the concept of honor I mentioned earlier, right?

It's like something that you can actually give something of a game theoretic foundation to. It's like, well, I'm going to insist on certain types of treatment, or certain types of commitment to my own word and so on, not because in some naive utility maximizing way that maximizes my utility, but because the kinds of interactions that it incentivizes and makes possible with other agents is a much better set. And so that's one way that you end up kind of recovering a bunch of interesting virtue ethics kind of philosophy out of things like game theory. But I'm curious where you would take going beyond game theory, or getting those insights, or how you would structure that kind of thinking.

It seems to be something, I mean historically speaking, it seems to be something that comes into being as a social equilibrium, rather than as an individual agent unilaterally doing it.

Well, how much of that is just that a lot of these things kind of inherently have to come from outside of the agency of the agents involved? And it's like there's either some founder, lawgiver figure or some kind of divine intervention of some kind, where the thing gets into the good equilibrium or the bad equilibrium not for any reason that has to do with the agency of the agents involved. And cuz the challenge suggested by that is that, well, if they could do it by their own agency, they can also kind of undo it by their own agency, or subvert it by their own agency.

Well, why would they? But if they have the power to change the equilibrium, then maybe one of the agents has an incen— like now you're playing a different game, right? It's no longer the original game. You're playing a game where at least one of the agents has the power to change the game. And in a way that potentially upsets that. And now you have a whole new set of problems and equilibria.

And so whenever you introduce the power onto the playing board of like the ability to change the thing or the ability to resolve these kinds of coordination problems, that is itself this overpowered thing that changes the game. And so my hunch is that as you run into that problem, at some point, you just have to sort of recognize this kind of from outside quality of our ability to get into these things.

But someone set that up in the first place, and they had—

they had some kind of large power.

I would say that, well, here's a claim: the functional ones come out of someone designing it, and the ones that just grow are often quite dysfunctional, or they're sort of one of the bad equilibria or the middling equilibria.

I think there is probably some kind of authority figure that would hold that together or something like that. Like—

There can be good equilibria that come out of the sort of natural process where you throw people into the situation and they'll find their way into the good equilibrium because it's just sort of rational and it's not that complicated of a problem. But I think historically when you're looking at the laws of a society, or the rules under which everyone's coordinating and the norms, they tend to be founded by particular political events where someone got a huge amount of power and reshaped things and built that.

And that's what I mean by this process of revelation, right? As far as anyone who's living under that is concerned, that person might as well be divinely inspired, right? They come in as this founder figure who lays down the laws and then everyone else is just kind of stuck in them.

Yeah, they can't do it against the will of everybody. Right, they do it because they've accrued the social authority where everyone accepts their authority, on whatever basis.

I see. And so that's the process that you're kind of interested in. I was trying to introduce that, but there is that crux figure who has to be the lightning rod of the equilibrium. And how do they end up in that position? What does it mean that that kind of person can exist? Like it can't be that anybody can just do that willy-nilly. But they do exist historically. That is where a lot of these things get figured out. But they end up in those positions of, I don't know, folk legitimacy because they credibly demonstrate their ability to do it well, and a goodwill to do it well.

Sure, there's many. There's many. Yeah, I mean we talked about that the other night in Jonathan's talk. But so what—

What kinds of theory are you working on that helps us better understand that process?

Yeah, which is stupid idea.

Yeah. Or better yet, how valuable it is for the overall economy to have that company or something.

Like what that would be — that's sort of what I imagine as the non-marginalist economics, is an economics where we can take the various parts and say, "Here's what this thing is doing, and its inputs and outputs, and why is that valuable? And what are its capital stocks and what are its liabilities?" in a way that is not as much about the buying and selling of the ownership shares, but about the underlying activity that it's doing.

Mhm. Do you think it's possible to have kind of crisp concepts there, or is this inherently a question of, broadly speaking, the humanities, where inherently it's kind of qualitative reasoning that is very hard to pin down?

Yeah, you can't easily measure the relative fitness.

Yeah, and do you think that relates to some of our earlier discussions about the question of interactions between humans and AI, or coalition dynamics one way or the other on those kinds of questions? Like do you think that with such theory, we could achieve better coordination against slippery slope arms races, or would maybe that would just give us a crisp reason to understand why we can't, or—

Right. Interesting. Yeah.

Yeah. Well, this sounds a lot like being stuck in a bad equilibrium in game theory, right? It's like in the prisoner's dilemma, it's like, "Oh, the other guy's going to defect. So I have to defect." Or at least the other guy's decision is independent of mine, so I should just defect.

Right?

That there were no such people.

Yeah, this is a very interesting topic actually. One of the things that I observed was that the arms race with China pre-existed any substantial results in China.

Like — and the arm— by which I mean the arms race in China, like the meme of the arms race in China, pre-existed any apparent attempts by the Chinese to dominate in AI or even to compete in AI. And so yeah, the Western labs were using, "Oh, we have to beat China," as a justification for competing among themselves in as uncoordinated way.

Yeah, and so right, this kind of self-destructive anticoordination is a very interesting phenomenon. And I think this is maybe getting to another interesting crux is — I'll ask, why do you think that it is possible to come to a crisper theory that would be able to avoid this? Like what evidence do we have that that's possible versus the alternative, which is that the crisper theory just formalizes why this always happens under any set — under any kind of — like maybe it's not just marginalist economics, right? It might be that this vulnerability exists, or some analogy to this vulnerability exists, within any sort of worldview of coordination.

Yeah, well, there's this really funny result in surveying various professions on game theory, and the economists do the worst, because they have this defect mindset.

Yeah. No, and I think that's true. Like certainly our society has been much more broadly than just in AI, torn apart by the sort of economic justification for looting, broadly speaking. Like there's just all these ways that the economics profession amounts to just apologia for various kinds of looting. And defection. And so that I very much take seriously. I guess where I start to become interested is—

…is like yeah, we could have a better basis for coordination. But can we have crisp theory that actually avoids this failure mode or some analogy to it? Or is this — is the existence of economics in this format the result of some kind of latent demand within the space of possible coordination for this kind of corruption, right? Like is it just that you always have corruption and then corruption comes up with ideology for itself? And then to propagate itself.

And, or — I think you can probably have healthier and less healthy versions and it's like, okay, maybe we're sort of further in the direction of being unhealthy on some of these dimensions relative to maybe where we were previously or other possible societies. But I guess I'm very skeptical that any of these problems will ever be permanently solved by better theory. I think at best you get a temporary sort of, oh, we all see things in a less corrupt way for a while and then someone figures out how to corrupt it. But—

but I'm curious what you make of that, like that sort of line of argument.

Well, it's a bet — it's certainly a bet against like final victory in—

in coordination. But I wouldn't say it's a bet against intellectual progress, cuz you could just as easily flip that around. It's like are we really going to bet against the intellect of the corrupters? Like they are also engaged in much progress, right?

Well, they massively improve our capabilities in some area, but that's different from massively improving our coordination. Or the possibilities of coordination, apart from there's technical means where, oh, coordination improves because now we can structure things in this new way where it's not reliant on this kind of relationship and instead we can lean on that kind of relationship. But that's different from better theory solving coordination in particular. I guess my view is that coordination in particular, because it's an adversarial game — it's the good equilibrium versus the corruption, right? And because it's an adversarial game, there's some way that the progress is symmetric. And—

You think so? Why?

Describe even the corruption process, you could — yeah, sure, we could describe even the corruption process perfectly, but what does that make it easier or harder to be corrupt? I don't know.

Well, okay, my claim would be that you can close off certain avenues, but it inherently kind of opens up other avenues. And we find it very difficult to imagine the sort of possibilities, because to some extent these things are sort of happening outside of our ability to theorize them. Like it's only in retrospect that you can diagnose the cancer once you've really seen the result and seen how it worked and so on. But at the time, there was sort of like no preexisting mechanism that could have recognized that that was cancer, because it occupied the blind spots of the system, and there were blind spots of the system, and sometimes you can't even tell that there's blind spots of the system. But my claim is there's always going to be blind spots.

I don't know.

Yeah. Well, I think part of the problem is that your epistemic environment — you can't assume an uncorrupted epistemic environment when you're dealing with these things, because the corrupters are going to be trying to mess with your epistemic environment and therefore the basis of your knowledge of these things. So as soon as you're trying to say, okay, we have theory that allows us to solve these problems, well, that's equivalent to saying there's some theory that could be turned the other way as well, right? Like if someone got control of that epistemic process or corrupted the epistemic process in some way where now it's not—

And this is almost what has happened in our society, is we have all these ideas of what it means to prevent corruption and prevent discoordination, like achieve coordination in society, and we have all these checks and balances. And those checks and balances now, largely speaking, exist to facilitate corruption.

Yeah, and so like how do you — I think the cancer is as smart as you are, right, is kind of the basic problem.

It's dumb, but it's powerful, right? And it's done by smart people. It's—

It's perpetrated by smart people.

No, I don't think it's a mind — I don't think it's a mind virus. Like it's—

it's that there's people with interests that diverge, right? Like someone finds a way to get more out of the system for themselves by making things a little worse for everyone else, and they decide to do that. Maybe they see it clearly, maybe they don't. But in deciding to do that, now they have more power and therefore more power towards the type of thinking that creates that kind of problem.

Right? And so as soon as you have this sort of — never mind the rational agents and the equilibrium, it's just that there's this kind of flywheel effect of the defector gets more power, and then now you're closer to an equilibrium of defection. And I think that this seems to be universal to life. Now, maybe it's because we just haven't discovered the true singleton operating system yet.

No, we're not, because the cancer systems die and are replaced. There's another asymmetry on the other end, which is that once things are sufficiently decayed, it's no longer able to enforce the decay, right? And so life is able to circumvent and come back. And so there's this cycle between growth and decay, where when the growth has happened for a while you've got a lot — a bunch of resources laying around that have been coordinated, like accumulated in coordination, there's suddenly this huge opportunity for subverting that coordination. And that can be a self-compounding process. But, on the other hand, once you've had self-compounding corruption for a while, now you're back in the Dark Age, now the only thing to do, the only way forward is to create virtue and to create life.

And so you get this balance between them, but you never get like a victory of one or the other, because I think it's like we would like things to be nice and coordinated and honest, because we're engaged in the coordination and the honesty. But that doesn't mean — and therefore we're going to apply our intelligence to solving things that way. But that doesn't mean that someone else applying their intelligence doesn't have other interests, right? That they actually get more out of corrupting the system in a particular way.

And so it's not quite so simple as to say intellectual progress, because the intellectual progress exists within the institutions under question and are influenced by the ways that corrupters can subvert these ideologies. Like I would even argue that maybe this is what happened with economics. Like why is economics so bad for coordination? It's because people figured out, oh, we can come up with a sort of intellectual sounding justification for corruption.

And maybe economics was actually a lot healthier before that happened.

I kind of think it was. And like, okay, let's close off that avenue with economics. That's certainly a worthy project. But I think that'll be temporary, even if you succeeded. And this comes back to the question of the singleton, right? This is why I think that singletons don't work. I don't think there actually is a final solution to corruption.

There's always more blind spots somewhere, and that, because it's an adversarial game where you can't assume that all the intelligence is on your side, you get these self-compounding corruption effects once you have a big, ripe utopian society where everything's coordinated. At that point there's this huge bounty for anyone who can corrupt it and crack it open from the inside. And so you get this feedback balance between growth and decay. And maybe this should be a theorem in your new science of coordination, but—

Yes.

Yes.

Right.

Yeah. I would say with respect to coordination paradigms and ideology and the social basis for coordination and so on, the next paradigm is usually better than the previous paradigm, but they get worse over time. And so — I don't mean the absolute progression is worse. I mean you establish a paradigm, it works for a while, but then it decays as people find more ways to subvert it.

And the result of this, anyways, to bring it all the way back to the beginning, the result of this is I think that, you know, very skeptical of things like, again, the coordination of humanity as humanity against these highly valuable technologies or potential political allies that could crack open that coordination. And so I expect that we're going to go into that world. And now, this, again, maybe the traitorous kind of reasoning that you're—

you're attacking as marginalist economics. And I recognize that that is a very valid criticism. I felt like that before, but just on reflection, I think I don't think it's true that we can actually occupy that equilibrium. And so it's something that I'm more interested in exploring, to say like how then do we theorize the not necessarily arms race,

…equilibrium, but the competitive equilibrium where there isn't any kind of global coordination. What is that actually going to look like and what kind of ethics comes out of that?

Yeah, or they're not about the competition. They're not about trying to win the competition.

No, I completely agree with that diagnosis of the concept of Moloch. Let me throw you something maybe strange here, which is that I think what he described in that process is not Moloch, but God. And the result of taking that definition of what is effectively God, which is, if you take a Landian view on theology, which is that—

Well, I do—

The basically the idea that — well, I can answer that question, I guess. I think good things exist in the world. I think the good things were created by competitive processes. And in fact, that's where the good things come from. Broadly speaking, the creative value in the world, the creation of new value and new good things comes out of competitive processes—

And cooperative processes, mostly competitive processes, in terms of—

I'm just saying in terms of the way that evolution has worked, the way that it's — it's which things work, right? Which things work is the crucial grounding. And that is where the good things come out of.

And so what Scott tried to do is take a sort of appreciation for how you can get this defection equilibrium, and say that that was the source — he generalized that way too far and generalized it to the point of basically denouncing competition as the source of good, broadly speaking, and made all competition into a form of treason against this sort of Edenic equilibrium that I think doesn't exist.

And in doing so, I think he is effectively engaged in some kind of rebellion against God, because he's in fact defining himself against the creative process that has created all this goodness. And what—

what ends up happening when you do that is that you get that kind of — I think you're right, that that worldview does summon into existence the kind of purely we must win the competition for its own sake kind of escalation that is sort of treasonous in the way that he described. Like I think there's this deep irony in that.

And this is one of the reasons that I try to attack the basis of that particular competition, because I don't think they're fighting over anything real. Like I don't think any of them are actually going to get control of the future light cone. I don't think there is any such thing as getting control of the future light cone. I don't think there's any such thing as astronomical stakes in terms of whether the future is going to be happy and utopian or horrible. I don't think that those are actually the possibilities on the table. And I think if people took those kinds of possibilities less seriously, they would also engage less in that kind of purely, let's call it Molochian competition, and more in the kind of competition that I think is much more valuable, which is let's pursue constructive projects even against other people, but for our own ends that are not about the competition.

And I think that's been kind of my angle on that discussion. Like I think the AI competition would be a lot healthier for everyone involved if they were pursuing their own ends rather than just purely the competitive end of winning.

Well, this is where I tried to — I did try to respond to Land's worldview, or the naive Landian kind of accelerationism on that front, the nihilistic accelerationism. I've sort of — I found that Pierre Teilhard de Chardin is a good antidote there. It's like a very similar worldview, but putting on the optimism glasses in a way. It's like, "Oh, this is actually beautiful and good and this is what creates good things. Let's actually take that seriously, and also love is the basis for all order in the universe and love is what's ultimately being created by this process." And that's also true. And so Teilhard de Chardin's The Phenomenon of Man is one good antidote.

Another good antidote is recognizing the importance of agency in the thing. Cuz again, I've mentioned a few times, in the naive nihilistic Landian worldview, there's a discounting of agency, the sense that we don't have control over the future, we're just on rails, right?

Yeah, but I'm saying there are stakes. Like we do have domains of agency that we should act within, and we should try to make what happens in those domains of agency as good as we can.

But I think one of the things that creates really bad equilibria and causes you to destroy the things in your domain of agency is to think that there's some bigger value outside of your domain of agency that you're going to try to seize, right? You sacrifice your actual domain of agency for the sake of an imaginary utopian endpoint. And that's what's going on with the AI lab competition, right? It's like they think they're going to get control of the future light cone and have astronomical stakes, which means that any amount of destruction of our actual domain of agency is justified on the utility calculus there.

And so I think that you have to be clear about what is your actual domain of agency and treat it as your beautiful garden, right? It's like, how do I make my actual domain of agency good? And this is where I've tried to provide the antidote to the problems of the Landian worldview.

Well, sometimes it is and sometimes it isn't. Like cuz sometimes you think it's illusory, but that's actually an illusion itself. And it depends on your own ambition, that—

The universe is up for grabs, but not by any coherent—

coordinating thing. No, I — well, this is the thing, this is the hard challenge, right? It's like to tell the difference between delusion and ambition. And I think it is epistemically very difficult. But I think on reflection, the light cone-eating singleton kind of idea is on the delusion end, not on the ambition end. I think that's not actually a real thing and I think that believing in it causes these sort of terrible coordination outcomes.

But not so much arbitrarily. Like the thing I—

Yeah, I think what I expect is that yes, life will continue to unfold and eat the universe and so on, and things will change on the margin depending on what we do at crux times, but not so much and not in any predictable way. Like you might get stuff like QWERTY, right, where maybe QWERTY is just locked in for all time, I don't know, but I think fingers are probably going to go away at some point, and then QWERTY will no longer be a thing. But things like that, right? Unix maybe is locked in for all time or whatever, right? English language maybe locked in for all time. But there's these things, but they tend not to matter, right? They're like these arbitrary little technical details that aren't the important things.

I think what you don't get is massive unbounded loss over the far future because it's just not really possible to maintain the kind of order that's going to — on that time scale or on that distance. And if it's not, then we can kind of dispense with those sorts of mythologies.

The only question, as far as I'm concerned, is do we sort of make it through the great filter or not? Or do we knock a big chunk off our potential by slowing ourselves down for another thousand-year dark age or something? Like those kinds of things suck. Let's not do that. But I guess I put it as a quip, like, if it keeps going, we win, right? That's sort of the best, I think the best far future outcome that we actually have control over, is whether we get anything there at all versus just kill ourselves in some way, which death does appear to be a possibility. You know, we could in fact wipe out all life on Earth if we did totally the wrong thing.

But yeah, I think it's just a way more realistic and therefore healthier view to say, "Okay, there are limits to coordination. There are fundamental limits to coordination. Therefore, the kind of astronomical stakes arguments don't work. Therefore, we should be focused on more immediate domains of capability, of agency, and what we would like to see in those domains of agency for their own sake and not for the sake of their future impact." And I think that is the only domain of value actually accessible to us, and if we sort of competed over that, I think we'd get much better outcomes in that sphere of value, which I think is the only one.

that you can drive — there's an assumption that you can drive coordination error down to zero. And if you can drive coordination error down to zero, then you get this huge infinite result—

No, but I'm not — I mean, historically where I came on to this kind of thinking was I was very much in the, let's say, orthodox Yudkowskian camp. I wrote a blog post called Capturing Gnon,

which was a sort of view on — it was like a Landian interpretation of Yudkowsky's project. And that blog post directly inspired Meditations on Moloch. If you read Meditations on Moloch, one of the things Scott cites that post that I wrote.

And—

and I chewed on this stuff for like 10 years of believing that there were astronomical stakes and that therefore, various things. And because of that overwhelming importance, you have to focus on how do we have to get there, how solid is this actually, and what does this actually look like, what possibilities are there, and all this. And eventually, I just realized that the whole thing had kind of divided by zero. Like, it was not actually a correct assessment of the kind of agency we had over the future, and I credited this to Nick Land, right? I got the antidotes from Nick Land, and I've tried to overcome the problems in that worldview as well, but—

So in terms of my historical evolution as a thinker, yeah, it's coming to believe that this sort of set of possibilities was impossible. And then I had to kind of go and update my whole value, cosmology, and so on as a result of that.

Now, it might be that after you get through all this, you can come up with a much better framing of it. Like, I agree, probably there's a much better framing that doesn't come as a result of explicitly, oh, that's closed off to us, so it drops out as a term, right? I think that's my response to the Yudkowskian view simply because I developed that idea as a response to the Yudkowskian view, and we're talking about these Yudkowskian AI labs, right?

But in terms of my actual worldview, how I would justify it more directly, I think there's probably a much healthier way to do that. We should probably talk about that sometime.

I think that's another discussion. But I think there is an optimistic, directly stated view of what kinds of agency are available to us, and why that's good, why it is that way, and what the good things we can do with that are. I think that you can just frame it all entirely positively. And I think that's a useful project. It's just not something I emphasize in response to the Yudkowskian stuff, because it's not as directly—

applicable to that. And maybe it's also more incomplete, right?

Yeah. Well, I think—

Maybe. I mean, these things always need more work. I guess the one big crux question that does matter is, does the integral blow up, or is the integral finite in terms of that long tail of future impact possibility?

Well, that's — maybe that's a way of saying the integral is finite, but—

No, because it's like you have this problem of — maybe this is, again, the problem with the expected utility framework, right? But if—

if you take anything like an expected utility framework, then if there's the possibility of an infinite utility, or a dominating utility,

that has non-vanishing possibility, then it blows up your integral, right?

Yeah, so let's get out of that framework. I don't really like that framework, either. But it's for reasons of tending to blow up in that way, some of which I think are fake, like this one, and some of which maybe are real, in the sense that if you define your utility function like that, it blows up, you know. But I guess that is the sort of — it sounds like, then, you're on my side of that problem, which is so let's not act as if the integral blows up.

Cause that's the mistake that I think the frontier AI labs are making, so they think the integral blows up.

Well, that sounds like fun. We should have that conversation. Yeah, I think we're probably out of time, though.

But, yeah, I mean, this responding to utilitarianism and providing a coherent alternative to utilitarianism does sound like a good project as well that we should do.

All right. Well, this was a lot of fun.

We wandered all over the place, but I think there's a coherent thread.

All right. Well, thanks a lot. Until next time.