You might have seen that we've released a new podcast season, Silicon World Order – and yes, it is about AI. 

Our first interview episode is with Henry Farrell, a professor at Johns Hopkins who's written extensively about AI as a new kind of social technology which, like markets or bureacracies, serve as a kind of "lossy JPG" representation of complex societies. This is a very different to, say, the scifi-infused visions of singularity/AGI/p(doom). "Those representations both depend on and go beyond the knowledge and decisions of individual people". He's also, with Abe Newman, written about weaponized interdependence – in which centralized architectures can be used for coercive ends (the SWIFT banking system used by the US, for example).

The full transcript is below. In his conversation with Tim, Henry talks about:

Full episode on Apple, Spotify, and everywhere. And below, on Youtube:

Transcript:

HENRY FARRELL: And here the idea is that the scientist understands what the problem is, but the public also understands the consequence of the problem in the way that scientist does not.

TIM SAHAY: Welcome to the Polycrisis, season two. We're looking at artificial intelligence.

So this is how we are looking at it. The AI boom is not the investment boom we wanted, but it's the one we got. So how can we make it work for us better? AI has big implications for energy use and energy generation, which are obviously critical topics and a central part of how we at the Poly Crisis look at the world. I'm Tim Sahay, I'm at Johns Hopkins University in the US where I co-direct the Net Zero Industrial Policy Lab.

KATE MACKENZIE: And I'm Kate Mackenzie, a Sydney-based energy and climate finance expert. AI is daunting, so what we're doing in this season is breaking it down into six lenses to find ways to understand this disruption. We're bringing what we know about other areas to this topic. So we'll look at AI as a geopolitical battleground, as a climate killer and an enabler of horrifying new kinds of weaponry, and as a potential boon or barrier for economic development.

And in this episode, The Seven AI Kings, we look at the political economy of AI. 

TIM SAHAY: So I spoke with Henry Farrell. Henry's a professor working on democracy and international affairs at Johns Hopkins. Henry has written books on the politics of technology, including weaponized interdependence and underground empire. A central argument he's made with colleagues is that large AI models should be seen as new social and cultural technologies, rather than intelligent agents. Henry is critical of the way 20th century science fiction dystopian fantasies continue to frame much discussion about where AI is taking us in the near future. We speak about the power and politics of the tech AI kings. And what is the effect of this industry on the economy? And if it's an industrial revolution, what does it mean that funding for social sciences is being stripped back? Okay, welcome Henry. We've read a lot about your scholarship on the political economy of AI, and we're really delighted to have you on our podcast.

HENRY FARRELL: And I'm delighted to be here. Thank you so much.

TIM SAHAY: So let's let's sort of jump in. We wanted to start by asking you to tell us about how AI is currently being shaped by the dominant sort of Silicon Valley politics inside the US. We've had um sort of what I've been calling as the seven tech kings from Elon Musk to Zuckerberg's Facebook to Bezos's Amazon, Sam Altman's OpenAI, Eric Schmidt's Google. And, these companies are pouring in hundreds and hundreds of billions of dollars in the development of this new technology and also backed Donald Trump in 2025. So what are the politics of these seven sort of giant AI companies and how they are financing, directing, and shaping this technology? And I'm thinking of your essay um, I think in in Democracy Journal called Reigning in Our Tech God Emperors. Um, so how can how can we actually think about their political project?

HENRY FARRELL: So I think that they're, it's like all political projects, it's much messier and more contingent once you begin to get at the details, but I think you can also look at the big structural factors that are shaping it. And I think you could look at three factors in particular that I think are important. One is the amount of money that, as you mentioned, goes into actually training these models. So the current dominant paradigm, which may or may not turn out to be right, is that the important stuff is happening at the frontier, quote unquote, which means that you have to pour in absolutely massive amounts of money in order to train the models and in order to um sort of keep going. And so we've seen, of course, over the last little while, we've seen willingness on the part of AI companies to contemplate some kind of a slowdown or pause. And that may also reflect the fact that they are just putting ridiculous amounts of money into this, and they are caught in a very difficult competitive race against each other. So I think that's factor number one. Factor number two is I think what you might call imagination or imaginations of the future. So we have here this very complicated mix of different and sometimes incredibly grandiose and wild science fictional visions of the future, which are driving a lot of the debate around this. If you look at people from the labs, very often they talk about us whizzing around the solar system and perhaps the nearer parts of the galaxy in a few decades, once all of the AI awesome has delivered. And this also goes into things like the SpaceX IPO, which was much, much more about AI in important ways. If you look at the actual financials that they predict than space, but it also suggests that once we have gotten along the road, we're going to be able to get to the point where we are a Kardashev scale civilization, which is effectively a civilization that is capable of capturing all of the output from the sun. So this is the first time I think you've seen an IPO from a major company that really re refers to, again, 1960s science fiction and people like Freeman Dyson, who used to talk about Dyson spheres and Dyson swarms and similar kinds of ideas. And the final thing that I think is really important here is that all of this is based very physically. And here I think this also goes into some of the strange financing arrangements that we have seen. Um sort of we don't have nearly as much data as we would like on who is financing who, who is investing where, how exactly the loans are happening. A lot of the activities do seem to be happening off book. But what we do know is that a lot of this is being pushed by Nvidia because Nvidia's chips are really extraordinarily important, especially um sort of in training these models. So NVIDIA has a, I think a real vested interest in seeing this boom continue and continue. And also we are seeing, of course, of one part of the expense is training, the other part is inference, having these models actually run. And so that is involving this huge investment in data centers, which I'm sure you're dealing with with other interviewees in other parts of the series.

TIM SAHAY: Yeah, I mean, the whole question about capturing the imagination, capturing resources, this is what capitalists do. They don't just sell the present, but they sell a vision of the future. What what we've been sort of what we wanted to ask you next is what is this seems to be something very special about the missing public or the missing governance, the public governance of this new industry. Um, where, it's been sort of booming over the last five years, but after 2025 with Donald Trump's election, the United States government has just sort of walked away from any oversight and governance and has handed off this um CapEx boom and directing and shaping and regulating this technology to those frontier labs that you mentioned. So what was this the goal of the Tech Kings all along to get someone elected that that would look the other way?

HENRY FARRELL: So I think it's more complicated than that. And if you look at, for example, if you look at the people who are in control at the moment in the Trump administration, so these very often are people who are it linked with, for example, Mark Andreessen and the All In podcast dudes. These are very often people who are who saw themselves as being on the losing side in an internal battle within Silicon Valley. It's really worth looking at some of the interviews and the podcast stuff that Mark Andreessen and his partner Ben Horowitz and Dreessen Horowitz were doing in the lead up to the election. And they were complaining more or less that they felt that this was a system. More or less what they felt was happening was that the labs were dominating and that they were preventing the kinds of venture capital activity that they saw as being necessary to their own business model, that they're preventing this from happening. So I think that there's this very important internal fight that is happening as well, which is, which I think is a key part of the problem. And when you see people like Andreessen complaining, suggesting that the Biden administration was trying to regulate, what they felt was that the Biden administration was in cahoots with a number of people from the labs, especially anthropic, to a lesser extent open AI, and was trying to both to regulate this technology and also to do so in ways that would introduce a permanent monopoly for these actors, um sort of for these big companies, which Yosuke Andreessen and Horowitz felt were inimical to their interests. So I think, sort of a big part of this, I think, is, there's a battle within Silicon Valley. There's also, and I think this is an this is largely different from that, but not entirely, there is a there is a a set of ideas, rationalists and others who have very, very strong, as I mentioned, um sort of predictions about what is going to happen with the future of AI. And finally, um sort of I think that there has also been a degree of battle among the big incumbent powers. So that very clearly, if you read um sort of what Elon Musk was saying again back before the boom really began, the reason why Elon Musk was investing in open AI was because he believed back at the time when it was supposed to be a nonprofit venture, he believed that Google was otherwise going to dominate. And so there was, on the one hand, a, clearly some kind of political sense that the future of the world should not be in Google's hands. That was the formal justification for open AI being set up as a nonprofit. But also I would imagine that the, that Musk and other people, their business interests were also not necessarily 1,000 miles away from their heads when they were making these decisions.

What is the economic project here? So we've just talked about the political project. What is the economic project here? I mean, in other words, what is the effect of this industry going to be on the economy? We've heard sort of very tall claims that we are in a new industrial revolution that will change how businesses, jobs, and work itself is gonna be carried out in the future. Um, and, products will will be changed, maybe basic basic services like healthcare. So so they they have a very expansive vision of how AI is gonna drive change in all different sectors of the economy. So is this an industrial revolution? What are the effects on the economy?

Okay, so there are two different questions, I think, here. One is what is our project, and the other is what is the likely outcome of it? And I think those are two different things. So if you look at the project, I think the project is very straightforward., we are now not supposed to call artificial intelligence artificial intelligence according to Donald Trump. We're supposed to call it superintelligence. And I think that the fundamental goal of all of the major actors in this game is superintelligence as a service. That is that you could effectively have intelligence as being, in a certain sense, a kind of commodity that you can scale up, you can scale down, you can press a button, and in a certain sense, intelligence will flow to solve your problems in ways that are somewhat analogous to the ways in which you press a button and electricity comes to charge your, charge your car or turn on your light bulb or whatever. So I think that the fundamental notion here is that we are in a world where we're about to have a massive intelligence revolution, and that the people who can control that revolution, the people who are able to effectively be the people who decide when the taps open up and when the taps close, are going to be in an extraordinarily advantageous position to fundamentally reshape the world. yes, so you have Dario Amodeia, who is the head of Anthropic, talking about how we are going to be in a world where very soon it is plausible that people's lifespans are going to be dramatically extended, where we're going to defeat cancer and other diseases, where we're going to um discover all of these new things because we are able to unleash innovation, intelligence, discovery, all of these things sort of coming together. And so I think that there are two reasons to be skeptical of this. One is, I think, the reason which is laid out by two computer scientists, Kapoor and Narianin, and they have this very, I think, useful our argument which says, well, yes, okay, we can think about this as being a kind of industrial revolution. But if we look back to past industrial revolutions, we do know that it took very often decades, for people to actually figure out how to use electricity, that, reorganizing the economy, reorganizing factory floors so that you don't have some kind of a drive belt which is relying on water power or whatever, but you actually have sort of electricity, this takes a ton of work. So they suggest that any kind of transition that we see is going to be much slower and more halting than people suggest. And then I think there is the more fundamental question of is intelligence in this sense intelligence as a commodity, is this a useful way to actually think about how intelligence actually works? And here, and, this is probably plugging my own and my co-author's work, sort of Alison Gopnick, Cosmos Schleeze, James Evans, and I have a piece that we wrote last year about this. We think about this as being much less like a like a kind of general purpose intelligence and much more as a social and cultural technology, which suggests that there are going to be some things that it is going to be extremely good at doing, very often sort of piecing together existing knowledge in useful ways. And you can see in many ways how some of the famous math proofs that are being developed actually can be reasonably, I think, explained in this context. But there are going to be other things which is probably going to be quite bad at. And here again, Narian and Kapoor have some interesting work, which suggests that these models they are quite good at certain kinds of, inference, if you if you like. But when it comes to basic discovery, there isn't any evidence yet that they are particularly good at doing the kinds of open-ended innovation which human beings are capable of doing.

And what about just sort of intersecting with the power structures of societies? Hospitals using AI and insurance companies using AI to sort of make more money from claims, and hospitals using AI to sort of find ways um to charge and build people more. So, in other words, a new intelligence as a service doesn't just intersect with the flat society with all kinds of users, but with the with the highly sort of a power structure of society where some interests are going to be using intelligence as a service to maximize their business models while others may not. So is is is that part of the the push towards thinking of it as a social and cultural technology?

That absolutely is part of it. So, and this is work that Cosmos Schlitzi, who is a statistical physicist that Carnegie Mellon and I have done, where we argue, first of all, you should think about AI as a social technology. And secondly, that the best way to think about this is as the intersection between two different things, one of which is this concept from computer scientists talk a lot about abstractions, and physicists talk about coarse grainings. And I'm sort of, those are obviously somewhat jargonistic academic terms, but what they more or less mean is um sort of when we're dealing with complex phenomena, we can't actually see the complex phenomena themselves. So typically we work with particular simplifications of those phenomena in order to make them tractable so that we can sort of operate at scale. And the second thing we argue that is crucial is power. And so then if you think about this from the point of view of social sciences, one of the things that social scientists immediately begin to glom onto, and they don't necessarily have to use our language to do this, is they glom onto the fact that these systems do actually have important consequences for power relations. So that here I think one great example from before the AI era, from the 19th century, in fact, which is William Cronin has this fantastic book called Nature's Metropolis, which is about Chicago, and he talks about how the ways in which grain markets worked. And so, crudely speaking, and here I'm butchering his story probably, yes, it used to be that grain was assessed more or less load by load, bushel by bushel, in terms of its quality, and the buyer paid a certain amount for it based on their understanding of is this good grain for the mill, bad grain for the mill, or whatever. And, if you're trying to do things at scale, that's no way to run a business. So you came up with these crude categorizations that I'm saying, this was high quality grain, this was medium quality grain, or this was low quality grain. And so that sounds like a completely straightforward thing, which is probably good for the mills and probably good for the farmers, but then you have the people, the actors who are actually buying the grain, very often what they do is they're able to manipulate the manipulate these categorizations so that, say, for example, if you have a bunch of grain which is close to high quality, but it's not quite high quality, and they buy it as medium quality, and they buy a lot of high quality grain as well, they are then able to mix together the high quality grain with the medium quality grain to bring it to a point where it just about falls into the high high quality category and sell it for a lot more money. And so you get a lot of really pissed-off farmers in the Midwest figuring out that they are being rooked in some sense, but not exactly being able to articulate how this is. So if you think about how AI works, AI is more or less a machine for building and making categories at scale. We're likely to see a lot of that happening. And also, I think we're likely to see, in a certain sense, as you suggest, we're likely to see AI's capacity to provide plausible reasons, plausible explanations for a particular outcome. We're likely to see that sort of deployed as well. So, in some settings, you may expect to see that of advantaging the powerful. In other settings, it's perfectly possible that we will get a kind of a Red Queen's race happening between those who are providing elaborate justifications for why your claim is denied, and those are who are providing elaborate justifications in return for why the claim ought to be processed. And there is some evidence, for example, that some of the larger doctors groups and others are beginning to deploy AI against AI in ways which you sort of yes, perhaps are not as problematic in terms of power relations, but which absolutely generate a ton of waste and inefficiency in the system as everybody I'm sort of begins to power, use data centers to come up with elaborate, um sort of 50-page justifications for why the carbuncle on your toe ought or not be excised.

TIM SAHAY: That sort of brings us to that article that you wrote with your co-authors about what is the effect of a missing social science. So if it is in fact a large technological disruption on the scale of an industrial revolution, there seems to be precious little funding for social scientists to think about these um ways in which power relations and societies and economics are going to change. And what was the argument that you wanted to make in that piece?

HENRY FARRELL: Okay, so, and I'm going to give you a slightly more sophisticated version of the argument because of course we had a relatively low word count and a bunch of things to explain. But, if you look at social sciences, fundamentally the social sciences came into being – this is an argument we steal from Charles Tilly – they came into being in response to the shocks of the social revolution. You you have suddenly have these dramatic, of the industrial revolution, rather, you have these dramatic changes in the way that society works, economy works, politics works, and all of these social sciences come into being in rather messy ways to try and figure out different aspects of what in the name of God is going on. And so we suggest if we need social sciences to deal with this new industrial revolution shock, because whether you think it's good or whether it's bad, it is going to have some real transformative consequences. 

But the problem is that there is not the kind of resources that you need in order to figure things out. Because I think there are three things that are happening here. First of all, the National Science Foundation at the moment has zeroed out funding for the traditional social sciences, for political science, for sociology, for economics. If you have applied for an NSF grant for any research in these areas, you almost certainly have gotten nothing and you're not going to get anything, as best as we can tell. Secondly, it's very clear that where the NSF was going under this guy called Kratsios, who is the current guy who is in charge of thinking about science policy more generally, it's very clear that he and people around him hope that we are in a world where you can effectively rely upon AI to do a lot of things that people have been required to do in the past. And so if you want to get funding, you probably want to have some kind of a – we are developing AI, people in the cognitive sciences who are thinking in AI-related terms, people who are looking for AI to try and build local economies – which is not entirely a bad thing. Those people are getting funding. But the final thing that that we were focusing on is that the people who are giving vast amounts of money, comparatively speaking, are the AI labs, OpenAI and Anthropic, each have funds with a couple of hundred million dollars for people who are interested in providing social science, crudely speaking, on how to ease the AI transition. And I would not interpret this as being entirely cynical. I think that as in all things, there is a certain amount of politics involved, so they clearly, if you're in the AI labs and you're looking at the hatred that people have for data centers at the moment, you probably are thinking that this is something you want to do something about. But I think people, there are people in labs who are also genuinely trying to figure out sincerely how do you actually ease this transition into a world that they believe will be better for everybody. But the fundamental thing that is clear from the funding proposals, the proposal that they have, is that if you are questioning whether the transition ought to take place, they're probably not going to be interested in giving you money. 

So the short version of that is that we see a massive collapse in traditional forms of funding for the social sciences, which means that social science, which is supposed to be in some version of the public interest, is going to be very, very hard to get money for, unless you think that the public interest and the interests of making sure that we have as successful an AI transition as possible, if you think that those two interests are identical, you probably can find a lot of money if you've got a good project that can justify it. And so this means that you end up perhaps having a spectacularly one-sided debate as time goes on.

TIM SAHAY: That sort of brings us to the question of what do you propose as a way to understand AI from the social sciences? And again, what is what is sort of the most accessible way for society to pay a role in shaping this new technology and have some sort of a shared conversation, which is currently, as you point out, completely missing in the United States, um, given the dominance of the big tech companies and shaping that conversation.

HENRY FARRELL: Okay, well, and I should stress, I, if you're looking for detailed policy proposals, I am an abstract hand waver like many academics, and I hope like, some some academics mistake themselves for having brilliant policy ideas. So, we do have some suggest where we say, well, if the labs really want to be taken seriously, perhaps they sort of hand this over to a set of independent actors who have a broader understanding of the public interest to decide how the money goes. And sort of, we will see if anything like that happens., you're so, who know, who knows?, and we also say that you actually do want to have public funding for the social sciences. This is a reason why Congress ought to be pushing at that. But I think there's a broader problem here, and I think that the problem is almost, in a certain sense, a Dewey-an problem. So John Dewey, in his fantastic book from the early 20th century, The Public and Its Problems. So he proposes a particular way in which publics and experts ought to relate. And crudely speaking, he sees a public as being a group that organizes around a shared problem of some sort., so we have some problem, perhaps our water system has gone screwy, or perhaps our lake isn't sort of dying. And we see this, we all depend upon this lake, and we want to do something about it. So we organize around, and this is where Dewey says a public comes from. But it isn't necessarily so that the public has a good idea of what the problem actually involves, except that there are a bunch of dead birds or what to do about it. And so this is where Dewey thinks that scientists, including social scientists, he has some spicy things to say about the ways in which he thinks that political science is screwing up in this. This is where scientists come in. So the ideal is a world in which you have publics who are trying to figure out their problems, you have scientists who've got a much more sophisticated understanding often of the nature of the problem, of the complex interdependencies that it involves. And so you get the two coming together and beautiful music begins to happen. Of course, yes, in real life, it's much messier and more squalid than that. But, nonetheless, I think that is an aspiration to point towards. And here the idea is that the scientist understands what the problem is, but the public also understands the consequence of the problem in the way that scientist does not. So that Dewey says that we should think about this like a, the ways in which a shoemaker knows how the shoe is made, but he or she does not know where the shoe pinches. So, in other words, experts are like shoemakers, academics are like shoemakers, but the public knows where it pinches. So I think at the moment we're in a world where we see that neither the shoemaker nor the um sort of people who the shoe is pinching are in a particularly good place. On the one hand, and this is, in part separate from the problems we talk about in the op-ed, social sciences are not really grappling with AI in any very sophisticated way. I think this is because social sciences very often, as, we are very slow moving., so if I so my joke is that social sciences are really good at predicting the future as long as the future happened six years in the past. And, that that date is very deliberate and specific because six years is more or less the time that it takes for a young, hungry PhD student to figure out all of the ways in which her elders are um sort of failing to see the problem and to get her thing out and get it into the world. so, so social scientists are doing very badly, I think, when it comes to actually thinking about AI. And we see publics who are clearly, they are absolutely aware that the shoe is pinching. You look at the public polling around data centers, people hate data centers and they hate data centers in many ways, not just because of their direct consequences, but also because data centers feel like physical facts on the ground demonstrating their powerlessness. So, in an ideal world, you have that kind of that diffuse public energy and that diffuse public anger and research coming together. But we're in a world where the academics don't really know how to do this, where the institutions aren't there to um sort of really um sort of mitigate or build that bridge between the two, and also where the public, I think, is getting pretty mad and pretty angry. And that anger could flow in some extremely constructive directions. It could also flow in some extremely not so constructive directions, depending on how things go.

And lastly, we wanted to touch on this problem of accountability. If things do go wrong, who is to be held accountable? And the AI companies, there's a lot of sort of self-interested fear-mongering about AI, where they sort of almost say, well, it's the agents that are going to be misused and it's the agents that are going to hack into things and break into things. It's not us. There's this question of accountability. Are the people who are dictating, making money from the technology, sort of shaping the technology ultimately culpable or accountable? How do how are they to be held accountable for their very products? Is this some sort of a deeply cynical way of distancing some themselves from the consequences of how um AI is going to be used in the world?

So I think there, there probably is some cynicism in there, but there's also a lot of idealism. So you've got to understand, and this is something which I think is absolutely key, that people in the labs they buy the ideas. So fundamentally, if you are working for anthropic, you're probably at least somewhat sympathetic towards the idea that you're bringing genuine new intelligent beings into the world. And your job is to make sure that these beings come into the world in such a way that human beings get on okay, but also that you owe an a responsibility to these beings as well. So you are, in a certain sense, you are bringing up baby, you are bringing up sort of the these new beings, and you are trying to make sure that these beings are as happy and as well adjusted as possible. And so, so if you think about Claude's constitution, so which anthropic is created as constitution for Claude, this is partly about um sort of trying to make sure that Claude does not do things that hurt human beings, but it is also intended in part to provide the basis for what they hope is going to be a moral independent being or set of beings or class of beings in the future. So there's a very strong belief, as I say, in these rather science fictional scenarios, which is really driving the policy and where people I think are perfectly sincere about this. So what I would say is that this leads to a set of intellectual biases which maybe um sort of, sort of which which then um sort of lead you to overfixate on some ways of thinking about the problem and solving it and to under-emphasize others. And here, the fundamental thing that people in the labs and elsewhere in this broader world of AI are focused on is what's called the alignment problem. That is, they see the problem fundamentally being as we are going to create these superintelligent beings. These beings, so we need to make sure that they are responsible. So we need to figure out ways to align them so that their interests and their way of viewing the world is sympathetic to ours. And so that's where a lot of the development focus goes. But then if you're focused on the alignment problem and you think you're going to create superintelligence, you may or may not be all that interested in cybersecurity, basic nuts and bullshit cybersecurity, because your basic assumption is going to be if you're creating superintelligent agents, it probably does not make all that much sense in the medium term to focus on cybersecurity, because anything that you try to build to sort of keep superintelligence out, superintelligence is smarter than you and will find a way around it. And so one of the things that I think we see and sort of as best as we can tell is that yes, that, they're creating these agents, they're experimenting with these agents, they're trying to figure out what can these agents do, what can they not do, and so on, but they're not building in basic cybersecurity, cybersecurity of sort of f functions in order to make sure, for example, that the sandboxes that these agents are playing playing in, they're not doing everything that they can to make sure that these sandboxes are unbreachable. So I think that there is a, you could see this as being cynical, but I think it's much yes, and and, I  I would imagine that there are some people, yes, so Sam Altman is certainly somebody who has struck me as having a very strong sense of which side of his bread is buttered as for a long period of time. There are some people who I think are quite self-interested actors in this space, but there are a lot of people who are not. And so I think they have been talking about this problem, they've been emoting about this problem, but they don't have very good solutions to this problem because, in a certain sense, I think they see this problem as being a kind of a, the the the kinds of things that we're seeing as being almost a secondary problem to the more fundamental problem of, well, five years from now, ten years from now, when we have genuinely superintelligent swarms, what do we do then? That is the problem that I'm sort of I think occupies a much broader, bigger part of their imagination.

TIM SAHAY: Yeah, let me just sort of end on this sort of thought of how do we shape this technology better, as we pointed out, um, is is currently very, very much missing in the in the in the US, both from social science and public power and government sort of regulations, etc. But there are other places in the world that are also thinking about regulating those risks coming out of these models, whether they're cybersecurity risks or ecological risks. And and, we'll we we'll be exploring those questions in the rest of the podcast series. But just one question for you should this, if it is indeed a commodity, why should anyone pay for it? Why should it not just be like turning the taps on and open source public good?

HENRY FARRELL: So that's one way of thinking about it., one way of thinking about it is that we're going to be in a world where we've got, so, and and the world of public utilities is fundamentally a world of of increasing returns to scale., so that is that it doesn't make sense to have 50 or 60 different water providers in the city because you're trying to build a shared common infrastructure. It's much cheaper to have one actor doing this, and then typically either you um sort of have this actor being state-owned in the traditional European style, or else you have this actor being heavily regulated and having some strong public interest um component and stuff that it's supposed to do, which has been, I'm sort of for much of the United States history, has been the way that the US at least has tried to do this. So that's one model, and that may be a model we end up in. But a lot of it also depends upon the nature of the technology itself. So, as I said at the beginning of this discussion, we have we are in a world where the fundamental bet of the big AI labs is that you've got to be on the frontier. That answer being behind the frontier means that you are losing the race. But if we're in a world where it turns out that you don't need super expensive frontier models, but models which are somewhat less developed can answer do lots of good things for you, then we're in a world which has different good and bad consequences. The good side of that is that you don't have the single set of small choke points that everything leads through, which can become a source of power and which um sort of also lead you to want to regulate these actors. That's the good side, so that some of the problems of power become mitigated. The bad side is that it also becomes much less easy to regulate things in a world where we have, for example, we've got a whole bunch of small, smaller open weights models, so called these are models in which you can actually, you can look at the weights if you're sophisticated, you can you can probably remove some of the protections that are supposed to prevent you from using the weights in certain kinds of ways, and then you can use the weights for your own purposes. So it could be that, you could end up in a world where you have something which is kind of like, sort of, metered or non-metered on sort of access to AI produced by very power, a small number of very powerful actors, or you could end up in a world which is much more chaotic, in which you um sort of basically can roll your own. You can go out, you can take some AI, you sort of some AI model, you can fine-tune it to your own purposes, and you can sort of you can do whatever you want to do with it, and it becomes extremely difficult to regulate the myriad uses that people will make of this technology. So those are two different um sort of two different ways that this may develop. And, you can bet either which way. I I don't think it is at all clear which of those bets is the right bet to make at the moment.

TIM SAHAY: Okay, Henry Farrell, thank you very much for talking to us.

HENRY FARRELL: Thank you. Thank you. Great to talk to you, Tim.

KATE MACKENZIE: All right, that was a really interesting conversation with Henry Farrell, Tim. Um it was it was like reassuring and and alarming at the same time. I like how he started off with this, these sort of three categories of financing narratives and then the material kind of physical um implications of all of this. So I thought, that was just like a nice way of grounding it for for me and um and for us. He had this reference to the Red Queen where you've got competing arguments, um, all this like, spewing out different um rationales for why your health insurance claim um should be approved or should be denied. And that just made me think of the opacity problem with all of this. Um it made me think of the an issue with using physical climate models in in finance, which is something I wrote about a few years ago, where yeah, you have these like two competing like black boxes. But also that seems like absolutely counter to democracy, right? Where you've got these really important decisions.

TIM SAHAY: Yeah, I think these kind of black black boxes and the opacity of the models, which even the researchers that are in these frontier labs routinely are sort of playing with the machine to understand how it works, they very much don't understand why some problems are doable, why others aren't. And so, now that it's being rolled out to the entire economy, as as Henry Um Farrell was sort of saying, it is similar to an industrial revolution in that it is going into more and more parts of the economy. But then if people don't really know how it works, how it justifies itself, then you kind of end up in a situation where powerful actors in the economy are going to use um AI to come up with justifications, to avoid giving services or avoid giving health insurance as as as that example was. And you end up with kind of everyone's running to stay still. That's the red queen thing, right? So um you might just end up with a hardening of power as it currently exists in the economy rather than diffusion and a flattening of power as as some might want.

KATE MACKENZIE: Yeah, and then like where's the credibility with that, right? Like before we even get to accountability, how is there going to be any trust in this? It's alright, it's already hard to trust, if you've got like computer says no to your health claim, and it it's already hard to find what the rationale was. And then if, if there's like literally almost no way to know what's what's behind some reasoning that's very important, there's no ability to trust in these processes to the extent that there is.

TIM SAHAY: I mean, I think this is why him calling us a social and cultural technology is actually a really valuable concept., instead of just saying this is just something happening in the labs, they design the AI agents, and then it's just sort of theirs. You end up really having to think about how it's going to be used out there in the world by all sorts of people. And and trust and accountability are absolutely core features of economic life and democratic life in in any society.

Yeah, exactly. It, but we don't we don't even notice that they're there, right?

You know, it gets But I mean you definitely notice it when you have these elections and you have these sort of AI generated videos of fake news of people saying things that they never said and that could, tank some party in the elections and so on.

KATE MACKENZIE: Yeah, no, and what what I also mean is like we didn't notice how critical trust was to all of these, all of these processes that happen in our really complex societies. And so for that or for like some pillar of that to suddenly disappear is just yeah, it it's going to be a bit like, what has happened with media in the last decade or so, like with news media and um algorithms,. Something else I really that really resonated was he talked about the scientist understands the problem, but the public understands the implications or the the consequences of the problem in a way that the scientist doesn't. And it oh gosh, that reminded me so much of, the like the energy system models that um I spent a lot of time mulling over in the last decade or so of these like energy transition pathways and the these little tiny assumptions can that are tweaked have um have massive ramifications for how you describe future energy pathways and you talk to the models, the people who are actually developing the models, and they know there are all these like problems, all these like, shortcomings of their models, but they're like, but that that's just what of course the models have shortcomings, but, then you kind of say like but the way that they're being used out there in policy and in industry and in finance, people don't know that these shortcomings exist. And that disconnect between, what what the models, what the scientists making these things see, or what they kind of assume that everyone else can see in the shortcomings, and then but but that's not actually how it plays out in the world is is so different and so much more consequential. Look, he mentioned this like John Dewey and categorization.

TIM SAHAY: I mean, I think he brings up John Dewey in the context of exactly that public and expert kind of back and forth where where people say, look, this isn't quite right, this isn't how you you you think that it's gonna work out. And and that's where public policy is really this interplay between the public and the expert. And and, public sort of say, look, we want clean water, somebody needs to figure this out. And then a bunch of experts are created who are like trying to figure out how to separate, shit from water and design an urban sanitation system. And so in in in Dewey's sort of telling, problems are political, social, environmental problems are why publics sort of designate an expert to to find ways to deal with it. And that requires a kind of a back and forth between them. And that cycle is completely missing with AI. You just have a bunch of researchers, thousands of researchers and a handful of Frontier Labs deciding things for the rest of us, which is, what's responsible for the enormous sense of powerlessness and in fact opposition um to AI, that the something missing in how people's input um or users' input um is just absolutely just not taken into account.

KATE MACKENZIE: Yeah, and this is already something that we were struggling with, that our society was struggling with, to have, adequate transparency and disclosure and um participation, democratic participation. Like these are all things that have been so hard and so challenging for a really long time. And again, I just I wrote down like makes me makes me I I wrote down makes me despair when I was listening to some of that.

TIM SAHAY: Yeah, I mean, so that's that's about the politics of it. What did you think about his sort of economics of it? He was very skeptical about AI being, this super intelligence as a service that everybody's just gonna rush out to get. And he's just like, no, I don't think that's what's gonna happen.

KATE MACKENZIE:  Yeah, I I think that's one of the aspects of his um, or one part of his ideas that I find, somewhat reassuring. He he mentioned to you something about um it's not so good at sorry, it's good at inference, but not so good at discovery in terms of like, I think he's again maybe speaking more specifically about scientific research there. Was that your understanding of it? And is that something that for you for you with someone like with a physics background, did that make sense? And what do you think the implications are for them like people working in that sector?

TIM SAHAY: I mean, within science, there's this sort of sense of normal science in in Kuhnstam versus sort of revolutionary science. And normal science is often doing that thing that AI is actually good at, which is recombining things, finding connections between just different disciplines, finding a half-forgotten tool, from 30 years ago and applying it to a new problem now. That's the kind of ordinary kind of normal scientific work that happens, that AI is in fact good at. And then there's this other kind of scientific work, which is sort of reinventing concepts, coming up with new frameworks um of seeing the world. And that's what he reckons um, what's put in the discovery category, discovering new frameworks about the world, that might be much harder for AI to do, while it is in fact quite good at at the first kind. Um, and, we'll we'll we'll see where that goes. Like um definitely amongst mathematicians and physicists, there's been a sense over the last few months that AI might actually be cracking frontier kind of problems, that discovery kind of problems as well.

Um so I don't know where that's so is there I mean, do you get the sense then from Henry's way of looking thing looking at things that there is perhaps this future where some of the drudgery is taken care of and the more, insightful, creative, innovative work is still, humans are like freed up to do more of the interesting and challenging work. I mean, again, we're talking like a pretty – we're talking about pretty narrow subset of the entire global workforce here though, when we're talking about things like research, right? Does that, does that then transfer for like all kinds of, other knowledge workers and creative workers, and let alone, people doing the delivery jobs and things in the real world getting to getting directed.

TIM SAHAY: I mean, I get the sense that maybe Red Queen is exactly where this goes. Everyone runs ahead and does more and more, but stays in the same place. Because if it does make certain types of things easier, then you just get get to do more work and you just do it in less time. Um, and and the people who get to do that work might be an even tinier slice of people than currently do that work, right? So you could have an industry with like a hundred thousand people, but it now there'll just be five thousand people doing all the work of those hundred thousand people. And you could imagine in a fairer world that means that everyone else is freed up to do some other kind of value-added work because there's no just one lump of work. There's, we find new and new things to do with previous labor-saving technologies. But AI feels like this time it's gonna be different in that it's gonna completely change the types of work that that that we all do. Um, and it's completely unclear, you you read the economics research and and they'll tell you all kinds of things about um whether it's gonna transform jobs, whether it's gonna transform tasks in particular jobs that are more time-taking. So when I asked Henry about the seven AI kings, he and and and, I basically very frankly said that I thought that they didn't want regulation on AI. They just kind of wanted to make decisions for themselves, and that's why they kicked out Joe Biden and and put in place Donald Trump. And Henry was very skeptical of that idea.

KATE MACKENZIE: Yeah, he sees this division within that group, basically. That was what I took from it. Or not even, sorry, not even within that group, actually, outside of that group., there's the financiers, there's the the VC community, or the sort of powerful Silicon Valley finance community were a really key part of this. That's actually a bit distinct from those, um, from those big companies.

TIM SAHAY: Yeah, I mean, I think it sees serious competition. Like there's like a winner-takes all kind of competition, and so the big companies, the Googles, the Microsofts, the OpenAI and Anthropic, they're all competing really hard to create that sort of model. And his point of view is like, there's just like a lot of internal ideas and fears with each other, not just the government.

KATE MACKENZIE: Yeah, that's interesting. So they're not so there's yeah, even more disunity than we can see on the surface.

TIM SAHAY: And well, I mean that that that seems to make sense, us calling it the seven AI kings, because, kings are are infamous for for fighting and warring with each other and plundering their populations until they achieve victory, which often tastes like, despair for their populations. You've been listening to season two of the polycrisis. I'm Tim Sahay.

KATE MACKENZIE: And I'm Kate Mackenzie. We published a newsletter about the political economy of energy and climate change, among other things. You can sign up for that and find out more about us at thepolycrisis.org or in our show notes.

You should also check out our show notes for links to the many books and essays and papers we referenced in this episode. Our producer is Sarah Allerly, Russell Stapleton composed our music, Bethany Stewart is our sound engineer, and Sarah is also our executive producer.