Speaker 1: Bloomberg Audio Studios, Podcasts, Radio News.
Speaker 2: Hello and welcome to another episode of the Odd Lots Podcast.
Speaker 2: I'm Joe Wattenthal.
Speaker 3: And I'm Tracy Alloway.
Speaker 2: Tracy, you know, there's a lot of like concerns about
Speaker 2: AI obviously these days, and if anyone who's like reasonably
Speaker 2: intelligent can like list tons like maybe like it's gonna
Speaker 2: go rogue and be smarter than us or maybe whatever,
Speaker 2: But I still think, like what or maybe there's just
Speaker 2: gonna be this like flood of disinformation and deep fakes,
Speaker 2: or maybe it's gonna put all journalists out of business,
Speaker 2: which is certainly plausible, But I think, you know, like
Speaker 2: something I think a lot about is just this idea
Speaker 2: that regardless of what happens, like we're going to be
Speaker 2: increasingly sort of trusting like a black box for answers
Speaker 2: that we really have no idea where those answers, however
Speaker 2: you want to describe that come from.
Speaker 3: Yes, absolutely, And this is something that's come up on
Speaker 3: the podcast a number of times.
Speaker 4: Now.
Speaker 3: I'm thinking way back to an episode we did that
Speaker 3: was basically about the black box of algorithms and how
Speaker 3: difficult it was to understand what goes into them and
Speaker 3: then what comes out. And then of course we recently
Speaker 3: did that episode on pricing and the idea of algorithmic pricing,
Speaker 3: the idea of building proxy consumer profiles, And you're right,
Speaker 3: the issue is we know that there's this new technology,
Speaker 3: We know that there's all this data floating around, but
Speaker 3: we don't entirely know how it is coming to the
Speaker 3: conclusions or creating the output that it actually is.
Speaker 2: You know, the pricing thing is interesting because you know,
Speaker 2: in a market economy, you know, you could argue it's like, oh,
Speaker 2: at any given moment, you know you're being served up
Speaker 2: the like the optimal price, right. And in theory, even
Speaker 2: with the most advanced algorithms and stuff like, maybe there's
Speaker 2: some like this price is happening at the best price
Speaker 2: for both the seller and the buyer, et cetera. But
Speaker 2: I think like people just have a sort of deep
Speaker 2: intuitive distrust about the fact that like, you know, you
Speaker 2: can't go there and like touch it and verify it
Speaker 2: and see like this is why this exists in the
Speaker 2: state that it is. And I think it's going to
Speaker 2: create a lot of I don't know, cultural apprehensions as
Speaker 2: more and more decisions and more and more things that
Speaker 2: are that affect our lives just seem to like emerge
Speaker 2: spontaneously out of the box.
Speaker 3: Absolutely. I'm thinking of all the people working at Chipotle
Speaker 3: who are going to have to answer questions about not
Speaker 3: just portion sizes now, but also questions from customers about
Speaker 3: are they getting the best price.
Speaker 2: Have you seen those awful videos that people are taking
Speaker 2: of the Chipotle workers.
Speaker 3: Yeah, I've seen some of us so vilely imagine, yes.
Speaker 2: It's so vile anyway, that's a that's a that's a
Speaker 2: separate thing. But yes, like all of these things, and
Speaker 2: you know it's it's not just with AI obviously, and
Speaker 2: like this sort of world exists in increasingly black boxes.
Speaker 2: You put in the support ticket, you try to like
Speaker 2: talk to someone in an embassy or a consulate or
Speaker 2: anything you do when you sort of like send out
Speaker 2: some requirement or some request to some bureaucracy or some company,
Speaker 2: and then it moves around. I was on a I
Speaker 2: had a flight recently that was delayed for nine hours,
Speaker 2: and there's like this palpable frustration that everyone feels that
Speaker 2: the person you know, standing at the gate like can't
Speaker 2: answer their questions and they can't get anyone to answer
Speaker 2: their questions, and it just sort of like, you know,
Speaker 2: everyone explodes and everyone knows it's not the gate agent's fault,
Speaker 2: but still, like, you know, there's just this frustration, like
Speaker 2: where is the answer to what's going on?
Speaker 1: Right?
Speaker 3: And you can't ask a single person because that single
Speaker 3: person doesn't, like the gate agent doesn't have the answers.
Speaker 3: I think what's happening is like society has organized itself
Speaker 3: in such a way as to devolve responsibility, and the
Speaker 3: creation of all this new technology is basically going to
Speaker 3: I guess, ramp all of that up, right, Like, so
Speaker 3: it might not even be the gate agent that you're
Speaker 3: asking in the future. It might be you trying to,
Speaker 3: like I don't know, ask the algorithm, like why it
Speaker 3: decided to bump you versus someone else or actually that
Speaker 3: already happens, right, There is an algo that dictates like
Speaker 3: who gets bumped from the plane and who doesn't. So yeah.
Speaker 2: The other big one, of course is health insurance and
Speaker 2: why some claims are suddenly denied and you never get
Speaker 2: this answer anyway. The world is already filled with systems
Speaker 2: in which we have some question and no one actually
Speaker 2: can sort of you know give you the answer.
Speaker 3: Yes, absolutely, and I think we might have the perfect guests.
Speaker 2: We do have the perfect guest. It's someone we've talked
Speaker 2: to multiple times on the podcast, one of the smartest
Speaker 2: guys around, Always interesting, always worth paying attention to. We're
Speaker 2: going to be speaking with Dan Davies. He is the
Speaker 2: author of the new book The Unaccountability Machine, Why Big
Speaker 2: Systems Make Terrible Decisions, and How the World Lost its Mind.
Speaker 2: So this should be really fun. Dan, thank you so
Speaker 2: much for coming back on the podcast.
Speaker 4: Oh thanks very much for inviting me.
Speaker 2: You know, before we get into the meat of your argument,
Speaker 2: The Unaccountability Machine, I'm actually curious about the second half
Speaker 2: of the title, how the World Lost its Mind, because
Speaker 2: I certainly feel like the world lost its mind. But
Speaker 2: I'm like a middle aged boomer, and I feel like,
Speaker 2: you know, anytime you get to my age, you're like, oh,
Speaker 2: the world's gone crazy, the world's gone mad, Like, you know,
Speaker 2: why is everything so nuts these days? Has the world?
Speaker 2: Do we actually know that the world's gone mad? Or
Speaker 2: is it just because like we're all sort of old
Speaker 2: and cranky now everything seems like the world's gone mad.
Speaker 4: Well, from your own perspective, you can never be sure. Yeah,
Speaker 4: but I think there's actually reasonable objective ways that you
Speaker 4: can check up on this, just by noticing that the
Speaker 4: world gets more complicated as it gets bigger, and it
Speaker 4: gets exponentially more complicated. And I mean that're in the
Speaker 4: literal mathematical sense, because the number of connections grows faster
Speaker 4: than the number of things, whereas our capability to understand
Speaker 4: the world, manage it, and make decisions doesn't necessarily grow exponentially.
Speaker 4: So this is the story, I would argue of economics,
Speaker 4: It's the story of any management book that's worth reading,
Speaker 4: because the central problem of management is the world is
Speaker 4: getting faster, more complicated, faster than you can process that complexity.
Speaker 4: What are you going to do about it? How are
Speaker 4: you going to reorganize? And you know, we've just been
Speaker 4: through a global financial crisis, we've just been through a
Speaker 4: political what Aanam two is called poly crisis. I think
Speaker 4: there's decent reasons to believe it's not just because we're
Speaker 4: getting older, and it is actually a crisis of the
Speaker 4: ability to make decisions matched up against the speed and
Speaker 4: complexity see of the decisions that we're having to take.
Speaker 3: So when we talk about the lack of accountability and
Speaker 3: making bad decisions. Give us some concrete examples of things
Speaker 3: that you have spoken about or written about in your book.
Speaker 3: What are you thinking about here?
Speaker 4: Well, I mean there are kind of there's there's little trivial,
Speaker 4: funny examples, and there's big, huge, serious examples. So for example,
Speaker 4: and apologize in advance because this is quite disgusting.
Speaker 3: Oh is it the squirrels?
Speaker 4: Would you rather I didn't talk?
Speaker 1: Yeah?
Speaker 3: You could. This is bad if there are children listening.
Speaker 3: Maybe child.
Speaker 4: At the start of this century there was a craze
Speaker 4: for squirrels as pets in Europe, and squirrels were being
Speaker 4: imported from North America in China to be pets, and
Speaker 4: they had to have the right paperwork. And so one
Speaker 4: day ad of four hundred of the poor little things
Speaker 4: showed up in Chippell Airport in Amsterdam without any paperwork
Speaker 4: and without a return address to send them back to.
Speaker 4: And it's difficult to know what the airline should have done,
Speaker 4: but you can't help thinking that there must have been
Speaker 4: a better solution than what they actually did do, which
Speaker 4: was that they threw all four hundred of them, except
Speaker 4: for one or two that escaped into an industrial shred
Speaker 4: and this caused an outrage. There were questions asked in
Speaker 4: the Dutch Parliament and people immediately started asking how did
Speaker 4: this happen? Who is responsible? And in fact, the press
Speaker 4: release from the airline apologizing for this is studied as
Speaker 4: a masterpiece of crisis pr in business schools. But when
Speaker 4: they went back to inquire, they ended up realizing that
Speaker 4: no one had ever really made the decision that that
Speaker 4: was what they were going to do. The government's biosecurity
Speaker 4: Ministry had set some standards for the importation of small animals,
Speaker 4: the airline had set some standards for compliance with that policy.
Speaker 4: The only people who were expected to make a decision
Speaker 4: about whether this was grotesque and couldn't be done or
Speaker 4: not were some low level employees in a shed at
Speaker 4: Chippell Airport. And frankly, people who work in sheds aren't
Speaker 4: usually going to be thinking that they're meant to be
Speaker 4: second guessing the government. And so what happened is that
Speaker 4: you had this phenomenon that turns up a lot of
Speaker 4: the time at all levels of organization, which is that
Speaker 4: something happened which nobody wanted, but was the predictable output
Speaker 4: of the system that they had created.
Speaker 2: Yeah, this is this is first of all that's grim,
Speaker 2: but also like when you describe it that way, you
Speaker 2: could see it because ultimately, like, all right, here's this
Speaker 2: awful thing that happened to three hundred and ninety eight squirrels.
Speaker 2: I guess in theory, someone had to I don't want
Speaker 2: to get I don't know dumped the bag to the shretter.
Speaker 4: I have not researched them.
Speaker 2: Yeah, but that's not a very satisfying answer. I mean, yes, okay,
Speaker 2: maybe there was someone who did the physical thing, and
Speaker 2: I guess the entire you know, operations of the airline
Speaker 2: and the port and the customs Bureau could like just
Speaker 2: blame that one person. But that's not a very satisfying
Speaker 2: conclusion I guess in terms of like how this actually happened.
Speaker 2: It's funny I brought this documentary up. I was thinking
Speaker 2: about this too, Like I'm with the destruction of the
Speaker 2: old Penn station in New York, which it's like this
Speaker 2: like extraordinary, like I guess, like Roman or Greek building,
Speaker 2: and they just tore it down to like build the
Speaker 2: pretty owes.
Speaker 3: Something hideous, Yeah, something hideous.
Speaker 2: In its place. And now the new Penn station is terrible,
Speaker 2: but it feels like the same thing. It's like, how
Speaker 2: did no one stop and say, like, wait, does this
Speaker 2: make any sense in the long in the big picture.
Speaker 4: Yeah. And the thing is they'd created a system on
Speaker 4: the assumption that squirrels would show up in ones and
Speaker 4: twos and they could be dealt with as individuals, and
Speaker 4: that therefore you would never get into this sort of
Speaker 4: situation because when you build a system, you're always building
Speaker 4: a model of the world, and if something happens which
Speaker 4: doesn't fit into your model in the world, your system
Speaker 4: might do something awful. And there's a sort of symmetry
Speaker 4: and a kind of resemblance here with much bigger and
Speaker 4: more grim things like the Boeing seven three seven max,
Speaker 4: like the light board scandal in financial markets. It's not
Speaker 4: so much that anyone sat down and said, let's form
Speaker 4: a conspiracy to manipulate interest rates, or let's build a
Speaker 4: place that crashes under certain circumstances. It's just that no
Speaker 4: one set things up so that that wouldn't happen.
Speaker 3: Do you think I'm afraid going for the squirrels are
Speaker 3: probably going to be our archetypal example of this, But
Speaker 3: do you think with the squirrels, for instance, the hyper
Speaker 3: specificity of the goals or the job rolls of everyone
Speaker 3: involved contributed to the outcome. So in the sense that
Speaker 3: you have you know, I guess, the Dutch Wildlife Department
Speaker 3: who is trying to protect Dutch wildlife. Then you have
Speaker 3: like the guys at the airport who are charged with
Speaker 3: actually carrying out these orders, and then you have the
Speaker 3: airline which is charged with like looking at the paperwork,
Speaker 3: do those tend to lead to when taken altogether, do
Speaker 3: those tend to lead to worse outcomes?
Speaker 4: Yeah? Absolutely. It's this kind of fragmentation of the decision
Speaker 4: which is a result of the industrialization of the decision
Speaker 4: and its industrialization literally in the atom Smith's sense that
Speaker 4: you don't have anyone in the pin factory building an
Speaker 4: entire pin. You have thirteen guys all performing one simple operation.
Speaker 4: And that's a much more productive way to do things.
Speaker 4: But then when you apply it to decision making, you
Speaker 4: have the problem that everyone assumes that everyone else is
Speaker 4: going to react when something unplanned happens. So this is,
Speaker 4: like I say, it's the central problem of every good
Speaker 4: management sex book. How do you deal with information? How
Speaker 4: do you get a drink from a fire hose. How
Speaker 4: do you stop yourself from being overwhelmed? The answer is
Speaker 4: always in some way or another, you build a system
Speaker 4: to make the decisions for you. But once you've built
Speaker 4: that system to make the decisions for you, you no
Speaker 4: longer feel ownership of that decision. Psychologically, you no longer
Speaker 4: feel like you're accountable for the decision, because if you
Speaker 4: were accountable, you might be able to change it. But
Speaker 4: if you're going to be held accountable for this thing,
Speaker 4: you haven't really moved that thing on, you haven't really
Speaker 4: delegated it to the system.
Speaker 2: You mentioned Boeing, And speaking of Boeing, there was a
Speaker 2: great blog post back in April from Steve Randy Waldman
Speaker 2: inter Fluidity, who I consider another one kind of in
Speaker 2: your category of people who have basically been writing interesting
Speaker 2: things on the internet for a very long time, and
Speaker 2: he has the title was seeing like a CEO and
Speaker 2: this idea that you know, when Boeing merged with McDonald douglas,
Speaker 2: the McDonald douglas CEO became an outside hire and he
Speaker 2: had to essentially gain some legibility into this new organization
Speaker 2: that he inherited, and that was the cause for some
Speaker 2: of this like streamlining and offshore or you know, offshoring,
Speaker 2: things like that. Talk to us like about you know,
Speaker 2: you mentioned Boeing and it's more you know, the crises
Speaker 2: there that's been going on several years. It's a more severe,
Speaker 2: serious issue than the squirrels. But how do you know,
Speaker 2: how do you think about what happened there?
Speaker 4: I think about it in very similar ways to Steve
Speaker 4: Randy Wilburn because the and you see it in Boeing,
Speaker 4: but it's visible to a greater or lesser extent in
Speaker 4: very very many companies, probably the majority of companies today
Speaker 4: that the C suite has an information environment which is
Speaker 4: almost completely composed of financial numbers, because the financial numbers
Speaker 4: are taken by them as objective facts. We can talk
Speaker 4: long and hard with accountants about how objective those financial
Speaker 4: numbers are and how many assumptions go into them, but
Speaker 4: they arrive on a spreadsheet looking very much like objective
Speaker 4: facts about the world. Things like engineering level, in principle
Speaker 4: are objective facts, but you have to do a lot
Speaker 4: more work to find them out and to know what's relevant.
Speaker 4: And then you have issues like culture and kind of
Speaker 4: the social environments which aren't even capable of being quantified,
Speaker 4: So there's always this tender and see, if you're trying
Speaker 4: to do that thing of manage your own capacity to
Speaker 4: manage your information flow, that you're going to concentrate on
Speaker 4: the things that look finite and look manageable, and that's
Speaker 4: always going to be the financial numbers, which can be
Speaker 4: a big problem because financial numbers can mislead. You know,
Speaker 4: you can create illusions in an accounting system the same
Speaker 4: way that you can with anything else.
Speaker 3: How did we end up here? And I'm thinking specifically
Speaker 3: to one particular development in the world of business, which
Speaker 3: is the creation of the limited liability company. And I
Speaker 3: guess the clues kind of in the name there, But
Speaker 3: what were the decisions or the trends that sort of
Speaker 3: came about in creating the current system.
Speaker 4: Well, I mean, the limited liability company is certainly a
Speaker 4: big kind of change if you think about these things
Speaker 4: in feedback terms and information terms, and you know, one
Speaker 4: of the big arguments of the BO is that we
Speaker 4: should be looking to the mathematics of information theory rather
Speaker 4: than the mathematics of optimization to explain and model some
Speaker 4: of these things. But a limited liability company is an
Speaker 4: information filter. It tells you that outcomes below a certain
Speaker 4: amount aren't going to affect you anymore, and that changes
Speaker 4: your information world, It changes what you care about. But
Speaker 4: then what I think really started doing the damage, so
Speaker 4: to speak, was the development of the leverage buyout in
Speaker 4: the nineteen seventies and the shareholder value movement, as it
Speaker 4: was kind of really kicked off by Milton Friedman's essay
Speaker 4: on the social responsibility of a business to increase its
Speaker 4: profits New York Times, nineteen seventy, but then built on
Speaker 4: by just the entire kind of two decades of business
Speaker 4: school research that followed from that. Because again, thinking about
Speaker 4: it in information terms, a leftage buyout is a massive,
Speaker 4: screaming signal. The requirement to make the payments on debts
Speaker 4: becomes a signal that swamps anything else you might be
Speaker 4: thinking of, because if you are a CEO and you've
Speaker 4: got LBO levels of debt, that's your priority. You can't
Speaker 4: think about anything that isn't related to servicing that debt.
Speaker 2: Let's go back and talk about the sort of big
Speaker 2: picture ideas. In your book, you talk a lot about
Speaker 2: this field called cybernetics, and cybernetics is a name sounds
Speaker 2: like something that they would have come up with in
Speaker 2: the early nineties, like the Wired magazine people like would
Speaker 2: get into like cybernetics in ninety one. But actually this
Speaker 2: field has been around at least since the nineteen forties.
Speaker 2: I'm surprised they even had a word like cybernetics in
Speaker 2: the nineteen forties. But what is cybernetics? And talk to
Speaker 2: us about the sort of general framework you use as
Speaker 2: a sort of to start talking about this stories in
Speaker 2: your book.
Speaker 4: Sure, I mean, you're right, it was its Second World
Speaker 4: War kind of talk. It's originally from a word meaning
Speaker 4: the man who steers the boat, So it's a cybernetics
Speaker 4: in that sense. And the first guy to use it
Speaker 4: was a scientist working on creating an automated gun site
Speaker 4: for the United States Air Force. And the idea here
Speaker 4: is that there is some quantity that is preserved in
Speaker 4: an automated gun site between the operator, the radar, the server,
Speaker 4: motors and that and all the components of that system,
Speaker 4: and that component is information. And so at the time
Speaker 4: this was Norbert Viena. I'm talking about the scientist in
Speaker 4: the automated gun sites. Another guy working in the same field.
Speaker 4: You might have heard of was Claude Shannon at Bell Labs,
Speaker 4: who was inventing information theory at Bell Labs and has
Speaker 4: some fundamental theorems there, And in many ways, science of
Speaker 4: cybernetics is information theory applied to control. So you might
Speaker 4: have an information theory kind of pie the piece of
Speaker 4: maths that tells you how much bandwidth you need to
Speaker 4: transmit a given signal, and the cybernetic interpretation of that
Speaker 4: math would be that it's telling you how much capacity
Speaker 4: you need to manage a system that's of similar kind
Speaker 4: of noise. So this was all made huge use of
Speaker 4: in controlling things where you have access to the whole
Speaker 4: information environment. So a lot of that early maths from
Speaker 4: the first cyberneticians has just kind of stayed with us
Speaker 4: through the invention of the electronic computer, and a lot
Speaker 4: of it is actually at work in really modern artificial intelligence.
Speaker 4: Meragai called ben Recht who works on recommendation algorithms, and
Speaker 4: he's quite upfront that a lot of his fundamental mathematical
Speaker 4: techniques are from the four seas and fifties just being
Speaker 4: applied in the context of massively more computing power. What
Speaker 4: I'm interested in is where you take those kinds of
Speaker 4: theorems and apply them in a slightly more unrigorous, slightly
Speaker 4: more metaphorical sense to situations of management and organization where
Speaker 4: you don't have access to the full information environment and
Speaker 4: you just have to say, we're going to think about
Speaker 4: this not in an optimizing economics kind of neoclassical economic sense,
Speaker 4: but we're going to think about this as a system
Speaker 4: that has to be kept under control, and we're going
Speaker 4: to say, well, how much resource do we need in
Speaker 4: order to stabilize this system as a system, Which is
Speaker 4: a kind of abstract way of looking at it, but
Speaker 4: it's the same fundamental problem of management. How do you
Speaker 4: get a drink from a firehouse? How do you match
Speaker 4: your own capacity to manage to the complexity of the
Speaker 4: thing that you're in charge of.
Speaker 3: Wait, can you give us a practical example of the
Speaker 3: application of cybernetics. I guess because it does sound to
Speaker 3: your point earlier, it sounds a little bit abstract in
Speaker 3: my mind.
Speaker 4: Well, I mean practical example I think is the history
Speaker 4: of the development of the corporation. So from the first
Speaker 4: days of the American railroads, which were probably the first
Speaker 4: really big corporate structures, the world ever saw. You have
Speaker 4: this problem that as the network builds out, it gets
Speaker 4: more complicated, and the ability of the head office to
Speaker 4: manage it doesn't grow faster. And you can try and
Speaker 4: solve that by adding more people. You can get a
Speaker 4: great improvement by adding wireless telegraphs. But fundamentally, at some
Speaker 4: point this railroad is going to grow big enough that
Speaker 4: you have to devolve. You have to split it into branches,
Speaker 4: and you have to give autonomy to some of the subsidiaries,
Speaker 4: because that's the only way that you can match the
Speaker 4: bandwidth of the management to the bandwidth of the control problem.
Speaker 4: So I'd say that what we always see in any
Speaker 4: big organization is that it grows, it gets more complicated,
Speaker 4: It tries to deal with that by adding more resources
Speaker 4: at head office, it ends up not being able to
Speaker 4: keep up, and then it reorganizes. And the reorganizations almost
Speaker 4: always either involve pushing responsibility down to the shop floor
Speaker 4: or down to the branches, or they involve spinning off
Speaker 4: parts of the business into a separate organization and giving
Speaker 4: up the task of controlling it at all.
Speaker 2: What are accountability sinks?
Speaker 4: The accountability sink is just it's a name for a
Speaker 4: particular move of cybern ethics that I notice a lot
Speaker 4: of these days, which is when you consciously break the
Speaker 4: feedback links from the subjects of a particular decision to
Speaker 4: yourself or to the unit that's kind of meant to
Speaker 4: be making it. So your gate agent Joe is just
Speaker 4: a classic example of the accountability sink, because they talk
Speaker 4: to you with the voice of a corporation and they
Speaker 4: say that this is the policy, there's nothing I can
Speaker 4: do to change it, and then you only able to
Speaker 4: talk back to them as a human being like yourself,
Speaker 4: so you can't get mad at them because it's not
Speaker 4: their decision they get Just to.
Speaker 2: Be clear, I did not get mad.
Speaker 3: There's a video of Joe out there.
Speaker 2: I just sat there and I closed my eyes, and
Speaker 2: I did stand around because I was sort of curious
Speaker 2: the gate banter. But I did not get bad.
Speaker 4: I just want to establish Yeah, but then you might
Speaker 4: ask them, and I will confess I've done this. I've
Speaker 4: asked them someone politely for the phone number of someone
Speaker 4: who I can call up who is responsible for that decision,
Speaker 4: and you know it's not the policy. You can't get
Speaker 4: that phone number. The whole point of this was to
Speaker 4: create a sink into which unpleasant feedback can be poured
Speaker 4: and does dissipated harmlessly. And when you start thinking about
Speaker 4: these things in terms of accountability sinks, you start seeing
Speaker 4: them everywhere. Because everywhere that there's a policy that can't
Speaker 4: be broken and no feedback to the person who could
Speaker 4: get the policy changed, that's an accountability sink. That's a
Speaker 4: way that someone has protected themselves from the consequences of
Speaker 4: their decisions, possibly at huge costs to the organization that
Speaker 4: they're working for, but possibly not.
Speaker 3: I guess I'll ask the obvious question, which is how
Speaker 3: do we break out of accountability sinks? And I think
Speaker 3: the frustration with everyone is that, you know, you feel
Speaker 3: powerless when you're caught in one, when you can't get
Speaker 3: the answer that you want, or when you can't speak
Speaker 3: to the decision maker and try to reason with them
Speaker 3: or explain why this might be a one off or
Speaker 3: a peculiar situation. And then it just feels like the
Speaker 3: idea of actually starting to break apart some of these
Speaker 3: sinks and move to more of an era of personal
Speaker 3: accountability the book stops here and all of that, Yeah,
Speaker 3: there we go. It just seems further and further away.
Speaker 4: Well, it is. And the horrible answer to your question,
Speaker 4: Tracy is that maybe we can't, or maybe as individuals
Speaker 4: we can't. And that's actually, in my view, potentially very
Speaker 4: bad news for society because all of this sink, you know,
Speaker 4: all of this negative emotion from people about the way
Speaker 4: the world is, goes into the sinks. But like any sink,
Speaker 4: it piles up, and it piles up, and then after
Speaker 4: a while it all spills out, and then suddenly we
Speaker 4: get things like Brexit in the UK, or kind of
Speaker 4: first go of Donald Trump in the USA. We get
Speaker 4: people who have used to being decided upon and used
Speaker 4: to being ignored, getting steadily more and more dissatisfied with
Speaker 4: the system, and then finally they start to use the
Speaker 4: only power left to them to just make use their
Speaker 4: votes in a way that says, I am no longer
Speaker 4: satisfied with this, this is no longer tolerable to me.
Speaker 4: I'm going to use my vote to tear the system apart.
Speaker 4: And so with all these things, can you can divert
Speaker 4: these things for a while, but it's at the cost
Speaker 4: of building up fragility.
Speaker 2: Yeah, I have to say, you know, I I didn't
Speaker 2: finish your book I read. I'm about halfway through, but
Speaker 2: it did leave me fairly nihilistic or pessimistic that it's like,
Speaker 2: these are these inexorable centrifugal or centripetal forces. I can
Speaker 2: never remember which is which, and they're pulling us into
Speaker 2: all of these sort of high stress decisions and it's
Speaker 2: really bad, and things are going to things are gonna
Speaker 2: keep breaking, and we're going to get angry or anger.
Speaker 2: We talked about AI in the beginning, and a sort
Speaker 2: of provocative idea that you talk about in your book
Speaker 2: is the idea of the corporation as it exists and
Speaker 2: as we've known about it as already a proto AI.
Speaker 2: So you go to chat GPT and you put in
Speaker 2: a request and something spits out and it's impressive whatever,
Speaker 2: but that actually this is just sort of a specific
Speaker 2: example of what the corporation has been for a long time.
Speaker 4: Absolutely, I mean, and this is the beauty of abstract mass.
Speaker 4: It describes things without you needing to know what they
Speaker 4: actually are. These are all just decision making systems. I
Speaker 4: had a conversation with someone at the European Commission the
Speaker 4: year before last, because in Europe they passed in Act,
Speaker 4: saying that if a decision is made affecting you, like
Speaker 4: to turn you down for health insurance, then if that's
Speaker 4: made by an algorithm, you have a right of explainability,
Speaker 4: So you have a right that someone can explain to
Speaker 4: you why that algorithm made that decision for you. And
Speaker 4: you know, I thought that's quite good, But it's kind
Speaker 4: of ironic that this decision is coming from the European Commission.
Speaker 4: So I asked the guy who works there, well, you know,
Speaker 4: when you make a decision like that, what right of
Speaker 4: explainability do I have from you? And the answer is
Speaker 4: ha hah, no, none at all. M All these things
Speaker 4: are basically working in the same way. The AI is
Speaker 4: working like the corporation, which is working like the government,
Speaker 4: and the same problems of information managements affect them all.
Speaker 4: But it's not as nihilistic as you think, in my view,
Speaker 4: because that means that these things can be so subject
Speaker 4: to the same kinds of solutions. You know, if we
Speaker 4: think about the original reason for building the accountability sink,
Speaker 4: it was that someone felt overwhelm us by information and
Speaker 4: so didn't feel that they were responsible for the decision
Speaker 4: and so wanted to cut the link of accountability. If
Speaker 4: you can put the AI in the loop in such
Speaker 4: a way that the decision maker is more able to
Speaker 4: manage their information flow, then they don't have so much
Speaker 4: need to break the feedback links because they've got more
Speaker 4: functional ways to deal with them.
Speaker 3: I have a theoretical I have a theoretical question, which is,
Speaker 3: what does accountability actually look like for a decision taken
Speaker 3: by an algorithm? Is it that, like we understand the
Speaker 3: factors that went into the model, and like the decisions
Speaker 3: within the model that spat out a particular outcome, or
Speaker 3: is it that the person who is using the ourg go,
Speaker 3: you know, decides to think more thoughtfully about how they're
Speaker 3: using it.
Speaker 4: It's a that's a really interesting question which I'm going
Speaker 4: to think for two seconds, delight and waffle before answering,
Speaker 4: which is that I think the basic definition of accountability
Speaker 4: in the decision sense, in my view, is that you're
Speaker 4: accountable for a decision exactly to the extent that you
Speaker 4: are able to change it. So, in terms of a
Speaker 4: decision made by an AI, it is accountable if it
Speaker 4: could be made to make a different decision by new
Speaker 4: information being provided to it. So the crucial thing is
Speaker 4: not so much having someone to point at and attribute
Speaker 4: moral responsibility to. The crucial thing is to have some
Speaker 4: link between the subject of the decision. Who can just say,
Speaker 4: let's review this, let's have a court of appeal. And
Speaker 4: you know, if the algo still thinks that this decision
Speaker 4: needs to be done, if the elgo still thinks I'm
Speaker 4: not an insurable risk, then maybe I've been Maybe I
Speaker 4: still don't agree with that, but at least I know
Speaker 4: I've been heard. It's not coming to me just simply
Speaker 4: as a one way communication channel.
Speaker 2: You know, making a more optimistic view. So you said
Speaker 2: something interesting or important, which is that for a company,
Speaker 2: the financial numbers are the closest thing, the closest form
Speaker 2: of information that is at least like objective in some sense.
Speaker 2: But then there's all these other things like how how
Speaker 2: well is your engineering team working together? How is the
Speaker 2: culture that are just inherently much more difficult. And they
Speaker 2: are all kinds of like consultants and other companies that
Speaker 2: try to like answer this for executives and you know,
Speaker 2: rank employees different things. Aaron Levy, you know, he's the
Speaker 2: CEO of Box and he's one of the few like
Speaker 2: tech CEOs who tweets some interesting stuff from time to time.
Speaker 2: But he said he had this recent thread about how
Speaker 2: his company is using AI, and he said, you know,
Speaker 2: the exciting thing is, you know, we have some data,
Speaker 2: but then we have all this unstructured data that exists
Speaker 2: in our company, and we've never been able to do
Speaker 2: with anything, probably like chat logs from customer support and
Speaker 2: all this. And he said, the exciting thing for them
Speaker 2: is the prospect of turning all of this sort of unstructured,
Speaker 2: unusable information that the company has into something that can
Speaker 2: be essentially searchable and that insights can be gleaned from it.
Speaker 2: Is there a story or is there a path in
Speaker 2: your view where artificial intelligence can actually make some of
Speaker 2: these other parts of a system more legible and more
Speaker 2: interactive and more concrete to the executives in a way
Speaker 2: that brings it those that other data on par with
Speaker 2: the financial data.
Speaker 4: I mean, I really hope. So, I mean, like kind
Speaker 4: of one thing that Aaron Levy could do that would
Speaker 4: be really radical would be to open up as much
Speaker 4: of that data as possible to the investors and let
Speaker 4: them pass it and have that as a main channel
Speaker 4: of communication of corporate performance. Rather than generally accepted accounting principles,
Speaker 4: because if you think about that phrase, that's an accountability
Speaker 4: sink right there. What are these accounting principles? They're generally accepted.
Speaker 4: Can I change them? No, that would not be generally accepted.
Speaker 4: What if this is completely irrelevant set of metrics to
Speaker 4: my business? Well, that you still have to do it
Speaker 4: in exactly this way, even if it's not presenting what
Speaker 4: you think is actually generating value. And in a world
Speaker 4: where we've got better ways of processing bulk information like that,
Speaker 4: then I think there's a real question about whether gap
Speaker 4: is something that we should be so fixated on, whether
Speaker 4: we should be thinking that the only way to report
Speaker 4: corporate performance is in a way that's optimized. Basically, for
Speaker 4: a guy with a greener eyeshade sitting at a desk
Speaker 4: in the beginning of the twentieth century, flip through printed
Speaker 4: reports and accounts, you know, that's not the way that
Speaker 4: we process information anymore. And so maybe that shouldn't be
Speaker 4: the way that we report information anymore, because, as you say,
Speaker 4: these assumptions that go into gap earnings start driving decisions
Speaker 4: and they were never meant to drive decisions.
Speaker 2: It's interesting Tracy now thinking about it in this financial sense,
Speaker 2: how many of these accountabilities sinks? Like even like performance benchmarks, right,
Speaker 2: it's like, oh, we beat the S and P or whatever.
Speaker 2: It's like, why the S and P etcetera. Well, you
Speaker 2: know it's there, right, we could point to it and
Speaker 2: we could say it. But like once you start thinking
Speaker 2: of them in all of the indices and measures we cite,
Speaker 2: you could see, Tracy, how they like serve that purpose
Speaker 2: of just like yeah, look this is what we measure against.
Speaker 3: Oh yeah, of course. I mean incentives matter, right, Like
Speaker 3: that's something that we say over and over again on
Speaker 3: this podcast and when it comes to accounting. Okay, just
Speaker 3: to push back a little bit, but like there is
Speaker 3: an argument to have standardized accounting rules so that we
Speaker 3: don't always end up with companies running off and creating
Speaker 3: community adjusted ebit dah and things like that. But on
Speaker 3: the subject of incentives, I wanted to ask, you know,
Speaker 3: I read David Graber's Bull Jobs this year and it's
Speaker 3: still sort of looming large in my memory, and I
Speaker 3: guess my question is how much overlap is there between
Speaker 3: the accountability issues that you describe in your book and
Speaker 3: the way specific jobs, especially middle management are structured. And
Speaker 3: I don't mean to trigger you because I know that
Speaker 3: you mentioned in the book that you got into it
Speaker 3: a little bit with Graber over a different subject. But
Speaker 3: if you want to talk about that too.
Speaker 4: Oh, I missed David so much because we used to
Speaker 4: wind each other up so badly, and I got into
Speaker 4: a different argument that's not mentioned into the book over
Speaker 4: bull jobs. Because it's just like, if you're saying that
Speaker 4: middle management is a bulk job, then you're saying that
Speaker 4: the Sarahbellum is a bull's organ The middle management exists
Speaker 4: precisely because of all the metrics, and all the financial
Speaker 4: and non financial metrics on the chief executive's dashboard are
Speaker 4: massive information reducing filters. The middle managers are the people
Speaker 4: who carry the knowledge of the ways in which those
Speaker 4: metrics can misrepresent reality and how to cure the problems
Speaker 4: which arise when they are. You know, when someone's in
Speaker 4: danger of making a decision. The first thing a bad
Speaker 4: company does before it creates something like the Libuil scandal
Speaker 4: or the seven three seven mags is thin out the
Speaker 4: ranks of its middle management. And this particularly without wanting
Speaker 4: to relate the gate arguments with David Graeber, particularly since
Speaker 4: he's not around any more to answer back. In his
Speaker 4: day job as an anthropologist, David was so subtle and
Speaker 4: intelligent about the ceremonial roles of elders and people who
Speaker 4: built census among hunter gatherers to decide on what bands
Speaker 4: they would do. And then when you have those exact
Speaker 4: same problem solving and dispute resolving jobs happening, you know,
Speaker 4: in the offices at Bloomberger at a law firm, suddenly
Speaker 4: he thinks that they're bulk jobs. So that's a bit
Speaker 4: of a personal hobby horse. But basically all those or
Speaker 4: very many of those roles are actually the preservation of
Speaker 4: the information systems and the memory of organizations.
Speaker 2: Who is Stafford Beer and why why does he loom
Speaker 2: so large in the story you tell about the world.
Speaker 4: Well, Stafford Bit was the father of management sebernetics. He
Speaker 4: was the guy who first said you can take the
Speaker 4: mathematics of information theory and apply it to industrial organization.
Speaker 4: He was also David Bowie's favorite management consultants. He was very,
Speaker 4: very influential on Brian Eno and the divilment of ambient music.
Speaker 4: He was a hippie. He tried to it's not clear
Speaker 4: that this was a joke, but he did try to
Speaker 4: invent a computing pond where the growth of algae would
Speaker 4: correspond to the solutions of differential equations.
Speaker 3: He's going to be my summer project. I got a
Speaker 3: pond with algae.
Speaker 4: Yeah. The problem was that he used to feed them
Speaker 4: iron filings to make them grow, and after a while
Speaker 4: the entire pond became magnetic. But he was just this crazy,
Speaker 4: larger than life figure who did these incredibly successful management
Speaker 4: consulting assignments, but just somehow never quite was able to
Speaker 4: get on with enough people in the corporate world to
Speaker 4: really get his ideas across. And then he ended up
Speaker 4: in Chile in nineteen seventy two with this incredibly romantic
Speaker 4: but ultimately doomed project to reinvent socialism for the twenty
Speaker 4: first century under the ide governments, which I mean realistically
Speaker 4: it was never going to work because the confuting resources
Speaker 4: were completely disproportionate to the task. But it never got
Speaker 4: a fair trial, obviously, because the Pinochet coup happened about
Speaker 4: eleven months after he started the project.
Speaker 2: So why do we talk a little bit more about
Speaker 2: like the current day? You know, I started in the
Speaker 2: conversation like kind of not disagreeing, but questioning the premise,
Speaker 2: like the world has lost its Has the world really
Speaker 2: lost its mind? Or just the three of us in
Speaker 2: this conversation getting old and angry? What do you see
Speaker 2: like when you like think about like applying. You know,
Speaker 2: we talked about Boeing and you mentioned the live war scandal.
Speaker 2: But how are these things when you look around the
Speaker 2: world today and you look around whatever apparent world losing
Speaker 2: its mind? What are you seeing?
Speaker 4: I think the number one thing I'm seeing, And this
Speaker 4: is a point where David Graeber had it absolutely right,
Speaker 4: is debt. You know, we have so many cases at
Speaker 4: presence of companies where you have plenty of people who
Speaker 4: know exactly what they need to do, what investments they
Speaker 4: want to make, but they can't have any plans which
Speaker 4: stretch out any further than the next debt repayment. And
Speaker 4: to a large extent, that's because of leverage buyouts and
Speaker 4: managements acting in anticipation of the risk of a leverage buyout.
Speaker 4: But this really is a degradation of the higher functions,
Speaker 4: the brain functions of the corporations of the anglosphere world.
Speaker 4: The practice of firing middle managers because you don't know
Speaker 4: what they do is also demonstrably, in my view, making
Speaker 4: corporations stupid. It might have been that at the start
Speaker 4: of the LBO boom in the seventies there were too
Speaker 4: many of these guys on soft jobs with country club
Speaker 4: memberships and private jets and whatnot, but we've clearly gone
Speaker 4: too far in the other direction in my view. And
Speaker 4: then we've got the frightening tendency of government organizations to
Speaker 4: outsource absolutely critical functions, which means that all of the
Speaker 4: knowledge of the systems that they're meant to be regulating
Speaker 4: and dealing with. What's an example of that, right, best
Speaker 4: example currently thinking, I think m just the question of
Speaker 4: infrastructure building, for example in the UK, where we have
Speaker 4: there's a river crossing in to the east of London
Speaker 4: which is being responsible for generating the largest pile of
Speaker 4: paper ever brought together in one place in the history
Speaker 4: of humanity. And the reason for that is that the
Speaker 4: people who are meant to be deciding what an appropriate
Speaker 4: level of consultation is don't know anything about environmental impact
Speaker 4: studies and building bridges anymore. So they commission results reports,
Speaker 4: the commission reports from professional services firms and professional services
Speaker 4: firms want to generate repeat business, and so you've got
Speaker 4: this situation where the people who are meant to be
Speaker 4: seeing the whole system as a system don't really understand
Speaker 4: it anymore because they've outsourced all of their engineering knowledge,
Speaker 4: all of their economic and environmental knowledge. And as a result,
Speaker 4: the UK or the London Department for Transports are going
Speaker 4: to be paying as much as ten times as much
Speaker 4: as it should reasonably cost to build a bridge over
Speaker 4: the River Thames.
Speaker 3: It's interesting that you single out debt as a sort
Speaker 3: of deciding factor versus share price, because this is the
Speaker 3: one we hear a lot about in the context of
Speaker 3: corporate short termism and people usually trying to hit specific
Speaker 3: share price metric that may or may not be tied
Speaker 3: into their compensation.
Speaker 4: Yeah. I think I'm right on that. I know that
Speaker 4: people disagree with me, but you know, we had share
Speaker 4: prices in the fifties and sixties and we didn't have
Speaker 4: this kind of problem of short termism. The difference is
Speaker 4: that now we've got takeovers, particularly private equity and LBOs,
Speaker 4: but also in general the use of debt in takeovers,
Speaker 4: and that makes the share price more salient because any
Speaker 4: incumbent management knows that if the share price falls, it
Speaker 4: makes them vulnerable to a takeover. So to my mind,
Speaker 4: I think it's not so much you know, the share
Speaker 4: price as the exaggerated importance of short term financial metrics,
Speaker 4: which is partly through the share price, but much more
Speaker 4: just simply because of leverage.
Speaker 2: One of the things that you know, you mentioned the
Speaker 2: UK bridge and it ten times more expensive then needs
Speaker 2: to be in a pile of paperwork. I mean, this
Speaker 2: is probably the number one thing that like, you know,
Speaker 2: people I follow on Twitter talk about all the time,
Speaker 2: which is just how hard it is to build anything
Speaker 2: in the United States and the interlocking systems of environmental
Speaker 2: regulations and nimbi's and everything else, and it's the big
Speaker 2: challenge of the IRA and it feels like listening to this,
Speaker 2: it's just accountability sink. After accountability sink is to blame.
Speaker 4: It's absolutely and it's accountability sinks being put up because
Speaker 4: the people who were meant to be taking the decisions,
Speaker 4: and who in the nineteen fifties and sixties did take
Speaker 4: the decisions, are no longer really able to They've not
Speaker 4: got the confidence of their decisions. They're not sure they're
Speaker 4: going to be able to defend them in litigation, and
Speaker 4: it's mainly because they've lost their executive functions with successive
Speaker 4: staff cuts and retirements and outsourcing contracts.
Speaker 2: Dan Davies, It's so great to catch up with you.
Speaker 2: It's fascinating conversation, fascination way of thinking about the world.
Speaker 2: Highly recommend everyone check out your book, The Unaccountability Machine.
Speaker 2: Thank you so much for coming back on outlag.
Speaker 4: Oh, thanks to so much.
Speaker 2: Oh is a pleasure, Tracy, I really I love talking
Speaker 2: to Dan. First of all, it's always I just like
Speaker 2: hearing his voice. You know this, I do think like
Speaker 2: accountability sinks is now going to be one of those
Speaker 2: phrases that I'm just going to now start seeing everywhere.
Speaker 2: And you know there's whole industry like McKinsey right, Like
Speaker 2: that's like, you know, it's like again someone else to
Speaker 2: lay off your workers, et cetera. Like you just start
Speaker 2: seeing how big that is everywhere.
Speaker 3: Well, this is what I was thinking. You know, you
Speaker 3: brought in the US example of building infrastructure or other
Speaker 3: energy products, and I kept thinking back to Jiggershaw and
Speaker 3: his point about the lack of institutional memory of how
Speaker 3: to build nuclear power plants. Right, it's not necessarily that
Speaker 3: it's so complicated to get environmental permits and things like that,
Speaker 3: although that is certainly part of it. But it's also
Speaker 3: that the people who used to do this haven't been
Speaker 3: doing it for a long time or are no longer around.
Speaker 3: And that kind of goes to Down's point about middle
Speaker 3: management being the sort of like what's the word I'm.
Speaker 2: Thinking connective tissue.
Speaker 3: Connective tissue is a good one of like institutional memory.
Speaker 2: Yeah, no, it's only true, you know, in defense of
Speaker 2: the the Nimbi's. You know, I keep mentioning this New
Speaker 2: York documentary and watching and I got to the yeah,
Speaker 2: I want to watch that, you gotta watch it. But
Speaker 2: I got to the episode where like really talks about
Speaker 2: like Robert Moses and just like plowing these big highways
Speaker 2: through neighborhoods and putting up these like terrible, like terrible
Speaker 2: housing projects that are like, you know.
Speaker 3: Sort of right, continuously prioritizing highways.
Speaker 2: That is someone who did not have the problem of
Speaker 2: Nimbi's or a million different interlocking constraints on him, Like
Speaker 2: there are our drawbacks to when someone has like too
Speaker 2: much autonomoy and it's sort of like, yeah, but now
Speaker 2: it does seem arguably we've gone too far in the
Speaker 2: other direction, in which everyone just clings to their accountability
Speaker 2: sink and can't get it right done.
Speaker 3: Everything is a collective decision therefore can be held responsible. Yeah,
Speaker 3: I feel like there must be a reasonable middle ground.
Speaker 3: And yet I don't know. I'm trying to think if
Speaker 3: I know of, like any organizations that have completely cracked
Speaker 3: like the nut just for a little while. Yeah, collectivism
Speaker 3: versus individual responsibility, I don't know. Well, on that note,
Speaker 3: shall we leave it there?
Speaker 2: Let's leave it there.
Speaker 3: This has been another episode of the All Thoughts podcast.
Speaker 3: I'm Tracy Alloway. You can follow me at Tracy Alloway and.
Speaker 2: I'm Joe Wisenthal. You can follow me at The Stalwart.
Speaker 2: Follow our guest Dan Davies. He's the author of the
Speaker 2: book The Unaccountability Machine, Why Big Systems Make Terrible Decisions
Speaker 2: and How the World Lost its Mind.
Speaker 3: Go check it out.
Speaker 2: His handle is at d squared Digest. Follow our producers
Speaker 2: Carmen Rodriguez at Carmen Arman, dash Ol Bennett at Dashbot
Speaker 2: and Kilbrooks at Kilbrooks. Thank you to our producer Moses Ondem.
Speaker 2: For more Oddlags content, go to Bloomberg dot com slash
Speaker 2: odd Lots, where we have transcripts, a blog, and a
Speaker 2: newsletter and you can chat about all of these topics
Speaker 2: twenty four to seven in our discord Discord dot gg
Speaker 2: slash odd lots.
Speaker 3: And speaking of personal accountability. If you like odd Lots,
Speaker 3: please leave us a positive review on your favorite podcast platform.
Speaker 3: And remember, if you are a Bloomberg subscriber, you can
Speaker 3: listen to all of our episodes absolutely ad free. All
Speaker 3: you need to do is connect your Bloomberg account with
Speaker 3: Apple Podcasts. In order to do that, you can find
Speaker 3: the Bloomberg channel on Apple Podcasts and follow the instructions there.
Speaker 3: Thanks for listening.