Speaker 1: All Zone Media.
Speaker 2: Hello and welcome to Better Offline. I'm your host ed Zichron.
Speaker 2: Now you may have forgotten, but about a month or
Speaker 2: two ago we had a huge, complete meltdown of the
Speaker 2: computer systems of the world when CrowdStrike failed. I did
Speaker 2: an episode on it, and well we've all just kind
Speaker 2: of forgotten about it. And today I'm joined by Christokal Walker,
Speaker 2: who's an author, journalist, lecturer and starting new column at
Speaker 2: The Guardian as well.
Speaker 1: Aren't you yeah for a whole weeknad, I'm a whole month.
Speaker 1: I'm taking over tech space, which will be very exciting
Speaker 1: for the month of September.
Speaker 2: But the reason I brought you on is you wrote
Speaker 2: a great article for The Independent back at the end
Speaker 2: of July about how CrowdStrike isn't the only cyber company
Speaker 2: that could trigger a glow, will melt down the second
Speaker 2: they fail. And this is a subject that fascinates me
Speaker 2: because I love disaster movies and also this article was terrifying,
Speaker 2: So why don't you walk me through it?
Speaker 1: Yeah, So, basically, back in July we had that odd
Speaker 1: outage that people might remember. Basically people woke up in
Speaker 1: Asia Australia and then eventually the UK and encountered blue
Speaker 1: screens of death on Microsoft Windows, which is something amazing
Speaker 1: that you know. I'm thirty five now and I remember
Speaker 1: that happening when I was a kid, and then it's
Speaker 1: never happened since, like for all that, you know, people
Speaker 1: make fun of Microsoft. Actually their PCs are decent, but
Speaker 1: then suddenly everything went to hell in the handbasket. So
Speaker 1: it turns out that, you know, CrowdStrike, which is one
Speaker 1: of the big service providers for kind of antivirus tools
Speaker 1: and software, had mist configured basically the thing that protects
Speaker 1: us so that it actually harmed us, which is just
Speaker 1: the ultimate in fantastic arriagories.
Speaker 2: And it turned out that it was actually within CrowdStrike,
Speaker 2: the thing that was bugged failed because the bug checking
Speaker 2: thing had a bug in it, which is so good.
Speaker 2: It's so good that we have everything built on.
Speaker 1: Yeah, and this is this is this is the thing,
Speaker 1: right it is? And I guess this is This is
Speaker 1: something that a lot of your listeners and others will
Speaker 1: chime with because you share a similar sensibility to me,
Speaker 1: which is that we have built a huge thing which
Speaker 1: is kind of like on a house of cards that
Speaker 1: is actually hiding the fact that humans are involved in this,
Speaker 1: and humans screw up frequently. Yet we think that actually,
Speaker 1: because this is wiz bang tech, you don't have to
Speaker 1: worry about it. It will worry of automation, Yeah, precisely,
Speaker 1: And yet it's not as some guy who's overworked, overtired
Speaker 1: and can't type properly and has fat fingers like you mean.
Speaker 2: Yeah, And a combination of private software companies like CrowdStrike,
Speaker 2: and then as you'll get into open source solutions that
Speaker 2: are a lot of people doing them for the love
Speaker 2: of the game, which is wonderful. But at the same time,
Speaker 2: we've got this patchwork system that holds up the Internet
Speaker 2: and a lot of the tech we rely on and
Speaker 2: we don't really know, and the fact that so much
Speaker 2: of it is automated is also terrifying.
Speaker 1: Yeah, And this is the thing that that update was
Speaker 1: sent out to millions of PCs over the course of
Speaker 1: an evening while people were sleeping, and people didn't realize
Speaker 1: that it was a massive issue with it until they
Speaker 1: started to wake up, and at that point it is
Speaker 1: essentially too late. And the best part of this story,
Speaker 1: I think, is that to fix the issue, and we
Speaker 1: have kind of fixed the issue a couple months on.
Speaker 1: For a lot of them, you had to actually get
Speaker 1: individual people to go out to either a computer or
Speaker 1: a terminal somewhere and put in the actual updates to
Speaker 1: unscrew up the problem that had existed initially, but you
Speaker 1: highlighted there to kind of the issue of how much
Speaker 1: of this is seen as kind of public utilities nowaday
Speaker 1: is like the plumbing of our kind of entire world,
Speaker 1: and yet it is kind of really rickety and hell
Speaker 1: together with scotch tape and a little bit of chewing gum.
Speaker 1: I did think a story a decade ago about how
Speaker 1: we had a similar issue with a thing called heartleaed,
Speaker 1: which was another co So basically it was another similar
Speaker 1: thing where there was an update to a thing called
Speaker 1: open SSL, which is the software tool that encrypts all
Speaker 1: the data that is sent through payment systems and passwords
Speaker 1: and so on and so forth. So whenever you see
Speaker 1: that kind of yellow lock on your web browser, that
Speaker 1: is usually running open SSL, and there was an issue
Speaker 1: with it, which essentially meant that all of the information
Speaker 1: that people thought was being shared in encrypted form was
Speaker 1: actually being shown in plain texts. So someone could in
Speaker 1: theory come along and snoop on everything that you're putting
Speaker 1: in there, from bank account details to passwords and so
Speaker 1: on and so forth. The reason why that happened is essentially,
Speaker 1: the Internet at the time and to a certain extent
Speaker 1: now or less so, was being run by two guys
Speaker 1: named Steve. The whole thing was developed by a guy
Speaker 1: called Steve Marquess and his friend who was also called Steve.
Speaker 1: They were kind of this weird transatlantic romance where they
Speaker 1: kept going open SSL. It was this initial volunteer run
Speaker 1: project that kind of became a key part of the
Speaker 1: Internet that frankly, these multi billion dollar companies used day in,
Speaker 1: day out, but they decided that they didn't really want
Speaker 1: to pay a penny for the upkeep and.
Speaker 2: Open SSL is one of the named kind of things
Speaker 2: that could break the entire Internet. From your article as.
Speaker 1: Well, Yeah, this is kind of the key part of
Speaker 1: the Internet's plumbing. And there are loads of these, right Like,
Speaker 1: this is the thing that we don't realize until things
Speaker 1: go wrong, and generally in it, people want to have
Speaker 1: ninety nine point nine nine nine percent up time until
Speaker 1: you have that kind of miniature final element where actually
Speaker 1: something does screw up and actually you start to recognize
Speaker 1: that this thing is held together pretty precariously. We don't
Speaker 1: realize it, but there is a sort of cabal of
Speaker 1: half a dozen or so companies whose job is essentially
Speaker 1: to keep this stuff running, and sometimes they do screw.
Speaker 2: Up and open SSL. How was that actually funded? Is
Speaker 2: it donations or Yeah?
Speaker 1: So Initially at the time, back in twenty fourteen, when
Speaker 1: heartle was Hartleed was kind of an issue. They were
Speaker 1: essentially relying on to nations. This was open source software
Speaker 1: and that it's kind of the basic principle of the web.
Speaker 1: We often forget about this. Actually it is hobbyists that
Speaker 1: set this thing up alongside a huge military industrial.
Speaker 2: Company XKCD comic, which is everything's held up by a
Speaker 2: guy called Runk.
Speaker 1: Yeah. Yeah, it turns out it was actually Steve but
Speaker 1: basically the same sort of thing, and that is the
Speaker 1: problem is they weren't fully funded. I did a follow
Speaker 1: up story back in twenty fourteen because Hertleed kind of
Speaker 1: drew the attention to this. And it comes back to
Speaker 1: that idea of how when you started our podcast here
Speaker 1: where you said this thing happened, this huge chaotic thing,
Speaker 1: and you've probably forgotten about it because it's been two
Speaker 1: months and we moved on same thing with heartleads. This happened,
Speaker 1: there was this huge outcry. They got a bit of
Speaker 1: funding around about a million dollars that was meant to
Speaker 1: kind of make them back on an even keel and
Speaker 1: so that they could be sustainable. Reality is kind of
Speaker 1: didn't work.
Speaker 2: What do you mean it didn't work.
Speaker 1: Well in the sense of they still had this issue
Speaker 1: happen again and again. So what was meant to be
Speaker 1: kind of a fronting up. I suppose of big tech
Speaker 1: companies saying, actually, you know what, we recognized that this
Speaker 1: hobbyist service is a vital part of our internet's running.
Speaker 1: We will fund it so it is sustainable. Didn't necessarily happen,
Speaker 1: not to the extent that we've had another heartlead from
Speaker 1: open SSL. But they do struggle still to kind of
Speaker 1: keep things going. And that is I think the big problem,
Speaker 1: which is the news agenda moves on. People forget very quickly,
Speaker 1: and because there is then not a problem for a
Speaker 1: little while longer, we kind of lurch from one catastrophic
Speaker 1: near miss to another.
Speaker 2: So open SSL what does it actually do? Though? I
Speaker 2: know you it's the little padlock on browsers. But what
Speaker 2: is its foundational point?
Speaker 1: Yeah, so it basically it shepherds across data from from
Speaker 1: a user to kind of a service provider. So if
Speaker 1: you think about it as kind of you input text
Speaker 1: on your laptop, your phone, wherever you are. It will
Speaker 1: then encrypt it. It will transfer it over to a
Speaker 1: payment provider, to your bank, to frankly also pretty much
Speaker 1: anywhere that you put a password in and it will
Speaker 1: ensure that that is encrypted all away. But there was
Speaker 1: an issue with the coding of it, which meant that actually,
Speaker 1: again comes back to fat fingers, some elements of what
Speaker 1: was being transferred went into kind of excess memory, which
Speaker 1: basically meant that bits of it were encrypted, but then
Speaker 1: large parts of it weren't. So if you were unlucky,
Speaker 1: then the bits that weren't encrypted could be your credit
Speaker 1: card details, your sort code, and your account number and
Speaker 1: a CBC number. And that's why there was kind of
Speaker 1: this big red flashing light back in a decade or
Speaker 1: so ago where people thought, you know what, actually, this exploit,
Speaker 1: if left unchecked, could become a massive issue and a
Speaker 1: real boon for cyber criminals. I think actually had it
Speaker 1: happened in twenty twenty four, we would have seen much
Speaker 1: more of a sort of significant issue in terms of
Speaker 1: second third or more ramifications because cybercriminals would have been
Speaker 1: all over that stuff.
Speaker 2: And if open SSL breaks again, and that just means
Speaker 2: that the Internet it is not really encrypted, but every
Speaker 2: place that uses it is kind of at risk every
Speaker 2: transaction on every place.
Speaker 1: Yeah, and that is that is the issue, that is
Speaker 1: the high wire act that is the Internet. And we've
Speaker 1: seen outages like this come and go time and time again.
Speaker 1: You know, we've had the crowd strike instant where you
Speaker 1: can actually get onto your desktop or your laptops. We've
Speaker 1: had outages on major payment platforms. We've had banks going offline,
Speaker 1: we've had social networks kind of disappearing for hours at
Speaker 1: a time. And invariably, this is just a very simple
Speaker 1: error that snowballs repeatedly and we're kind of doomed to
Speaker 1: repeat it. And I guess the challenge is, like, how
Speaker 1: do we put the web on a firmer footing that
Speaker 1: prevents this from happening again and again.
Speaker 2: It feels like funding the open SSL movement of foundation
Speaker 2: would probably be a good start, But let's I imagine
Speaker 2: and that's not happening. We need to build the computer
Speaker 2: that makes pictures of Garfield with a gun exactly. This
Speaker 2: is the thing.
Speaker 1: Commercial interests always come into this, and the reality is,
Speaker 1: as you and I both know, and as many of
Speaker 1: our listeners will know, companies tech companies in particular, will
Speaker 1: take action when they realize that the spotlight is on
Speaker 1: them and that there is this kind of intrinsic demand
Speaker 1: for them to do that as soon as it's off,
Speaker 1: they're actually doing the same old thing. They are happy
Speaker 1: with the status quo as it is.
Speaker 2: Which is crazy as well, because what worries me about
Speaker 2: open SSL is that nobody will really be to blame
Speaker 2: and thus nobody will really feel responsible. They might kick
Speaker 2: in some money here and there. Google especially very reliant
Speaker 2: on them, but I just don't see them doing it.
Speaker 1: Yeah, and this is the thing that the only people
Speaker 1: that I really noticed when I was reporting out that
Speaker 1: story a decade ago who felt any kind of like
Speaker 1: guilt or personal alarm kind of just like even responsibility
Speaker 1: for it were actually those two Steves, like they were
Speaker 1: they will gut it. The story I did, Yeah, the
Speaker 1: story I did for bus you back then was it
Speaker 1: was very difficult to report out because they had been
Speaker 1: taken out of not to overly stereotype, but they were
Speaker 1: like super tech, nuity people like they they were very
Speaker 1: happy being in the background tinkering with this thing. They
Speaker 1: realized that it was important and they took their jobs
Speaker 1: very seriously, but they had never been put in a spotlight,
Speaker 1: and they were initially very wary of speaking to me
Speaker 1: because suddenly this thing happened. And it's over the course
Speaker 1: of like hours that they got kind of thrust into
Speaker 1: the limelight. They had the daily mail knocking at their door,
Speaker 1: which was one of the reasons why they were the
Speaker 1: super wary of talking to me. So it took actually
Speaker 1: a few days of winning them over and saying, you
Speaker 1: know what, like this isn't going to be a hit job,
Speaker 1: Like I'm not looking to kind of hold you up
Speaker 1: and say this is the person responsible for this happening.
Speaker 1: It's more I wanted to tell the story of why
Speaker 1: this has happened, why it's an issue, and why we
Speaker 1: shouldn't have the ability for kind of slight errors in
Speaker 1: upkey do cause catastrophic effects.
Speaker 2: It almost feels like people getting mad at the homeless
Speaker 2: to some extent where it's like, oh, this person is
Speaker 2: on the street and there are problems that are happening
Speaker 2: around them, and they are and you blame the person
Speaker 2: who is the victim here. You blame the fact that
Speaker 2: when you look at the Internet right now and it's instability,
Speaker 2: you're like, oh, well, these open source people who are
Speaker 2: doing it for free, it's therefore because they should have
Speaker 2: fucking they should have been better at doing this thing
Speaker 2: for free. That holds up the entire versus the fact
Speaker 2: that the problem is that the entire Internet relies on
Speaker 2: this underfunded group of people, and it really is. I
Speaker 2: just want to be clear for listeners and Chris you
Speaker 2: of course know this. When I say this holds up
Speaker 2: the entire Internet, I do actually mean that. It's very
Speaker 2: easy to fall vout to hyperbole sometimes, but this is
Speaker 2: genuinely that level. When Heartbleod happened, it sounds like it
Speaker 2: could have been truly catastrophic.
Speaker 1: Good. It really good. And this is the thing that
Speaker 1: we overlook is either there are kind of not amateurs,
Speaker 1: because these people are super professional in terms of what
Speaker 1: they do and they take their jobs around seriously, but
Speaker 1: they are either not paid or they're paid a pittance,
Speaker 1: particularly in comparison to the total compensation packages that you
Speaker 1: see washing around Silicon Valley. And yet there is this
Speaker 1: kind of super extractive approach from big tech companies of
Speaker 1: we will kind of roll in whatever it is that
Speaker 1: you provided us soften open source. And this is the
Speaker 1: kind of big secret right of A large part of
Speaker 1: big techs success is they rely on these open source
Speaker 1: developments that have kind of underpinned key parts of their tech,
Speaker 1: and if things go wrong, they can always shift the
Speaker 1: blame onto those open source things and say, well, this
Speaker 1: isn't actually our fault, this is the fault of our supplier,
Speaker 1: the thing that came second or third order down the line.
Speaker 2: And you've got this big movement in cloud as well
Speaker 2: towards like composable architecture, which involves a lot of slotting
Speaker 2: in open source solutions as well. It's just it almost
Speaker 2: feels like we need a big tech mutual aid thing
Speaker 2: for open source. I wish that. I don't think there's
Speaker 2: any way we could get a government to do this,
Speaker 2: but I think they should force big tech to put
Speaker 2: like a percentage of revenues, not profits, into open source
Speaker 2: and have very defined lairs for them, because otherwise you
Speaker 2: get situations where I don't know, the entire Internet is
Speaker 2: underpinned by two steeves.
Speaker 1: Yeah, exactly, And I think this is this is the
Speaker 1: unfortunate thing is that should be the lesson that we learned.
Speaker 1: It should have been a lesson that we learned from
Speaker 1: heart bleed. It should have been the lesson that we
Speaker 1: learned from xe utils, which was another issue that we
Speaker 1: encountered relatively recently, where there was it turned out, we
Speaker 1: believe a bad actor kind of deliberately inserting milicious code
Speaker 1: into another thing that kind of underpinds large parts of
Speaker 1: our digital lives. Turns out that the volunteers that were
Speaker 1: running that couldn't keep track of it. One of them
Speaker 1: literally talked about their burnout and how they've kind of
Speaker 1: taken a step away from the project. And yeah, we
Speaker 1: always see these things kind of passing by in the
Speaker 1: rearview window. We say, oh, you know what, isn't that
Speaker 1: such a shame? We ought to do something about that,
Speaker 1: And then we move on to the next thing, and
Speaker 1: we don't.
Speaker 2: Pay attention taking a step back. What was exe utils
Speaker 2: for the American listeners, ex for British and Canadian what
Speaker 2: happened there?
Speaker 1: Yeah, this is kind of again another bit of free software.
Speaker 1: This was back in sort of spring of twenty twenty four.
Speaker 1: A kind of malicious hacker had, we think, basically socially
Speaker 1: engineered their way into the upkeep of this bit of
Speaker 1: open source utility, which is essentially designed to kind of
Speaker 1: compress data. So the idea was that it would kind
Speaker 1: of take a big file, chunk it up, make it smaller,
Speaker 1: get rid of the bits that you don't need. And
Speaker 1: it was kind of in the same way as you
Speaker 1: talked about cloud architecture slotting in lots of really useful
Speaker 1: open source tools, this is a similar thing where you
Speaker 1: could slot in exit utails x utails into whatever you're
Speaker 1: building and it would be fantastic. This guy had kind
Speaker 1: of offered to volunteer at a time when the original developers,
Speaker 1: the custodians of this tool were feeling very burned out,
Speaker 1: said I will help. The original person then took their
Speaker 1: eyes off the ball. This malicious actor started putting in
Speaker 1: back doors intentionally the ways of accessing kind of the
Speaker 1: data within. And it was only spotted basically by a
Speaker 1: Microsoft developer who happened to come across.
Speaker 2: This, and I said, and just to be clear, though
Speaker 2: exit utils looks like it's a big part of Linux,
Speaker 2: which is a bit which people who use the smosftware
Speaker 2: may not realize is basically underpinning most server architects. Like
Speaker 2: a ton of server architecture, a ton of web architecture.
Speaker 1: Right, yeah, so servers, web servers, cloud hosting tools, lovely webcams,
Speaker 1: basically anything that connected. Probably your fridge, if you have
Speaker 1: an Internet of Things fridge, if you are that frivolous,
Speaker 1: then that will be connected in some way to Linux.
Speaker 1: I hope not. I hope that your diet coke is
Speaker 1: not being kept cold by an IoT fridge.
Speaker 2: And there's someone who knows me so. But so this
Speaker 2: Microsoft developer found it. And so it turns out that
Speaker 2: just the corruption of open source happen like it can
Speaker 2: happen as well with these open source projects, particularly I
Speaker 2: imagine when they're underfunded and the people get burned out.
Speaker 1: Yeah, and this is this is the thing is again
Speaker 1: it's another example of a kind of hobby project that
Speaker 1: turned into something bigger. Nobody who has the money either
Speaker 1: realized or decided that it was important enough to fund
Speaker 1: directly the people involved who are often you know, again,
Speaker 1: I don't want to stereotype, and I don't want to
Speaker 1: kind of make them seem too much like a victim here,
Speaker 1: But these folks are often super humble, super helpful, just
Speaker 1: trying to keep their heads above water, essentially because they've
Speaker 1: made something that has proved very, very useful, and they
Speaker 1: don't want to trouble people by shouting for help. In
Speaker 1: In this case, same thing happened. Single person in charge
Speaker 1: of this tool didn't want to shout too loudly about
Speaker 1: the problems that it would cause him in his life.
Speaker 1: He kind of took a step away, decided to get
Speaker 1: help from someone. Turned out they were bad, and nobody
Speaker 1: decided to shout about it. So again this one was
Speaker 1: This one was more deliberate rather than the fucker.
Speaker 2: I would also say they are victims. These people are
Speaker 2: doing some of the most important work in the world
Speaker 2: while san Dhar Pashai gets two hundred million dollars a year.
Speaker 2: That I see these people as heroes and victims at
Speaker 2: the same time.
Speaker 1: Yeah, and I agree, I think. I think what I
Speaker 1: mean by that is they would not want to be
Speaker 1: seen as either the hero or the victim in the piece, right.
Speaker 1: They don't. They don't they have I think to be
Speaker 1: involved in open source software more generally, you have to
Speaker 1: have a very kind of uh it's almost quaint, right,
Speaker 1: You have to be very, very community minded, very kind
Speaker 1: of I am doing this for the good of everybody,
Speaker 1: but I also don't want the praise for it, and
Speaker 1: so either kind of being presented as like this hero
Speaker 1: defending us against all of the bad stuff, or being
Speaker 1: presented as the victim who is you need pity. I
Speaker 1: think that's the thing they don't want. They don't want right,
Speaker 1: but they just want money.
Speaker 2: And I think we get back to the systemic problem then,
Speaker 2: because I don't see them as like any kind of
Speaker 2: pathetic thing or indeed, I mean that I think that's
Speaker 2: something heroic about what they're doing. But I think what
Speaker 2: they're doing is cool. I think what the problem is
Speaker 2: the systemic lack of support for them. We blame these things,
Speaker 2: we don't but people may blame these projects for breaking
Speaker 2: Oh it didn't work as well as it should, But
Speaker 2: it turns out that it's just we we put all
Speaker 2: of this pressure and these requirements on these people and
Speaker 2: on these projects and then don't give them the sport
Speaker 2: at all. So naturally, I'm going to say to my listeners,
Speaker 2: please go and fund your open source movements. Brought on
Speaker 2: Molly with Wikipedia. Molly Molly White of course about Wikipedia
Speaker 2: very early on in the show. Fund these projects because
Speaker 2: they deserve it. But the funny thing is is that
Speaker 2: some of the sometimes I've seen very stupid idiots say
Speaker 2: things like, well, if they were fun, if they were
Speaker 2: private and corporate entities, they'd be fined and nothing would
Speaker 2: go wrong. Except we look at CrowdStrike and it's the
Speaker 2: complete opposite.
Speaker 1: Yeah, And the worst thing is with CrowdStrike is it's
Speaker 1: they thought they knew better, right, like this is this
Speaker 1: is the key thing. They thought that they were doing
Speaker 1: everything perfectly. They kind of crowed about how good their
Speaker 1: tools were, how well they could protect people, and then
Speaker 1: they didn't. And this is kind of the endemic thing
Speaker 1: is you you can't introduce profit to the equation because
Speaker 1: if you do your kind of you're looking to cut corners.
Speaker 1: Now we still don't really fully you know why this happened,
Speaker 1: and crowdstrikeer is still taking a hit to its business.
Speaker 1: It still has the threat of legal action from those
Speaker 1: customers who were affected, and they were an awful lot.
Speaker 1: You know, airlines were knocked off for basically an entire
Speaker 1: week in the United States, you couldn't get anywhere on
Speaker 1: some airlines because the systems were just so completely broken.
Speaker 1: But the idea that you can just kind of throw
Speaker 1: money at it through a big tech lens doesn't really
Speaker 1: work because you need that idea, that kind of ethos
Speaker 1: of I'm doing this not for profit, not for myself
Speaker 1: and not for the company that I work for, but
Speaker 1: I'm doing it for kind of the greater good. And
Speaker 1: I think the problem if you brought this into a
Speaker 1: Google or even a crowd strike or whatever, is that
Speaker 1: you end up looking at the bottom line and realizing, actually,
Speaker 1: I need to acquire customers. I need to keep them.
Speaker 1: That's going to be my focus, not just making good
Speaker 1: stuff and making it work.
Speaker 2: So onto profit seeking entities. One of the others you
Speaker 2: mentioned is Fastly, so fastly. Let's why do you walk
Speaker 2: me through Fastly? Because I know there are other companies
Speaker 2: in this realm too.
Speaker 1: Yeah, so fastly is it's kind of what you would
Speaker 1: call an edge cloud provider. So that is basically an
Speaker 1: attempt to try and bring the internet speeds up a
Speaker 1: little bit, make them a bit quicker. So the idea
Speaker 1: of bringing files out are commonly used, or websites that
Speaker 1: they're commonly used closer to where the users want to
Speaker 1: request them. The thing that people often overlook is that,
Speaker 1: you know, the web is essentially still a data transmission system,
Speaker 1: and so you have to If I was to pull
Speaker 1: up a YouTube video from my home in the UK,
Speaker 1: it would be very silly for me to put that
Speaker 1: request through YouTube servers in United States because I would
Speaker 1: have to send the request to the United States, the
Speaker 1: request would have to be fulfilled, YouTube would have to
Speaker 1: go looking for the video, It would then have to
Speaker 1: send the video back to me, and then it would
Speaker 1: have to be played. Now we're really talking about kind
Speaker 1: of a fraction of a second there, but it can
Speaker 1: be done quicker by serving it closer to me physically.
Speaker 2: A content delivery network like vastly exactly.
Speaker 1: So that is what Vastly does. The problem is that
Speaker 1: it went wrong around about three years ago, again like
Speaker 1: crowd strike, a misconfigured file got pushed out of the
Speaker 1: company systems. Because fastly is used by Amazon, by read,
Speaker 1: by Twitch, by the UK government, by PayPal, all of
Speaker 1: those platforms were affected, which is kind of a big issue.
Speaker 1: This guy you know who runs Fastly, hugely wealthy man
Speaker 1: and you know, has done an awful lot of good
Speaker 1: makes the Internet faster. But the problem is a game.
Speaker 1: It is a private company. It is a single point
Speaker 1: of failure for many many platforms and many websites that
Speaker 1: we use day in, day out, and so if something
Speaker 1: goes wrong, it goes really really wrong.
Speaker 2: And what's weird about that is you'd think that Amazon,
Speaker 2: for example, would have their own CDN.
Speaker 1: And they do have their own CDNs in some ways,
Speaker 1: but they still the part of the thing is these
Speaker 1: companies are so sprawling and these services that they provide
Speaker 1: are so huge that they tend to try and bucket
Speaker 1: them in different ways. And so while you know, the
Speaker 1: fast the element went down, they still had other bits,
Speaker 1: but it was kind of very much concentrate on Fastly.
Speaker 1: So that's why we had those outages there.
Speaker 2: And there are other companies like this, like Achimaize the
Speaker 2: other one, where if they buckle or fall, just chunks
Speaker 2: of the Internet fall offline.
Speaker 1: Yeah, and again it goes back to around about maybe
Speaker 1: got the late nineties early two thousands. We took a
Speaker 1: series of decisions that essentially decided we are going to
Speaker 1: take this thing that was previously like a kind of
Speaker 1: hobbyist's home developed by frankly amateurs but actually kind of worked.
Speaker 1: And we're going to turn this into like a massive
Speaker 1: profit making machine, and we're going to privatize large parts
Speaker 1: of it, and we're going to simultaneously have you know,
Speaker 1: big business and also kind of you know, public goods
Speaker 1: and services being transacted on it. And we've kind of
Speaker 1: existed in that awkward space forever, and you've done episodes
Speaker 1: in the past about loads of parts of social media
Speaker 1: and the fact that there is this kind of challenge
Speaker 1: of this is as Eezylon Musk's favorite raisor de facto
Speaker 1: public square, but it is based on essentially private land.
Speaker 1: And as soon as you kind of take what was
Speaker 1: initially kind of like an educational base communications network and
Speaker 1: you turn it into something that is for profit, you
Speaker 1: really complicate things in a way that means you have
Speaker 1: single points of failure and a lot of banks on
Speaker 1: those things working, and when they don't, it causes big.
Speaker 2: Issues and it's it. It is a bit worrying, and
Speaker 2: I try not to do too much fud on this show,
Speaker 2: but this is the stuff that actually keeps me up
Speaker 2: at night. This is the thing, especially as we have
Speaker 2: the increasing electricity use of AI as, especially as we
Speaker 2: have any basic strain on these companies that hold up
Speaker 2: the Internet. The other thing I think about is what
Speaker 2: if there are problems with I mean, we've seen this
Speaker 2: tons of times with Amazon Web Services, with Microsoft as
Speaker 2: your Google Cloud and so on and so forth. They
Speaker 2: feel like also a huge point of failure.
Speaker 1: Yeah, and you see kind of rumblings of this right,
Speaker 1: Like down detector is constantly pinging with things. Down Detector
Speaker 1: is kind of the website that everybody goes to whenever
Speaker 1: something either isn't working or isn't responding, to see whether
Speaker 1: or not other people are noticing these sorts of issues.
Speaker 1: It's strange, right because it's like we have and it
Speaker 1: happens every month or two. We have kind of like
Speaker 1: pretty significant tremors that put cracks in our walls, and
Speaker 1: we kind of go, oh, you know what, let's just
Speaker 1: plaster them up and I'll be okay, well okay, and
Speaker 1: we kind of overlook it, and it's it's I suppose
Speaker 1: the question is to what extent are those tremors kind
Speaker 1: of like the pre warnings of like a massive rupture,
Speaker 1: A huge kind of that is going to affect things
Speaker 1: or are we able to just kind of keep it
Speaker 1: ticking over and we have occasional outages and we fix
Speaker 1: it and that's okay. Yeah.
Speaker 2: I was speaking with Burial in the other day. He's
Speaker 2: an anti monopoly expert and he kind of made this point. Though.
Speaker 2: We also have absolutely no public kind of measurement of
Speaker 2: success or efficacy or indeed safety with any of these
Speaker 2: cloud providers. We have it for power plants, we have
Speaker 2: it for sewerage, we have it for water. We don't
Speaker 2: treat despite those utilities, but really cloud services are utilities too,
Speaker 2: and we just don't. We have no idea, We don't know,
Speaker 2: and we have no quality standards. So who even knows
Speaker 2: as they push these massive data centers whether they stay up.
Speaker 2: And it terrifies me, it really does.
Speaker 1: Yeah. I think what's interesting is any data that we
Speaker 1: do have is also provided by them, and it's kind
Speaker 1: of it's bundled into marketing materials. Right They say we
Speaker 1: have like six digit up time, which is that kind
Speaker 1: of six nines after ninety nine points to highlight how
Speaker 1: how well they maintain their services and how likely it
Speaker 1: is that you will never encounter an outage. But the
Speaker 1: reality is even that kind of point, not not one percent,
Speaker 1: over a long course of time, can be quite a
Speaker 1: significant outage. And if it's the thing, you know, if
Speaker 1: it's if it's an outage that happens that is, you know,
Speaker 1: keeping a hospital online or keeping your banking system online
Speaker 1: at a time when everybody needs it. Even the smallest
Speaker 1: outage on these kind of too big to fail services
Speaker 1: can be huge, and we don't we don't realize them because,
Speaker 1: as you say, there is no centralized record of this
Speaker 1: is when we've had outages. This is when we've had issues.
Speaker 1: They just come along every couple of months. They kind
Speaker 1: of grab the attention. In the case of crowd strike,
Speaker 1: it grabbed the attention because it was quite so massive
Speaker 1: and quite so visual and visceral. But then we move
Speaker 1: on and we forget about it, and actually we're only
Speaker 1: ever reminded the next time, and by that point we're
Speaker 1: so far beyond it that we forget actually how significant
Speaker 1: it was. I mean, people couldn't go to work on
Speaker 1: that Friday because they couldn't use their computers.
Speaker 2: Yeah, it's just really terrifying. This is the actual crisis,
Speaker 2: and I feel as if it's almost it feels like
Speaker 2: screaming into the void at times. One of the reasons
Speaker 2: I wanted to do this episode was because of this,
Speaker 2: because I don't think most people realize how brittle everything is.
Speaker 2: You've got, oh the way that most transactions are in
Speaker 2: crypto on the Internet, that's by two steves and everything
Speaker 2: holding everything up is like a patchwork of a few
Speaker 2: companies that are pretty much do not have They don't
Speaker 2: get held accountable until something breaks. It's it's very bad,
Speaker 2: but let's get it. Let's make it worse. So the
Speaker 2: last two you brought up in your article, I can
Speaker 2: and vera sign when you talk about why, they're also
Speaker 2: very worrying.
Speaker 1: Yeah, so I currently is it's how do I describe this? Basically,
Speaker 1: I can't at its heart run to what are called DNS,
Speaker 1: the domain name system, which is kind of the address book.
Speaker 1: So you type in a URL to your web browser
Speaker 1: that is not machine readable, so it gets converted into
Speaker 1: an IP address, which is a bunch of digits essentially,
Speaker 1: and that gets rooted through what is called the DNS,
Speaker 1: which is essentially a massive address book, and it's run
Speaker 1: many of them, not all of them, three of the
Speaker 1: kind of dozen or so that exists are run by
Speaker 1: i can, which is a sort of nonprofit that is
Speaker 1: one of the kind of earliest major organizations and involved
Speaker 1: in kind of the early web, and also VeriSign, which
Speaker 1: is kind of a private company. So if these things
Speaker 1: go offline, then like everything breaks, because if the DNS,
Speaker 1: if the kind of the address system of the Internet
Speaker 1: of the Web is corrupted in some way. I don't
Speaker 1: know about you, but I don't remember the IP address
Speaker 1: of like the BBC News website.
Speaker 2: I remember my own phone number and nobody else's. That's
Speaker 2: that should tell you everything. And just to be clear,
Speaker 2: every website you visit, without exception, is actually just the
Speaker 2: an IP address which has then gone through DNS. That's
Speaker 2: that's good.
Speaker 1: Yeah. And so unless you are I don't know, some
Speaker 1: sort of amazing memory, powerful individual who can remember every
Speaker 1: single IP address.
Speaker 2: Who's also who also knows them because we don't get
Speaker 2: exposed to them by the nature of the DNS system. DNS.
Speaker 1: Yeah, you just you just type it and it works,
Speaker 1: and that is you know, it's one of those things.
Speaker 1: It's we've we've traded off convenience for actually understanding how
Speaker 1: this technology works, which is great because it works, but
Speaker 1: if it doesn't work, then we're in real trouble. And
Speaker 1: I think that is if you think about kind of
Speaker 1: the economic impacts of crowd striking, the outages because they
Speaker 1: couldn't people couldn't get onto their devices. Think about what
Speaker 1: happens if people can get onto their devices, but they
Speaker 1: don't know how to access their bank or they don't
Speaker 1: know how to access the websites that they need for
Speaker 1: day to day working. That is the really interesting thing.
Speaker 1: And you know, I CAN is nonprofit. It has around
Speaker 1: about four hundred staff, so like it is well staffed.
Speaker 1: This isn't too Steves and a dog, but it is.
Speaker 1: I suppose four hundred seems fewer people than you need
Speaker 1: for something as important as this, right when you consider
Speaker 1: the huge numbers that are employed by big tech companies,
Speaker 1: you would think that I CAN would have.
Speaker 2: More than Well, the un has thirty six thousand people
Speaker 2: working for it, and this is probably the size that like,
Speaker 2: this is a probably a little bit more important than
Speaker 2: the u N if you really think about it.
Speaker 1: Yeah, the website would not work without these things, and
Speaker 1: so that is the.
Speaker 2: Email would email break as well?
Speaker 1: That's a good question. I think it would be.
Speaker 2: If you were accessing through through the webit world.
Speaker 1: Yeah, and also presumably I would I don't fully know
Speaker 1: the answer, but I would presume that actually, yeah, because
Speaker 1: you're putting in a a kind of domain name something
Speaker 1: at something dot com or dot co, dot UK or
Speaker 1: dot net or whatever, that it would still be routed
Speaker 1: through the same systems.
Speaker 2: So yeah, And a quick Google says that that's the
Speaker 2: case too. This is how I learned things. And also
Speaker 2: another website I wonldn't be able to access the DNS
Speaker 2: was down. That's that's so good.
Speaker 1: Yeah, exactly, this is the thing you would sometimes It's
Speaker 1: happened when I used to work prior to journals, I
Speaker 1: used to work in an office and sometimes the like
Speaker 1: the actual router would fail and you would just kind
Speaker 1: of be stuck there twiddling your thumbs and things, Well,
Speaker 1: what can I do? Like imagine that but everybody in
Speaker 1: the world, all at once unable to do the most
Speaker 1: basic stuff. And think about how reliant we are on
Speaker 1: all of the the internet connected services and tools that
Speaker 1: we use, and then think about what would be the
Speaker 1: impact if all of those stopped suddenly and we didn't
Speaker 1: know what to do afterwards.
Speaker 2: And it says here in your article. There's thirteen of
Speaker 2: the largest DNS servers a run by i CAN. So
Speaker 2: three of the three of the thirteen largest run by
Speaker 2: i CAN. So if you took if someone took out
Speaker 2: I CAN, it would still function, but I imagine there'd be
Speaker 2: a massive outage just kind of connecting the bits.
Speaker 1: Yeah, so they all have different route service, which is
Speaker 1: kind of like the they have kind of the original
Speaker 1: phone books as it were. You can get copies of
Speaker 1: copies of copies of copies, which are increasingly less reliable.
Speaker 1: It kind of generally seems to work geographically, so it
Speaker 1: would probably affect parts of of the world rather than
Speaker 1: the entire world, depending on which way you were served
Speaker 1: through in terms of which quote unquote phone book you got,
Speaker 1: which root server. But it's it's kind of a huge issue,
Speaker 1: and the problem is we don't fully understand and wouldn't
Speaker 1: fully understand until it happened, what the impact could be
Speaker 1: because we know, okay, if it affected those three servers,
Speaker 1: those three root servers, fine, but is there something on
Speaker 1: the other root servers, or the websites or the back
Speaker 1: ends of the organizations that operate the other root servers
Speaker 1: that relies on those root servers to get access to
Speaker 1: Like it's kind of could the domino effect start to
Speaker 1: play out here where actually one pretty significant error anyway
Speaker 1: could actually spread further and further and further.
Speaker 2: It's yeah, it's almost as if everyone It's would be
Speaker 2: like if they one forgot how to speak. Yeah, you
Speaker 2: could perhaps write letters, but speaking was off the table.
Speaker 2: It's terrifying. And again, three of them are held up
Speaker 2: by nonprofits. It's which is good, but all of them
Speaker 2: should be. It's so strange. As countries we can all
Speaker 2: get together to go to war or help support a
Speaker 2: war perhaps, but we genocide, I guess in that case.
Speaker 2: But we are in this situation where it's fucking we
Speaker 2: can't put the money together to support the literal way
Speaker 2: that people communicate online.
Speaker 1: Which is because we get through Yeah.
Speaker 2: It must works right now.
Speaker 1: Yeah, it works right now, and when it goes wrong,
Speaker 1: it hasn't gone completely wrong one hundred percent all over
Speaker 1: the world, and so we kind of go, well, that's
Speaker 1: a whoopsie, Okay, we can deal with it and move on.
Speaker 1: Hopefully it won't happen again. Fingers crossed. Let's hope that's
Speaker 1: all okay, and that the way that it's kind of working.
Speaker 1: That's the status quote a minute.
Speaker 2: I guess there's nothing. It's one of these I like
Speaker 2: to end episodes by being like, what can regular people do?
Speaker 2: It doesn't feel like we can.
Speaker 1: And you have read some of my journalism, you know
Speaker 1: that I'm one of the most pessimistic people. I do
Speaker 1: a radio slot here in the UK where I introduce
Speaker 1: tech stories to people who don't necessarily know lots about tech,
Speaker 1: and every single week I get harangued by the hosts
Speaker 1: because I always end with depressing notes, and unfortunately that
Speaker 1: is the case here.
Speaker 2: I think, Yeah, I think that something I like to
Speaker 2: come back to though, is knowledge is power. I think
Speaker 2: that I wonder if there is This is one of
Speaker 2: the dumber things I've thought up, but I wonder if
Speaker 2: there is actually a way of most people downloading the
Speaker 2: phone book of DNA with DNS phone book just distributed.
Speaker 1: It could be crypto far off. I can be on
Speaker 1: a blockchain.
Speaker 2: Bit of crypto. And it is funny as well, because
Speaker 2: you see all of this AI bullshit and you got
Speaker 2: the crypto bullshit and they're like, yeah, this is the future.
Speaker 2: This is so cooled important. Objectively, DNA is cool, like
Speaker 2: this stuff is that it's actually insane. The Internet works
Speaker 2: at all.
Speaker 1: Yeah, Like I wrote a book called The History of
Speaker 1: the Internet in bite sized Trucks, And as I said,
Speaker 1: I'm thirty five. I kind of I joined the Web
Speaker 1: when I was about ten or eleven, and yeah.
Speaker 2: I'm thirty eight, by the way, so we're right there.
Speaker 1: They go kind of got interested in it and found
Speaker 1: it incredible, but forgot that, like I lost that wonder
Speaker 1: because we don't see how it works anymore. You don't
Speaker 1: see the crankshafts, you don't see the gears working in
Speaker 1: the way that you used to. Kids nowadays don't know
Speaker 1: how to store files on a computer because they just
Speaker 1: have cloud storage. It's just always accessible easily there. They
Speaker 1: don't have to structure a file system or something like that.
Speaker 1: And so we take it for granted that these things
Speaker 1: work and we just assume that like it's all okay.
Speaker 1: But actually, yeah, knowledge is power. And knowing that there
Speaker 1: is a person behind this, Knowing that there is a
Speaker 1: system behind this and kind of getting a sense a
Speaker 1: little bit of how it works means that you understand
Speaker 1: more perhaps when things go wrong, and importantly, you can
Speaker 1: kind of advocate maybe for how to make sure that
Speaker 1: it doesn't go wrong again in the future.
Speaker 2: Chris, thank you so much for joining me. Where can
Speaker 2: people find you?
Speaker 1: They can unfortunately find me on X that stokel that
Speaker 1: is my am. I'm going down with that ship long
Speaker 1: li SD Okay.
Speaker 2: Yeah, all right, thank you so much. Chris. You've been
Speaker 2: listening to Better Offline. You know where to find me.
Speaker 2: There's the same thing that comes on after it that
Speaker 2: you'll complain because they haven't changed it in a wow,
Speaker 2: thank you for listening everyone, and then it's gonna say
Speaker 2: thank you for listening again. Thank you for listening to
Speaker 2: Better Offline.
Speaker 3: The editor and composer of the Better Offline theme song
Speaker 3: is Metasowski. You can check out more of his music
Speaker 3: and audio projects at Metasowski dot com, M A T
Speaker 3: T O S O W s ki dot com. You
Speaker 3: can email me at easy at Better Offline dot com,
Speaker 3: or visit Better Offline dot com to find more podcast
Speaker 3: links and of course my newsletter. I also really recommend
Speaker 3: you go to chat dot where's youreaed dot at to
Speaker 3: visit the discord, and go to our slash.
Speaker 2: Better Offline to check out our reddit. Thank you so
Speaker 2: much for.
Speaker 3: Listening Better Offline is a production of cool Zone Media.
Speaker 3: For more from cool Zone Media, visit our website cool
Speaker 3: zonemedia dot com, or check us
Speaker 1: Out on the iHeartRadio app, Apple Podcasts, or wherever you
Speaker 1: get your podcasts.