Speaker 1: Hey, it's Os Valoscian, host of tech Stuff. This week,
Speaker 1: we want to do something a little bit different and
Speaker 1: share an episode of a show that's new to the
Speaker 1: Kaleidoscope and iHeart Podcast family. It's called kill Switch. Kill
Speaker 1: Switch is a technology survival guide to modern culture. Host
Speaker 1: Dexter Thomas answers questions big and small, putting back the
Speaker 1: curtain on the systems we rely on daily, as well
Speaker 1: as the more absurd corners of the Internet. So here's
Speaker 1: an episode of kill Switch.
Speaker 2: Quick question, do you know about the xzutil's backdoor hack?
Speaker 3: So what?
Speaker 2: Or wait? Wait, wait, wait, wait what the EXU tills
Speaker 2: back door? I have no idea what you're talking about.
Speaker 1: I don't know what that is.
Speaker 2: This is something almost nobody's heard of, but in the
Speaker 2: spring of twenty twenty four we narrowly avoided a complete
Speaker 2: technological disaster. So you've never heard of this though?
Speaker 1: Nope?
Speaker 3: Yeah, I just searched it up on Wikipedia and it
Speaker 3: seems way too dune.
Speaker 2: To read about. These aren't just random people. These are
Speaker 2: other journalists people in general who keep up with the news.
Speaker 3: Okay, I was like, wait, what did I miss? And
Speaker 3: I feel bad, but I guess maybe I'm not the
Speaker 3: only one, what journalist who doesn't know what this is about.
Speaker 2: Even they didn't really know about what could have been
Speaker 2: the biggest hack in the history of the Internet.
Speaker 3: If this had not been caught, then this would have
Speaker 3: been a skeleton key that would have allowed these attackers
Speaker 3: to break into tens of millions of incredibly important servers
Speaker 3: around the world. We probably would have had airlines not working,
Speaker 3: trading halted ATM's not working, bank's not working, people not
Speaker 3: able to get their money. You'd have a huge loss
Speaker 3: of credibility of technology in people's lives.
Speaker 2: Alex Doamos is a cybersecurity expert. Specifically, he's the chief
Speaker 2: information security Officer or CISO at a cybersecurity company called
Speaker 2: Sentinel One, and he's the former CISO at Facebook. He's
Speaker 2: also a lecturer in the computer science department at Stanford
Speaker 2: and this attempted hack is something that is still keeping
Speaker 2: him up at night.
Speaker 3: It's fallen out of popular discussion, but among people in
Speaker 3: security we're still talking about it. It uncovered a real
Speaker 3: fundamental weakness that terrifies lots of people who have responsibility
Speaker 3: in this area and.
Speaker 2: What scares them most, and this should scare US two
Speaker 2: is that this was caught by complete chance.
Speaker 3: We just got lucky, like one dude got really bored
Speaker 3: and noticed a tiny little change in the speed of
Speaker 3: one program executing and pulled the thread, and on the
Speaker 3: end of this thread was a humongous, ticking time bomb.
Speaker 3: It was one dude, and he should never have to
Speaker 3: buy a beer for himself ever again. Underus freuend I'm
Speaker 3: raising a toast to you right now. This is just water,
Speaker 3: but I wish it was more.
Speaker 2: I'm afraid Kaleidoscope and iHeart podcast is killed switch. I'm
Speaker 2: Dexter Thomas. I'm sorry. I'm If you've never heard about this,
Speaker 2: that's no reason to feel bad, but if it hadn't
Speaker 2: been caught, it absolutely would have affected you. What kind
Speaker 2: of activity were talking about here.
Speaker 3: Well, we really don't know, and because we don't know
Speaker 3: who the attackers are, we don't know whether that would
Speaker 3: have been used for really quiet surveillance. It could have
Speaker 3: been used for national security intelligence gathering purposes, It could
Speaker 3: have been used for a humongous heist of hundreds of
Speaker 3: millions or billions of dollars of cryptocurrency, or it could
Speaker 3: have been used as part of a massive cyber attack
Speaker 3: to shut down millions of computers and cause massive disruptions.
Speaker 2: One of the main reasons that this potential attack isn't
Speaker 2: talked about much is because the details are kind of technical. Well,
Speaker 2: some of the details are a lot of this stuff
Speaker 2: is really just basic human behavior. It's stuff that you
Speaker 2: or I could do if we really wanted, and it
Speaker 2: shows us that sometimes the best hacks are the simplest ones.
Speaker 2: Let me break it down for you. In late March
Speaker 2: of twenty twenty four, Andres Freud, who's an engineer at Microsoft,
Speaker 2: was sitting at his desk doing his job when he
Speaker 2: discovered a malicious piece of code in this little known
Speaker 2: tool called xdu tilS. This code created a method that
Speaker 2: would allow hackers to access a lot of different computers.
Speaker 2: Maybe right now you're thinking, okay, so why is this
Speaker 2: the problem for me? I mean, I don't use xdu tilS,
Speaker 2: so they couldn't get on my computer. And yeah, maybe
Speaker 2: you've never heard of XCU tills. Actually I hadn't either,
Speaker 2: and I did what most people do when they don't
Speaker 2: understand something about a computer, call an expert. But it
Speaker 2: turns out that this really well respected expert he found
Speaker 2: out about XU tills when I did.
Speaker 3: Yeah, So I personally had not heard of xdu tilS
Speaker 3: before this, even.
Speaker 2: You really, yeah, I had definitely not heard it XU tills.
Speaker 2: I figured you would have hearing that you had not
Speaker 2: heard of it before all this happened. Frankly, that's a
Speaker 2: little bit more scary to me. Now, So why did
Speaker 2: this backdoor into a program that no one seems to
Speaker 2: know about? Matters so much? And what is xzu tills.
Speaker 3: This is the brilliance of what these attackers did. XCU
Speaker 3: tilS is an ingredient to an ingredient to an ingredient
Speaker 3: to something really important. So the thing that they wanted
Speaker 3: to have a backdoor into is a really important program
Speaker 3: called open ssh. So this is something that every tech
Speaker 3: he has heard of.
Speaker 2: All right, but what if you're not a techie. So
Speaker 2: in order to understand the XU tills hack, we do
Speaker 2: need to back up and understand something that xdu tilS
Speaker 2: is used in this thing called open ssh.
Speaker 3: This is the program that the majority of Unix like systems,
Speaker 3: especially Linux, also Max and some other operating systems allow
Speaker 3: you to access them remotely over the Internet.
Speaker 2: I'll get to the open later. But SSH stands for
Speaker 2: secure show, and let's just focus on secure right now.
Speaker 2: If you think of the difference between posting a tweet
Speaker 2: online and dming someone, you're actually kind of halfway there.
Speaker 2: Open Ssh allows you to communicate with a remote computer
Speaker 2: just like you were sitting there right in front of it.
Speaker 2: So even though you're far away, if you want to
Speaker 2: send a message, or install programs or delete files, you
Speaker 2: know that the connection is safe and that nobody else
Speaker 2: can see what you're doing or tamper with that connection.
Speaker 3: So you know, when you see like people in the
Speaker 3: matrix typing really fast, see a lot of text right
Speaker 3: if somebody is doing that remotely, it's probably over open ssh.
Speaker 2: You might think this doesn't matter for you because you
Speaker 2: don't use open ssh, but you do because that's what
Speaker 2: you use to connect to systems running Linux around the world.
Speaker 3: Linux has become the standard operating system for the cloud.
Speaker 3: So when you talk to Google, you're talking a Linux system.
Speaker 3: When you talk to Facebook, you're talking to a Linux system.
Speaker 3: When you talk to Apple, you're probably talking to a
Speaker 3: Linux system. Right now, the system that we're talking to
Speaker 3: each other with almost certainly is running Linux. So the
Speaker 3: vast majority of systems you talk to in the cloud
Speaker 3: are running Linux.
Speaker 2: Linux is used for Apple's iCloud, for social media sites
Speaker 2: like Facebook, Instagram, for YouTube, for Twitter, It's used for
Speaker 2: the New York Stock Exchange. Gamers use it when they
Speaker 2: run Steam or they play games online, and the list
Speaker 2: goes on. The vast majority of the Internet runs on
Speaker 2: Linux and open Ssh. Make sure that it's you logging
Speaker 2: in and not somebody else.
Speaker 3: When you log in and you get your mail, the
Speaker 3: server that holds your mail hash on it. The server
Speaker 3: that holds your social media content has SSH. The servers
Speaker 3: that have your banking information of Ssh. It's the door
Speaker 3: by which you get into these systems.
Speaker 2: So open ssh is incredibly important to the Internet and
Speaker 2: all the cloud systems that we rely on, and because
Speaker 2: of that, it has a lot of eyes on it.
Speaker 2: Trying to hack open Ssh directly would pretty much be impossible.
Speaker 2: Someone would catch you pretty quick.
Speaker 3: People pay a lot of attention to it, a lot
Speaker 3: of people run their code scanners on it, a lot
Speaker 3: of people look for bugs in it, and so it
Speaker 3: has been a while since open ssh has had itself
Speaker 3: a humongous security flaw in it. If you just join
Speaker 3: the open ssh project and said, hey, I'm a new
Speaker 3: guy that nobody ever knew. Here's my code. Everybody would
Speaker 3: be super suspicious, right, and whoever these bad guys are,
Speaker 3: they know that. So what they did was they looked
Speaker 3: at open ssh and they looked at its dependency graph,
Speaker 3: what we call They looked at all the stuff that
Speaker 3: goes into opensh and what they saw was open ssh
Speaker 3: depends on other things.
Speaker 2: This is where XU tills comes in. Xeu tills is
Speaker 2: one of the things that open Ssh depends on. What
Speaker 2: does xutil actually do. It's a compression library.
Speaker 3: So it's just a library that is used to make
Speaker 3: data that comes in smaller so that if you're moving
Speaker 3: like a big file back and forth, it can fit
Speaker 3: down a smaller pipe. Right, you might be talking to
Speaker 3: a server on a satellite link, you might be talking
Speaker 3: over a modem. Right, you might be talking over a
Speaker 3: cell phone, and so you want your big file to
Speaker 3: fit into a smaller pipe.
Speaker 2: If you've ever used the zip file on your computer,
Speaker 2: you get the general idea. Smaller files can be transferred faster,
Speaker 2: which is important when you're dealing with so much data
Speaker 2: flowing back and forth. Xdutils allows open ssh to be
Speaker 2: both safe and fast, But that's the trick. By inserting
Speaker 2: a backdoor into xu tills, the hackers created a way
Speaker 2: to access anything being transmitted via open ssh. That meant
Speaker 2: they could not only read supposedly secure messages, but remotely
Speaker 2: run code on any server that uses open ssh. And
Speaker 2: since basically the entire Internet uses this thing, once you're
Speaker 2: in there, you can do anything you want.
Speaker 3: You could have used it for a bunch of very
Speaker 3: quiet surgical attacks over a multi year period, or you
Speaker 3: could have done one humongous, big bane where you knock
Speaker 3: out a huge chunk of the Internet.
Speaker 2: All at once. But how did hackers get access to
Speaker 2: xdutails in the first place. Well, remember when I promised
Speaker 2: to tell you about the open in open ssh. Open
Speaker 2: Ssh and also Linux are open source programs. This means
Speaker 2: that anyone can look at the source code because it's
Speaker 2: open and it's posted publicly. The idea is that if
Speaker 2: everyone works together on the code, it'll be better and
Speaker 2: the public benefit, and so anyone's free to look at
Speaker 2: the code, to learn from the code, or even to
Speaker 2: remix it for their own use. And even if you
Speaker 2: have no interest in all that nerd stuff, you still
Speaker 2: use versions of open source code every day on basically
Speaker 2: all of your devices.
Speaker 3: When you're running open source software, which people don't understand.
Speaker 3: Basically everybody is, right, So what kind of phone do
Speaker 3: you have? Do you have an iPhone or Android?
Speaker 2: I actually have an Android?
Speaker 3: Yeah, okay, Android. A humongous chunk of that code is
Speaker 3: open source, right right, And that is code that is
Speaker 3: maintained by volunteers that you have no idea who those
Speaker 3: people are. Google has no idea who those people are, right.
Speaker 3: Google collects all this code from around the internet, they
Speaker 3: package it all up, and then they put it on
Speaker 3: a phone, or they send it to Samson, and Samson
Speaker 3: puts on the phone.
Speaker 2: And before we get any further, iPhone people, this applies
Speaker 2: to you too. Your iPhone uses a lot of open
Speaker 2: source code also. And don't get me wrong, this is
Speaker 2: not a bad thing.
Speaker 3: It's great because it's free and it makes the phone cheaper,
Speaker 3: and it's cool that we all get to contribute. But
Speaker 3: the flip side is is that, yes, OpenSSH itself gets
Speaker 3: lots of love, The Linux kernel gets lots of love, right,
Speaker 3: But something like xcu tills, which is this tiny little
Speaker 3: component over here, does not get lots of love and
Speaker 3: xCD details at the time was maintained by one person.
Speaker 3: That one dude was then manipulated to giving up control
Speaker 3: of it, and the person he gave up control of
Speaker 3: it too, turned out to be a totally fake persona
Speaker 3: do not exist.
Speaker 2: This is where we get to the human part of
Speaker 2: the story. The one guy who was maintaining xeu tills,
Speaker 2: his name was lost to Culin. He'd been maintaining xeu
Speaker 2: tills since two thousand and nine and he was the
Speaker 2: sole maintainer for the project. He wasn't being paid for it.
Speaker 2: He was a volunteer. That's usually how open source projects go.
Speaker 2: In twenty twenty two, Lusta Collins started to get a
Speaker 2: lot of requests to make updates to the code. Throughout
Speaker 2: the year, multiple accounts, seemingly out of nowhere, started complaining
Speaker 2: that Colin wasn't working fast enough and implying that if
Speaker 2: he wasn't interested in doing this anymore, maybe he wasn't
Speaker 2: the guy for the job and the pressure was getting
Speaker 2: to him. In June of twenty twenty two, Colin wrote
Speaker 2: in a public note quote, I haven't lost interest, but
Speaker 2: my ability to care has been fairly limited. Mostly due
Speaker 2: to long term mental health issues, but also due to
Speaker 2: some other things. He also went on to remind people
Speaker 2: that quote, it's also good to keep in mind that
Speaker 2: this this is an unpaid hobby project. Thankfully, right about
Speaker 2: that time, a new programmer had come into help. This
Speaker 2: new person's name was Gia Tan. Colin seemed a little
Speaker 2: relieved that finally someone wasn't just complaining but helping. In
Speaker 2: that same note from June, he wrote that he'd been
Speaker 2: working a bit with Gatan on execu utils to address
Speaker 2: all of those complaints, and he said about gia quote,
Speaker 2: perhaps he will have a bigger role in the future.
Speaker 2: We'll see. Over the course of a few years, gia
Speaker 2: Tan really started to gain Lasa Collins trust. Gia Tan
Speaker 2: was the ideal contributor. He didn't just help when he
Speaker 2: was asked to, but he would offer to take on
Speaker 2: more work, and by twenty twenty four, Colin had made
Speaker 2: gia Tan a co maintainer on the project, which allowed
Speaker 2: him to add code without needing approval.
Speaker 3: This is a human attack, right. It all happened in
Speaker 3: the open, but the way they did it was they
Speaker 3: created these fake personas were one guy super Friendly and
Speaker 3: one guy's a jerk, and the jerk basically is abusing
Speaker 3: the person who's maintaining the software and saying, oh, I
Speaker 3: need this change, I need this change. You're so slow.
Speaker 3: Why are you so slow? And remember this guy's not
Speaker 3: getting paid, right like, And so eventually basically bully this
Speaker 3: guy to say, oh, I'm tired of doing this. I
Speaker 3: don't want to do it anymore. And then the nice
Speaker 3: guy's like, oh, well you know, I'll do it for you.
Speaker 3: I'll take over, man, let me take this burden.
Speaker 2: For you, right, very convenient, right.
Speaker 3: And this took several years, and so this shows you
Speaker 3: kind of the long play. They're willing to spend months
Speaker 3: and months and months and in fact years building these
Speaker 3: personas because like, look, if you just created an account
Speaker 3: and you're like, hey, i've got code, take it, that
Speaker 3: wouldn't work. So what these people figured out is that
Speaker 3: you have to create these personas. They have to seem real.
Speaker 3: You have to make posts, you have to contribute legit stuff,
Speaker 3: you've got to create kind of a history, build a relationship.
Speaker 3: You have to build a relationship. And so the guy
Speaker 3: who maintains it gives it up of like, oh, thank
Speaker 3: you so much. For taking this burden from me because
Speaker 3: look at these jerks. Now, of course he doesn't know
Speaker 3: that the jerks worked for the same team, or maybe
Speaker 3: you're even the same person as the nice guy, right,
Speaker 3: And then he hands it over to this nice guy
Speaker 3: who's a friend of his, and then the friend takes
Speaker 3: it over and then does a bunch of legitimate stuff,
Speaker 3: and then in the middle of all that legitimate stuff
Speaker 3: inserts a very very subtle backdoor.
Speaker 2: I've seen this back door talked about using the phrase sophisticated,
Speaker 2: that it was very sophisticated. Yes, in some ways it
Speaker 2: sounds sophisticated, but in some ways it sounds like it
Speaker 2: kind of wasn't because a lot of it just revolved
Speaker 2: around getting somebody to give them some access.
Speaker 3: The code was sophisticated, the method of getting in there
Speaker 3: was very human. It was bugging a guy until he
Speaker 3: gave up control. Yes, right, just being a nuisance, Just
Speaker 3: being a nuisance. So who was behind those fake personas
Speaker 3: We don't know for sure, but Alex has a theory
Speaker 3: that's after the break, over the course of years, the
Speaker 3: one guy maintaining this very important tool called Xzeu Tills
Speaker 3: last of Colin was being bullied and manipulated online to
Speaker 3: give a persona called Gia Tan a lead role in
Speaker 3: handling the code. But who is Gia Tan? Everybody's been
Speaker 3: asking this question of like who did this, Who's behind this?
Speaker 3: Most of the names have kind of an Asian origin, right,
Speaker 3: So there's accounts like Jagar Kumar. The key one is
Speaker 3: Gia Tan, which is like, could be Chinese, could be Korean.
Speaker 3: Most of the either the names or the technical indicators
Speaker 3: point to Asia, right, So the time zones that this
Speaker 3: person was working into are kind of the East Asian
Speaker 3: time zone, so it's like Beijing or Korea. The names
Speaker 3: are Asian. Everything points to Asia, which makes a lot
Speaker 3: of people think it's Russia actually because it's just too perfect, right.
Speaker 3: WHI is like it's somebody spent three years doing all
Speaker 3: this work, and then you're like, like, let's say your Chinese.
Speaker 3: Are you gonna use like a Chinese name as your
Speaker 3: fake name? Are you going to spend three years but
Speaker 3: then work in your normal time zone? And the generally
Speaker 3: the only actor who has shown this level of patients
Speaker 3: who's been willing to spend three years working on a
Speaker 3: back door like this. The only people who have ever
Speaker 3: done that is either the United States or the SVR.
Speaker 3: So Russia okay, yeah, are really the only groups where
Speaker 3: you've seen people spend years kind of doing this kind
Speaker 3: of work. And a lot of people don't think it
Speaker 3: would be the US doing something like this, that they
Speaker 3: would never mess with something this important because also the
Speaker 3: thing the Russians really like to do is blame other
Speaker 3: people right again, because we never got to the point
Speaker 3: of again used. Usually attribution is done after something's used,
Speaker 3: and so it's a lot easier to figure out because
Speaker 3: then you can ask quebono right who benefits. But like
Speaker 3: all of these indicators pointing specifically to kind of China
Speaker 3: or Korea makes you think it's just a little too obvious.
Speaker 2: A major theory in cybersecurity circles is that Giatan isn't
Speaker 2: one person, it's potentially multiple people, but likely Russian hackers
Speaker 2: working for the SVR, which is Russia's foreign intelligence service,
Speaker 2: and that they tried to cover their tracks even if
Speaker 2: it wasn't consistent.
Speaker 3: The guys who worked for the professionals will change their
Speaker 3: time zones specifically around who either what allows them to
Speaker 3: avoid detection or specifically around whatever they're doing. For attribution.
Speaker 2: Well, there were some times that the time zones actually
Speaker 2: pointed to an Eastern European time zone or time or
Speaker 2: another time zone, right.
Speaker 3: Yeah, I mean there's there is a little mixed right,
Speaker 3: So somebody could be working from the eastern side of Russia,
Speaker 3: or they could be waking up early in Moscow Saint
Speaker 3: Petersburg and then they slipped right.
Speaker 2: In other words, they might have just slipped up and
Speaker 2: forgot to change their time zones because remember this happen
Speaker 2: been over the course of years. Maybe somebody had an
Speaker 2: off day and forgot to change the computer settings. But
Speaker 2: Alex has another reason for suspecting Russia over China.
Speaker 3: Chinese hackers, for the most part, work very rigorous hours.
Speaker 3: You can almost always tell when Chinese hackers are working
Speaker 3: because they work office hours. They work eight to five,
Speaker 3: eight to six. Really, it's like very regular, yeah okay,
Speaker 3: Whereas it's much harder to do time zone stuff based
Speaker 3: for the Russians because they they will work whatever hours
Speaker 3: they need to work. You know that that scene in
Speaker 3: like one of the Born movies where it's like the
Speaker 3: club scene. I always think about this with the Russia.
Speaker 3: There's like a club scene in Russia, and it's like
Speaker 3: you think it's the middle of the night and he
Speaker 3: walks out, it's like ten am or something. Right, It's like,
Speaker 3: that's what I think about with Russian hackers. Whereas like
Speaker 3: for China, it's amazing because it's like, oh, six pm
Speaker 3: in Beijing, you know, it's like you know, everybody goes home.
Speaker 2: To hacking stops.
Speaker 3: Yeah, or like Chinese Chinese New Year, Lunar New Year,
Speaker 3: everybody goes home, go sees their parents in the village
Speaker 3: or whatever, like hacking stops. It's amazing.
Speaker 2: And in the case of exit the UTILS, looking at
Speaker 2: the timing when this ga Tan was submitting code, there's
Speaker 2: a bunch of submissions during Lunar New Year, but during
Speaker 2: big Eastern European holidays like Christmas, crickets. But that leaves
Speaker 2: a question, what's the motive? Why would the restern SVR
Speaker 2: want to do this.
Speaker 3: So open everybody uses. That's why this is so powerful
Speaker 3: is you don't have to have a specific target in mind,
Speaker 3: which is why you'd also spend three years doing it.
Speaker 3: Because let's say you're at the SVR, you know, no
Speaker 3: matter what war you're involved with, no matter what target,
Speaker 3: you're going after openness station is going to be useful.
Speaker 3: So this is probably a team of the SVR who
Speaker 3: they don't know what's going to be used for. They're
Speaker 3: just they know they're going to get a medal.
Speaker 2: You'll be able to use this at some point, Yeah,
Speaker 2: who knows for what?
Speaker 3: And the US does the same thing, right, Like, there's
Speaker 3: people whose job it is to get the capability, and
Speaker 3: it's other guy's job who understand the geopolitics, who understand
Speaker 3: the intelligence to use it.
Speaker 2: But thankfully last spring, I'm just Freud. The Microsoft engineer
Speaker 2: was able to discover the back door, but this was
Speaker 2: all by chance. He wasn't looking for it.
Speaker 3: He works on a database called Postgress, so he doesn't
Speaker 3: work on xdutils. He works on Postgress is a big
Speaker 3: open source database program that Microsoft uses in their Azure cloud.
Speaker 3: So I'm guessing that's why Microsoft pays him. And in
Speaker 3: the next version of Debian, so a popular Linux distribution,
Speaker 3: Postgress was running a little bit slower, just tiny, tiny,
Speaker 3: a little bit tiny, tiny.
Speaker 2: Little bit right, so tiny, like how much slower?
Speaker 3: Like in one specific circumstance, it was taking a couple
Speaker 3: of milliseconds longer to do.
Speaker 2: Something right, So like a millisecond.
Speaker 3: Yeah, but like if you're a database guy, that's a lot, right.
Speaker 3: And so he is super looking into what is going on,
Speaker 3: and he realizes, oh, it's not actually postgress that's doing this,
Speaker 3: it's open as his age. And so he could have
Speaker 3: stopped there because he could have been like, oh, well,
Speaker 3: it's not my problem, right, it's not my thing, and
Speaker 3: then maybe nobody would have looked at it right, Like
Speaker 3: you could see an open source is that people pass
Speaker 3: problems to each other all the time, right, so it
Speaker 3: is this is like I think a normal like a
Speaker 3: normal person, even open source developer would have been like, oh, okay,
Speaker 3: I looked at this, it's not me. I'm gonna let
Speaker 3: it go. But he did not let it go. He
Speaker 3: ended up digging into okay, well what changed and open
Speaker 3: as the stage and then he looks into open a
Speaker 3: stah and sees this code. And so what the What
Speaker 3: the attackers that did is they created what's called a
Speaker 3: no bus back door nobody butt us.
Speaker 2: No bus or nobody butt us is a way of
Speaker 2: creating a backdoor into something where nobody but us or you.
Speaker 2: The hackers have the key.
Speaker 3: They want a skeleton key that only they can use,
Speaker 3: but no bus back doors nobody but us back doors
Speaker 3: arm are actually hard to sneak in because they're pretty
Speaker 3: like obviously sketchy.
Speaker 2: So instead of doing everything all at once, they'd delivered
Speaker 2: multiple patches in multiple different places, little things here and
Speaker 2: there that wouldn't raise suspicion if you looked at one
Speaker 2: or two or three of them, but layered on top
Speaker 2: of each other, it created a key that only they
Speaker 2: could use.
Speaker 3: And so because they did all this stuff to kind
Speaker 3: of obfuscate it and make it super secret, they actually
Speaker 3: created the performance impact that Undris saw and then went
Speaker 3: way out of his way to pull. And then he
Speaker 3: posts in a public post, Guys, this is super sketchy, right, Like,
Speaker 3: look at this code. There's no good argument for what's going.
Speaker 2: On here, right, So, I mean I kind of have
Speaker 2: to wonder about what the implications for this are. I mean,
Speaker 2: this clearly it almost worked. Do you think there's hackers
Speaker 2: out there saying okay, yeah, yeah, let me change my
Speaker 2: approach and maybe this is the way to do it.
Speaker 3: Yeah. I Mean what I'm afraid of is we haven't
Speaker 3: found any other ones like this, So what I thought
Speaker 3: would happen is at the time I'm like, oh, man,
Speaker 3: we'll have one or two more of these, because everybody
Speaker 3: started looking and then nobody else found any other ones,
Speaker 3: which terrifies me.
Speaker 2: You think there's more like this out there.
Speaker 3: I think it's quite possible. It's more like this. Yeah,
Speaker 3: Like if anybody has an idea, two or three other
Speaker 3: people have had the idea, right, So I can't imagine
Speaker 3: these are the only people who are like, oh, I'm
Speaker 3: gonna go bully some maintainer of one of the five
Speaker 3: thousand libraries on Linux to go take it over or
Speaker 3: submit a patch. I can't imagine there aren't other ones. Now,
Speaker 3: are they in OpenSSH or are they something much more subtle?
Speaker 3: I don't know. I mean this would have been both
Speaker 3: in kind of one of the worst possible places, and
Speaker 3: it would have been a skeleton key that only this
Speaker 3: attacker could have used, which is like kind of the
Speaker 3: worst case scenario. It's also the hardest level of difficulty, right,
Speaker 3: these people picked the hardest level. Thing. If you want
Speaker 3: to do something much much simpler is you go after
Speaker 3: a much lesser used service that's specifically at the target
Speaker 3: that you're going after. If you're going after a specific
Speaker 3: target and you're like, oh, they use this specific this
Speaker 3: one specific service that's much less popular, that doesn't have
Speaker 3: all these eyeballs on it, then you don't have to
Speaker 3: be as tricky.
Speaker 2: There haven't been any like this in OpenSSH but there
Speaker 2: have been other attempts that the Open Source Security Foundation
Speaker 2: and the Open JavaScript Foundation have found that use similar
Speaker 2: social tactics. One project received emails from accounts asking to
Speaker 2: be designated as project maintainers despite having little prior involvement,
Speaker 2: and two other projects saw very similar suspicious patterns. This
Speaker 2: kind of social engineering is really effective because you don't
Speaker 2: have to manipulate code. You just manipulate the person who
Speaker 2: has their hands on the code. And it's only going
Speaker 2: to get easier to do and harder to detect.
Speaker 3: Now we're at the point where with AI, like you
Speaker 3: could be fake now and I have no idea if
Speaker 3: you really exist or vice versa.
Speaker 2: Wait are you are you suggesting that doing something like
Speaker 2: this might be a little bit easier because somebody could
Speaker 2: fake that they actually exist. Oh yeah, with a phone
Speaker 2: conversation or a video conversation.
Speaker 3: Oh yeah, we're already seeing that from the ransomware actors.
Speaker 3: It's easy for phone right, So you're already seeing them
Speaker 3: fake people's voices. So people are getting phone calls from
Speaker 3: like their CEO. The CEO goes on CNBC for two minutes,
Speaker 3: they get their voice from CNBC. They plug it into
Speaker 3: a AI voice library, and then you call and like, hey,
Speaker 3: it's Bob, I need you do a million dollar transfer. Right,
Speaker 3: So that kind of stuff, and now you see real
Speaker 3: time video too. It's not perfect, but it's getting there.
Speaker 2: Yeah.
Speaker 3: The trick, by the way, if this happens to any
Speaker 3: of your listeners, the trick is you can ask people
Speaker 3: to move, touch things in the background, do three sixty
Speaker 3: on the head. It's harder for them to do ears
Speaker 3: forever reason, but they'll get there, right. So, like if
Speaker 3: I asked you to take your glasses off, it'd be
Speaker 3: very hard for the model, Like take your glasses off.
Speaker 2: By the way, hold on, for those of y'all listening
Speaker 2: at home, I took my glasses off here, just double
Speaker 2: check it.
Speaker 3: Oh you kind of frozen me when you did that.
Speaker 3: That's sketch man, it's sketch af As my students say, sorry,
Speaker 3: they keep me on my Stanford since But you know, in.
Speaker 2: The future, though, it is going to be easier to
Speaker 2: spoof people's personalities and stuff like that. So these things
Speaker 2: that you're suggesting right now they work now, are they
Speaker 2: going to work in a year? So?
Speaker 3: I mean the good thing about this is open source
Speaker 3: developers have become much more paranoid, right, So people have
Speaker 3: become much more paranoid about new people. And there's a
Speaker 3: downside of that, right that if you're trying to get
Speaker 3: into open source it's harder. There have become projects where
Speaker 3: it's like, okay, great, let's meet up in person. If
Speaker 3: somebody is willing only to communicate with you an email,
Speaker 3: then you have to be kind of sketched out now.
Speaker 3: So there have been some changes since this. I think
Speaker 3: people have been more paranoid. There's been a bunch of
Speaker 3: work on the flip side of AI is that traditional
Speaker 3: code scanning tools pre AI code scanning tools are not
Speaker 3: extremely good at detecting this kind of malicious code. But
Speaker 3: there is some hope that some of the newer AI
Speaker 3: based code scanning tools could could do this kind of
Speaker 3: stuff at scale. The flip side is is AI is
Speaker 3: really good at writing code, so you know, do you
Speaker 3: not have to be SVR level anymore to be able
Speaker 3: to write a backdoor. That's good.
Speaker 2: That's probably true as well, it's open source too much
Speaker 2: of a risk in the age of AI, and can
Speaker 2: we protect ourselves from another hack like this? That's after
Speaker 2: the break? So this and I want to get back
Speaker 2: into kind of the play by play here, but a
Speaker 2: lot of this hinges on open source. So and I
Speaker 2: think one of the really kind of concerning things about
Speaker 2: this entire thing that happened or almost happened is the
Speaker 2: fact that it basically happened in broad day. Yes, and
Speaker 2: it happened because this is open source. The thing about
Speaker 2: open source, I think, is when you start to explain
Speaker 2: it to somebody who's never heard of it. Are you
Speaker 2: familiar with the galaxy brain meme?
Speaker 3: Yeah?
Speaker 2: Do you know what I'm talking about? Yeah? So I
Speaker 2: feel like this is this is like that Galaxy brain meme,
Speaker 2: where at the very top, when you tell somebody to
Speaker 2: open source, the response is this is a terrible idea.
Speaker 2: Everybody can see the code. Yeah, And then you get
Speaker 2: a little bit further down, it's, oh, this is a
Speaker 2: great idea. Everybody can see the code, and then they
Speaker 2: hear about xutails. When we get down to the bottom,
Speaker 2: and it's a terrible idea. Everybody can see the code.
Speaker 2: What's the true galaxy brain take on this for open source?
Speaker 3: I mean people go back and forth. So one of
Speaker 3: the ideas is that if you can see all the code,
Speaker 3: you can see all the bugs. Right. Is the idea
Speaker 3: that because it's open source, that it should be more
Speaker 3: secure than closed source because you can see the flaws, right.
Speaker 3: I don't think that has empirically turned out to be true. Right,
Speaker 3: And so I I think what I would say is
Speaker 3: I'm a big proponent of oen source. I think it's great.
Speaker 3: I think it has a humongous economic benefit to the world.
Speaker 3: The truth is is the entire kind of cloud competing
Speaker 3: revolution we're all living through only exists because of open
Speaker 3: source software. So that's an incredible thing. That's a wonderful thing.
Speaker 3: Open source is great from an economic perspective. It is
Speaker 3: great from an innovation perspective. We should not pretend that
Speaker 3: it magically solves trust and security problems. And if you're
Speaker 3: a company that's relied upon open source, you have a
Speaker 3: ethical and moral obligation to deal with the security aspects
Speaker 3: of it. And it contribute back. And I do think
Speaker 3: that is something that's gone lost is that people have
Speaker 3: just kind of assumed somebody else is dealing with it,
Speaker 3: and everybody assumed somebody else is doing the security work,
Speaker 3: and that turns out not to be true.
Speaker 2: You know. I think that really gets the core of
Speaker 2: what a lot of this is because if somebody sees
Speaker 2: XU tills there was a potentual security flaw on that. Okay, well,
Speaker 2: I don't care about that. What's that? Oh, well, you
Speaker 2: know it's involved with open as a stage. Well, I
Speaker 2: don't use that either. I don't have that app on
Speaker 2: my phone. I don't know what you're talking about. And
Speaker 2: in this weird way, I feel like the more and
Speaker 2: more technology actually starts to become just magic, that things
Speaker 2: just work. Yeah, we are less and less actually tech literate.
Speaker 2: All the stuff that was science fiction even ten years ago,
Speaker 2: two years ago, frankly, is it's just normal now.
Speaker 3: Yeah.
Speaker 2: And so we're able to do so much with technology
Speaker 2: just regular people things we just do with our phone
Speaker 2: every day, that we've become really removed from the technology itself,
Speaker 2: and so less and less of us fewer and few
Speaker 2: of us actually know how to use a computer. Yeah,
Speaker 2: and so this feels totally removed from us this is like, oh,
Speaker 2: this is some weird nerd shit. I'd like, I don't
Speaker 2: use that nerd program doesn't affect me.
Speaker 3: Yeah, no, you're totally right. I mean, I tell my
Speaker 3: Stanford students. Security is one of the best fields to
Speaker 3: get into professionally because it's the only part of computers
Speaker 3: it gets worse every year. Everything else magically gets better. Man,
Speaker 3: So you could find yourself in any other field being
Speaker 3: made irrelevant, But if you get into security, you have
Speaker 3: job security for life. Because every year I've been in it,
Speaker 3: it's gone worse. And one of the reasons is because
Speaker 3: you say it's nerdsh it. But even the nerds we
Speaker 3: get the normal median nerd gets further and further away
Speaker 3: from the truth, the reality of what's going on on computers.
Speaker 3: So when I learned how to program, I learned assembly language, right,
Speaker 3: I learned how to write like the lowest level languages.
Speaker 3: And then you know, they stopped teaching assembly language unless
Speaker 3: you took special classes and you learn like in Python, right,
Speaker 3: like a very high level language that you don't even
Speaker 3: you know, you don't learn how to like do memory management.
Speaker 2: Right, I mean, pithon. And it's even just to bring
Speaker 2: this down like Python. For a casual person, you can
Speaker 2: look at it. You can kind of tell what's going on.
Speaker 2: Basically looks like English. Yeah, assembly is letters and numbers
Speaker 2: right right.
Speaker 3: But the nice thing about assembly is it's the truth
Speaker 3: of the matter. Right. It has a one to one
Speaker 3: matching to what the processor itself is doing. And from
Speaker 3: a security perspective, if you look at it, is the
Speaker 3: reality of what a security flaw is is seen in
Speaker 3: the assembly.
Speaker 2: Right.
Speaker 3: In Python, you get further, you get abstracted away, you
Speaker 3: get further from the reality of what's actually going on
Speaker 3: on the computer. Now what you see it's incredibly powerful.
Speaker 3: It's incredibly cool, and so I'm not gonna crap on
Speaker 3: it because I think it's an incredibly good thing for people.
Speaker 3: But you look at like Claude three point seven code.
Speaker 3: You know, this new Claude model, and you see people
Speaker 3: on Twitter who don't know anything about computers and they're
Speaker 3: able to program now because they can go into there
Speaker 3: and they can say, build me software that does X.
Speaker 3: And that is going to be terrible for security. It's
Speaker 3: super cool for people's economic opportunities because any big you
Speaker 3: can become a program er now. But man, are people
Speaker 3: in security gonna love it because now you don't need
Speaker 3: to know anything about how can work, and you're just
Speaker 3: going to ask the AI system to build it for you.
Speaker 3: And I see it with my students. Stanford students like
Speaker 3: one of the top computer science programs in the world,
Speaker 3: and you can graduate and not actually really understand how
Speaker 3: operating systems work. I apologize to Sandford Computer Science department, right,
Speaker 3: but really like, you can have a totally productive career
Speaker 3: in Silicon Valley and not really understand what's going on
Speaker 3: three or four layers down. In fact, it's better for
Speaker 3: you not to write. It's better because you're at the
Speaker 3: high level where you're much more productive. Right, You're much
Speaker 3: more productive having the AI do the work for you.
Speaker 3: You're much productive having get hub Copilot help you rewrite stuff.
Speaker 3: You're much more productive using all the cloud intermediation layers.
Speaker 3: And so that's one of the reasons why security gets
Speaker 3: worse every single year is that we add these layers
Speaker 3: of abstraction that make things easier for people. And AI
Speaker 3: is the ultimate abstraction layer, because now you can talk
Speaker 3: to computers and plain English and have them do incredibly
Speaker 3: complex things.
Speaker 2: The thing about this whole story. I mean, I'm thinking
Speaker 2: about you know, we're in a time right now where
Speaker 2: anything bad happens or almost happens. Netflix documentary, Hulu documentary,
Speaker 2: it's a true crime podcast. At some point. I don't
Speaker 2: see that happening with this. This is something that, as
Speaker 2: you were saying, was almost it truly could have been catastrophic. Yeah,
Speaker 2: but it's also kind of boring. It's money you don't think.
Speaker 3: I could get. I could sell ten episodes to Netflix
Speaker 3: on this.
Speaker 2: If you can hire me as a producer, I'd love
Speaker 2: to help. But you see what I'm saying is it
Speaker 2: takes a while to even explain what the heck we're
Speaker 2: talking about. Yeah, and I think that comes back to
Speaker 2: some of this in the same way that this vulnerability
Speaker 2: was introduced via social engineering. A lot of this is
Speaker 2: social I mean a lot of your work you probably
Speaker 2: think about this. How do you get people to care
Speaker 2: about something like this?
Speaker 3: I mean, so that's that's a challenge. That's one of
Speaker 3: the biggest challenges. If you're like a chief information security officer,
Speaker 3: one of your big jobs is getting the rest of
Speaker 3: the company to care about security. Ciso's we have a
Speaker 3: reputation of being the people who say no all the time.
Speaker 3: So I was the CISO of Facebook and I once
Speaker 3: walked into a meeting with a bunch of other vps
Speaker 3: and somebody literally said, like, oh shit, Sumus this year,
Speaker 3: like hey guys, I can hear you. I can hear you,
Speaker 3: and like, no, no, it's not you. It's just like
Speaker 3: whenever you come like it's because you're telling us like
Speaker 3: there's a coup in Turkey or something terrible, and like
Speaker 3: because I was just the bare or bad news, right.
Speaker 3: But this is what's a real challenge for my colleagues,
Speaker 3: and it's a real challenge for us as a society.
Speaker 3: People don't want to think that the systems that they
Speaker 3: rely upon are fragile. And I think that's like a
Speaker 3: real problem.
Speaker 2: What do we learn from this? What is it? Let
Speaker 2: me just say, because I don't I personally don't think
Speaker 2: just being out and talking to people, if I was
Speaker 2: trying to if I try to tell somebody, hey, yeah, man,
Speaker 2: what do you think about the XDUTILS thing? Have you?
Speaker 3: You know, hey, buddy, you what's up?
Speaker 2: Yeah? Has has it changed anything about how? Yeah? Has
Speaker 2: that changed anything about how you go about your life?
Speaker 2: People can tell me no, So I got to ask
Speaker 2: somebody who's actually closer to this. Has this changed how
Speaker 2: you approach things? Has this changed how the industry approaches things?
Speaker 2: Has this changed how I mean the theory that you're
Speaker 2: putting out is that this is a state actor? Has
Speaker 2: this changed how national security is being looked at?
Speaker 3: So for companies that know what they're doing, it has
Speaker 3: changed that they approach open source. For a handful of
Speaker 3: really big you know, like the Googles, the Metas, the Amazon's,
Speaker 3: the Microsoft's, the really big tech companies that do a
Speaker 3: lot of open source work, they are looking more carefully
Speaker 3: at open source for security companies that do this work.
Speaker 3: We're investing in software and AI that can do this
Speaker 3: work for us. But it has not changed anything massively. Right,
Speaker 3: We're still running Linux, We're still all pulling in fifty
Speaker 3: thousand packages. We have these fwomongkeys, depend t graphs. The
Speaker 3: truth is, you can't just pivot all these things. Right.
Speaker 3: It has made us more concerned about these problems. When
Speaker 3: you talk to Cisows, my colleagues and I we're all
Speaker 3: more concerned. But we can't magically pivot off of the
Speaker 3: infrastructure we have built over a decade. I do not
Speaker 3: think we've dealt with the fact that if you get
Speaker 3: on the subway in the morning and you look around,
Speaker 3: most of the people on that train in their pocket.
Speaker 3: Exutills is in their pocket. Every single person in there,
Speaker 3: hundreds of copies of their Social Security number is sitting
Speaker 3: on servers that would have been backdoored by this attack.
Speaker 3: That's how you can think of it, right, So that's
Speaker 3: how close we came.
Speaker 2: Man. So just some closing thoughts here. Again, the reason
Speaker 2: that most people don't know about what was almost the
Speaker 2: biggest hack in the history of the Internet is because
Speaker 2: this is really hard to describe to a non technical audience.
Speaker 2: I mean, when you say EXEU or linux or open ssh,
Speaker 2: people's eyes just rolling the back of their heads. But
Speaker 2: we can't allow tech literacy to be a barrier to
Speaker 2: understanding how the world works, and the truth is, even
Speaker 2: beyond all the tech jargon, a lot of these things
Speaker 2: are very human and they're not so hard to understand.
Speaker 2: And so that's one of the things that we're really
Speaker 2: trying to do here on kill Switch, as we keep
Speaker 2: doing these episodes, is to open it up so that
Speaker 2: more people are able to feel like they're part of
Speaker 2: the conversations that affect all of us. And that is
Speaker 2: it for this one, for real. Thank y'all so much
Speaker 2: for listening to kill Switch. You can hit us up
Speaker 2: at kill Switch at Kaleidoscope dot NYC if you got
Speaker 2: any thoughts or if there's anything you want us to
Speaker 2: cover in the future, and you can hit me at
Speaker 2: dex Digi that's the d e x d igi on
Speaker 2: Instagram or blue Sky if that's more your thing. And
Speaker 2: if you like the episode, you know, take that phone
Speaker 2: out of the pocket and leave us a review. It
Speaker 2: helps people find the show, which in turn helps us
Speaker 2: keep doing our thing. And this thing is hosted by
Speaker 2: me Dexter Thomas. It's produced by Sena Ozaki, darl Luk Potts,
Speaker 2: and Kate Osborne. Our theme song is by Kyle Murdoch,
Speaker 2: who also makes the show from Kaleidoscope. Our executive producers
Speaker 2: are oz Va Lashin, mungesh Hat Togadur, and Kate Osborne.
Speaker 2: From iHeart, our executive producers are Katrina Norville and Nikki E. Tour.
Speaker 2: That's it for this time. Catch on the next one.