Speaker 1: Welcome to tech Stuff, a production from I Heart Radio.
Speaker 1: Hey there, and welcome to tech Stuff. I'm your host,
Speaker 1: Jonathan Strickland. I'm an executive producer with I Heart Radio
Speaker 1: and a love of all things tech and last week,
Speaker 1: Amazon's US East one cloud region had a bit of
Speaker 1: an outage, and the effects were widespread. Amazon delivery services
Speaker 1: were affected. A lot of deliveries just couldn't be made
Speaker 1: because the whole system that underlies that the computer system
Speaker 1: was affected. Computer games like Player Unknowns Battlegrounds became unavailable.
Speaker 1: People discovered that some of their home automation devices weren't
Speaker 1: working properly. Room Baz went berserk and rose up against
Speaker 1: their human owners. Even down at Walt Disney World, guests
Speaker 1: found themselves struggling with systems like Genie Plus, or even
Speaker 1: just making a park reservation so that they could visit
Speaker 1: a theme park. Also, I kind of made up the
Speaker 1: roomba thing. So today I thought I would talk a
Speaker 1: little bit about the history of Amazon Web Services, what
Speaker 1: it actually does, why it's such a big deal for
Speaker 1: Amazon the company, and why when there's an outage it
Speaker 1: has such a widespread effect. Now, the history of Amazon
Speaker 1: Web Services or AWS goes back a couple of decades
Speaker 1: and it is tied closely with the general rise of
Speaker 1: cloud computing. So first, let's define cloud computing just so
Speaker 1: that we have a common language. Now, if you were
Speaker 1: to go to Google and query the terms cloud computing definition,
Speaker 1: you would likely get something like the following quote. The
Speaker 1: practice of using a network of remote servers hosted on
Speaker 1: the Internet to store, manage, and process data either than
Speaker 1: a local server or a personal computer end quote. So,
Speaker 1: at its most simplest form, cloud computing is when you
Speaker 1: access computational resources that are on someone else's computer and
Speaker 1: you use the Internet to do it. So, if you
Speaker 1: use any sort of cloud storage like one drive or
Speaker 1: drop box or any of a thousand others, what you
Speaker 1: are actually doing is saving files to special data servers
Speaker 1: that are in some massive server farm somewhere in the world,
Speaker 1: probably not too far from where you are, Or maybe
Speaker 1: you're actually saving that one file two servers that are
Speaker 1: in a few different massive server farms. Though you wouldn't
Speaker 1: necessarily be aware of any of this, because that would
Speaker 1: be going on in the background, and it would be
Speaker 1: a matter of redundancy to make sure that your file
Speaker 1: remains available even if something should happen to any one
Speaker 1: particular machine. So when you access that file, what you're
Speaker 1: doing is connecting back to one of those servers that
Speaker 1: holds that particular file, and you might download the file
Speaker 1: to your local machine, so you're just retrieving it, or
Speaker 1: depending on the type of file you're accessing and the
Speaker 1: type of service you're using, you might be able to
Speaker 1: do stuff like make changes to that file through a
Speaker 1: web based client. So if you were to create a
Speaker 1: document in Google Docs, for example, that would follow that
Speaker 1: kind of cloud computing model. It's one of the simplest
Speaker 1: manifestations of cloud computing and is effectively cloud storage with
Speaker 1: a little bit of editing thrown in. But cloud computing
Speaker 1: can go far beyond just storing files. There are cloud
Speaker 1: based services that allow developers to build out an app environment.
Speaker 1: They might do this so that a distributed team, you know,
Speaker 1: people who aren't working all in the same location can
Speaker 1: simultaneously work on the same code and create test environments
Speaker 1: to make sure that the app performs as expected before
Speaker 1: or they deploy the app to end users, you know,
Speaker 1: to customers. Other cloud services serve as an actual deployment platform,
Speaker 1: so not just to develop, but to deploy the gifts,
Speaker 1: developers the assets that they need to push out an
Speaker 1: app and handle user interactions. So some apps might quote
Speaker 1: unquote live natively on your device. Right, You download a
Speaker 1: file to whatever you're using, whether it's a computer or
Speaker 1: smartphone or tablet or whatever it is, and then all
Speaker 1: the processes and all the data could be contained right
Speaker 1: there locally on your machine. Um that's like the old
Speaker 1: form of computing. But increasingly we're seeing apps that rely
Speaker 1: on the cloud for functionality. So games could have stuff
Speaker 1: like leaderboards or ways that you can compete or cooperate
Speaker 1: with other players in real time. A weather app needs
Speaker 1: to fetch data from servers to tell you what the
Speaker 1: weather will be, like your your own doesn't magically know
Speaker 1: what the weather is going to be. Even a lot
Speaker 1: of home automation apps will communicate back with a web
Speaker 1: server somewhere rather than handle everything right there in your home.
Speaker 1: In fact, that's a sticking point for a lot of
Speaker 1: home automation folks, right they don't necessarily want to have
Speaker 1: the cloud part of the infrastructure. They would prefer to
Speaker 1: have their home be kind of a self contained system.
Speaker 1: You see this a lot with people who have security
Speaker 1: systems where they would prefer to have something that was
Speaker 1: completely contained within their own home, as opposed to having
Speaker 1: their security system become a surveillance tool for a company
Speaker 1: that may or may not be working in conjunction with, say,
Speaker 1: law enforcement. That's become a big issue, but that's a
Speaker 1: matter for a different podcast. Now. Building out these kinds
Speaker 1: of systems is expensive because you need the physical facilities right,
Speaker 1: You need the actual buildings, and they have to be
Speaker 1: large enough to hold all the servers that are designed
Speaker 1: to make your app work right, and then the facilities
Speaker 1: also have to be designed themselves to allow those servers
Speaker 1: to operate. That means building out stuff like cooling systems
Speaker 1: so that your machines don't overheat. So you know, it's
Speaker 1: not just enough to have a place where you store
Speaker 1: all the computers. You've gotta have it be appropriate for that.
Speaker 1: You know, it needs to be dry, free of dust, cooled,
Speaker 1: that kind of stuff. You also need to maintain those systems.
Speaker 1: You have to repair or replace components as they fail,
Speaker 1: because we all know technology does fail at some point
Speaker 1: for a variety of reasons. That's also why you need
Speaker 1: to make more than the bare minimum to run your operations, right,
Speaker 1: you need to do more than just the basics. You
Speaker 1: need backups for redundancy so that if and when a
Speaker 1: specific machine goes down, others can take its place seamlessly
Speaker 1: without affecting the end user. So in our Google Docs experience,
Speaker 1: like if you were to create a document that's not
Speaker 1: just sitting on one server that Google owns, it's actually
Speaker 1: on multiple servers um multiple machines, and if one of
Speaker 1: those machines goes down, you can still get access to
Speaker 1: your file. Also when you make changes to it. Essentially
Speaker 1: you're making changes on one file on one machine that
Speaker 1: that machine then sends out a message to all the
Speaker 1: other machines that have that same file so that they
Speaker 1: can all be updated with the newest version. UM. I've
Speaker 1: done episodes about you know, that kind of background stuff
Speaker 1: in in the Google Docs world. So moving on, Essentially,
Speaker 1: cloud computing utilizes network connections to allow you, other people, organizations,
Speaker 1: companies to rely on machines that are hosted somewhere else,
Speaker 1: and that frees you up considerably. You don't have to
Speaker 1: invest in buying hard drives so that you can save
Speaker 1: all your files. You can just subscribe to a service
Speaker 1: to get some cloud storage companies don't have to keep
Speaker 1: themselves out with massive computer systems, uh complete with an
Speaker 1: I T department to support those computer systems. They can
Speaker 1: just you know, spend some money to use a cloud
Speaker 1: computing service owned by someone else and then host all
Speaker 1: their operations through that. Though, I should add a lot
Speaker 1: of companies take a more hybrid approach, so they have
Speaker 1: some systems typically really like mission critical systems are sometimes
Speaker 1: ones that require a great deal of privacy and security.
Speaker 1: They might run those on premises or on prem and
Speaker 1: then rely on other stuff, like the more administrative stuff
Speaker 1: for cloud systems. Cloud computing started becoming a term, a
Speaker 1: buzz term really around I mean, the idea was older
Speaker 1: than that, but it was starting to really get circulation
Speaker 1: around twenty ten or so. But the seeds, like I said,
Speaker 1: we're planted earlier. So let's let's take a look at
Speaker 1: Amazon specifically, because it played a big part in this.
Speaker 1: So way back in two thousand, Amazon was scrambling to
Speaker 1: keep up with some scaling issues, and this is something
Speaker 1: that you hear about with startups pretty frequently. A new
Speaker 1: startup is usually a fairly small company, and it's nimble,
Speaker 1: and it's agile, and it might offer a small range
Speaker 1: of services or products, or it might only serve a
Speaker 1: relatively small region or both. I think companies like Lift
Speaker 1: and Uber those launched in just a couple of cities
Speaker 1: early on, right, so they were able to grow in
Speaker 1: a controlled manner. Well, if customer demand is high and
Speaker 1: investors are pouring money into the startup, it makes some
Speaker 1: sense to try and grow the company and expand operations.
Speaker 1: But growing ads new challenges and making sure that the
Speaker 1: things you offer are able to scale up and meet
Speaker 1: demand is a non trivial matter. That's the situation Amazon
Speaker 1: was in around two thousand. One of the things the
Speaker 1: company was exploring was building out merchant sites for other
Speaker 1: companies but still using the Amazon platform. So, for example,
Speaker 1: Amazon might partner with a retail company like Target to
Speaker 1: provide an online store, but use Amazon's infrastructure underlying that store,
Speaker 1: and this would bring in a new stream of revenue
Speaker 1: for Amazon, and it would mean these retail companies could
Speaker 1: rely on Amazon's platform rather than having to build out
Speaker 1: us an online store all of their own. So Amazon
Speaker 1: called this merchant dot com. But it turned out building
Speaker 1: merchant dot com was pretty challenging. It was one thing
Speaker 1: to manage Amazon's rapid growth, but it was another to
Speaker 1: build out products that could immediately scale to fit the
Speaker 1: needs of established companies like Target. The initial result was
Speaker 1: a product that had so many interconnected moving parts and
Speaker 1: features that it was difficult for a user to navigate
Speaker 1: and actually use. And I'm sure all of you out
Speaker 1: there know that if a tool is hard to use,
Speaker 1: most people don't bother with it. Right. You might get
Speaker 1: it and try and think this is too much hassle,
Speaker 1: so you would rather go without or find some of
Speaker 1: their alternative. Well, in two thousand two, Amazon began building
Speaker 1: out Amazon dot Com Web Service. Now this would not
Speaker 1: quite be the same thing as Amazon Web Services, despite
Speaker 1: the similar names. It was much more simple than that.
Speaker 1: It used a SOAP and XML interface. And by SOAP,
Speaker 1: I don't mean the stuff you use to get clean. Now,
Speaker 1: if you're not a developer, those things probably sound a
Speaker 1: little confusing, so let's clear it up. SOAP is a
Speaker 1: messaging protocol which originally stood for Simple Object Access Protocol
Speaker 1: and XML means extensible markup Language. It is a language
Speaker 1: so weird to say this, so it's like a machine
Speaker 1: readable and human readable language that's used to create sets
Speaker 1: of rules for document encoding. So this this is a
Speaker 1: language we used to define rules as opposed to you know,
Speaker 1: programming something together. These allowed developers to create processes that
Speaker 1: can run on pretty much any machine that has HTTP
Speaker 1: installed on it. UH. That way, you could create a
Speaker 1: process that can run on Windows device is or Mac
Speaker 1: os or Lenox, all that kind of stuff without having
Speaker 1: to program a specific version for each operating system. So
Speaker 1: Amazon's version of this allowed for a pretty limited amount
Speaker 1: of development around creating processes that could access the Amazon
Speaker 1: product catalog. This would allow web developers to create an
Speaker 1: interface on their own web page that would utilize Amazon's store,
Speaker 1: with the idea that people could buy a product right
Speaker 1: there from that web page instead of having to navigate
Speaker 1: over to Amazon dot com itself, and the developers would
Speaker 1: earn a small commission on every sale made through that
Speaker 1: you know, web page based point of sale. It's just
Speaker 1: a tiny dip of the toe in the cloud based infrastructure. Also,
Speaker 1: Amazon noticed that developers were I mean, this happens all
Speaker 1: the time. Developers were taking that tool and making stuff
Speaker 1: that Amazon had not anticipated or intended, and nothing necessarily bad,
Speaker 1: but like some were making games where they would show
Speaker 1: use this this methodology to show a picture of an
Speaker 1: Amazon product, and it was up to you to guess
Speaker 1: what that product was. That kind of thing. So they
Speaker 1: were gamifying certain elements of this and that kind of
Speaker 1: got wheels turning over at Amazon. This happens all the time.
Speaker 1: Whenever you create anything and you give it to developers,
Speaker 1: they immediately figure out ways to misuse it, I mean
Speaker 1: use it creatively anyway. Around the same time, Amazon executives
Speaker 1: began to realize that their various development teams were running
Speaker 1: into the same problems over and over. Namely, each team
Speaker 1: working on a different internal project would need to go
Speaker 1: through the same basic steps before they could do any
Speaker 1: serious work on the project itself, which involved things like
Speaker 1: establishing systems to handle compute operations, UH, storage solutions to
Speaker 1: hold all the data, and also database solutions to organize everything,
Speaker 1: and a clear picture began to emerge. Amazon's teams were
Speaker 1: having to reinvent the wheel with every new project, and
Speaker 1: the original projection for seeing a project go from start
Speaker 1: to finish was supposed to be three months. That was
Speaker 1: the goal for Amazon, but it turned out that just
Speaker 1: building out the infrastructure to allow a project team to
Speaker 1: actually start developing their project would take three months, so
Speaker 1: everything was running behind schedule. The lesson that the executives
Speaker 1: took from this is that it would be a worthwhile
Speaker 1: endeavor to establish an centralized internal system that could support
Speaker 1: the compute, database, and storage needs of all these different
Speaker 1: project teams. So it would need to be a system
Speaker 1: that could compartmentalize and contain each project so that every
Speaker 1: one of them would have the resources that the teams needed.
Speaker 1: It meant building out virtual machines and figuring out ways
Speaker 1: to create redundancy, and it was a matter of necessity
Speaker 1: for Amazon in order for those internal teams to get
Speaker 1: out of that you know, three month projection goal. But
Speaker 1: it also meant Amazon was building up something that could
Speaker 1: potentially end up being a service that the company could
Speaker 1: offer to others. It would take a little bit longer
Speaker 1: for that to come about. Over time, Folks at Amazon
Speaker 1: again to think of this effort as creating something almost
Speaker 1: like an operating system, but for the Internet rather than
Speaker 1: for a computer or a mobile device. These ideas began
Speaker 1: to first take shape around two thousand three, when Amazon
Speaker 1: executives were attending a company retreat. It would be another
Speaker 1: few years before the earliest version of Amazon's web services
Speaker 1: would launch. All Right, we're gonna take a quick break.
Speaker 1: When we come back, we'll talk more about Amazon Web Services. Okay,
Speaker 1: we left off in two thousand three. Let's put this
Speaker 1: in perspective. If there anything like me, you might say,
Speaker 1: all right, well that's less than twenty years ago. I
Speaker 1: get it. But we let's think about other things that
Speaker 1: were going on. Right, So, two thousand three was a
Speaker 1: year before Facebook would launch at Harvard, let alone expand
Speaker 1: beyond it. In fact, it was about three years before
Speaker 1: Facebook would get out of the phase where it was
Speaker 1: only available to college students. Two thousand three was two
Speaker 1: years before You Too blaunched. It was four years before
Speaker 1: Apple would introduce the iPhone, and it was just two
Speaker 1: years after we had had the dot com crash that
Speaker 1: had whited out numerous web based companies. So this was
Speaker 1: very early on in thinking about cloud computing and operations
Speaker 1: at this kind of scale. The company began to invest
Speaker 1: in building out data centers, you know, these huge facilities.
Speaker 1: The whole thousands of servers and engineers developed and tweaked
Speaker 1: database management services to coordinate and partition these machines effectively Meanwhile,
Speaker 1: the product development teams would work on new products to
Speaker 1: expand what Amazon could do for customers. So in two
Speaker 1: thousand three, Andy Jesse, who would go on to become
Speaker 1: the CEO of Amazon as of July five of this year,
Speaker 1: he became the project lead for Amazon Web Services. He
Speaker 1: had suggested to Jeff Bezos that Amazon could take the
Speaker 1: systems the company had been developing for internal use and
Speaker 1: then open those up as a product for other companies.
Speaker 1: And he was essentially pitching cloud computing to Jeff Bezos,
Speaker 1: and he got the go ahead. In two thousand four,
Speaker 1: Jesse's team had a beta version of this product that
Speaker 1: was ready for testing, and over the following two years
Speaker 1: they would refine and tweak that product until in two
Speaker 1: thousand six, AWS was ready to launch its first initial product.
Speaker 1: Now this would not be Amazon Web Services as a
Speaker 1: cohesive whole, but rather a single product called Simple Storage
Speaker 1: Services or S three, which debuted on March fourteen, two
Speaker 1: thousand six. Now, Amazon described AS three as a tool
Speaker 1: that would let developers save and retrieve quote any amount
Speaker 1: of data at any time from anywhere on the web
Speaker 1: end quote. So this was a cloud storage product. It
Speaker 1: is a cloud storage product it still exists. The experience
Speaker 1: of merchant dot com had, however, taught Amazon de olopers
Speaker 1: a pretty valuable lesson, which I would summarize as just
Speaker 1: because he can doesn't mean you should. Now. Granted, I
Speaker 1: usually use that phrase to criticize vocalists who do irritating
Speaker 1: vocal runs during their songs um Mariah carry, but in
Speaker 1: this case, I'm talking about the issue of feature creep. Now.
Speaker 1: Feature creep is this tendency to throw in extra features
Speaker 1: and options into a product just because he can. These
Speaker 1: features don't necessarily contribute to the usefulness of that product.
Speaker 1: In fact, more often than not, they can cause a
Speaker 1: product to be jan kie and hard to navigate. The
Speaker 1: Amazon developers didn't want S three to fall into that trap,
Speaker 1: and so early on the team decided that the only
Speaker 1: thing that needed to be done was to make sure
Speaker 1: the storage service was as good as it could be
Speaker 1: and just avoid including any extraneous options. Their motto was
Speaker 1: quote the system should be made as simple as possible,
Speaker 1: but no simpler end quote. That's also a good point.
Speaker 1: A bare bones approach is sometimes the best one, but
Speaker 1: you do still need the bones to be there. The
Speaker 1: architecture of the product can be described as objects, buckets,
Speaker 1: and keys. Objects are essentially data, and that data could
Speaker 1: be just about anything. S three doesn't care what the
Speaker 1: data is. It could be video files, it could be
Speaker 1: a game, it could be a database, it could be music,
Speaker 1: it could be whatever. The objects have metadata that describes
Speaker 1: what the object is and when it was last modified. Next,
Speaker 1: you've got your buckets, and this is a kind of
Speaker 1: classification system. So imagine you've got these objects, that is, files,
Speaker 1: and you've got a lot of different types of them
Speaker 1: that belong to a lot of different things. So you
Speaker 1: might have a bucket for specific kind of file like
Speaker 1: music files, or more likely, you might organize buckets according
Speaker 1: to specific projects. So one project might have all its
Speaker 1: objects sorted into one or more buckets that belong to
Speaker 1: that project alone. Now, keys are a kind of I
Speaker 1: D for each object inside bucket, and each object has
Speaker 1: one key, so you can find any object inside S
Speaker 1: three if you have two pieces of information, the bucket
Speaker 1: it is in and the key for the object. So
Speaker 1: keys are used mainly for retrieval and you know that
Speaker 1: kind of thing. The Amazon developers created a storage system
Speaker 1: that was priced at fifteen cents per gigabyte of storage
Speaker 1: space per month. At least at launch, it is significantly
Speaker 1: cheaper than that now and this tells you that Amazon
Speaker 1: has scaled the service dramatically, and considering we're well into
Speaker 1: the era of big data, that's a good thing for developers.
Speaker 1: So today, Amazon's S three standard storage has three different
Speaker 1: tiers of cost, which depends on how much storage you're
Speaker 1: actually using, Like how much data do you have in
Speaker 1: the system. So let's say that you have fifty terabytes
Speaker 1: or less in S three standard, that would mean that
Speaker 1: you are looking at two point three since per gigabyte
Speaker 1: per month. Uh, if you've got more than five hundred
Speaker 1: terabytes stored, that's on the other end of the scale,
Speaker 1: then you're paying two point one since per gigabyte per month.
Speaker 1: And yeah, that adds up for companies that need to
Speaker 1: store a lot of data. Anyway, I bring it up
Speaker 1: to help illustrate how much things have changed. Fifteen cents
Speaker 1: per gig per month is way, way, way, way, way
Speaker 1: more expensive than two point three cents per gig per month.
Speaker 1: Oh and I should also mention that S three today
Speaker 1: offers several other storage products that have other features and
Speaker 1: costs associated with them. But this is not meant to
Speaker 1: be an ad for S three, so we're just gonna
Speaker 1: leave that for now. Anyway, S three right now, the
Speaker 1: gate was successful. In fact, just two months after launch,
Speaker 1: Amazon saw that demand had exceeded their projections by act
Speaker 1: of one hundred. Today, there are more than one hundred
Speaker 1: trillion objects stored in buckets in S three, and the
Speaker 1: fact that the product could scale up to accommodate that
Speaker 1: number of objects attests to good design decisions that were
Speaker 1: made early on. Yeah, the organization system is simple, but
Speaker 1: that simplicity also meant that S three could grow on demand,
Speaker 1: which it did. In August two thousand six, Amazon launched
Speaker 1: a new cloud based service, and this one was called
Speaker 1: and still is called Amazon Elastic Compute Cloud or e
Speaker 1: C two, And as the name suggests, this product offers
Speaker 1: up a different element of computing, the actual compute part.
Speaker 1: That is, this is a system that would allow customers
Speaker 1: the chance to tap into on demand computing power. Now,
Speaker 1: developers who had a great idea but who lacked the
Speaker 1: money or space or both to build out a computer
Speaker 1: facility could subscribe to e C two and lean on
Speaker 1: Amazon's systems to do the work for them. Like S three.
Speaker 1: This idea had its roots back in two thousand three.
Speaker 1: A couple of Amazon engineers, Chris Pinkham and Benjamin Black,
Speaker 1: had authored a memo suggesting a product that could give
Speaker 1: developers the chance to run software on Amazon computer systems
Speaker 1: specifically designated for that task. Around this same time, Amazon
Speaker 1: introduced Simple que Services or Amazon s q S. This
Speaker 1: is a type of message que and by message I
Speaker 1: mean the kinds of communications that go from service to service.
Speaker 1: So let's say you're running an app on your phone
Speaker 1: and the app might in the background send a request
Speaker 1: to a remote server to get access to some data,
Speaker 1: and that would be a message. So s q S
Speaker 1: is a platform that queues up messages so that the
Speaker 1: back end of a system can respond appropriately to requests
Speaker 1: that should give the end user a seamless experience. Now
Speaker 1: there's a lot more to s q S than that,
Speaker 1: but I think that's simple X A nation will serve
Speaker 1: us well enough for this episode. So these products S three,
Speaker 1: e C two and s q S kind of became
Speaker 1: the backbone for what would grow into Amazon Web Services
Speaker 1: as a whole. And there are a lot of other
Speaker 1: focused products in that suite, but generally speaking, each one
Speaker 1: is meant to be really good at doing something specific
Speaker 1: without having that feature creep issue come into play. Amazon
Speaker 1: got the jump on other big companies like Google and
Speaker 1: Microsoft when it came to offering up cloud based computing products.
Speaker 1: This gave Amazon the chance to establish a dominant position
Speaker 1: in the market. I mean, when you're effectively the only
Speaker 1: game in town, it's you know, not hard to become dominant.
Speaker 1: But today these other companies, Microsoft and Google and lots
Speaker 1: more have their own cloud computing services available. Still, Amazon's
Speaker 1: head start meant that the company still has a very
Speaker 1: strong presence. According to Synergy Research Group, Amazon's share of
Speaker 1: the cloud computing market is thirty two percent, or nearly
Speaker 1: one third of the entire market. That's more than Microsoft
Speaker 1: and Google's products combined together, those companies make up about
Speaker 1: of the market. So about a third of all the
Speaker 1: cloud computing business that's going on out there is going
Speaker 1: through Amazon. And like I said, that includes tons of
Speaker 1: different things from apps on your phone to video games
Speaker 1: to Walt Disney World's virtual ticketing system. Now, I'm not
Speaker 1: going to say that as long as a WS is
Speaker 1: running smoothly, everything should go well, because all the products
Speaker 1: that are built on top of a w S still
Speaker 1: need to have a good design. I mean, it's possible
Speaker 1: to make a really lousy product that's using AWS, and
Speaker 1: it's not the fault of AWS if that product is lousy.
Speaker 1: But as we saw last week, when things get harry
Speaker 1: on AWS, all the products that rely on those services
Speaker 1: they can be affected. So last week, at approximately ten
Speaker 1: thirty am on Tuesday, December seven, twenty one, a w
Speaker 1: S had what we in the tech biz call a
Speaker 1: whoop see. It was a whoop see that lasted between
Speaker 1: five to seven hours, depending upon the services you were
Speaker 1: relying upon. And because a WS has this massive presence
Speaker 1: in the market, and because so many big companies rely
Speaker 1: on it in order to make their stuff work, that
Speaker 1: whoop see had a pretty big footprint. According to Amazon,
Speaker 1: the issue was that there was a glitch in some
Speaker 1: crucial networking hardware. And this hardware is in charge of
Speaker 1: hosting what Amazon called foundational services, including stuff like e
Speaker 1: C two, but also it handled stuff like Amazon's Domain
Speaker 1: name service. Now, this service is kind of like the
Speaker 1: liaison that connects human readable u r L addresses with
Speaker 1: machine readable addresses, and without it, you can tell your
Speaker 1: browser to go to that particular website all you like,
Speaker 1: but it ain't happening because the liaison is on like
Speaker 1: a five to seven hour offee break, and the machines
Speaker 1: have no idea what you're on about. Anyway, the AWS
Speaker 1: internal system became overwhelmed, and that's something that usually doesn't happen.
Speaker 1: Usually there's this cross network scaling system that kicks in
Speaker 1: and meets increased demand. But this glitch essentially caused a
Speaker 1: massive game of telephone within the AWS system, and it
Speaker 1: overloaded all the circuits. To use a somewhat flimsy analogy,
Speaker 1: so the glitch triggered what Amazon called quote a large
Speaker 1: surge of connection activity that overwhelmed the networking devices between
Speaker 1: the internal network and the main AWS network, resulting in
Speaker 1: delays for a communication between these networks end quote. So
Speaker 1: it's almost like a classic denial of service attack, only
Speaker 1: Amazon kind of did it to itself. I guess we're
Speaker 1: being fair, we would say the glitch cost it. Now.
Speaker 1: A delay in communication normally just means you have an
Speaker 1: irritating experience like lag, right, and you can manage that
Speaker 1: you usually, but you know, it just makes whatever you're
Speaker 1: doing more difficult. Except a lot of systems have time
Speaker 1: out features, in which if there is a long enough
Speaker 1: delay between sending a message and getting a response, you
Speaker 1: reach a failed state, and that happened a lot last Tuesday.
Speaker 1: What made matters more difficult was that Amazon's own real
Speaker 1: time monitoring services rely on those internal AWS systems. I
Speaker 1: mean that's how AWS even got started, right, I mean
Speaker 1: it was Amazon building out its own infrastructure and then
Speaker 1: offering up those capabilities to other companies. So that meant
Speaker 1: the mitigation teams who were working to fix stuff didn't
Speaker 1: have all their real time monitoring tools available as they
Speaker 1: were tackling the problem, so that slowed down the recovery
Speaker 1: quite a bit. Amazon has since apologized to customers for
Speaker 1: this outage, and the reps now say that the company
Speaker 1: is working to distribute its service Health Dashboard across multiple regions,
Speaker 1: so that should something similar happen in the future, the
Speaker 1: fixed should theoretically happened much more quickly. Uh So, Yeah,
Speaker 1: this is another way for us to realize that we
Speaker 1: have put a tremendous amount of trust and dependence upon
Speaker 1: cloud services. And it's another reminder that sometimes like you
Speaker 1: could have designed everything yourself as good as it can
Speaker 1: possibly be. You can have an incredible app, but if
Speaker 1: the technology that powers that app goes down, it doesn't
Speaker 1: matter how good your product is, right it, you know,
Speaker 1: and since you don't control that, since you are dependent
Speaker 1: upon a cloud uh provider, then if the cloud provider
Speaker 1: has problems, that's really a big blow to your own
Speaker 1: business plans. It's one of the reasons why companies really
Speaker 1: debate on what services they want to put on the
Speaker 1: cloud versus on premises. Um It's it's a complicated thing too,
Speaker 1: because scaling is such a tricky issue. Most companies you
Speaker 1: know that aren't like huge fortune five companies don't have
Speaker 1: the assets necessary to be able to scare al at
Speaker 1: least not to the massive scales that we're seeing in
Speaker 1: the global Internet space. Anyway, I hope you found this
Speaker 1: episode interesting as we talked about a WS and what
Speaker 1: happened last week. If you have suggestions for topics I
Speaker 1: should cover on future episodes of tech Stuff, please reach
Speaker 1: out to me. The best way to do that is
Speaker 1: on Twitter. The handle for the show is text Stuff
Speaker 1: H s W and I'll talk to you again really soon.
Speaker 1: Text Stuff is an I Heart Radio production. For more
Speaker 1: podcasts from My Heart radio, visit the I heart Radio app,
Speaker 1: Apple podcasts, or wherever you listen to your favorite shows.