Speaker 1: Welcome to tech Stuff, a production from iHeartRadio. Hey there,
Speaker 1: and welcome to tech Stuff. I'm your host, Jonathan Strickland.
Speaker 1: I'm an executive producer with iHeart Podcasts. And how the
Speaker 1: tech are yet. So let's take a little literary trip.
Speaker 1: In Anthony Burgess's a clockwork Orange, the extremely wicked protagonist
Speaker 1: it's putting it lightly. At one point early early in
Speaker 1: the novel, reflects on the nature of permanence. He thinks
Speaker 1: the reader might not remember what milk bars were like
Speaker 1: due to quote things changing so scory these days and
Speaker 1: everybody very quick to forget, newspapers not being read much
Speaker 1: neither end quote. Alex in this case is saying that
Speaker 1: the combination of the world changing very quickly scory is
Speaker 1: derived from a Slavic word meaning swiftly or quickly, and
Speaker 1: people having short memories means that referencing something that happened
Speaker 1: even just a few years ago might mean you're met
Speaker 1: with blank stares because the world has moved on. Now
Speaker 1: take that same sentiment and crank it up to eleven
Speaker 1: when you talk about the Internet in general and the
Speaker 1: Web in particular. So, on the one hand, we know
Speaker 1: that the rule of thumb is that once something gets
Speaker 1: posted online, that's kind of it, right, it's sort of
Speaker 1: perpetually online. Like that's kind of the joke. Like once
Speaker 1: it's up, it's up, and you can take it down,
Speaker 1: but there's going to be a copy of it somewhere.
Speaker 1: So even if the originator tries to take down whatever
Speaker 1: the stuff was, somebody's got it. But on the other hand,
Speaker 1: we also know that so much stuff gets added every
Speaker 1: single day to the Internet. There's actually a colossal mountain
Speaker 1: of content out there that just keeps getting bigger moment
Speaker 1: by moment, and everything that came before it can end
Speaker 1: up getting buried in the process. And sometimes stuff can
Speaker 1: be added and taken down without anyone being the wiser. Now,
Speaker 1: on top of that, web pages obviously can change. A
Speaker 1: website might adopt a new format or style, might incorporate
Speaker 1: new technologies and interfaces that are added to web browsers,
Speaker 1: or it might choose to remove sections that once might
Speaker 1: have been relevant but maybe now not so much. Or
Speaker 1: entire websites could disappear as servers go offline or companies
Speaker 1: go bankrupt, or you know, web administrators just lose interest.
Speaker 1: The entire spectrum of human output can be found on
Speaker 1: the web. Not every instance of human output, but an
Speaker 1: example of everything is out there. Everything from deep philosophical
Speaker 1: musings to the most banal posts you know, which often
Speaker 1: revolve around what someone is having for lunch. All of
Speaker 1: that finds its way to the Internet. And while you
Speaker 1: might argue that a lot of it, or perhaps even
Speaker 1: most of it, is it really worth the time it
Speaker 1: takes to consume, let alone keep it around. There is
Speaker 1: undeniably a huge amount of valuable data out there too,
Speaker 1: but there's no guarantee that it will stay there or
Speaker 1: remain easily findable. And that's where today's topic comes in.
Speaker 1: I wanted to talk about a project that began back
Speaker 1: in nineteen ninety six. It's a project that aims to
Speaker 1: preserve as much of the Internet as possible and little
Speaker 1: slices of time, little snapshots. Not only does that mean
Speaker 1: you can potentially dig up something that hasn't been online
Speaker 1: for years, but also you can get a look at
Speaker 1: what different sites were like in various eras of the Web.
Speaker 1: It could be a really eye opening experience to see
Speaker 1: something like Amazon and what it looked like, you know,
Speaker 1: shortly after it launched, compared to what it looks like today.
Speaker 1: So we are going to talk about the Internet Archive. Now.
Speaker 1: To do that, we need to talk a little bit
Speaker 1: about the people who founded the ding dang darn thing,
Speaker 1: and that would be Brewster Kale and Bruce Gilliat. So
Speaker 1: Klee graduated from m with a degree in computer science
Speaker 1: and engineering. After he graduated, he joined fellow MIT graduate
Speaker 1: Danny Hillis, who had created a company called Thinking Machines.
Speaker 1: So this was a super computer company. His team specialized
Speaker 1: in building massively parallel computer systems, mostly with the aim
Speaker 1: of building machines for AI research and development. So yeah,
Speaker 1: Calee was working on the challenges of providing AI researchers
Speaker 1: with the compute power they need, decades before our current
Speaker 1: AI explosion. Bruce Gilliot is also a computer scientist, and
Speaker 1: that's just about all I know about him. I mean,
Speaker 1: I know he is, or at least was married, and
Speaker 1: I also know he owned a series of very impressive
Speaker 1: houses in the San Francisco and San Jose areas because
Speaker 1: it made the news whenever he sold one or bought
Speaker 1: a new one. But other than that, there's precious little
Speaker 1: information about him that I could find, which is somewhat ironic.
Speaker 1: When you consider that he has dedicated a lot of
Speaker 1: time and effort to preserving information on the Internet. He
Speaker 1: would go on to co found the company called Alexa
Speaker 1: Internet with Brewster Kale, but that's getting ahead of ourselves.
Speaker 1: So most of my story will center around Kale simply
Speaker 1: because out of the two co founders, he's the one
Speaker 1: who acted more as the face of the efforts, and Gileat,
Speaker 1: from what I can tell, has just been really good
Speaker 1: about kind of maintaining a very personal private life. So
Speaker 1: I don't mean to diminish Gileat's contributions, but at the
Speaker 1: same time, you know, I can only cover what I
Speaker 1: can find. So in nineteen eighty nine, Kale, along with
Speaker 1: a colleague named Harry Morris, created an innovative tool for
Speaker 1: the blossoming Internet. Now remember this is the Internet. It's
Speaker 1: not the Worldwide Web. It didn't exist yet the Web
Speaker 1: the Internet did, and the tool they created was called
Speaker 1: the Wide Area Information Server or ways WAIS. So people
Speaker 1: could create a web server. They could host documents on
Speaker 1: their web servers. But finding these documents was really hard
Speaker 1: because you didn't necessarily have hyperlinks connecting one document to
Speaker 1: others and vice versa. You didn't have an easy way
Speaker 1: of even navigating through different documents from one to the next.
Speaker 1: So it was almost a case that you needed to
Speaker 1: know where something was and what it was called first,
Speaker 1: and then you could go to the relevant server and
Speaker 1: retrieve that document. Otherwise the document would just remain quietly
Speaker 1: sitting on some server somewhere and no one would know
Speaker 1: about it. Now, that is antithetical to the entire purpose
Speaker 1: of a wide area information sharing system, because, I mean,
Speaker 1: the name tells us the whole purpose of this technology
Speaker 1: is to allow information to be widely shared. Jeremy Norman's
Speaker 1: History of Information lists ways as quote the first Internet
Speaker 1: publishing system, just predating Gopher and the World Wide Web
Speaker 1: end quote. In a recorded presentation to some Xerox employees,
Speaker 1: Kale laid out personal perspective on what he wants from
Speaker 1: his experience on the Internet. So first up, he said
Speaker 1: he wanted his own personal information to be easily accessible
Speaker 1: by him. Specifically, not that it should be easily accessible
Speaker 1: to everybody, but specifically to him. He wanted the ability
Speaker 1: to get access to all the different stuff he generates,
Speaker 1: like articles and such, and to make it really easy
Speaker 1: to do that. He also wanted the ability for publishers
Speaker 1: to get their work to him. So in Kal's mind,
Speaker 1: the best approach would be for published works that are
Speaker 1: relevant to his interests to find their way to him,
Speaker 1: as opposed to Kale having to go out and hunt
Speaker 1: down these published works himself. And he pointed out this
Speaker 1: is what publishers want too, because you wouldn't publish something
Speaker 1: unless he wanted folks to actually read it. He also
Speaker 1: said that he wanted this technology to be usable anywhere.
Speaker 1: He wanted people to be able to access it no
Speaker 1: matter what kind of device they were relying on. Now
Speaker 1: he was specifically referencing laptops at the time, but he
Speaker 1: was also saying that portable computer systems, essentially things that
Speaker 1: would become smartphones and tablets, were on the horizon and
Speaker 1: that these needed to be able to access that stuff too.
Speaker 1: And he said that he wanted people to be able
Speaker 1: to use what he had learned should he choose to
Speaker 1: share the information, that if he had come up with
Speaker 1: something that was useful and he wanted to share that,
Speaker 1: he wanted other people to be able to access that.
Speaker 1: Cale didn't say that people should be compelled to share,
Speaker 1: but if they wanted to it should be possible to
Speaker 1: do so. Ways was Cale's attempt to bring these ideas
Speaker 1: to life. In that presentation to the Xerox employees, he
Speaker 1: defined ways as electronic publishing. He further defined that term
Speaker 1: to mean the distribution of information. So whether the end
Speaker 1: user was to look at this information on a computer
Speaker 1: screen or they just chose to print out the information
Speaker 1: and then read it that way, that was beside the point.
Speaker 1: Electronic publishing was all about how information got from the
Speaker 1: originator to the end user. That's what made it e
Speaker 1: publishing that it was publishing over wires. Now, one thing
Speaker 1: Cale introduced in this presentation to Xerox was this concept
Speaker 1: of conducting searches using natural language. This concept is one
Speaker 1: that we're really familiar with today. You enter a query
Speaker 1: into a search bar. You describe what it is that
Speaker 1: you want to know or learn about, or have access to,
Speaker 1: or retrieve or whatever. This search engine brings back search
Speaker 1: results that are ordered by some kind of relevance depending
Speaker 1: upon the search engines, you know, various algorithms. How the
Speaker 1: search engine determines relevance really depends upon the system itself,
Speaker 1: of course, Like you could run the same search across
Speaker 1: different search engines and get very different results based upon
Speaker 1: that methodology of determining relevance. If the system believes it's relevant,
Speaker 1: it may or may not be relevant to what you
Speaker 1: actually want. Like hopefully the two are aligned. If it's
Speaker 1: a really good search engine, then you're going to get
Speaker 1: something that is actually meaningful to you. Anyway, Ways was
Speaker 1: kind of following in that approach back before there was
Speaker 1: a World Wide Web, you know, when you just needed
Speaker 1: a way to find stuff that was being stored on
Speaker 1: these Internet servers and to be able to retrieve these
Speaker 1: documents to make use of them. Otherwise you had this
Speaker 1: incredibly powerful communications tool, but it was so challenging to
Speaker 1: use in a meaningful way that the information stored there
Speaker 1: would be not that useful. I think of it akin
Speaker 1: to imagine that there's this one remote library and it's tiny,
Speaker 1: but it has the world's only copy of some text.
Speaker 1: But this libraries in the middle of nowhere. It's really
Speaker 1: hard to get to the fact that that library has
Speaker 1: that document would not be terribly useful to most people,
Speaker 1: and so it might as well not have the document
Speaker 1: at all. That's kind of what Ways was trying to
Speaker 1: do is solve this problem of making it easier to
Speaker 1: get access to this wealth of information that Kale saw
Speaker 1: was only going to get more complex and more full
Speaker 1: of data. Well, we'll move away from Ways, because we
Speaker 1: could do a full episode about that. I will say
Speaker 1: that Cale and Morris, the founders of Ways, the guys
Speaker 1: who created the Ways technologies, would actually leave Thinking Machines
Speaker 1: and they would found a spinoff company just called Ways Incorporated.
Speaker 1: And it was around this point when the mysterious Bruce
Speaker 1: Gilliot joined the team. And while the Worldwide Web would
Speaker 1: debut in the early nineties, which really opened up accessibility
Speaker 1: to information on the Internet for a lot of people,
Speaker 1: most of them for the first time, Ways would continue
Speaker 1: to remain relevant. In fact, it was relevant enough that
Speaker 1: in nineteen ninety five AOL would come calling with an
Speaker 1: offer to purchase the company for a cool fifteen million dollars.
Speaker 1: If we adjust that for inflation today's money, that would
Speaker 1: be around thirty million bucks around that ballpark. Now, a
Speaker 1: lot of the folks that Ways Incorporated would split off
Speaker 1: to create new companies after this acquisition, and within a
Speaker 1: year that included Cale and Gileat, who went on to
Speaker 1: found a new company called Alexa Internet and you might think, huh, Alexa,
Speaker 1: you mean like the same name as the Amazon Digital Assistant,
Speaker 1: And yes, exactly that, because, as it would turn out,
Speaker 1: Amazon would ultimately acquire Alexa Internet just a few years
Speaker 1: after it was founded. But the name derived from the
Speaker 1: Library at Alexandria, the ancient library of Egypt that at
Speaker 1: one point housed one of the world's largest collections of
Speaker 1: accumulated knowledge. Now around forty eight BCE, Julius Caesar Julie
Speaker 1: Baby and his boys they barged into Alexandria, and as
Speaker 1: a consequence of their rowdy invasion, the library caught fire
Speaker 1: and much of the collection burned. Sadly, that was not
Speaker 1: the only indignity. In fact, it wasn't the first indignity
Speaker 1: that the library suffered that would impact its relevance. Further
Speaker 1: conflicts a couple of centuries later pretty much wiped out
Speaker 1: whatever had been left from the previous calamities, and the
Speaker 1: Library of Alexandria became kind of a touchstone for folks
Speaker 1: who have stressed the importance of access to knowledge and
Speaker 1: the protection of that knowledge, and that the consequences that
Speaker 1: could follow from the loss of such knowledge can be
Speaker 1: really dire. See also like the Middle Ages the Dark Ages,
Speaker 1: for example, that loss of knowledge is a really terrible thing.
Speaker 1: So the impetus for Alexa Internet was that Cale and
Speaker 1: Gillat wanted, in the words of the Web Design Museum quote,
Speaker 1: to develop advanced web navigation that would continually improve itself
Speaker 1: on the basis of user generated data end quote, which
Speaker 1: is a pretty advanced idea for nineteen ninety six when
Speaker 1: the Web was still very young and the general public
Speaker 1: was still just trying to get a grip on exactly
Speaker 1: what the Web and by extension, the Internet were. One
Speaker 1: of the first tools that Alexa Internet developed was a
Speaker 1: browser toolbar. So installing this toolbar into a browser would
Speaker 1: give the user's access to a sort of crowd powered
Speaker 1: recommendation engine. In some ways, it's not that different from
Speaker 1: sites like dig and Reddit that would later rely on
Speaker 1: the user community to actually work and to recommend links
Speaker 1: to really interesting sites. This toolbar would recommend the sites
Speaker 1: to users based upon how the overall community was browsing.
Speaker 1: So the more people who were using this toolbar, the
Speaker 1: more information was going into where they were going, and
Speaker 1: thus you would get different recommendations. So if a lot
Speaker 1: of people were navigating to a specific site for whatever reason,
Speaker 1: you might get a recommendation to do the same. It
Speaker 1: was an attempt at an organic way for folks to
Speaker 1: suggest websites, kind of like a word of mouth campaign,
Speaker 1: and Alexa Internet would also provide meta information about websites
Speaker 1: to users if they wanted it. Meta information is information
Speaker 1: about information, so this would include stuff like how many
Speaker 1: web pages were part of an overall website, or how
Speaker 1: many other websites were pointing back to the one you
Speaker 1: were on, and so forth. A lot of the stuff
Speaker 1: that Alexa Internet could tell you would reflect a specific
Speaker 1: web page's relevance. It's the same sort of information that
Speaker 1: search engines like Google would take into account when deciding
Speaker 1: relevance for search results. And that meant that it didn't
Speaker 1: take very long for Amazon to come around with an
Speaker 1: offer to purchase Alexa Internet. I'll talk about that more,
Speaker 1: as well as the development of the Internet Archive after
Speaker 1: we come back from this quick break to thank our sponsors.
Speaker 1: So Amazon in nineteen ninety nine takes a look at
Speaker 1: Alexa Internet and says, Wow, this is pretty incredible. This
Speaker 1: little company has created some means of checking for stuff
Speaker 1: like relevance and metadata that could be really really useful
Speaker 1: for us, And so Amazon made an offer that Alexa
Speaker 1: Internet couldn't refuse to acquire the company for the and
Speaker 1: slee some of two hundred and fifty million dollars in
Speaker 1: Amazon stock in May of ninety nine. So this is
Speaker 1: a little different than the earlier deal we talked about
Speaker 1: where AOL bought you know, the Ways Incorporated, because they
Speaker 1: bought it with two hundred and fifty million dollars with
Speaker 1: a stock. If we just treated that like it was
Speaker 1: a cash exchange, then if we had just for inflation,
Speaker 1: that's like around four hundred and sixty nine million dollars
Speaker 1: worth of stock. But that's not really how you deal
Speaker 1: with the value here, right. You have to think about
Speaker 1: how much was the stock worth back in nineteen ninety
Speaker 1: nine versus how much is the stock worth today? I
Speaker 1: checked and I saw that in May of nineteen ninety nine,
Speaker 1: Amazon stock was trading for around two dollars eighty nine
Speaker 1: cents per share. These days, it's closer to one hundred
Speaker 1: and eighty dollars per share. Plus. Between that time, Amazon
Speaker 1: had two different stock splits. There was a two to
Speaker 1: one split in late ninety nine, and there was a
Speaker 1: twenty to one stock split in twenty twenty two. When
Speaker 1: you factor all that up, that two hundred and fifty
Speaker 1: million dollars in stock ends up being a ton of wealth.
Speaker 1: Like it's just a huge amount. It would take a
Speaker 1: lot of calculating to get an estimate, and even then
Speaker 1: it wouldn't really be accurate just say that deal is
Speaker 1: worth a lot. So anyway, the important thing with the
Speaker 1: Internet Archive is that Cale and Gileat, through their work
Speaker 1: and creating tools for Alexa Internet, found themselves able to
Speaker 1: create snapshots of the Web. So they were using Alexa
Speaker 1: Internet to have a commercial business, and they established the
Speaker 1: Internet Archive as a way of preserving information that had,
Speaker 1: at some point or another found its home on the Internet.
Speaker 1: So they were using Alexa Internet tech to crawl the
Speaker 1: young Web in order to index everything, which is a
Speaker 1: necessary step if you want to give people access to
Speaker 1: the various documents posted on the web. We first have
Speaker 1: to know what is there and where is it. To
Speaker 1: do that, you've got to index everything. And then they said, well,
Speaker 1: now that we are able to index this, we could
Speaker 1: actually download these little snapshots and keep them. And according
Speaker 1: to the Internet Archive, that would be important because the
Speaker 1: average lifespan for a new web page was not very long,
Speaker 1: So contrary to our belief that once something is posted
Speaker 1: to the Internet, it's there forever, the archive found that
Speaker 1: on average, new web pages stuck around for about seventy
Speaker 1: seven days, which means it's less than three months, and
Speaker 1: then puff they would disappear, like maybe they would change drastically,
Speaker 1: maybe they would just go away. Now, imagine that you
Speaker 1: were to walk into a brick and mortar library, but
Speaker 1: then you found out that on average the books in
Speaker 1: that library would only stick around for three months before
Speaker 1: being lost forever. And think of all the knowledge that
Speaker 1: would disappear on a regular basis and ongoing basis. It
Speaker 1: would be impossible to calculate the impact of that kind
Speaker 1: of reality. It would be like losing the Library of
Speaker 1: Alexandria regularly every three months. So Cale had come to
Speaker 1: the conclusion that knowledge should be preserved and made available
Speaker 1: for posterity. This is similar to an idea that was
Speaker 1: proposed by Stuart Brand back in the nineteen eighties. It's
Speaker 1: a complicated idea that typically gets boiled down to the
Speaker 1: saying information wants to be free. That's actually an oversimplification
Speaker 1: of what Brand was really communicating. But his point was
Speaker 1: that information's value is kind of like a paradox. The
Speaker 1: information could be incredibly valuable, right, it could be absolutely critical,
Speaker 1: and therefore it could be expensive, but the cost of
Speaker 1: distributing information was consistently declining. It was getting easier and
Speaker 1: cheaper to share information, and the benefits of making information
Speaker 1: accessible are typically pretty tremendous. But information is only accessible
Speaker 1: if someone is able to hold onto that info. Otherwise
Speaker 1: it's lost. Right, The Internet was such a volatile thing
Speaker 1: that there was no guarantee that what you saw today
Speaker 1: would be available tomorrow. In the days before the dynamic web,
Speaker 1: it wasn't really unusual for someone to establish a web page,
Speaker 1: to publish that page, and then later on to wipe
Speaker 1: the slate clean or you know, otherwise alter vast portions
Speaker 1: of that page in order to use that same web
Speaker 1: landscape to host a totally different document. So the old
Speaker 1: stuff would just disappear. And so Calee and Gilliat created
Speaker 1: the Internet Archive, a nonprofit organization dedicated to the archival
Speaker 1: of information across the Internet. And I think most people
Speaker 1: are familiar with it from the web wayback machine, but
Speaker 1: that's just one part of what the Internet Archive does.
Speaker 1: As stated in the Library of Congress, the mission of
Speaker 1: the Internet Archive was quote offering permanent access for researchers,
Speaker 1: his story and scholars to historical collections that exist in
Speaker 1: digital format end quote. Cale and Gilliat founded the Internet
Speaker 1: Archive the same year they founded Alexa Internet. So that's
Speaker 1: nineteen ninety six. And it wasn't easy. And why is that? Well,
Speaker 1: you got to think about the challenge you face if
Speaker 1: you want to archive everything on the Internet, or at
Speaker 1: least everything that you're allowed to archive on the Internet.
Speaker 1: We'll come back to that a couple of times. So,
Speaker 1: for one thing, you need to create a way to
Speaker 1: capture the content of a web page and to preserve
Speaker 1: that for posterity. And you need a way for people
Speaker 1: to access those archived web pages and to navigate them.
Speaker 1: So Alexa Internet would end up developing these technologies and
Speaker 1: commercializing them in various ways, and the Internet Archive was
Speaker 1: made possible through these tools. So you could think of
Speaker 1: Alexa Internet as being the funding machine for Internet Archive
Speaker 1: in the beginning, at least as far as the tools
Speaker 1: Internet Archive would use in order to achieve its mission. Now,
Speaker 1: on the capturing front, Alexa Internet created a web crawler.
Speaker 1: So for applications like web search engines, primarily web search engines,
Speaker 1: web crawlers are the soldiers that they send out. A
Speaker 1: web crawler's job is to index content across the Internet
Speaker 1: and to capture information about what the various web pages
Speaker 1: on the Internet are actually about. It's complicated, right. You
Speaker 1: could just have a directory of web pages that's based
Speaker 1: off the title of the web pages, but title and
Speaker 1: content are not always in alignment. So web crawlers are
Speaker 1: all about following the various branching pathways across the web.
Speaker 1: They crawl through the web, in other words, indexing every
Speaker 1: page as they do. So. Not everyone, however, wants their
Speaker 1: web page indexed. So you can actually include some HTML
Speaker 1: language in your web page that indicates that it's off
Speaker 1: limits for indexing, and appolite web crawlers such as the
Speaker 1: ones that Alexi Internet was using, will honor those instructions
Speaker 1: and it will not index that page. But other pages
Speaker 1: that lack this specific instruction of hey, don't index this,
Speaker 1: they're fair game. I like to think of web crellers
Speaker 1: kind of like Doctor Strange from the Marvel Universe the
Speaker 1: Cinematic Universe in particular, they all want. He uses his
Speaker 1: time manipulation abilities to see where all the different possible
Speaker 1: pathways can lead to. The web crellers do that across
Speaker 1: the web. They explore all the nooks and crannies. They
Speaker 1: follow each link that even the ones that no one
Speaker 1: ever clicks on, they follow those two. And you know,
Speaker 1: hats off to web crellers for doing that to build
Speaker 1: out these indices, because without it, web search wouldn't work,
Speaker 1: and Alexa Internet wouldn't have been a thing anyway. Alexa
Speaker 1: Internet and by extension, the Internet Archive used several different
Speaker 1: web crallers over the years, but they all basically do
Speaker 1: the same thing, or they they you know, more accurately.
Speaker 1: They all aimed to achieve the same results. So the
Speaker 1: crawler starts with seed URLs. This is like the starting
Speaker 1: point where you let them go, and then they follow
Speaker 1: each link and they download documents to the archives servers.
Speaker 1: The crawlers also reference the links to ensure that they're
Speaker 1: not double dipping on a specific crawl. So if you
Speaker 1: have a ton of different sites that are all linking
Speaker 1: to the same document, like let's say that someone has
Speaker 1: published something, and hundreds of other resources on the internet
Speaker 1: reference that published document, Well, That means there's all these
Speaker 1: different pathways that lead to the same destination, right, and
Speaker 1: it would be somewhat wasteful to capture this exact same
Speaker 1: document multiple times during the same crawl, so there's cross
Speaker 1: referencing that happens in order to prevent that from occurring.
Speaker 1: This process does work, but it also has limitations. So
Speaker 1: for one thing, these crawls they do create snapshots of
Speaker 1: the web in intervals, So if you use the wayback machine,
Speaker 1: we'll talk more about that in a second. You'll see
Speaker 1: that the history of a web page consists of a
Speaker 1: series of dates from which the Internet archive first received
Speaker 1: a snapshot of that page, and it leads all the
Speaker 1: way up to the most recent reference of that page,
Speaker 1: the most recent snapshot. The various dates and the wayback
Speaker 1: machine are not necessarily relevant to any major changes that
Speaker 1: happened on the web page itself. This is just when
Speaker 1: the web crawlers went to that particular web page. So
Speaker 1: it may be immediately after a massive change has been implemented,
Speaker 1: it may be well after. In fact, there might be
Speaker 1: a point where between webcraller visits a web page has
Speaker 1: changed a couple of times. Well, that means that the
Speaker 1: ones that are happening in between those changes aren't going
Speaker 1: to be captured. It's just whatever was there the first
Speaker 1: time the web crawler came through, and whatever was there
Speaker 1: the next time the web craller came through. So interesting
Speaker 1: thing is that if a particular page does have a
Speaker 1: ton of other links pointing to it, that page is
Speaker 1: more likely to have very frequent snapshots throughout its history,
Speaker 1: because again, through subsequent crawls, there are various routes that
Speaker 1: take web crallers through that web page, so they're more
Speaker 1: likely to capture a snapshot of it. For pages that
Speaker 1: have fewer links pointing to them, maybe there aren't that
Speaker 1: many other web pages out there that cite this particular page,
Speaker 1: they're more likely to have sporadic updates throughout their history.
Speaker 1: You might pull up a page in the Wayback machine
Speaker 1: and see that there's only maybe half a dozen captures
Speaker 1: of that particular page, and that means that there could
Speaker 1: be a lot of changes that were missed in between visits.
Speaker 1: So not everything gets captured in the Internet archive. I
Speaker 1: think that some people work under the mistaken presumption that
Speaker 1: anything that was ever published to the web is captured
Speaker 1: and archived. There that's not the case. It's whatever was
Speaker 1: there when the web crawlers came through it. So, because
Speaker 1: even the Internet Archive is not a perfect record of
Speaker 1: everything that's ever happened on the web, other elements, like
Speaker 1: I said, could also be lost to time due to
Speaker 1: the complexity of web navigation. For example, so when web
Speaker 1: designers started to incorporate things like flash, which really is
Speaker 1: no longer a thing but it was for a while,
Speaker 1: or JavaScript, then the web callers that were being used
Speaker 1: to index the web, a lot of them just couldn't
Speaker 1: navigate these types of tools that were made through flash
Speaker 1: or JavaScript. So while human users could, and they could,
Speaker 1: you know, interact with interfaces that had these tools created
Speaker 1: through these various methods, web collers couldn't. And that meant
Speaker 1: that if a website used like tools that were made
Speaker 1: in JavaScript to act as the interface, the web creller
Speaker 1: might only be able to index the homepage, but not
Speaker 1: any of the other links branching off from the homepage
Speaker 1: because it couldn't navigate that same interface. So there's a
Speaker 1: lot of stuff from that era that's lost to the
Speaker 1: Internet Archive as well, simply because the crawlers just could
Speaker 1: not navigate those pages. They were never captured. And like
Speaker 1: I said, if you happen to have the instruction, the
Speaker 1: HTML instruction not to index the site, well then that's
Speaker 1: not going to be there either. Now let's move on
Speaker 1: to another challenge, which is the storing of these files.
Speaker 1: Indexing everything was one thing. How do you store everything
Speaker 1: that can be indexed on the web in an archive?
Speaker 1: That's what we're going to come back and explore after
Speaker 1: we take another quick break to thank our sponsors. Okay,
Speaker 1: so the Internet archive, how do you store all the
Speaker 1: information that you find across the web. Well, the big
Speaker 1: one for web pages was that you had to figure
Speaker 1: out where do you store and how do you organize
Speaker 1: snapshots of the web so that one you have a
Speaker 1: record of them, and two you can find what you're
Speaker 1: looking for. You can navigate to the specific instance that
Speaker 1: you're looking for. Keep in mind again, the archives not
Speaker 1: capturing everything. As I said before the break, there's a
Speaker 1: lot of stuff that web crawlers could not access for
Speaker 1: one reason or another. Those things would be either off
Speaker 1: limits or inaccessible and thus would not be in the archive.
Speaker 1: But everything else was still fair game. So to store
Speaker 1: and organize everything, Alexa Internet created a new file format
Speaker 1: called an ARC file. ARC ARC files contain information about
Speaker 1: all the stuff that's inside them, the metadata of the Internet.
Speaker 1: So again, metadata is data about data. It makes the
Speaker 1: small files inside the larger ARC files all self identifying,
Speaker 1: so there's no need to actually build out an index.
Speaker 1: The self identifying information includes stuff like the URL for
Speaker 1: the file, like what the URL for that particular document is,
Speaker 1: how big the document is when it was retrieved, and
Speaker 1: other stuff like that. Each ARC file would have a
Speaker 1: capacity of around one hundred megabytes, and it was possible
Speaker 1: for a single website to span multiple ARC files. I mean,
Speaker 1: there's some big websites out there that have been around
Speaker 1: for a long time, so yeah, sometimes a single ARC
Speaker 1: file would just be a portion of that website. At first,
Speaker 1: the Internet archives stored all this information on magnetic tape,
Speaker 1: So you would do this indexing of the web, all
Speaker 1: these snapshots, and you would save it to magnetic tape.
Speaker 1: I remember I used to work for a company, a
Speaker 1: consulting firm that had magnetic tape backups. So it was
Speaker 1: my job, one of my jobs to occasionally back up
Speaker 1: all the data on our network to tape, and I
Speaker 1: would have to swap tapes out and label them and
Speaker 1: everything and archive them properly. The Internet Archive worked under
Speaker 1: the same idea. It would capture a snapshot of all
Speaker 1: the files across the web, save them to tape, and
Speaker 1: that was how the Internet Archive kept track of things
Speaker 1: for about three years. But eventually activity on the Internet
Speaker 1: was such that that was not going to do it.
Speaker 1: There were too many users who wanted to be able
Speaker 1: to access things that were stored or saved within the
Speaker 1: Internet Archive, and this method just couldn't keep up with
Speaker 1: demand and necessity, as we all know, is the mother
Speaker 1: of invention. So the Internet Archive needed an alternative way
Speaker 1: to store these snapshots. And of course, the Web was
Speaker 1: really growing dramatically, which is putting it lightly, and there
Speaker 1: was a real need to step things up considerably. So
Speaker 1: to that end, the staff at Internet Archive developed a
Speaker 1: storage system they called the PetaBox PetaBox, and it was
Speaker 1: called the PetaBox because it could house a petabyte of information.
Speaker 1: A petabyte, in case you're curious, is a million gigabytes. Now,
Speaker 1: the most recent data I have about the PetaBox storage
Speaker 1: system actually comes from December twenty twenty one, so it's
Speaker 1: a few years out of date. But at that time,
Speaker 1: the Internet Archive was using two hundred and twelve petabytes
Speaker 1: of storage, which is a lot that wasn't all the
Speaker 1: Wayback Machine. However, only around fifty seven petabytes of that
Speaker 1: was for the Wayback Machine. The rest was for other
Speaker 1: things like archiving various forms of digital media as well
Speaker 1: as what Internet Archive references as quote unquote unique data. Anyway,
Speaker 1: the page on Internet Archive site says that the data
Speaker 1: centers there are four of them that house the petabyte
Speaker 1: storage system, don't use air conditioning, which helps keep electric
Speaker 1: bills down. They actually let the heat from the data
Speaker 1: storage devices provide heating for the buildings that they're stored
Speaker 1: in and that you know, this is all part of
Speaker 1: a strategy to keep things at low cost but high
Speaker 1: usability and high efficiency. So that's really the big requirements
Speaker 1: for the PetaBox system. It has to be efficient. It
Speaker 1: cannot require too much power to operate any single PetaBox.
Speaker 1: Another requirement is that each rack of hard drive storage
Speaker 1: has to hold a ton of hard drives. We're talking
Speaker 1: like one hundred plus terabytes worth of hard drive space.
Speaker 1: Another requirement is that to serve as an administrator, it
Speaker 1: needs to be easy like it can't be complicated to
Speaker 1: administrate this storage system, and according to Internet Archive, the
Speaker 1: structure of this is such that you need about one
Speaker 1: administrator for every petabyte worth of data, so you know,
Speaker 1: that's like two hundred administrators. Essentially, the whole goal was
Speaker 1: to create systems that were relatively inexpensive, relatively efficient, and
Speaker 1: relatively easy to use. At least from an administrative perspective.
Speaker 1: That's really tall order. It's hard to meet all those
Speaker 1: but the folks at Internet Archive made it happen, and
Speaker 1: it was such a useful approach to storage and to
Speaker 1: being able to organize the files within storage so that
Speaker 1: you didn't have to build out indices that ultimately Internet
Speaker 1: Archive would deploy this same strategy for other organizations and institutions. Okay,
Speaker 1: but that's all about, you know, collecting and storing all
Speaker 1: the information across the Internet. How do you access it?
Speaker 1: How is a user? How is a researcher? Are you
Speaker 1: able to tap into this? Because again, unless accessibility is easy,
Speaker 1: then there's not much point to doing this. You're just
Speaker 1: making a record that nobody can reference. Well, I would
Speaker 1: argue the most famous of the ways to access information
Speaker 1: contained within the Internet Archive is the wayback machine, which
Speaker 1: is specifically for web pages. The Internet Archive first introduced
Speaker 1: the wayback Machine in two thousand and one, and the
Speaker 1: way it works is pretty simple. There's a little it's
Speaker 1: kind of like a search bar, but it's a urlbar.
Speaker 1: You put in a URL for the web page that
Speaker 1: you're interested in, and the wayback machine pulls up the
Speaker 1: snapshots that are contained within the archive if there are
Speaker 1: any snapshots. As I mentioned earlier, not everything is in there,
Speaker 1: but if it is in there, you will see options
Speaker 1: available to you to look at the page at different
Speaker 1: points in history. One thing I like to do is
Speaker 1: look back at how famous web pages have changed in
Speaker 1: their design over the years. If you put in something
Speaker 1: like really big like CNN dot com, you can see
Speaker 1: how the look and interface of that site has transitioned
Speaker 1: during different eras across the web. I also used to
Speaker 1: do this with the old website I worked for houstuffworks
Speaker 1: dot com. I mean that's where tech stuff gets the
Speaker 1: stuff and its name is from HowStuffWorks dot com. I
Speaker 1: like using the wayback machine to look at what the
Speaker 1: site looked like when I first joined, which was a
Speaker 1: in February two thousand and seven. In case you're curious.
Speaker 1: It looks entirely different now than how it looked back then,
Speaker 1: and through the wayback Machine you can see what it
Speaker 1: looked like back then. Also, these days, the wayback machine
Speaker 1: is the only way I can see some of the
Speaker 1: articles I wrote for that site, because the articles have
Speaker 1: been either deleted or more likely rewritten over the time. Now.
Speaker 1: To be fair to how stuff works, a lot of
Speaker 1: my writing was in the computers and electronics sections, and
Speaker 1: obviously things change in those fields very quickly, and something
Speaker 1: that was relevant fifteen years ago is definitely not relevant today.
Speaker 1: So you have to replace old stuff on a regular basis.
Speaker 1: But it is kind of sad that a lot of
Speaker 1: my work, a lot of my work for the first
Speaker 1: you know, ten years of my career doing this kind
Speaker 1: of stuff, is not accessible unless you use something like
Speaker 1: the wayback Machine. Now, one super neat thing about the
Speaker 1: wayback machine is that you can still follow links that
Speaker 1: are on pages, like if the archive has those linked
Speaker 1: assets also in the archive, then you're going to be
Speaker 1: shown a record, and the record will be one that
Speaker 1: was captured closest in time with the first page that
Speaker 1: you were originally on. This sounds complicated, Let me give
Speaker 1: an example, it makes it way easier. So let's say
Speaker 1: that I visit the web capture the snapshot for HowStuffWorks
Speaker 1: dot COM's homepage on February nineteenth, two thousand and seven.
Speaker 1: By the way, this snapshot on feb nineteenth, two thousand
Speaker 1: and seven is the closest date to when I started
Speaker 1: working at that company that's in the archive. The actual
Speaker 1: date when I started the website was not captured on
Speaker 1: that day. Anyway, By clicking around on this homepage, I
Speaker 1: can actually follow links and it'll pull up archived links
Speaker 1: of archived articles, which is really neat. And when I
Speaker 1: did that, at one point, I clicked on a link
Speaker 1: for more information or related articles to how helicopters work.
Speaker 1: That page, the related page was actually archived on February
Speaker 1: twenty second, two thousand and seven. So one was on
Speaker 1: February nineteenth, the other was February twenty second, but the
Speaker 1: link still worked. Right. Yes, these were two different pages
Speaker 1: that were archived on two different days, but the nature
Speaker 1: of the archive allows those links to still work between
Speaker 1: the two, which is neat because I'm not just popping
Speaker 1: around through a web of links. I'm also kind of
Speaker 1: time traveling, right, I'm looking at a timeline of snapshots
Speaker 1: that are all still interlinked together, even if they were
Speaker 1: captured on different days. I think that's really cool. Now
Speaker 1: it gets even more cool when you think about the
Speaker 1: scale of this project. So, according to the Internet Archive itself,
Speaker 1: the archive contains eight hundred and thirty five billion with
Speaker 1: a B web pages, And as I mentioned earlier, that
Speaker 1: just makes up part of all the data that's stored
Speaker 1: on Internet Archive servers, because the organization is also home
Speaker 1: to more than forty four million books and other texts,
Speaker 1: fifteen million audio recordings, more than ten million videos, and
Speaker 1: more than a million different pieces of software. Again, some
Speaker 1: of this stuff might not be recorded anywhere else. There
Speaker 1: may not be duplicates or copies of some of this
Speaker 1: stuff anywhere else. While you might have things like Blu
Speaker 1: ray DVDs or whatever of some of those videos, others
Speaker 1: might not have anything. And history is filled with instances
Speaker 1: of media companies generating stuff or others, you know, independent
Speaker 1: people too, generating stuff but not keeping a copy for posterity,
Speaker 1: and then it's here and it's gone. Sometimes that's on purpose.
Speaker 1: Sometimes it's a statement, like you make something ephemeral for
Speaker 1: that very reason. Other times it's out of convenience, Like
Speaker 1: there are stories about how the BBC would regularly reuse
Speaker 1: tapes and tape over previous programming because there was no
Speaker 1: thought about preservation or a home theater industry. So there
Speaker 1: are entire eras of stuff like Doctor Who that are
Speaker 1: just gone or believed to be gone because the BBC
Speaker 1: would just tape over old tapes and so you lost
Speaker 1: whatever was on there originally. That's why things like the
Speaker 1: Internet Archive exist is to avoid that in the case
Speaker 1: of stuff that's stored across the Internet, to make sure
Speaker 1: that there is an accessible record of those things and
Speaker 1: that they don't just disappear. In two thousand and seven,
Speaker 1: the state of California recognize the Internet Archive as an
Speaker 1: official library, which was important it's not just an honorarium.
Speaker 1: It would allow the nonprofit organization to receive federal funding,
Speaker 1: which is a pretty important development for the longevity of
Speaker 1: the program. But while the usefulness of the organization is
Speaker 1: beyond question, the methods that the Archive has used this
Speaker 1: have not always been met with universal approval. For example, recently,
Speaker 1: the Internet Archive has been embroiled in a pretty nasty lawsuit.
Speaker 1: It's called the Hatchet versus Internet Archive suit, and it
Speaker 1: revolves around a group of publishers that object to how
Speaker 1: the Internet Archive scans physical books for the purposes of
Speaker 1: lending them out as digital copies. Publishers are in the
Speaker 1: business of publishing and selling copies of books, but for years,
Speaker 1: libraries have existed in order to get copies of various
Speaker 1: books and to make them available for lending. So libraries
Speaker 1: have to purchase the books or have them donated to
Speaker 1: the library, and then makes those books available to lend
Speaker 1: out to members of the library. The Internet Archive has
Speaker 1: a controlled digital lending program to handle this sort of thing,
Speaker 1: only we're talking about digital formats, not a physical copy
Speaker 1: of a book. This is where things get tricky because obviously,
Speaker 1: if you, as a American citizen at least, if you
Speaker 1: go out and buy a copy of a book, you
Speaker 1: can do whatever you like with your copy of that book,
Speaker 1: apart from making your own copies of it and then
Speaker 1: selling those. You can't do that. That's copyright infringement. But
Speaker 1: if you own a physical copy of a book, you can.
Speaker 1: You can keep it for yourself. You could lend it
Speaker 1: to a friend and let them read it, they return
Speaker 1: it to you later. You could give the book away.
Speaker 1: You could resell your copy to someone else, even if
Speaker 1: you're selling it for a fraction of what the book
Speaker 1: is going for in bookstores. You could do that. You
Speaker 1: could even burn the darn thing if you're so inclined.
Speaker 1: Just don't do that. Don't burn books. But all of
Speaker 1: those things are permitted with your personal copy of the book. However,
Speaker 1: a digital copy, well, now we're starting to talk about
Speaker 1: different rules. So yes, you can lend out a physical
Speaker 1: copy of a book. That's allowed. That's fair use. But
Speaker 1: actually it's not even fair use. That's under laws of property.
Speaker 1: But we won't get into all that. A digital copy
Speaker 1: is a lot trickier because it's easy to replicate, much
Speaker 1: easier than replicating a physical copy of a book, and
Speaker 1: so different rules have developed to handle digital information compared
Speaker 1: to stuff that's in our physical meat space. So this
Speaker 1: lawsuit argues that the Internet Archive first digitized physical books
Speaker 1: without permission from the publishers, and that that was problem
Speaker 1: number one. There's been some different arguments about that, like
Speaker 1: if there was no ebook equivalent of the copy of
Speaker 1: the book, if the publishers had not digitized that, that's
Speaker 1: slightly different than if the publishers also offer an electronic
Speaker 1: version of the physical books they sell. But the other
Speaker 1: problem is that the Internet Archive received donations and funding
Speaker 1: that in part stemmed from the practice of lending out
Speaker 1: digitized books, So the publisher said that made the Internet
Speaker 1: Archives activities a commercial enterprise. In twenty twenty three, a
Speaker 1: judge found in favor of the publishers, saying that the
Speaker 1: Internet Archive failed to argue that their work fell under
Speaker 1: the principles of fair use. Again, getting into fair use,
Speaker 1: that's a whole thing, but generally speaking, fair use covers
Speaker 1: a relatively narrow set of use cases in which the
Speaker 1: copying are the use or the distribution of a copyrighted
Speaker 1: work does not count as copyright infringement. But it has
Speaker 1: to meet certain criteria, and it's only ever decided in
Speaker 1: a court of law. It's not something that's just you
Speaker 1: can apply to proactively. It's something that you use in
Speaker 1: a defense if you're brought up on charges of copyright infringement.
Speaker 1: So by the time you're actually talking fair use, it's
Speaker 1: already pretty late in the game. But anyway, this particular
Speaker 1: lawsuit is under appeal. The Internet Archive recently made final
Speaker 1: arguments in the case I have not seen anything about
Speaker 1: the case being decided one way or the other since then,
Speaker 1: so I'm not really sure which way it's going. Again.
Speaker 1: I didn't see anything about a decision made, but then
Speaker 1: most the articles about this are about the initial trial
Speaker 1: that happened in twenty twenty three, so hopefully I will
Speaker 1: find some follow up on this at some point. But
Speaker 1: there's no denying the Internet Archive has done a tremendous
Speaker 1: amount of work in the field of knowledge preservation and
Speaker 1: knowledge accessibility. Without the Internet Archive, there's no way of
Speaker 1: knowing how much information would be lost to us forever.
Speaker 1: Stuff that could have been incredibly useful or even just
Speaker 1: diverting could be gone, and we'd never have a way
Speaker 1: of retrieving it again. And I am very thankful that
Speaker 1: an organization like the Internet Archive exists. If you're not
Speaker 1: familiar with it, if you never used it, I recommend
Speaker 1: you check it out and explore the Internet Archive. Look
Speaker 1: at some of the things that are in that archive,
Speaker 1: like some of the books, some of the recordings. There's
Speaker 1: some great stuff. I think there's like a quarter of
Speaker 1: a million live performances archived just on the Internet Archive,
Speaker 1: like live music performances. That alone is super cool. Anyway,
Speaker 1: I hope you found this episode informative and entertaining. I
Speaker 1: hope you check out Internet archive. I also very much
Speaker 1: hope that you are all well and I will talk
Speaker 1: to you again really soon. Tech Stuff is an iHeartRadio production.
Speaker 1: For more podcasts from iHeartRadio, visit the iHeartRadio app, Apple Podcasts,
Speaker 1: or wherever you listen to your favorite shows.