Speaker 1: Welcome to Textuff, a production from my Heart Radio. Hey there,
Speaker 1: and welcome to tech Stuff. I'm your host, Jonathan Strickland.
Speaker 1: I'm an executive producer with I Heart Radio and I
Speaker 1: love all things tech and listener David reached out to
Speaker 1: me on Twitter and said, I would like to hear
Speaker 1: an episode on search engine spiders. He is our hero.
Speaker 1: You gotta David, and if you get the spider, he
Speaker 1: is our hero. Reference let me know. So we're gonna
Speaker 1: talk about the development of search engines and how they
Speaker 1: work from admittedly a pretty high level, because to go
Speaker 1: into great detail would probably take three or four episodes plus.
Speaker 1: Different search engines use slightly different strategies in order to
Speaker 1: index and rank search results. And the reason I'm doing
Speaker 1: all of that is because if we just talked about spiders,
Speaker 1: it would be a fairly short episode. But what the
Speaker 1: heck is a search engine spider. Well, the index the
Speaker 1: contents of the world wide Web. You need to search
Speaker 1: around and find what's there first, right, you can't return
Speaker 1: results without first knowing what is out there in the
Speaker 1: first place. So a search engine spider is a bot
Speaker 1: that does this. It crawls through the Worldwide Web. Thus
Speaker 1: the whole spider name. We'll learn more about what's actually
Speaker 1: going on a little bit later, but to understand search,
Speaker 1: we need a few more basics. So keep in mind
Speaker 1: that all the stuff we see online, whether it's a
Speaker 1: web page or it's a web service or whatever it
Speaker 1: may be, it ultimately sits on a computer that is
Speaker 1: connected to the Internet infrastructure, so it's connected to routers,
Speaker 1: which then connected to various servers and and domain name
Speaker 1: servers and all that kind of stuff. If you visit
Speaker 1: the house Stuff Works homepage, that's the site for the
Speaker 1: company that I used to work for, don't work for
Speaker 1: them anymore, But that website consists of pages that are
Speaker 1: on a computer in a data center. If you happen
Speaker 1: to know the u r L for the site, so
Speaker 1: you happen to know how stuff works dot com, you
Speaker 1: can type that into a browser u r L bar
Speaker 1: address bar, and the browser will then take care of
Speaker 1: sending the appropriate message to that computer. In this case,
Speaker 1: we will call it a server, and the server will
Speaker 1: then return the appropriate information to your browser the web
Speaker 1: page maybe the home page for how Stuff Works in
Speaker 1: this case, and then you'll see the website. But all
Speaker 1: of that requires that First, you have to know that
Speaker 1: there's a site there at all. Plus you have to
Speaker 1: know the U r L for it, and you might
Speaker 1: not have that information. Before there even was a Worldwide Web,
Speaker 1: there was a need to know where you could find
Speaker 1: stuff on the Internet. Now, remember, the Internet is older
Speaker 1: than the Web, and the Internet and the Web or
Speaker 1: not the same thing. The Web exists on top of
Speaker 1: the Internet. It consists of a lot of other stuff
Speaker 1: besides the Web, right, like email and FTP servers. In fact,
Speaker 1: we need to really talk about FTP servers. FTP stands
Speaker 1: for File Transfer Protocol. So these are computers that house
Speaker 1: certain files on them and through FTP. Through this protocol
Speaker 1: that allows for files to transfer from one computer to
Speaker 1: another across a network connection. People can thus access files
Speaker 1: that can transfer them from the server to their own computer,
Speaker 1: which in this case we would call a client. But again,
Speaker 1: FTP is really only useful if you know the address
Speaker 1: of the servers where the stuff is that you want. Right,
Speaker 1: you can't just use FTP to pull a file out
Speaker 1: of nowhere. You have to contact the proper server and
Speaker 1: pull the relevant file from that server. Enter an Emtaj,
Speaker 1: who in nineteen nine was a graduate student at McGill
Speaker 1: University in Montreal, Canada. He also worked as a systems
Speaker 1: administrator for the School of Computer Science at the university,
Speaker 1: and he was running into a challenge. It was his
Speaker 1: job to locate software for professors, for staff, for students
Speaker 1: at the university, but there was no easy way to
Speaker 1: know where all the various files were on the network
Speaker 1: of public FTP servers. Emtage decided there needed to be
Speaker 1: a way to get a snapshot of which public FTP
Speaker 1: servers had which files. There needed to be some sort
Speaker 1: of directory, and since servers were popping up more frequently
Speaker 1: as more people began to develop stuff for the Internet,
Speaker 1: there also needed to be a good way to search
Speaker 1: those lists to find something specific. Otherwise it would be
Speaker 1: like reading through an entire phone book to find out
Speaker 1: which person or business corresponded to a phone number you
Speaker 1: happen to have seen. Let's say the phone number was
Speaker 1: eight six seven five three oh nine, and you don't
Speaker 1: know that that's Jenny's phone number. You just know it's
Speaker 1: the number. So instead of calling the number and asking, Hey,
Speaker 1: whose number is this, you get a phone book and
Speaker 1: you start searching for eight three o nine to find
Speaker 1: the corresponding name that is not efficient. In fact, in
Speaker 1: the early days, information about servers frequently had no other
Speaker 1: real channel to get to users other than word of mouth.
Speaker 1: So there was a really good chance that there was
Speaker 1: stuff that was relevant to you that you just had
Speaker 1: no way of knowing about because you had to hear
Speaker 1: it from somebody else first. Imtash, along with a couple
Speaker 1: of other folks like Bill Healen and J. Peter Deutsch,
Speaker 1: began building a tool to solve this problem. They ended
Speaker 1: up calling this tool Archie, which actually was not a
Speaker 1: nod to the comic book character from Archie Comics. Instead,
Speaker 1: Archie was a somewhat shortened form of the word archives.
Speaker 1: They created programs that could look through the repositories of
Speaker 1: public FTP sites and get an inventory of the files
Speaker 1: stored on those servers or as documented in the book
Speaker 1: A Rough Guide to the Internet by Nicholas West Quote,
Speaker 1: it combined a script based data gatherer which accessed listings
Speaker 1: from anonymous sites with a script which matched regular expressions
Speaker 1: which could retrieve file names matching a user query end
Speaker 1: quote simple right now. In case you're like me and
Speaker 1: what I just quoted sounded a little bit confusing. One
Speaker 1: it really boils down to is to say they made
Speaker 1: a computer program that followed some fairly simple rules. The
Speaker 1: program made note of the file titles that were on
Speaker 1: various FTP servers, kind of like a list of contents,
Speaker 1: and they noted which files were on which servers. Another
Speaker 1: part of the program arranged those findings into a database
Speaker 1: not that much different from the types of spreadsheets you've
Speaker 1: probably worked with in the past. Imtaj and crew also
Speaker 1: created a tool that would allow them to search this database.
Speaker 1: Before long, other people began to hear that he had
Speaker 1: this database and that they would ask him, Hey, can
Speaker 1: you do a search for me, and they would give
Speaker 1: him the search terms, and it started taking up a
Speaker 1: lot of his time. So in an effort to streamline things,
Speaker 1: he programmed a user interface or UI that would allow
Speaker 1: people to conduct their own searches. They could just log
Speaker 1: into this tool and then type in the file that
Speaker 1: they were looking for and it would return the results
Speaker 1: for them. So as long as they were sure about
Speaker 1: the specific file they needed, they would get the results. Now,
Speaker 1: most resources generally agree that Archie was the first real
Speaker 1: search engine on the Internet, but it wasn't a web
Speaker 1: search engine, the Web didn't exist yet. It wasn't long
Speaker 1: before a couple of other tools followed. In some researchers
Speaker 1: with the University of Minnesota develop a new tool to
Speaker 1: organize and discover documents stored on servers, and the tool
Speaker 1: was called the Gopher protocol. Servers were data repositories called
Speaker 1: Gopher holes eventually that's what they were called anyway, and
Speaker 1: Gopher organized everything into a hierarchical text based menu system.
Speaker 1: So this was a specific strategy that was built on
Speaker 1: top of the Internet. It was kind of think of
Speaker 1: it as being in parallel with the Web. It predated
Speaker 1: the Web, but the Web and Gopher would exist at
Speaker 1: the same time, but they were not the same thing.
Speaker 1: This was a different strategy in order to serve information
Speaker 1: across the networks. Before long, like in more researchers developed
Speaker 1: a search function to work on top of Gopher. Because again,
Speaker 1: if you didn't know where something actually quote unquote lived
Speaker 1: in the Gopher network, you would never be able to
Speaker 1: find it unless you were just lucky. So this search
Speaker 1: tool was called Veronica, and that is pretty cute because
Speaker 1: Veronica is a character in the Archie comics books. And
Speaker 1: while the search engine Archie did not pull its name
Speaker 1: from Archie Comics. Veronica was a nod to the older
Speaker 1: search engine as well as a nod to the comic book,
Speaker 1: so it almost kind of retroactively made Archie relate back
Speaker 1: to the comics. Later, computer geeks assigned a backronym to Veronica.
Speaker 1: This is an acronym that you create after you've already
Speaker 1: named a thing. So you've given the thing a name,
Speaker 1: and then you're thinking, okay, well, what can we say.
Speaker 1: Each of those letters stands for that's a backronym, And
Speaker 1: in this case, the revisionist name was very easy, rodent
Speaker 1: oriented net Wide Index to Computer Archives, or Veronica super cute.
Speaker 1: What Veronica did was fairly primitive. It created a database
Speaker 1: of every file and every directory on every go for
Speaker 1: a server that was connected to the Internet, and it
Speaker 1: would update dynamically as more servers joined the network. That
Speaker 1: approach worked fairly well when there was still a relatively
Speaker 1: small number of servers to keep track of, But as
Speaker 1: more servers came online and joined this Gopher network, with
Speaker 1: more documents stored on each server, it started to get
Speaker 1: a really you know, challenging to manage Veronica, a secondary
Speaker 1: Gopher search tool kind of addressed this problem, and this
Speaker 1: one also took its name from a character from Archie comics,
Speaker 1: jug Head. This one didn't create a full database of
Speaker 1: everything that was on the Gopher network. Instead, as a user,
Speaker 1: you would have to designate which Gopher server you wanted
Speaker 1: to search, so you had to at least have some
Speaker 1: general idea of where it was you needed to look.
Speaker 1: But if you did know that, it was a much
Speaker 1: faster approach than trying to search everything on the network
Speaker 1: as a whole. Gopher had a major problem, and that
Speaker 1: was that was becoming increasingly less efficient and easy to navigate.
Speaker 1: The larger it got, it didn't scale well. Meanwhile, at
Speaker 1: the same time that Gopher was growing, a guy named
Speaker 1: Tim berners Lee over at CERN. You know that's the
Speaker 1: research facility that oversees stuff like the large Hadron collider. Well,
Speaker 1: he was developing a different approach to storing and sharing
Speaker 1: information across networks. Tim and his team at CERN developed
Speaker 1: a protocol called Hypertext Transfer Protocol or h t t
Speaker 1: P and Hypertext Markup Language or h t mL. Both
Speaker 1: of these kind of grew out of stuff that CERN
Speaker 1: had been using internally for a while. Now I'm guessing
Speaker 1: those terms sound familiar to you, guys. These are the
Speaker 1: two components that really formed the basis of web pages
Speaker 1: and the Worldwide Web. The markup language acts as the
Speaker 1: set of instructions on how a computer, or more specifically,
Speaker 1: how a browser is to interpret and display documents, eventually
Speaker 1: including stuff like images and sound files. Although initially the
Speaker 1: web was strictly text based and browsers were text based
Speaker 1: as well. H t t P is the set of
Speaker 1: rules and the processes through which a client that being
Speaker 1: your web browser, can request a specific document from a server,
Speaker 1: and how the server can then send that requested document
Speaker 1: to the browser. The server since the h t m
Speaker 1: L files to the client, and the client interprets those
Speaker 1: HTML files in order to display the relevant web page
Speaker 1: to the user. Hypertext refers to text that has a
Speaker 1: link to some other text, and you can think of
Speaker 1: it kind of like a footnote in a book. The
Speaker 1: hypertext has an asterisk that corresponds to another piece of
Speaker 1: information somewhere that is also marked by an asterisk, except
Speaker 1: in this case the asterisks are invisible. It's highlighted text
Speaker 1: or or text in a different color, or it's underlined.
Speaker 1: It's designated in some way to be different from all
Speaker 1: the rest of the text. That's what lets you know
Speaker 1: it's hypertext and it's linked to something else. Hypertext documents
Speaker 1: connect to one another through hyperlinks. Those documents don't even
Speaker 1: have to be on the same server. They can be
Speaker 1: on opposite sides of the world. So this means you
Speaker 1: can build a reference in one hypertext document to content
Speaker 1: that's found on a totally different hypertext document. Clicking on
Speaker 1: that hypertext activates the link. It sends a command to
Speaker 1: the browser, which then relays that command to the server
Speaker 1: that the client wants to see specific linked information, and
Speaker 1: the server returns that. You can also link the locations
Speaker 1: that are within the same page of a document, or
Speaker 1: specific locations on other pages. It doesn't have to just
Speaker 1: be click on this and you go to a new
Speaker 1: web page. It might be click on this and you
Speaker 1: skip down, you know, a significant number of paragraphs to
Speaker 1: get to the relevant information. Really, all the link is
Speaker 1: doing is telling the browser where some specific point in
Speaker 1: a specific document happens to be and how to get there.
Speaker 1: It's kind of like if you were reading a book
Speaker 1: that said I want to know more, skipped a page
Speaker 1: to nineteen and read the third paragraph or or sometimes
Speaker 1: I compare it to those old choose your own adventure
Speaker 1: books where you get to the bottom of a page
Speaker 1: and you have to make a decision, and based on
Speaker 1: which decision you make, you have to turn to a
Speaker 1: specific page to pick up the story again. Well, you
Speaker 1: can quickly see how this would be really useful. Let's
Speaker 1: say I want to make a web page that includes
Speaker 1: directions and how to perform a particular process. We're gonna
Speaker 1: call it baking a soup FLA and the steps I
Speaker 1: list include references to other, maybe slightly less involved processes
Speaker 1: that are part of this, and I don't go into
Speaker 1: explaining how those work. Let's say, like I talk about
Speaker 1: cracking eggs, but I don't tell you the best way
Speaker 1: to crack an egg. However, I could create hypertext links
Speaker 1: to other sets of instructions, maybe a specific page just
Speaker 1: on different ways to crack an egg, And that way
Speaker 1: you could go and look that up if you weren't confident.
Speaker 1: So if you don't know how something works, you can
Speaker 1: click on that other link and go to a page
Speaker 1: let's dedicated to that in order to learn more. And yes,
Speaker 1: I just described how the web works in general, which
Speaker 1: is something I'm sure you all know at least at
Speaker 1: some level, even if it's not you know, a formal one.
Speaker 1: But if you've ever been on say Wikipedia, reading up
Speaker 1: on a topic and saw hypertext link and thought, yeah,
Speaker 1: I should find out what this term means. I don't
Speaker 1: understand it. So you click on that and you go
Speaker 1: follow that so you can get better understanding, then that's
Speaker 1: the use case I'm referring to. And there's a conversation
Speaker 1: we could have about what actually goes on when you
Speaker 1: click a link, but that would require a deeper dive
Speaker 1: into how the web and by extension, how the Internet
Speaker 1: works on a very technical level, and I think that
Speaker 1: goes beyond the scope of what we're trying to do
Speaker 1: in this episode. So I'm going to simple five, perhaps
Speaker 1: to a ludicrous degree, and say that a link contains
Speaker 1: within it the information about where another document, or even
Speaker 1: a specific point within another document exists, and activating that
Speaker 1: link by clicking on it in a browser initiates a
Speaker 1: sequence that results in the browser requesting that specific document
Speaker 1: from the appropriate server, which then sends that document to
Speaker 1: the browser so that you can see it. A lot
Speaker 1: more is going on to make this happen, but let's
Speaker 1: just stick with that high level view. So the pair
Speaker 1: of h T t P and HTML evolved the same
Speaker 1: time that Gopher was establishing itself, and some people stuck
Speaker 1: with Gopher, but it just really never took off the
Speaker 1: same way that the web did with HTML and HTTP.
Speaker 1: The protocol and the markup language is what the web
Speaker 1: is built upon, and we call it a web because
Speaker 1: of that interconnectivity of documents. You can build out a
Speaker 1: dock and then link that document to another doc which
Speaker 1: might be linked to a dozen others, and by following
Speaker 1: those links, you can navigate from one document to the next.
Speaker 1: It really is similar to what happens to a lot
Speaker 1: of people when they visit Wikipedia and they just start
Speaker 1: following all sorts of links. But you might already see
Speaker 1: a challenge with that kind of design. It works great
Speaker 1: if you've got a centralized person or institution that's building
Speaker 1: out the web, adding pages in a very logical way
Speaker 1: and linking to them in a very logical way. But
Speaker 1: one of tim berners Lee's major goals was to create
Speaker 1: a democratized system that didn't depend upon a centralized authority.
Speaker 1: People should be able to build their own web pages
Speaker 1: and host them on their own servers. But how would
Speaker 1: anyone else find them if there were no links going
Speaker 1: into those pages. If the web pages are made and
Speaker 1: hosted independently of the first few pages on the web,
Speaker 1: where is the connective tissue? The original solution wasn't a
Speaker 1: search engine. It took a it more of a hands
Speaker 1: on approach. I'll explain more, but first let's take a
Speaker 1: quick break. In the early days, when people first started
Speaker 1: building documents to host on the web, in other words,
Speaker 1: the earliest web pages, Tim Burner's leave would take it
Speaker 1: upon himself to create an index hosted on cerns own server.
Speaker 1: Someone might send him a message saying that they had
Speaker 1: built in are hosting a new web page, and they
Speaker 1: could include the address or U R L. Burners Lee
Speaker 1: would then add a hypertext link to a growing catalog
Speaker 1: of those kind of links on a page hosted by CERN.
Speaker 1: So if you visited cerns site, you could navigate to
Speaker 1: that index and see the links to the other sites.
Speaker 1: Tim came to call this a virtual library. He and
Speaker 1: a group of volunteers oversaw its evolution. They organized it
Speaker 1: into different areas of interest, with subject matter experts overseeing
Speaker 1: specific categories, and a lot of these early pages belonged
Speaker 1: to scientific research organizations or universities or publications, and all
Speaker 1: that makes sense. CERN is the organization that oversees the
Speaker 1: large Hadron Collider after all, So it's no surprise that
Speaker 1: the early web really focused on science and academia. Also,
Speaker 1: it's good to mention that the web in those early
Speaker 1: days again was text based. Browsers were text based too,
Speaker 1: that would not really change until another year like nine.
Speaker 1: Mike Gopher's design, the virtual library approach worked fairly well
Speaker 1: when the Web was still small in scale. According to
Speaker 1: the Virtual Library website, in August nine there were about
Speaker 1: twenty web servers in existence total. A little more than
Speaker 1: a year later, in October of nineteen, it was more
Speaker 1: than two hundred web servers, so growth was still fairly modest,
Speaker 1: but things kind of took off after that. By January
Speaker 1: nine six, there were more than one hundred thousand web servers.
Speaker 1: The following year there were more than six hundred fifty thousand.
Speaker 1: It was growing so fast, and maintaining an index was
Speaker 1: becoming increasingly more difficult, particularly by doing it, you know, manually.
Speaker 1: The virtual library was taking shape around the same time
Speaker 1: that students were building the Veronica search engine for gophers,
Speaker 1: so all this was happening around the same time. I
Speaker 1: know it sounds like I'm going strictly chronologically, but that's
Speaker 1: just too not too helpful. We have to remember this
Speaker 1: is all happening simultaneously. So as the web grew and
Speaker 1: became more complex, indices were growing as well. Just navigating
Speaker 1: an index to find what you wanted would become a challenge,
Speaker 1: particularly if you weren't thinking in the same way as
Speaker 1: the people who had organized the index. This is where
Speaker 1: taxonomy comes in. Taxonomy refers to a system of classification.
Speaker 1: A taxonomy is a set of rules we use to
Speaker 1: organize stuff, and there is no one way to do
Speaker 1: it correctly. So I'll give a simple example. Let's say
Speaker 1: you've got a class of students and you have them
Speaker 1: all divide up into smaller groups. You give each group
Speaker 1: a pile of documents, the same documents per group, but
Speaker 1: you tell the students it's their job to organize those documents. Well,
Speaker 1: one group decides that they're going to organize all the
Speaker 1: documents by alphabetizing them by title. The title of each
Speaker 1: document will determine how they fall in the pile, so
Speaker 1: there's are all in alphabetical order. Another group decides that
Speaker 1: they're gonna bundle their documents that all cover the same
Speaker 1: subject matter together, and they'll alphabetize within subjects. So they
Speaker 1: might have a stack that's just about biology, another stack
Speaker 1: that's about chemistry, another one about material science, and so on.
Speaker 1: A third group bundles their docum it's together by author.
Speaker 1: They put all of the same author's works together, and
Speaker 1: then maybe they alphabetize the authors. Yeah. Another group focuses
Speaker 1: on publication date. They arrange all their documents in order
Speaker 1: of when they were published. So you can imagine combinations
Speaker 1: of these approaches as well, right, such as ordering documents chronologically,
Speaker 1: but then if two documents were published on the same date,
Speaker 1: you then alphabetize them. That kind of thing. You can
Speaker 1: think of those different sets of rules, and you have
Speaker 1: to determine which rules are most important, right, which one
Speaker 1: you do first, and which is secondary. Well, it's important
Speaker 1: that these taxonomy's, however you construct them, are consistent or
Speaker 1: else it becomes a chaotic mess. But even a well
Speaker 1: organized and maintained taxonomy can still be a challenge for
Speaker 1: someone new coming into the system. And that's really what
Speaker 1: I'm getting at here. A comprehensive index will seem incredibly
Speaker 1: overwhelming to someone who's unfamiliar with the system's taxonomy, and
Speaker 1: it will still seem like finding a specific document or
Speaker 1: web page is an impossible task. Once that index grows
Speaker 1: to a large enough size, clearly a search tool would
Speaker 1: be a big solution to that problem. If you can
Speaker 1: type a query into a search engine and you can
Speaker 1: get the results you want, you don't have to comb
Speaker 1: through an enormous index hoping that you're thinking in the
Speaker 1: same way as the custodians of that index. But in
Speaker 1: addition to those challenges was one of scale building. This
Speaker 1: index required a lot of volunteer effort because again it
Speaker 1: was done by hand. The next step toward the development
Speaker 1: of a search engine was a project that was tackled
Speaker 1: by an m I T student, Matthew Gray. That was
Speaker 1: the student. He designed a program he called the Worldwide
Speaker 1: Web Wanderer, and the purpose of this program was to
Speaker 1: automatically navigate across the web, cataloging the web's growth by
Speaker 1: registering new websites, web pages, and web servers. The Worldwide
Speaker 1: Web Wanderer is arguably the earliest automated spider or web
Speaker 1: crawler designed for the Web. When Gray developed the program
Speaker 1: back in the web browser Mosaic, which was the first
Speaker 1: popular browser designed for Windows, was just a couple of
Speaker 1: months old. Mosaic was also a graphical browser, and while
Speaker 1: it wasn't the first graphical browser, it was the first
Speaker 1: popular one available to the average person outside of places
Speaker 1: like CERN. Gray wanted to automatically detect new websites for
Speaker 1: discovery purposes, but before long the number of sites was
Speaker 1: growing so quickly that he shifted his attention to charting
Speaker 1: the growth of the web in general. Gray wrote his
Speaker 1: program using the Perl computer language. That's pe r L,
Speaker 1: And here's a quick refresher on computer languages. We know
Speaker 1: that typically computers process information in machine code, which mostly
Speaker 1: for our cases means binary data, and that means all
Speaker 1: the information going through the machines ultimately breaks down into
Speaker 1: zeros and ones, and you can think of that like
Speaker 1: a light switch flipped either off or on. Now, a
Speaker 1: single offer on is easy, but if we want to
Speaker 1: represent more complex ideas processes that kind of thing, we
Speaker 1: need a lot of bits. A single alpha numerical character
Speaker 1: in the as key code requires seven bits, so you
Speaker 1: need seven strings of zeros or ones just to represent
Speaker 1: a letter, number or symbol and as key. So you
Speaker 1: can imagine that programming in machine code would be really
Speaker 1: tough for most humans because it would be so easy
Speaker 1: to mistype a zero or a one, or to skip one.
Speaker 1: And if you're, you know, typing out a really long sequence,
Speaker 1: it's easy for you to overlook a zero or a one,
Speaker 1: And that's why people developed programming languages. A programming language
Speaker 1: creates a layer of abstraction between the programmer and the
Speaker 1: machine or system of machines. It acts as a sort
Speaker 1: of interpreter. It takes the intentions of the programmer and
Speaker 1: turns them into processes a computer can respond to. Some
Speaker 1: programming languages are closer to machine code. Those are low
Speaker 1: level programming languages. They're typically really challenging to work with,
Speaker 1: and others are more abstract and thus easier for us
Speaker 1: to work with, and these are called high level programming languages.
Speaker 1: Pearl falls into that high level category. Now. Originally, Gray's
Speaker 1: program would seek out links on web pages and then
Speaker 1: note the web servers that were hosting those pages before
Speaker 1: following the link over, and then it would repeat that process,
Speaker 1: and the program was really just automating the process that
Speaker 1: we would do manually if we were to look at
Speaker 1: a web page, see a link, and click on it.
Speaker 1: The program saught the links embedded in pages and then
Speaker 1: act made those links to explore whichever documents were pulled
Speaker 1: up as a result, and then would repeat that process
Speaker 1: while building out this index, kind of leaving a trail
Speaker 1: of where it had been. The program built what Gray
Speaker 1: called the wand DECKX W A and d e X.
Speaker 1: It was an index of web servers that we're joining
Speaker 1: the Internet. Not long after launching the Wanderer than Gray
Speaker 1: built in additional capability of capturing the u r l's
Speaker 1: that it was going through in addition to just the
Speaker 1: web servers, so you can think like originally he was
Speaker 1: just like, I wonder how many web servers are connected
Speaker 1: to the Internet, how many are connected to it today,
Speaker 1: how many will be connected tomorrow? That kind of thing,
Speaker 1: and just sort of keeping track of how the web
Speaker 1: was growing. Then he thought, I want to know actually
Speaker 1: the u r ls that exists too, so he's keeping
Speaker 1: track of both. This didn't go totally smoothly, however. The
Speaker 1: Wanderer was an energetic little spider. It would move through
Speaker 1: links throughout the day. It would index the same pages
Speaker 1: hundreds of time in the process, and this started to
Speaker 1: cause network lag across the Internet, and it meant that
Speaker 1: people who were just trying to navigate to those pages
Speaker 1: were experiencing long delays as a result, and that made people,
Speaker 1: let's say, a little miffed at Mr Gray, and he
Speaker 1: was able to modify the spiders operation so it wouldn't
Speaker 1: cause so much of a disruption, but that early enthusiastic
Speaker 1: mistake kind of created a tough environment for other people
Speaker 1: who wanted to create similar tools that would allow for
Speaker 1: fully fledged searching on the Internet, and the concept of
Speaker 1: spiders had kind of a big old X next to
Speaker 1: it in the minds of many people. It became synonymous
Speaker 1: with this idea of lag and just bad network performance.
Speaker 1: The Wanderer didn't index sites for content either. It was
Speaker 1: more about tracking the growth of the Internet as a whole.
Speaker 1: It wasn't so much concerned with what was on pages,
Speaker 1: so it didn't create a means to search for specific
Speaker 1: web pages or subject matter. Meanwhile, over at the University
Speaker 1: of Geneva, a developer named Oscar near Strats developed a
Speaker 1: tool that could search lists of websites and return results
Speaker 1: based on a query. This tool didn't actually survey the
Speaker 1: web as a whole. Instead, it would access lists other
Speaker 1: organizations had made, such as the Virtual Library, so it
Speaker 1: would reformat those lists as entries into a database and
Speaker 1: that's what could be searched. It was called the W
Speaker 1: three Catalog. It still wasn't quite a search engine as
Speaker 1: we think of them today. Martin Costa developed another tool
Speaker 1: in late which he called the Archie like Indexing for
Speaker 1: the Web or ali Web as the name suggests. He
Speaker 1: was taking inspiration from the Gopher search tool of Archie.
Speaker 1: Ali Web needed web administrators to provide the location for
Speaker 1: their site index files so that they could be included
Speaker 1: in the ali web search database. Users could also create
Speaker 1: dis cryptions for the websites and add in keywords to
Speaker 1: help with search. And this brings us into the world
Speaker 1: of metadata. Metadata is information about information. So you've got
Speaker 1: the core information of a web page, what we might consider,
Speaker 1: you know, the content of the web page, the stuff
Speaker 1: that you and I would actually read if we went there.
Speaker 1: But then you've got the metadata and that describes the
Speaker 1: information in some meaningful way. Now, if you were in
Speaker 1: a physical library, the metadata would be the stuff that
Speaker 1: you used to help locate where in that library a
Speaker 1: specific book should be, and that could include stuff such
Speaker 1: as the author name, the publication date, the subject matter,
Speaker 1: that kind of thing. And it was a pretty good
Speaker 1: idea ALI web but not many people knew about it,
Speaker 1: and those who did know about it, not a lot
Speaker 1: of them went through the trouble of submitting the information
Speaker 1: to ALI webs, so it didn't really see much widespread use.
Speaker 1: Also in and then extending into was the development of
Speaker 1: another search tool, and this was the brain child of
Speaker 1: a guy named Jonathan Fletcher. He was a grad student
Speaker 1: at the University of Sterling in Scotland. Fletcher's approach combined
Speaker 1: the strategies of his predecessors. He built a web crawler
Speaker 1: to find and index web pages. He designed the database
Speaker 1: to be searchable and he called it jump Station. Unfortunately,
Speaker 1: his efforts were limited by the budget that he got
Speaker 1: from his university. He didn't have the resources to really
Speaker 1: build out a tool that could index all of a
Speaker 1: website's contents, so instead he designed jump Station to parse
Speaker 1: web page titles and headers, and that would still help
Speaker 1: people find pages that, at least according to the title
Speaker 1: and header were focused on whatever the area of interest was.
Speaker 1: But it would also mean that other pages that might
Speaker 1: have critically relevant information about the subject could be overlooked
Speaker 1: because those terms just weren't in the title or header.
Speaker 1: We're getting closer to the search engines that would most
Speaker 1: resemble what we think of today, which, let's be honest,
Speaker 1: is primarily Google, and we will learn more about those
Speaker 1: after we take another quick break. I think you could
Speaker 1: argue pretty convincingly the jump station was the first true
Speaker 1: web search engine as we have come to understand them,
Speaker 1: though it was limited since it couldn't crawl through and
Speaker 1: index all the contents of a page. The next name
Speaker 1: on our journey is one a lot of people will recognize,
Speaker 1: and that is Yahoo. But Yahoo didn't start off as
Speaker 1: a search engine. Rather, Yahoo was originally a web directory.
Speaker 1: It started in as just a list of websites that
Speaker 1: the founders of the site that would be Jerry Young
Speaker 1: and David Filo they thought were interesting. There this is
Speaker 1: a cool website. I want more people to know about it,
Speaker 1: so I'm gonna include did on my web page about
Speaker 1: cool websites. So Yahoo started off as another curated list
Speaker 1: of web pages. The search tool aspect of Yahoo would
Speaker 1: follow in The search tool worked on the sites that
Speaker 1: were curated in the human curated Yahoo directory, but if
Speaker 1: a site wasn't in that directory, it wouldn't show up
Speaker 1: in search results. So someone would have had to have
Speaker 1: found the website already and then included it within Yahoo's
Speaker 1: growing directory for it to register as a result. Following
Speaker 1: Yahoo were a couple of other notable names. There was
Speaker 1: info Seek and web Crawler. Webcrawlor was the first search
Speaker 1: engine I remember using. In fact, I stuck with web
Speaker 1: Crawler for a long time, even after the infamous Google
Speaker 1: emerged and started making waves. Web color did something that
Speaker 1: other search engines had not yet done. It's index was
Speaker 1: looking at the full content of a web page, including
Speaker 1: the meta data on that page. So let's talk about
Speaker 1: that for a second. Web spiders, when you get down
Speaker 1: to it, are just bots that follow links, but some
Speaker 1: web spiders can also make a full index of the
Speaker 1: content found at each links destination, essentially scanning all the
Speaker 1: text that's within a web page and indexing it so
Speaker 1: that that content is searchable and that searchable index forms
Speaker 1: as a result of all this, and it can bring
Speaker 1: back any results of any pages that contain a specific word.
Speaker 1: Let's use an example. It makes it easier. So let's
Speaker 1: say you're in a literature class and you're having a
Speaker 1: real hard time understanding Milton's Paradise Lost. So you're looking
Speaker 1: for some resources to help you get a better handle
Speaker 1: on things. You go to a search engine on the Internet.
Speaker 1: It doesn't really matter which one, and you type in
Speaker 1: Paradise lost Milton analysis. You're trying to really cut down
Speaker 1: on anything that might just mention paradise or loss or
Speaker 1: anything like that. You really want to focus on this.
Speaker 1: This part of the search engine is the UI or
Speaker 1: the user interface, right, This is the part that we
Speaker 1: as humans interact with in order to tell the engine
Speaker 1: what it is we're looking for. The search engine then
Speaker 1: goes and consults it's index of the web. So no
Speaker 1: matter which search engine you're using, it's not a representation
Speaker 1: of every single web page that exists. It's every web
Speaker 1: page that exists within that engine's index. So each search
Speaker 1: engine has its own index, or in some cases search
Speaker 1: engines are powered by other engines. It may be sharing
Speaker 1: an index with another search engine, but it looks for
Speaker 1: documents in that index that contain the words that you
Speaker 1: have submitted in the UI, then it has to return
Speaker 1: those results to you, which also means that the search
Speaker 1: engine has to determine which of those search results are
Speaker 1: likely to be the most relevant to your query. This
Speaker 1: is actually harder to do that, and it sounds if
Speaker 1: a surge engine is only looking for documents that happened
Speaker 1: to contain the words that you've submitted, you could get
Speaker 1: back pages that have little to no relevance to what
Speaker 1: you actually wanted. Plus, some web page administrators, especially back
Speaker 1: in the early days, we're really trying to game the system.
Speaker 1: They might use tricks in order to get more people
Speaker 1: to come to that web page. And it might be
Speaker 1: because their web pages had banner ads on them and
Speaker 1: so more people visiting the page meant more money, or
Speaker 1: maybe they just wanted bragging rights. Because some of you
Speaker 1: guys might remember this. It used to be back in
Speaker 1: the day that one of the standard features you would
Speaker 1: see on web pages was the ever present web counter
Speaker 1: that would tell you how many people had visited that
Speaker 1: website since it had been created. And a few folks
Speaker 1: were hoping to just spread malware by tricking people to
Speaker 1: visiting a website and downloading some malicious program. And then
Speaker 1: there were also link farms. These were sites that were
Speaker 1: just one long list of links to other sites. More
Speaker 1: on why that's important in just a second. One trick
Speaker 1: was to include just a ton of different popular search
Speaker 1: terms on a page, even if the page had nothing
Speaker 1: to do with any of those search terms, and you
Speaker 1: could even hide that. You can make the text and
Speaker 1: background the same color, so a human visiting the website
Speaker 1: and looking at it through a standard browser wouldn't see
Speaker 1: anything because the background color in the text is the
Speaker 1: same color. They see whatever the content of the web
Speaker 1: page was, but they wouldn't see all these hidden keywords.
Speaker 1: But a computer would totally see it. It would ignore
Speaker 1: the fact that the font and the background color are
Speaker 1: the same and it would just pick up on the text.
Speaker 1: So you would end up having these false returns on
Speaker 1: search results because those keywords were there in the page,
Speaker 1: they just weren't relevant to whatever the content was. Other
Speaker 1: administrators would put keyword dumps into web page meta data,
Speaker 1: so wouldn't show up on the page itself at all.
Speaker 1: It would all be in the background. Following a search
Speaker 1: result like that would be really frustrating because you wouldn't
Speaker 1: actually get whatever it was you were looking for, you
Speaker 1: would get something else. It was a bait and switch.
Speaker 1: So building search engines meant not only did the developers
Speaker 1: need to figure out how to build in disease that
Speaker 1: could grow as the Web was growing, they also had
Speaker 1: to figure out how to defeat strategies that were intended
Speaker 1: to game the system. How can you make sure the
Speaker 1: people who are using your search engine are actually getting
Speaker 1: the stuff that they want, because if they're not getting
Speaker 1: the stuff they want, they're gonna bounce. They're never going
Speaker 1: to use your search engine again. I'm gonna use Google
Speaker 1: as the example for this, because i mean, let's be honest,
Speaker 1: Google is dominant in that space. It's almost like it's
Speaker 1: the only game in town. But just know that all
Speaker 1: search engines, in general, we're all trying variations on this
Speaker 1: kind of general philosophy. Google's approach used a tool that
Speaker 1: they called page rank, which, as the name suggests, would
Speaker 1: take the documents that came back from any given search,
Speaker 1: then rank those search results before presenting them to the user.
Speaker 1: So if you went to Google and you typed in
Speaker 1: Paradise Lost Milton analysis, Google would consult its own index
Speaker 1: of the web, and it would look for stuff like,
Speaker 1: are the search terms showing up in the page? Is
Speaker 1: examples of words that are close together, because that might
Speaker 1: indicate that this result is more relevant. Right, if these
Speaker 1: words are all kind of next to each other, it's
Speaker 1: more likely to be what the person was looking for,
Speaker 1: as opposed to, Yeah, all four of those words are
Speaker 1: showing up on this page, but they're so far apart,
Speaker 1: then maybe this isn't even related to what the person
Speaker 1: was looking for. That was part of page rank. The
Speaker 1: tool also would look at things like the title of
Speaker 1: the page and maybe even the header, but it mostly
Speaker 1: ignored the metadata because you know, search in gen designers
Speaker 1: were picking up on the tricks people were using in
Speaker 1: order to get more clicks. At the same time, the
Speaker 1: search algorithm would assign ranks to pages based on a
Speaker 1: few other points of criteria. The algorithm attempted to figure
Speaker 1: out how reputable every page was, and it did so
Speaker 1: in a couple of different ways. One was to look
Speaker 1: at which other sites were linking to that page. If
Speaker 1: the other sites that were linking to it were considered
Speaker 1: generally reputable, that would improve the results page rank score.
Speaker 1: So in our case, and this is a totally unrealistic example.
Speaker 1: But let's say we've searched that Paradise Lost Milton analysis
Speaker 1: and all we got back our three results, but Google
Speaker 1: has to rank those results as one, two, and three.
Speaker 1: One of those results is from a website dedicated to
Speaker 1: Paradise Lost, the literary work and has literary analysis on it,
Speaker 1: and it sits on a server that belongs to a
Speaker 1: prestigious university. Let's say that the second result is coming
Speaker 1: from a literary discussion site. It doesn't belong to a university,
Speaker 1: but it does have critical analysis and an entry specifically
Speaker 1: on Paradise Lost. And let's say that the third result
Speaker 1: is Billy Bob's Homespun Guide to Milton and Crab Trap
Speaker 1: Maintenance or something. Now, the algorithm is not smart enough
Speaker 1: to actually read each of these sites as a human
Speaker 1: would and judge them and analyze them and weigh the
Speaker 1: value of each one, but it can see that the
Speaker 1: university server is, you know, it belongs to a university.
Speaker 1: It's generally treated as the property of a recognized authority,
Speaker 1: and so it sees that other reputable sites are also
Speaker 1: linking to that university's web pages, and to that Milton
Speaker 1: page in particular. So it assigns that result a very
Speaker 1: high page rank, saying it's probably pretty darn good. That
Speaker 1: also means it's going to appear higher on the list
Speaker 1: of search results. Meanwhile, Billy Bob's is likely to appear
Speaker 1: at the bottom of that list because very few people
Speaker 1: are linking to it. It might be hosted on just
Speaker 1: some server somewhere that happens to host a whole hodgepodge
Speaker 1: of different web pages, and the page that's on that
Speaker 1: site that has just sort of literary analysis discussions on it,
Speaker 1: that one appears in the middle. Now, could Billy Bob's
Speaker 1: page actually be the best resource? Yes, it could be,
Speaker 1: but without a human or maybe a really incredibly advanced
Speaker 1: AI to review the contents of that page and to
Speaker 1: really understand them, the ranking approach seemed like the best
Speaker 1: way to quickly organize results to give the best chance
Speaker 1: that the returns were going to be relevant to the user. Now,
Speaker 1: in that example I just gave, I mentioned three results. However,
Speaker 1: if you were to really perform that search, because I
Speaker 1: did it before I recorded this episode, you would get
Speaker 1: millions of results. In fact, just for a laugh, I
Speaker 1: went to Google typed in Paradise lost Milton Analysis, and
Speaker 1: I got quote about three point eight million results end quote,
Speaker 1: that happened in less than one second. Page rank becomes
Speaker 1: really important when you get to that level of response,
Speaker 1: when you get to that many results, if you're talking
Speaker 1: about that enormous amount of information, you really want the
Speaker 1: most relevant choices to be near the top to save
Speaker 1: yourself time. And that has created some pretty bad habits
Speaker 1: for us as users. By the way, we've become so
Speaker 1: used to search engines returning the most relevant results right
Speaker 1: at the top that we don't necessarily bother to look
Speaker 1: beyond the first few sites. There are a lot of
Speaker 1: resources out there that have estimates on how many people
Speaker 1: actually bother to ever go past the first page results,
Speaker 1: and some of them even say that as much as
Speaker 1: of all web traffic will just go to results that
Speaker 1: appear on the first page for any given search, and
Speaker 1: that means that all the other results that appear after
Speaker 1: page one are sharing just five percent of the web traffic.
Speaker 1: So when I did that Paradise law search, that first
Speaker 1: page of results had nine websites linked to it, plus
Speaker 1: a few videos. That means somewhere around three million, seven
Speaker 1: nine web pages are sharing just five percent of leftover
Speaker 1: traffic that go to that first page. So they might
Speaker 1: include incredible resources that are even more relevant than the
Speaker 1: stuff that appears on page one, but very few people
Speaker 1: are going to that. That's one bad habit that we've
Speaker 1: all developed through using these search engines. On the flip side,
Speaker 1: that message is that it's really important to get your
Speaker 1: page to show up on that first screen of results.
Speaker 1: If you're bill holding a web page about a specific thing,
Speaker 1: you want to be on that first page because otherwise
Speaker 1: you're gonna have to hope people find your your website
Speaker 1: through some other means, you know, outside of search. That
Speaker 1: gave birth to the industry of s e O or
Speaker 1: search engine optimization, which is a constantly evolving set of
Speaker 1: practices that web designers try to follow in order to
Speaker 1: rank better in search. And whenever a search engine, which
Speaker 1: again these days we mostly just mean Google, whenever Google
Speaker 1: makes a change in its algorithm, it can really upset
Speaker 1: the apple cart, and it can push everyone back to
Speaker 1: the drawing board. It can completely jumble up who appears
Speaker 1: at the top of search results. Now, all of that
Speaker 1: is another kettle of fish, So I'm going to leave
Speaker 1: off of s e O and go back to that
Speaker 1: on some other day, but more Germane to our episode
Speaker 1: here is that the spiders, those web crawling bots, or
Speaker 1: what build out those indices that search engines used to
Speaker 1: give us the results we ask for. There are some
Speaker 1: things I did not cover, such as tags that web
Speaker 1: developers can use to make sure that search engines just
Speaker 1: pass over their websites or sometimes just pages within their
Speaker 1: websites without adding them to an index, so they'll never
Speaker 1: show up in search. But we could go over that
Speaker 1: in a future episode. Two. For now, it's kind of
Speaker 1: time to wrap things up, So guys, I hope you
Speaker 1: enjoyed this episode. If you have suggestions for future topics,
Speaker 1: whether it's a specific technology, a trend in tech, a
Speaker 1: person in tech. Maybe it's a company you want to
Speaker 1: know more about, let me know, draw me a line
Speaker 1: on Facebook or Twitter. The handle of both of those
Speaker 1: is text stuff H s W and I'll talk to
Speaker 1: you again really soon. Text Stuff is an I Heart
Speaker 1: Radio production. For more podcasts from my Heart Radio, visit
Speaker 1: the I Heart Radio app, Apple Podcasts, or wherever you
Speaker 1: listen to your favorite shows. Two