WEBVTT - Web Spider Is Our Hero 0:00:04.400 --> 0:00:12.520 Welcome to Textuff, a production from my Heart Radio. Hey there, 0:00:12.520 --> 0:00:16.240 and welcome to tech Stuff. I'm your host, Jonathan Strickland. 0:00:16.320 --> 0:00:18.880 I'm an executive producer with I Heart Radio and I 0:00:19.000 --> 0:00:23.840 love all things tech and listener David reached out to 0:00:23.840 --> 0:00:26.040 me on Twitter and said, I would like to hear 0:00:26.079 --> 0:00:30.160 an episode on search engine spiders. He is our hero. 0:00:30.760 --> 0:00:33.800 You gotta David, and if you get the spider, he 0:00:33.920 --> 0:00:36.600 is our hero. Reference let me know. So we're gonna 0:00:36.640 --> 0:00:40.920 talk about the development of search engines and how they 0:00:40.960 --> 0:00:44.479 work from admittedly a pretty high level, because to go 0:00:44.520 --> 0:00:48.240 into great detail would probably take three or four episodes plus. 0:00:48.320 --> 0:00:52.640 Different search engines use slightly different strategies in order to 0:00:52.880 --> 0:00:57.640 index and rank search results. And the reason I'm doing 0:00:57.680 --> 0:01:00.760 all of that is because if we just talked about spiders, 0:01:01.200 --> 0:01:03.880 it would be a fairly short episode. But what the 0:01:03.920 --> 0:01:07.720 heck is a search engine spider. Well, the index the 0:01:07.800 --> 0:01:11.559 contents of the world wide Web. You need to search 0:01:11.600 --> 0:01:16.560 around and find what's there first, right, you can't return 0:01:16.680 --> 0:01:19.319 results without first knowing what is out there in the 0:01:19.360 --> 0:01:23.479 first place. So a search engine spider is a bot 0:01:24.120 --> 0:01:28.200 that does this. It crawls through the Worldwide Web. Thus 0:01:28.240 --> 0:01:31.679 the whole spider name. We'll learn more about what's actually 0:01:31.680 --> 0:01:34.119 going on a little bit later, but to understand search, 0:01:34.600 --> 0:01:37.400 we need a few more basics. So keep in mind 0:01:37.640 --> 0:01:40.440 that all the stuff we see online, whether it's a 0:01:40.440 --> 0:01:43.840 web page or it's a web service or whatever it 0:01:43.840 --> 0:01:47.600 may be, it ultimately sits on a computer that is 0:01:47.600 --> 0:01:52.320 connected to the Internet infrastructure, so it's connected to routers, 0:01:52.360 --> 0:01:56.000 which then connected to various servers and and domain name 0:01:56.080 --> 0:01:59.080 servers and all that kind of stuff. If you visit 0:01:59.120 --> 0:02:02.440 the house Stuff Works homepage, that's the site for the 0:02:02.440 --> 0:02:04.440 company that I used to work for, don't work for 0:02:04.480 --> 0:02:08.600 them anymore, But that website consists of pages that are 0:02:08.720 --> 0:02:12.520 on a computer in a data center. If you happen 0:02:12.560 --> 0:02:15.160 to know the u r L for the site, so 0:02:15.280 --> 0:02:17.880 you happen to know how stuff works dot com, you 0:02:17.919 --> 0:02:20.679 can type that into a browser u r L bar 0:02:20.800 --> 0:02:23.760 address bar, and the browser will then take care of 0:02:23.960 --> 0:02:27.800 sending the appropriate message to that computer. In this case, 0:02:27.840 --> 0:02:30.639 we will call it a server, and the server will 0:02:30.720 --> 0:02:34.240 then return the appropriate information to your browser the web 0:02:34.280 --> 0:02:36.560 page maybe the home page for how Stuff Works in 0:02:36.600 --> 0:02:39.720 this case, and then you'll see the website. But all 0:02:39.760 --> 0:02:42.520 of that requires that First, you have to know that 0:02:42.639 --> 0:02:45.240 there's a site there at all. Plus you have to 0:02:45.240 --> 0:02:47.119 know the U r L for it, and you might 0:02:47.280 --> 0:02:51.919 not have that information. Before there even was a Worldwide Web, 0:02:52.160 --> 0:02:54.360 there was a need to know where you could find 0:02:54.400 --> 0:02:58.280 stuff on the Internet. Now, remember, the Internet is older 0:02:58.320 --> 0:03:00.200 than the Web, and the Internet and the Web or 0:03:00.280 --> 0:03:04.320 not the same thing. The Web exists on top of 0:03:04.360 --> 0:03:07.120 the Internet. It consists of a lot of other stuff 0:03:07.160 --> 0:03:12.120 besides the Web, right, like email and FTP servers. In fact, 0:03:12.760 --> 0:03:16.560 we need to really talk about FTP servers. FTP stands 0:03:16.600 --> 0:03:21.200 for File Transfer Protocol. So these are computers that house 0:03:21.520 --> 0:03:26.440 certain files on them and through FTP. Through this protocol 0:03:26.560 --> 0:03:30.160 that allows for files to transfer from one computer to 0:03:30.320 --> 0:03:34.560 another across a network connection. People can thus access files 0:03:34.560 --> 0:03:37.880 that can transfer them from the server to their own computer, 0:03:38.000 --> 0:03:41.240 which in this case we would call a client. But again, 0:03:41.680 --> 0:03:45.400 FTP is really only useful if you know the address 0:03:45.520 --> 0:03:49.520 of the servers where the stuff is that you want. Right, 0:03:49.960 --> 0:03:52.480 you can't just use FTP to pull a file out 0:03:52.520 --> 0:03:55.920 of nowhere. You have to contact the proper server and 0:03:55.960 --> 0:04:00.960 pull the relevant file from that server. Enter an Emtaj, 0:04:01.200 --> 0:04:04.920 who in nineteen nine was a graduate student at McGill 0:04:05.040 --> 0:04:09.000 University in Montreal, Canada. He also worked as a systems 0:04:09.040 --> 0:04:13.280 administrator for the School of Computer Science at the university, 0:04:13.320 --> 0:04:16.040 and he was running into a challenge. It was his 0:04:16.200 --> 0:04:20.719 job to locate software for professors, for staff, for students 0:04:20.760 --> 0:04:23.080 at the university, but there was no easy way to 0:04:23.160 --> 0:04:26.640 know where all the various files were on the network 0:04:26.760 --> 0:04:31.080 of public FTP servers. Emtage decided there needed to be 0:04:31.200 --> 0:04:34.839 a way to get a snapshot of which public FTP 0:04:35.080 --> 0:04:39.559 servers had which files. There needed to be some sort 0:04:39.600 --> 0:04:43.760 of directory, and since servers were popping up more frequently 0:04:43.960 --> 0:04:48.080 as more people began to develop stuff for the Internet, 0:04:48.360 --> 0:04:50.360 there also needed to be a good way to search 0:04:50.520 --> 0:04:54.680 those lists to find something specific. Otherwise it would be 0:04:54.720 --> 0:04:57.479 like reading through an entire phone book to find out 0:04:57.480 --> 0:05:00.520 which person or business corresponded to a phone number you 0:05:00.600 --> 0:05:03.200 happen to have seen. Let's say the phone number was 0:05:03.279 --> 0:05:06.000 eight six seven five three oh nine, and you don't 0:05:06.040 --> 0:05:08.440 know that that's Jenny's phone number. You just know it's 0:05:08.440 --> 0:05:11.560 the number. So instead of calling the number and asking, Hey, 0:05:11.600 --> 0:05:13.960 whose number is this, you get a phone book and 0:05:14.000 --> 0:05:16.720 you start searching for eight three o nine to find 0:05:16.760 --> 0:05:20.960 the corresponding name that is not efficient. In fact, in 0:05:21.000 --> 0:05:24.559 the early days, information about servers frequently had no other 0:05:24.720 --> 0:05:29.000 real channel to get to users other than word of mouth. 0:05:29.160 --> 0:05:32.800 So there was a really good chance that there was 0:05:32.839 --> 0:05:35.680 stuff that was relevant to you that you just had 0:05:35.720 --> 0:05:38.000 no way of knowing about because you had to hear 0:05:38.040 --> 0:05:41.680 it from somebody else first. Imtash, along with a couple 0:05:41.720 --> 0:05:44.360 of other folks like Bill Healen and J. Peter Deutsch, 0:05:44.760 --> 0:05:48.200 began building a tool to solve this problem. They ended 0:05:48.279 --> 0:05:52.680 up calling this tool Archie, which actually was not a 0:05:52.720 --> 0:05:56.800 nod to the comic book character from Archie Comics. Instead, 0:05:57.040 --> 0:06:01.200 Archie was a somewhat shortened form of the word archives. 0:06:01.920 --> 0:06:05.600 They created programs that could look through the repositories of 0:06:05.720 --> 0:06:09.680 public FTP sites and get an inventory of the files 0:06:09.720 --> 0:06:13.599 stored on those servers or as documented in the book 0:06:13.800 --> 0:06:17.320 A Rough Guide to the Internet by Nicholas West Quote, 0:06:17.800 --> 0:06:22.440 it combined a script based data gatherer which accessed listings 0:06:22.440 --> 0:06:26.600 from anonymous sites with a script which matched regular expressions 0:06:26.600 --> 0:06:29.960 which could retrieve file names matching a user query end 0:06:30.040 --> 0:06:34.279 quote simple right now. In case you're like me and 0:06:34.480 --> 0:06:38.160 what I just quoted sounded a little bit confusing. One 0:06:38.240 --> 0:06:40.719 it really boils down to is to say they made 0:06:40.800 --> 0:06:44.800 a computer program that followed some fairly simple rules. The 0:06:44.880 --> 0:06:47.720 program made note of the file titles that were on 0:06:47.880 --> 0:06:51.520 various FTP servers, kind of like a list of contents, 0:06:51.920 --> 0:06:55.680 and they noted which files were on which servers. Another 0:06:55.680 --> 0:06:58.839 part of the program arranged those findings into a database 0:06:59.360 --> 0:07:02.080 not that much different from the types of spreadsheets you've 0:07:02.080 --> 0:07:05.520 probably worked with in the past. Imtaj and crew also 0:07:05.640 --> 0:07:08.960 created a tool that would allow them to search this database. 0:07:09.440 --> 0:07:12.040 Before long, other people began to hear that he had 0:07:12.080 --> 0:07:14.840 this database and that they would ask him, Hey, can 0:07:14.880 --> 0:07:16.480 you do a search for me, and they would give 0:07:16.520 --> 0:07:18.960 him the search terms, and it started taking up a 0:07:18.960 --> 0:07:22.040 lot of his time. So in an effort to streamline things, 0:07:22.320 --> 0:07:26.320 he programmed a user interface or UI that would allow 0:07:26.360 --> 0:07:29.440 people to conduct their own searches. They could just log 0:07:29.480 --> 0:07:32.200 into this tool and then type in the file that 0:07:32.240 --> 0:07:34.960 they were looking for and it would return the results 0:07:35.000 --> 0:07:37.960 for them. So as long as they were sure about 0:07:38.000 --> 0:07:41.480 the specific file they needed, they would get the results. Now, 0:07:41.560 --> 0:07:45.239 most resources generally agree that Archie was the first real 0:07:45.480 --> 0:07:49.120 search engine on the Internet, but it wasn't a web 0:07:49.320 --> 0:07:53.200 search engine, the Web didn't exist yet. It wasn't long 0:07:53.240 --> 0:07:58.160 before a couple of other tools followed. In some researchers 0:07:58.160 --> 0:08:01.200 with the University of Minnesota develop a new tool to 0:08:01.440 --> 0:08:05.840 organize and discover documents stored on servers, and the tool 0:08:05.960 --> 0:08:10.840 was called the Gopher protocol. Servers were data repositories called 0:08:11.000 --> 0:08:14.800 Gopher holes eventually that's what they were called anyway, and 0:08:14.880 --> 0:08:19.440 Gopher organized everything into a hierarchical text based menu system. 0:08:19.520 --> 0:08:23.040 So this was a specific strategy that was built on 0:08:23.080 --> 0:08:25.320 top of the Internet. It was kind of think of 0:08:25.320 --> 0:08:29.400 it as being in parallel with the Web. It predated 0:08:29.440 --> 0:08:31.840 the Web, but the Web and Gopher would exist at 0:08:31.880 --> 0:08:34.240 the same time, but they were not the same thing. 0:08:34.640 --> 0:08:38.679 This was a different strategy in order to serve information 0:08:38.720 --> 0:08:45.360 across the networks. Before long, like in more researchers developed 0:08:45.400 --> 0:08:48.600 a search function to work on top of Gopher. Because again, 0:08:49.240 --> 0:08:52.319 if you didn't know where something actually quote unquote lived 0:08:52.800 --> 0:08:56.160 in the Gopher network, you would never be able to 0:08:56.200 --> 0:08:59.520 find it unless you were just lucky. So this search 0:08:59.600 --> 0:09:03.400 tool was called Veronica, and that is pretty cute because 0:09:04.080 --> 0:09:08.719 Veronica is a character in the Archie comics books. And 0:09:08.800 --> 0:09:12.400 while the search engine Archie did not pull its name 0:09:12.520 --> 0:09:15.840 from Archie Comics. Veronica was a nod to the older 0:09:15.880 --> 0:09:19.160 search engine as well as a nod to the comic book, 0:09:19.200 --> 0:09:23.920 so it almost kind of retroactively made Archie relate back 0:09:23.960 --> 0:09:29.560 to the comics. Later, computer geeks assigned a backronym to Veronica. 0:09:29.920 --> 0:09:33.080 This is an acronym that you create after you've already 0:09:33.160 --> 0:09:35.080 named a thing. So you've given the thing a name, 0:09:35.320 --> 0:09:37.360 and then you're thinking, okay, well, what can we say. 0:09:37.400 --> 0:09:41.120 Each of those letters stands for that's a backronym, And 0:09:41.160 --> 0:09:45.600 in this case, the revisionist name was very easy, rodent 0:09:45.840 --> 0:09:51.400 oriented net Wide Index to Computer Archives, or Veronica super cute. 0:09:51.920 --> 0:09:56.040 What Veronica did was fairly primitive. It created a database 0:09:56.160 --> 0:10:00.199 of every file and every directory on every go for 0:10:00.280 --> 0:10:02.960 a server that was connected to the Internet, and it 0:10:02.960 --> 0:10:07.040 would update dynamically as more servers joined the network. That 0:10:07.080 --> 0:10:10.599 approach worked fairly well when there was still a relatively 0:10:10.679 --> 0:10:14.440 small number of servers to keep track of, But as 0:10:14.520 --> 0:10:18.200 more servers came online and joined this Gopher network, with 0:10:18.280 --> 0:10:22.199 more documents stored on each server, it started to get 0:10:22.200 --> 0:10:26.800 a really you know, challenging to manage Veronica, a secondary 0:10:26.840 --> 0:10:30.040 Gopher search tool kind of addressed this problem, and this 0:10:30.080 --> 0:10:33.520 one also took its name from a character from Archie comics, 0:10:34.040 --> 0:10:37.840 jug Head. This one didn't create a full database of 0:10:37.960 --> 0:10:42.040 everything that was on the Gopher network. Instead, as a user, 0:10:42.360 --> 0:10:45.920 you would have to designate which Gopher server you wanted 0:10:45.960 --> 0:10:48.320 to search, so you had to at least have some 0:10:48.480 --> 0:10:51.640 general idea of where it was you needed to look. 0:10:52.360 --> 0:10:54.199 But if you did know that, it was a much 0:10:54.240 --> 0:10:57.920 faster approach than trying to search everything on the network 0:10:57.920 --> 0:11:01.760 as a whole. Gopher had a major problem, and that 0:11:01.920 --> 0:11:06.079 was that was becoming increasingly less efficient and easy to navigate. 0:11:06.600 --> 0:11:11.360 The larger it got, it didn't scale well. Meanwhile, at 0:11:11.360 --> 0:11:14.240 the same time that Gopher was growing, a guy named 0:11:14.400 --> 0:11:17.959 Tim berners Lee over at CERN. You know that's the 0:11:18.000 --> 0:11:22.120 research facility that oversees stuff like the large Hadron collider. Well, 0:11:22.160 --> 0:11:26.240 he was developing a different approach to storing and sharing 0:11:26.280 --> 0:11:30.199 information across networks. Tim and his team at CERN developed 0:11:30.240 --> 0:11:34.160 a protocol called Hypertext Transfer Protocol or h t t 0:11:34.360 --> 0:11:39.040 P and Hypertext Markup Language or h t mL. Both 0:11:39.040 --> 0:11:41.240 of these kind of grew out of stuff that CERN 0:11:41.320 --> 0:11:44.600 had been using internally for a while. Now I'm guessing 0:11:44.640 --> 0:11:47.240 those terms sound familiar to you, guys. These are the 0:11:47.280 --> 0:11:51.040 two components that really formed the basis of web pages 0:11:51.120 --> 0:11:54.760 and the Worldwide Web. The markup language acts as the 0:11:54.800 --> 0:11:58.440 set of instructions on how a computer, or more specifically, 0:11:58.480 --> 0:12:04.040 how a browser is to interpret and display documents, eventually 0:12:04.040 --> 0:12:07.880 including stuff like images and sound files. Although initially the 0:12:07.920 --> 0:12:11.640 web was strictly text based and browsers were text based 0:12:11.640 --> 0:12:15.240 as well. H t t P is the set of 0:12:15.360 --> 0:12:19.320 rules and the processes through which a client that being 0:12:19.400 --> 0:12:23.680 your web browser, can request a specific document from a server, 0:12:24.280 --> 0:12:27.600 and how the server can then send that requested document 0:12:27.640 --> 0:12:30.400 to the browser. The server since the h t m 0:12:30.520 --> 0:12:33.440 L files to the client, and the client interprets those 0:12:33.559 --> 0:12:37.200 HTML files in order to display the relevant web page 0:12:37.240 --> 0:12:42.040 to the user. Hypertext refers to text that has a 0:12:42.120 --> 0:12:45.480 link to some other text, and you can think of 0:12:45.520 --> 0:12:48.360 it kind of like a footnote in a book. The 0:12:48.480 --> 0:12:52.840 hypertext has an asterisk that corresponds to another piece of 0:12:52.880 --> 0:12:57.440 information somewhere that is also marked by an asterisk, except 0:12:57.480 --> 0:13:02.600 in this case the asterisks are invisible. It's highlighted text 0:13:02.720 --> 0:13:05.360 or or text in a different color, or it's underlined. 0:13:05.840 --> 0:13:08.480 It's designated in some way to be different from all 0:13:08.520 --> 0:13:10.719 the rest of the text. That's what lets you know 0:13:10.960 --> 0:13:14.800 it's hypertext and it's linked to something else. Hypertext documents 0:13:14.840 --> 0:13:18.800 connect to one another through hyperlinks. Those documents don't even 0:13:18.840 --> 0:13:20.640 have to be on the same server. They can be 0:13:20.679 --> 0:13:23.240 on opposite sides of the world. So this means you 0:13:23.240 --> 0:13:27.679 can build a reference in one hypertext document to content 0:13:27.760 --> 0:13:31.640 that's found on a totally different hypertext document. Clicking on 0:13:31.840 --> 0:13:35.840 that hypertext activates the link. It sends a command to 0:13:36.360 --> 0:13:39.440 the browser, which then relays that command to the server 0:13:39.800 --> 0:13:43.880 that the client wants to see specific linked information, and 0:13:44.040 --> 0:13:47.600 the server returns that. You can also link the locations 0:13:47.640 --> 0:13:50.080 that are within the same page of a document, or 0:13:50.200 --> 0:13:53.440 specific locations on other pages. It doesn't have to just 0:13:53.520 --> 0:13:55.559 be click on this and you go to a new 0:13:55.559 --> 0:13:57.880 web page. It might be click on this and you 0:13:58.120 --> 0:14:02.000 skip down, you know, a significant number of paragraphs to 0:14:02.040 --> 0:14:05.360 get to the relevant information. Really, all the link is 0:14:05.400 --> 0:14:09.920 doing is telling the browser where some specific point in 0:14:09.960 --> 0:14:12.680 a specific document happens to be and how to get there. 0:14:13.000 --> 0:14:14.640 It's kind of like if you were reading a book 0:14:14.640 --> 0:14:17.080 that said I want to know more, skipped a page 0:14:17.080 --> 0:14:20.400 to nineteen and read the third paragraph or or sometimes 0:14:20.440 --> 0:14:22.760 I compare it to those old choose your own adventure 0:14:22.800 --> 0:14:25.000 books where you get to the bottom of a page 0:14:25.080 --> 0:14:27.160 and you have to make a decision, and based on 0:14:27.200 --> 0:14:28.960 which decision you make, you have to turn to a 0:14:28.960 --> 0:14:32.280 specific page to pick up the story again. Well, you 0:14:32.280 --> 0:14:35.360 can quickly see how this would be really useful. Let's 0:14:35.360 --> 0:14:37.400 say I want to make a web page that includes 0:14:37.520 --> 0:14:40.560 directions and how to perform a particular process. We're gonna 0:14:40.600 --> 0:14:43.640 call it baking a soup FLA and the steps I 0:14:43.760 --> 0:14:48.160 list include references to other, maybe slightly less involved processes 0:14:48.240 --> 0:14:50.960 that are part of this, and I don't go into 0:14:51.000 --> 0:14:54.240 explaining how those work. Let's say, like I talk about 0:14:54.280 --> 0:14:56.080 cracking eggs, but I don't tell you the best way 0:14:56.080 --> 0:14:59.440 to crack an egg. However, I could create hypertext links 0:14:59.520 --> 0:15:03.120 to other sets of instructions, maybe a specific page just 0:15:03.320 --> 0:15:06.040 on different ways to crack an egg, And that way 0:15:06.040 --> 0:15:08.440 you could go and look that up if you weren't confident. 0:15:08.760 --> 0:15:10.720 So if you don't know how something works, you can 0:15:10.720 --> 0:15:13.280 click on that other link and go to a page 0:15:13.320 --> 0:15:17.240 let's dedicated to that in order to learn more. And yes, 0:15:17.560 --> 0:15:20.600 I just described how the web works in general, which 0:15:20.640 --> 0:15:23.240 is something I'm sure you all know at least at 0:15:23.280 --> 0:15:25.920 some level, even if it's not you know, a formal one. 0:15:26.240 --> 0:15:29.400 But if you've ever been on say Wikipedia, reading up 0:15:29.400 --> 0:15:32.360 on a topic and saw hypertext link and thought, yeah, 0:15:32.520 --> 0:15:34.840 I should find out what this term means. I don't 0:15:34.920 --> 0:15:37.880 understand it. So you click on that and you go 0:15:38.040 --> 0:15:40.400 follow that so you can get better understanding, then that's 0:15:40.400 --> 0:15:43.400 the use case I'm referring to. And there's a conversation 0:15:43.480 --> 0:15:46.440 we could have about what actually goes on when you 0:15:46.520 --> 0:15:49.960 click a link, but that would require a deeper dive 0:15:50.000 --> 0:15:53.160 into how the web and by extension, how the Internet 0:15:53.200 --> 0:15:55.680 works on a very technical level, and I think that 0:15:55.720 --> 0:15:57.640 goes beyond the scope of what we're trying to do 0:15:57.680 --> 0:16:00.840 in this episode. So I'm going to simple five, perhaps 0:16:00.840 --> 0:16:04.760 to a ludicrous degree, and say that a link contains 0:16:04.800 --> 0:16:08.480 within it the information about where another document, or even 0:16:08.560 --> 0:16:13.160 a specific point within another document exists, and activating that 0:16:13.240 --> 0:16:16.160 link by clicking on it in a browser initiates a 0:16:16.200 --> 0:16:20.240 sequence that results in the browser requesting that specific document 0:16:20.400 --> 0:16:24.040 from the appropriate server, which then sends that document to 0:16:24.120 --> 0:16:26.440 the browser so that you can see it. A lot 0:16:26.480 --> 0:16:29.280 more is going on to make this happen, but let's 0:16:29.280 --> 0:16:32.280 just stick with that high level view. So the pair 0:16:32.360 --> 0:16:36.000 of h T t P and HTML evolved the same 0:16:36.040 --> 0:16:40.480 time that Gopher was establishing itself, and some people stuck 0:16:40.520 --> 0:16:44.480 with Gopher, but it just really never took off the 0:16:44.520 --> 0:16:47.760 same way that the web did with HTML and HTTP. 0:16:48.320 --> 0:16:51.440 The protocol and the markup language is what the web 0:16:51.560 --> 0:16:54.720 is built upon, and we call it a web because 0:16:54.760 --> 0:16:57.920 of that interconnectivity of documents. You can build out a 0:16:58.000 --> 0:17:01.160 dock and then link that document to another doc which 0:17:01.200 --> 0:17:04.080 might be linked to a dozen others, and by following 0:17:04.119 --> 0:17:07.000 those links, you can navigate from one document to the next. 0:17:07.320 --> 0:17:09.680 It really is similar to what happens to a lot 0:17:09.720 --> 0:17:12.240 of people when they visit Wikipedia and they just start 0:17:12.320 --> 0:17:15.440 following all sorts of links. But you might already see 0:17:15.440 --> 0:17:19.320 a challenge with that kind of design. It works great 0:17:19.400 --> 0:17:23.399 if you've got a centralized person or institution that's building 0:17:23.400 --> 0:17:27.000 out the web, adding pages in a very logical way 0:17:27.440 --> 0:17:30.119 and linking to them in a very logical way. But 0:17:30.280 --> 0:17:33.399 one of tim berners Lee's major goals was to create 0:17:33.400 --> 0:17:37.879 a democratized system that didn't depend upon a centralized authority. 0:17:38.040 --> 0:17:40.360 People should be able to build their own web pages 0:17:40.520 --> 0:17:43.120 and host them on their own servers. But how would 0:17:43.200 --> 0:17:46.600 anyone else find them if there were no links going 0:17:46.760 --> 0:17:50.119 into those pages. If the web pages are made and 0:17:50.200 --> 0:17:52.960 hosted independently of the first few pages on the web, 0:17:53.440 --> 0:17:57.800 where is the connective tissue? The original solution wasn't a 0:17:57.880 --> 0:18:01.080 search engine. It took a it more of a hands 0:18:01.119 --> 0:18:04.480 on approach. I'll explain more, but first let's take a 0:18:04.560 --> 0:18:16.240 quick break. In the early days, when people first started 0:18:16.280 --> 0:18:19.520 building documents to host on the web, in other words, 0:18:19.800 --> 0:18:23.399 the earliest web pages, Tim Burner's leave would take it 0:18:23.440 --> 0:18:27.879 upon himself to create an index hosted on cerns own server. 0:18:28.359 --> 0:18:31.760 Someone might send him a message saying that they had 0:18:31.920 --> 0:18:34.640 built in are hosting a new web page, and they 0:18:34.680 --> 0:18:38.360 could include the address or U R L. Burners Lee 0:18:38.440 --> 0:18:41.600 would then add a hypertext link to a growing catalog 0:18:41.840 --> 0:18:45.000 of those kind of links on a page hosted by CERN. 0:18:45.240 --> 0:18:47.960 So if you visited cerns site, you could navigate to 0:18:48.000 --> 0:18:50.919 that index and see the links to the other sites. 0:18:51.600 --> 0:18:55.600 Tim came to call this a virtual library. He and 0:18:55.640 --> 0:18:59.399 a group of volunteers oversaw its evolution. They organized it 0:18:59.440 --> 0:19:03.080 into different areas of interest, with subject matter experts overseeing 0:19:03.160 --> 0:19:06.560 specific categories, and a lot of these early pages belonged 0:19:06.560 --> 0:19:11.280 to scientific research organizations or universities or publications, and all 0:19:11.359 --> 0:19:14.320 that makes sense. CERN is the organization that oversees the 0:19:14.400 --> 0:19:17.240 large Hadron Collider after all, So it's no surprise that 0:19:17.359 --> 0:19:21.480 the early web really focused on science and academia. Also, 0:19:21.640 --> 0:19:24.280 it's good to mention that the web in those early 0:19:24.359 --> 0:19:28.639 days again was text based. Browsers were text based too, 0:19:28.760 --> 0:19:32.360 that would not really change until another year like nine. 0:19:34.160 --> 0:19:38.159 Mike Gopher's design, the virtual library approach worked fairly well 0:19:38.359 --> 0:19:41.399 when the Web was still small in scale. According to 0:19:41.440 --> 0:19:46.119 the Virtual Library website, in August nine there were about 0:19:46.160 --> 0:19:50.639 twenty web servers in existence total. A little more than 0:19:50.680 --> 0:19:54.120 a year later, in October of nineteen, it was more 0:19:54.119 --> 0:19:58.800 than two hundred web servers, so growth was still fairly modest, 0:19:58.960 --> 0:20:03.800 but things kind of took off after that. By January 0:20:04.000 --> 0:20:09.360 nine six, there were more than one hundred thousand web servers. 0:20:09.720 --> 0:20:13.480 The following year there were more than six hundred fifty thousand. 0:20:13.560 --> 0:20:17.000 It was growing so fast, and maintaining an index was 0:20:17.040 --> 0:20:23.000 becoming increasingly more difficult, particularly by doing it, you know, manually. 0:20:23.400 --> 0:20:26.480 The virtual library was taking shape around the same time 0:20:26.520 --> 0:20:29.600 that students were building the Veronica search engine for gophers, 0:20:29.640 --> 0:20:31.720 so all this was happening around the same time. I 0:20:31.760 --> 0:20:35.880 know it sounds like I'm going strictly chronologically, but that's 0:20:35.880 --> 0:20:39.840 just too not too helpful. We have to remember this 0:20:39.880 --> 0:20:43.439 is all happening simultaneously. So as the web grew and 0:20:43.520 --> 0:20:48.320 became more complex, indices were growing as well. Just navigating 0:20:48.359 --> 0:20:52.240 an index to find what you wanted would become a challenge, 0:20:52.359 --> 0:20:55.120 particularly if you weren't thinking in the same way as 0:20:55.160 --> 0:20:58.399 the people who had organized the index. This is where 0:20:58.440 --> 0:21:02.880 taxonomy comes in. Taxonomy refers to a system of classification. 0:21:03.160 --> 0:21:05.560 A taxonomy is a set of rules we use to 0:21:05.720 --> 0:21:08.879 organize stuff, and there is no one way to do 0:21:08.960 --> 0:21:12.840 it correctly. So I'll give a simple example. Let's say 0:21:12.960 --> 0:21:16.240 you've got a class of students and you have them 0:21:16.280 --> 0:21:20.200 all divide up into smaller groups. You give each group 0:21:20.520 --> 0:21:23.960 a pile of documents, the same documents per group, but 0:21:24.040 --> 0:21:28.680 you tell the students it's their job to organize those documents. Well, 0:21:28.720 --> 0:21:31.080 one group decides that they're going to organize all the 0:21:31.119 --> 0:21:34.840 documents by alphabetizing them by title. The title of each 0:21:34.920 --> 0:21:38.640 document will determine how they fall in the pile, so 0:21:39.240 --> 0:21:42.320 there's are all in alphabetical order. Another group decides that 0:21:42.320 --> 0:21:45.160 they're gonna bundle their documents that all cover the same 0:21:45.200 --> 0:21:50.320 subject matter together, and they'll alphabetize within subjects. So they 0:21:50.400 --> 0:21:53.840 might have a stack that's just about biology, another stack 0:21:53.920 --> 0:21:57.240 that's about chemistry, another one about material science, and so on. 0:21:57.920 --> 0:22:01.159 A third group bundles their docum it's together by author. 0:22:01.440 --> 0:22:04.480 They put all of the same author's works together, and 0:22:04.480 --> 0:22:08.800 then maybe they alphabetize the authors. Yeah. Another group focuses 0:22:08.880 --> 0:22:12.880 on publication date. They arrange all their documents in order 0:22:12.920 --> 0:22:16.840 of when they were published. So you can imagine combinations 0:22:16.840 --> 0:22:20.600 of these approaches as well, right, such as ordering documents chronologically, 0:22:20.640 --> 0:22:23.880 but then if two documents were published on the same date, 0:22:23.920 --> 0:22:26.479 you then alphabetize them. That kind of thing. You can 0:22:26.520 --> 0:22:28.760 think of those different sets of rules, and you have 0:22:28.800 --> 0:22:32.600 to determine which rules are most important, right, which one 0:22:32.720 --> 0:22:36.400 you do first, and which is secondary. Well, it's important 0:22:36.440 --> 0:22:40.720 that these taxonomy's, however you construct them, are consistent or 0:22:40.760 --> 0:22:43.640 else it becomes a chaotic mess. But even a well 0:22:43.840 --> 0:22:47.560 organized and maintained taxonomy can still be a challenge for 0:22:47.680 --> 0:22:51.040 someone new coming into the system. And that's really what 0:22:51.040 --> 0:22:55.520 I'm getting at here. A comprehensive index will seem incredibly 0:22:55.600 --> 0:22:59.840 overwhelming to someone who's unfamiliar with the system's taxonomy, and 0:23:00.040 --> 0:23:03.480 it will still seem like finding a specific document or 0:23:03.560 --> 0:23:07.560 web page is an impossible task. Once that index grows 0:23:07.600 --> 0:23:11.880 to a large enough size, clearly a search tool would 0:23:11.920 --> 0:23:14.320 be a big solution to that problem. If you can 0:23:14.359 --> 0:23:17.600 type a query into a search engine and you can 0:23:17.640 --> 0:23:20.240 get the results you want, you don't have to comb 0:23:20.320 --> 0:23:23.639 through an enormous index hoping that you're thinking in the 0:23:23.680 --> 0:23:27.240 same way as the custodians of that index. But in 0:23:27.359 --> 0:23:31.199 addition to those challenges was one of scale building. This 0:23:31.320 --> 0:23:34.879 index required a lot of volunteer effort because again it 0:23:34.920 --> 0:23:38.320 was done by hand. The next step toward the development 0:23:38.320 --> 0:23:41.800 of a search engine was a project that was tackled 0:23:41.840 --> 0:23:45.359 by an m I T student, Matthew Gray. That was 0:23:45.400 --> 0:23:49.399 the student. He designed a program he called the Worldwide 0:23:49.600 --> 0:23:53.679 Web Wanderer, and the purpose of this program was to 0:23:53.760 --> 0:23:58.240 automatically navigate across the web, cataloging the web's growth by 0:23:58.280 --> 0:24:03.560 registering new websites, web pages, and web servers. The Worldwide 0:24:03.560 --> 0:24:07.879 Web Wanderer is arguably the earliest automated spider or web 0:24:07.880 --> 0:24:11.640