WEBVTT - The Common Crawl 0:00:00.160 --> 0:00:07.080 Brought to you by Toyota. Let's go places. Welcome to 0:00:07.280 --> 0:00:13.360 Forward Thinking. Hey, they're in Loving and Forward Thinking, the 0:00:13.520 --> 0:00:16.639 podcast that looks at the future and says, Kitty McGee's 0:00:16.680 --> 0:00:21.160 in Dublin Town upon the Crawl. I'm Jonathan Strickland and 0:00:21.280 --> 0:00:25.360 I'm Joe McCormick, and today we're gonna be talking about 0:00:25.400 --> 0:00:28.159 the crawl. We are talking about the crawl, not a 0:00:28.200 --> 0:00:30.760 pub crawl. No, sadly, not a pub crawl, which is 0:00:31.240 --> 0:00:33.280 what I was referring to in the lyric. But that's 0:00:33.320 --> 0:00:35.760 not what we're talking about today. The krawl. What is that? 0:00:35.960 --> 0:00:39.360 Is that the name of a movie that was like 0:00:39.080 --> 0:00:41.320 a like a fantasy movie from the eighties, or it 0:00:41.320 --> 0:00:45.600 sounds like a I'm thinking krawl. Oh yeah, that's a 0:00:45.640 --> 0:00:50.160 science fiction fantasy film with a phenomenal one I might 0:00:50.200 --> 0:00:53.960 add phenomenal science fiction fantasy film. Okay, So why would 0:00:53.960 --> 0:00:56.200 we be talking about a crawl that's not a pub 0:00:56.320 --> 0:00:58.920 crawl and not a sci fi fantasy movie. And it's 0:00:58.960 --> 0:01:02.000 not the future of baby these crawling right. No, it 0:01:02.120 --> 0:01:05.000 has something to do with the Internet. Yes, it has 0:01:05.040 --> 0:01:08.880 everything to do with the Internet Web in particular. Actually, uh, 0:01:08.959 --> 0:01:12.480 and here's a funny little little tidbit of information that 0:01:12.520 --> 0:01:14.960 you probably already knew, but you might be a little 0:01:14.959 --> 0:01:17.479 bit fuzzy on. Wait, what's the difference between the Web 0:01:17.520 --> 0:01:20.320 and the Internet. Because when I say the Internet, most 0:01:20.319 --> 0:01:22.600 of the time, what I'm talking about is the place 0:01:22.640 --> 0:01:26.280 where people leave comments and argue about things, which would 0:01:26.280 --> 0:01:29.960 be the Web mostly. Right. So Internet is the network 0:01:29.959 --> 0:01:32.280 of networks of computers. Right. So You've got all these 0:01:32.280 --> 0:01:36.440 different computer networks that then connect to a larger backbone 0:01:36.880 --> 0:01:40.520 that allow all these various networks to interact and communicate 0:01:40.560 --> 0:01:44.040 with one another. That is the Internet. The Worldwide Web 0:01:44.240 --> 0:01:47.960 is one thing that sits on top of this network 0:01:48.000 --> 0:01:52.240 of networks, other things being email and FTP servers and 0:01:52.960 --> 0:01:56.440 other stuff that uses the Internet as its method of 0:01:56.520 --> 0:02:00.160 transmitting data to and from different computers. But the Old 0:02:00.200 --> 0:02:02.960 Wide Web is often what we think of with the 0:02:03.000 --> 0:02:06.200 Internet because it is a very forward facing part of 0:02:06.240 --> 0:02:08.600 the Web, or the Internet rather, Right. One way to 0:02:08.639 --> 0:02:12.200 think about the Web is that it's a gigantic collection 0:02:12.320 --> 0:02:16.679 of interactive documents. Yeah, exactly. Yeah. Some of those documents 0:02:16.720 --> 0:02:21.200 are very static and they don't change frequently or at all. 0:02:21.680 --> 0:02:25.000 Some of them are more like programs. Yeah, yeah, some 0:02:25.080 --> 0:02:27.360 of them are more like like white boards, where you 0:02:27.400 --> 0:02:29.280 know stuff is being put up and taking down and 0:02:29.280 --> 0:02:32.440 put up and taking down constantly. So some of them 0:02:32.480 --> 0:02:35.200 linked to lots of other documents, some do not, yep, 0:02:35.320 --> 0:02:39.040 so some are applications. Right, So you've you've got this 0:02:39.400 --> 0:02:43.360 massive number of documents. And when we say massive, uh, 0:02:43.440 --> 0:02:47.680 it's hard to put it all into context. First of all, 0:02:47.720 --> 0:02:52.839 if you talk about all the information that we have created, 0:02:53.320 --> 0:02:56.160 not us, but humanity, humanity itself, you have the three 0:02:56.200 --> 0:03:00.000 of us have done our share, but no humanity overall. 0:02:59.800 --> 0:03:02.080 All the information that has been created, well back in 0:03:02.120 --> 0:03:04.400 two thousand and twelve, that was estimated to be at 0:03:04.480 --> 0:03:09.480 two point eight zetta bytes two point eight trillion gigabytes, 0:03:10.480 --> 0:03:16.760 trillion gigabytes. It's bigger than my hard drive significantly. So yeah, 0:03:17.040 --> 0:03:20.560 if you're hard drive can hold two point eight zetta bytes, 0:03:21.200 --> 0:03:24.320 I need to see your gaming rig sir. I think 0:03:24.400 --> 0:03:27.519 I have downloaded two point eight zeta bytes of Pirated 0:03:27.560 --> 0:03:30.680 anime before I was gonna say I have two point 0:03:30.680 --> 0:03:34.280 eight zeta bytes of Skyrim mods. But so no, not 0:03:34.400 --> 0:03:38.000 all of this data is necessarily available for access on 0:03:38.040 --> 0:03:40.320 the web, right, This is just data that we have created. 0:03:40.720 --> 0:03:43.640 So let's let's narrow it down and look at the 0:03:43.680 --> 0:03:46.920 information that's actually on the web. So the web has 0:03:46.960 --> 0:03:51.520 between ten billion and one trillion documents on it. Now 0:03:51.600 --> 0:03:55.160 that's a huge range, but it tells you that it's 0:03:55.200 --> 0:03:57.760 hard to make an estimate about something that one is 0:03:57.800 --> 0:04:01.520 so big and two is rapidly evolving. Right, there are 0:04:01.520 --> 0:04:04.920 always things being added to it and deleted from it. Yeah, 0:04:05.000 --> 0:04:07.960 you have servers that go offline from the Internet. If 0:04:07.960 --> 0:04:11.440 those servers had web pages on them, those, unless they've 0:04:11.480 --> 0:04:14.720 been mirrored onto other servers, are no longer accessible. They 0:04:14.720 --> 0:04:18.400 have they have left the web. Other people deleting their 0:04:18.400 --> 0:04:23.359 MySpace accounts. Why would you do that? Look, I have 0:04:23.520 --> 0:04:29.000 so few friends there, but so many awesome bands. Uh yeah, 0:04:29.040 --> 0:04:32.560 so seriously true moment. Does my Space still exist? Yes, yes, 0:04:32.640 --> 0:04:37.640 it's largely How recently did you check? Probably? Probably then 0:04:37.880 --> 0:04:41.479 eight months ago. Let's look it up. Yeah, because it's 0:04:41.480 --> 0:04:44.280 a it's a music discovery site more than anything else. Now, 0:04:44.320 --> 0:04:47.440 Oh yeah, here we go MySpace dot com. Oh oh, 0:04:47.480 --> 0:04:49.760 it's it's breaking my browser home. I was about to say, 0:04:49.760 --> 0:04:51.720 why did you go to that? You realize that my 0:04:51.800 --> 0:04:55.480 Space is like the home of the auto play music file. Right, No, 0:04:55.880 --> 0:04:58.680 we just talked about how that's like my least favorite thing. Well, 0:04:58.760 --> 0:05:04.800 let's not. Let's not invoke the auto playing music gods. Yeah. So, 0:05:04.800 --> 0:05:07.360 so the reason why we're even talking about how much 0:05:07.400 --> 0:05:10.120 information is on the web and how many documents there 0:05:10.160 --> 0:05:12.800 are out there is that the web. You can think 0:05:12.800 --> 0:05:17.320 of the web as representing the world's largest database of information, 0:05:17.400 --> 0:05:21.839 and that information spans every topic imaginable. Yeah, and there's 0:05:21.960 --> 0:05:24.719 lots of great stuff out there that might be really 0:05:24.760 --> 0:05:28.039 relevant to you, might have answers to questions that you have, 0:05:28.279 --> 0:05:30.680 or it might just be very interesting to you. But 0:05:31.600 --> 0:05:34.400 a strange question that you may never have considered is 0:05:35.760 --> 0:05:38.640 how do I get the stuff that I want to 0:05:38.720 --> 0:05:41.160 get from the web? I mean, you know, how you 0:05:41.200 --> 0:05:43.400 get it in practical terms while you go you sit 0:05:43.440 --> 0:05:47.120 down at Google and you type in terms or Google yeah, 0:05:47.240 --> 0:05:50.080 or you or you maybe have some kind of aggregator, 0:05:50.200 --> 0:05:52.680 like a friend on social media or some kind of 0:05:52.760 --> 0:05:59.400 things content writer, or perhaps you have received the direct 0:05:59.400 --> 0:06:03.159 to your l of a website that you wish to visit. Ye, 0:06:03.600 --> 0:06:05.880 might you might have one in particular in mind that 0:06:05.960 --> 0:06:09.440 you go too frequently, and so like dinosaur Comics Dot 0:06:09.440 --> 0:06:13.560 com awesome should example. Yeah, so they're all all these 0:06:13.600 --> 0:06:16.200 different ways. But let's say that you want to use 0:06:16.240 --> 0:06:19.440 the web to do something dion just visiting a particular 0:06:19.520 --> 0:06:22.080 web page if you know the U r L, that's 0:06:22.080 --> 0:06:25.920 pretty simple. But what if you're you're just trying to 0:06:25.960 --> 0:06:29.320 find something. Yeah, maybe that you don't even know what 0:06:29.440 --> 0:06:31.520 that thing is, or you know what that thing is, 0:06:31.560 --> 0:06:35.240 but nobody has gathered that and placed it into an 0:06:35.240 --> 0:06:38.760 easily digestible piece of information. So, in other words, let's 0:06:38.800 --> 0:06:42.520 say that you're looking at some sort of statistical uh 0:06:43.160 --> 0:06:45.160 result that you want to know. You want to know 0:06:45.279 --> 0:06:50.520 the percentage of people who drove red cars in two 0:06:50.560 --> 0:06:55.920 thousand and twelve who ended up getting speeding tickets, and 0:06:55.960 --> 0:06:57.680 you know this this sort of thing like, there may 0:06:57.720 --> 0:07:00.599 be a web page out there that has that specific 0:07:00.680 --> 0:07:03.720 answer on it, but there may not be. However, there 0:07:03.839 --> 0:07:07.520 may be the data out there that exists across multiple 0:07:07.560 --> 0:07:11.239 web pages and multiple places that could answer that question 0:07:11.320 --> 0:07:15.440 for you, but there's no easy way for the average 0:07:15.480 --> 0:07:19.200 person to be able to collect and coalate all that data, 0:07:19.280 --> 0:07:23.600 analyze it and get to a meaningful answer, especially not quickly, 0:07:23.680 --> 0:07:27.040 because if you wanted to go through the entire Internet 0:07:27.080 --> 0:07:29.720 to try to find that information, it would take you 0:07:29.720 --> 0:07:32.400 a minute. Yeah, it would take quite some time. So 0:07:32.640 --> 0:07:36.680 what we wanted to do is tough. Sometimes well, again, 0:07:36.720 --> 0:07:39.080 depending upon what it is you're looking for, right, because 0:07:39.160 --> 0:07:43.120 in some cases you may have very little information and 0:07:43.280 --> 0:07:45.280 it may take you some time just to make sure 0:07:45.320 --> 0:07:49.240 that the information that you do have is worthy of consideration. 0:07:49.880 --> 0:07:52.640 Or you may have the opposite problem. Let's say you 0:07:52.640 --> 0:07:54.160 want to look at anything that has to do with 0:07:54.200 --> 0:07:59.280 about cats, good grief, You're gonna have so much information 0:07:59.360 --> 0:08:02.280 on the inner and on the web that relates to 0:08:02.360 --> 0:08:05.560 cats that finding the you know, separating that the signal 0:08:05.680 --> 0:08:08.600 from the noise would take you a really long time. 0:08:08.680 --> 0:08:13.680 So and uh, this problem is already has a solution, 0:08:13.880 --> 0:08:18.360 and that is why we are today talking about web crawlers. Yeah. 0:08:18.400 --> 0:08:20.920 And web crawlers are something that have been around for 0:08:21.000 --> 0:08:23.440 about as long as the web has been around, because 0:08:23.480 --> 0:08:26.720 people realized early on that in order to make the 0:08:26.720 --> 0:08:31.720 web really user friendly, especially once it grew beyond a 0:08:31.760 --> 0:08:36.280 collection of you know, three computers, right, Yeah, three computers 0:08:36.320 --> 0:08:40.360 with twelve web pages altogether, Once you get past all 0:08:40.360 --> 0:08:42.000 of that and you get to a point where it 0:08:42.080 --> 0:08:45.760 really is growing rapidly. You need a way to navigate 0:08:45.840 --> 0:08:48.280 through the web and find the stuff you're interested in. 0:08:48.600 --> 0:08:51.400 You need an index. Yeah, you have to have that 0:08:51.480 --> 0:08:53.840 index because otherwise the only other option you have is 0:08:53.880 --> 0:08:57.440 to know the address of a particular web page and 0:08:57.480 --> 0:08:59.839 then to just follow whatever links that web page have 0:09:00.120 --> 0:09:02.319 is to have, and then once you hit a dead end, 0:09:02.640 --> 0:09:04.520 you've got to backtrack. And you know, it's kind of 0:09:04.520 --> 0:09:06.480 like a choose your own adventure book, And it's a 0:09:06.559 --> 0:09:09.520 choose your own adventure book that's that isn't even connected 0:09:09.559 --> 0:09:13.280 to all the pages that you need. Right, So indexing 0:09:13.440 --> 0:09:17.320 is a way of creating a means to find web 0:09:17.360 --> 0:09:23.360 pages about any given keyword. Right, And again, this is 0:09:23.400 --> 0:09:27.880 a big, big job. You can't expect this to be 0:09:27.960 --> 0:09:31.920 something that only humans are doing under human power. It 0:09:31.960 --> 0:09:35.880 would take way too long and it would be exhausting. 0:09:36.000 --> 0:09:39.600 So there have to be automated ways to index web pages. 0:09:39.640 --> 0:09:43.200 Well yeah, I mean, just consider the ridiculousness of the alternative. 0:09:43.840 --> 0:09:47.439 So let's say you are searching for a term and 0:09:47.520 --> 0:09:53.360 that term is, I don't know, lobster baseball. Somewhere out there, 0:09:53.400 --> 0:09:56.280 there might be a page about lobster baseball, but it 0:09:56.320 --> 0:09:59.680 would not be a good way to find it. To say, well, 0:09:59.720 --> 0:10:03.120 I'm going to ping every web server in the world 0:10:03.800 --> 0:10:07.360 and see if it's offering any public pages that say 0:10:07.440 --> 0:10:10.640 lobster baseball on them. Yeah, that would not especially you know, 0:10:10.720 --> 0:10:13.880 as the Web grows and gets larger and larger and larger, 0:10:14.040 --> 0:10:17.240 that task becomes impossible. It would just it would take 0:10:17.280 --> 0:10:21.360 your computer longer than your lifespan to complete the job, 0:10:21.480 --> 0:10:24.320 especially considering that, as we mentioned before, the web is 0:10:24.360 --> 0:10:28.800 constantly changing, so we would have new web servers joining 0:10:28.840 --> 0:10:32.200 while you're still doing this pinging operation, which means you 0:10:32.280 --> 0:10:34.440 just have you know, you've added more that you have 0:10:34.520 --> 0:10:38.240 to ping before you're done. You never finish. So what's 0:10:38.280 --> 0:10:41.960 the solution, Well, web crawlers would be would be the solution, Joe. 0:10:42.240 --> 0:10:45.720 Web crawlers and search engines are our favorite things here 0:10:45.800 --> 0:10:47.960 at Health Tough Works. I mean, if if it weren't 0:10:47.960 --> 0:10:52.760 for them, our jobs would be significantly more difficult. So, uh, 0:10:52.840 --> 0:10:56.200 let's say that you've got all right, So to break 0:10:56.200 --> 0:10:58.520 it down, we've got web servers that have web pages 0:10:58.559 --> 0:11:02.160 on them, right, we have it's a computer somewhere out there. 0:11:02.600 --> 0:11:05.520 It's got a public facing document that it will show 0:11:05.559 --> 0:11:07.960 you if you ask for right, and your browser is 0:11:07.960 --> 0:11:10.240 the way that you ask for it right, So your 0:11:10.280 --> 0:11:13.960 browser is your conduit to getting the information that's stored 0:11:14.120 --> 0:11:18.080 on other computers that maybe on completely different networks, on 0:11:18.120 --> 0:11:22.719 another on another part of the world even and the 0:11:22.760 --> 0:11:24.800 fact that you have a browser that is what allows 0:11:24.840 --> 0:11:27.960 you to have the access to that document that exists 0:11:27.960 --> 0:11:31.120 on that other page. But those servers can have really 0:11:31.160 --> 0:11:34.360 funky names. Um, the web pages may not have a 0:11:34.400 --> 0:11:37.800 title that is is identical to what it is you're 0:11:37.800 --> 0:11:40.760 looking for, but the information may be in that page. 0:11:41.160 --> 0:11:44.200 Sure for to to use my prior example, dinosaur comics 0:11:44.240 --> 0:11:47.520 dot com used to be known only as quantz dot com. 0:11:47.520 --> 0:11:50.560 Perfect with the QW the way that you sometimes spell 0:11:50.640 --> 0:11:55.120 words much yes, exactly the way the way words are 0:11:55.120 --> 0:11:59.040 never spelled in English. Uh yeah, I I My example 0:11:59.040 --> 0:12:00.800 of my notes I wrote is that let's say that 0:12:00.800 --> 0:12:03.680 you're looking for funny cat memes. The funniest memes happen 0:12:03.679 --> 0:12:05.680 to be on a page that has the title things. 0:12:05.840 --> 0:12:09.480 FDR definitely didn't say, Well, the title of the page 0:12:09.520 --> 0:12:11.719 wouldn't tell you that there are cat memes on there. 0:12:11.720 --> 0:12:15.160 You would need something to have searched that page to 0:12:15.320 --> 0:12:19.400 understand what actually appears on that page, the context within 0:12:19.880 --> 0:12:22.520 which it appears, and to be able to serve that 0:12:22.640 --> 0:12:25.400 up to you. And that's really where the crawlers come in. 0:12:25.480 --> 0:12:28.280 They they build out these indexes of words and where 0:12:28.320 --> 0:12:32.120 to find those words on the web like uh, they 0:12:32.200 --> 0:12:35.800 use lots of They use well, actually pretty simple software. 0:12:36.720 --> 0:12:39.800 They are often referred to as either robots or spiders, 0:12:40.040 --> 0:12:42.480 and they're called spiders because they crawl the web. That 0:12:42.559 --> 0:12:47.160 is good. Yeah, alright, So here's where we mentioned that 0:12:47.200 --> 0:12:53.920 most of these terms. So wait, are we the flies? Good? 0:12:54.160 --> 0:12:59.640 Good question? I mean, I think the uh cams, I'm 0:12:59.679 --> 0:13:03.960 not so. So here's where we mentioned that a lot 0:13:04.000 --> 0:13:07.600 of these terms were all invented around the same time. 0:13:07.720 --> 0:13:10.760 And boy, when we when we go with a metaphor, 0:13:10.880 --> 0:13:15.040 we just go whole spider. So um so all right, 0:13:15.080 --> 0:13:18.599 So spiders typically start by traveling to web servers that 0:13:18.720 --> 0:13:22.040 have lots of traffic, the ones that are the most popular, 0:13:22.600 --> 0:13:25.400 and they explore the most popular web pages and start 0:13:25.440 --> 0:13:28.559 to build up the index of words of those web pages. 0:13:28.600 --> 0:13:31.920 Then all the links that are on those popular web pages, 0:13:32.000 --> 0:13:35.800 the spiders start to follow those links and index those 0:13:35.840 --> 0:13:39.319 pages in turn, and then do the same thing over 0:13:39.400 --> 0:13:41.760 and over and over again, so they just you know, 0:13:41.800 --> 0:13:44.600 it's it is like a spider web or a crack 0:13:44.640 --> 0:13:46.880 in the glass where you see its splintering over and 0:13:46.920 --> 0:13:49.679 over while the glass shatters. The same sort of thing. 0:13:49.880 --> 0:13:54.240 It's following all those potential pathways, and they can hold 0:13:54.520 --> 0:13:57.720 hundreds of pages open at a time. We're talking like 0:13:57.920 --> 0:14:04.439 three hundred pages a second. So yes, more than Google 0:14:04.520 --> 0:14:07.000 Chrome will allow me to have opened before my computer 0:14:07.080 --> 0:14:10.400 says listen, I give up. Uh. So, depending on the 0:14:10.440 --> 0:14:13.040 crawler of the spiders will index these pages based upon 0:14:13.240 --> 0:14:16.080 which words appear in the page and where those words 0:14:16.120 --> 0:14:20.480 actually appear in that page, like in what context. So 0:14:20.720 --> 0:14:23.080 you may remember in the early days of the web, 0:14:23.160 --> 0:14:27.600 before web search engines got really sophisticated, that some people 0:14:27.720 --> 0:14:30.480 would make a web page and then just litter the 0:14:30.520 --> 0:14:33.240 bottom of the page with tons of random words that 0:14:33.320 --> 0:14:36.760 we're doing really well in search, mostly because they had 0:14:36.800 --> 0:14:39.160 ads served on the page of the type that they 0:14:39.240 --> 0:14:42.640 got money from per page view. So I might have 0:14:42.680 --> 0:14:46.920 a talk Brittney Spear, right, Yeah, it would often be 0:14:47.080 --> 0:14:50.280 celebrity rumors and gossip that kind of stuff, and just 0:14:50.480 --> 0:14:54.800 random recipe. Yeah, yeah, it'd be weird stuff, like totally 0:14:54.960 --> 0:14:57.200 some of it would be disturbing to read. You're like, wow, 0:14:57.240 --> 0:14:59.760 I can't believe that that. To know that this particular 0:14:59.800 --> 0:15:02.040 turn m is a very popular search term is disturbing 0:15:02.320 --> 0:15:06.640 others would Yeah. Yeah, I'm more of an A C, 0:15:06.800 --> 0:15:09.320 D C kind of guy myself, so I'm I'm with 0:15:09.360 --> 0:15:12.120 you there. So anyway, Uh, you know, this was a 0:15:12.160 --> 0:15:16.720 way of fooling search engines into into indexing that page 0:15:16.840 --> 0:15:20.720 on multiple indexes so that it would appear no matter 0:15:20.760 --> 0:15:23.000 what search you put in, your page would pop up. 0:15:23.400 --> 0:15:26.320 You as a as saying, if you're assuming you are 0:15:26.320 --> 0:15:28.920 the one who are administering this web page, you have 0:15:28.960 --> 0:15:31.960 no ethics, Like, you don't care if people come to 0:15:32.000 --> 0:15:34.680 your page and are completely disappointed because it has nothing 0:15:34.720 --> 0:15:36.440 to do with the search term they put in there. 0:15:36.480 --> 0:15:39.080 You just want to get those sweet sweet clicks. You 0:15:39.160 --> 0:15:41.160 just need the page views because you need to pay 0:15:41.200 --> 0:15:46.280 the bills, right, So, uh, search engines and spiders got 0:15:46.320 --> 0:15:49.320 more sophisticated, so they were able to look for the 0:15:49.400 --> 0:15:52.480 placement of words where it fell in the page, whether 0:15:52.600 --> 0:15:54.840 or not it appeared more than once within a page, 0:15:54.840 --> 0:15:58.960 to understand if a page really was about that particular 0:15:59.040 --> 0:16:01.720 search term, or if it was just one of those 0:16:01.720 --> 0:16:05.320 things where the word happened to appear once, it may 0:16:05.400 --> 0:16:08.440 be a saying or a quote that has very little 0:16:08.440 --> 0:16:10.480 to do with the actual substance of the rest of 0:16:10.520 --> 0:16:13.360 the page. You know, this would help the search engine 0:16:13.440 --> 0:16:16.880 rank the page in search right. Right, So, the final 0:16:16.960 --> 0:16:21.600 product of of these spiders doing this indexing is called 0:16:21.600 --> 0:16:26.520 a crawl and and it's essentially a lightweight copy of 0:16:26.720 --> 0:16:30.360 the Worldwide Web that's built to be much more easily 0:16:30.360 --> 0:16:33.160 searched than the whole web itself. Uh and and a 0:16:33.240 --> 0:16:36.960 crawl usually consists, therefore, of this huge cash of data 0:16:37.000 --> 0:16:39.880 about the web, including like the text of each page 0:16:39.960 --> 0:16:44.360 it's spiders encountered, the code that constructed those pages h T, M, 0:16:44.480 --> 0:16:47.800 L or et cetera, uh, and a certain amount of 0:16:48.000 --> 0:16:50.560 metadata um you know, certainly the pages r L and 0:16:50.640 --> 0:16:53.080 maybe the tags. As we discussed, that's not always as 0:16:53.160 --> 0:16:56.360 useful as it used to be due to uh scammy stuff. 0:16:56.360 --> 0:17:00.360 But yeah, so so creating a crawl is a huge 0:17:00.440 --> 0:17:03.920 project in terms of time and computer equipment and drive 0:17:03.960 --> 0:17:08.800 space and spider programming and just sheer Internet bandwidth. Right. So, 0:17:09.080 --> 0:17:11.760 for the longest time, this is something that was really 0:17:11.880 --> 0:17:18.640 only accessible by big corporations like Google or Microsoft, Yahoo that. Yeah, 0:17:18.680 --> 0:17:22.480 we're talking huge companies that have the computer power and 0:17:22.520 --> 0:17:25.000 the bandwidth to pull this sort of stuff off on 0:17:25.040 --> 0:17:28.359 a on a regular basis. And while those are incredibly 0:17:28.440 --> 0:17:31.280 useful for us as consumers, if we are looking for 0:17:31.320 --> 0:17:33.919 a specific piece of information that happens to live somewhere 0:17:33.920 --> 0:17:35.920 on the web page, if we want to do more 0:17:35.960 --> 0:17:39.520 of a big data analysis something where we need to 0:17:40.200 --> 0:17:44.800 collate the information across multiple, perhaps hundreds or thousands of 0:17:44.840 --> 0:17:49.040 web pages, it's not easy, right. We don't have those 0:17:49.080 --> 0:17:51.560 tools for the most part, right right, Because when you 0:17:51.600 --> 0:17:55.000 go to Google, you can't access that level of information. 0:17:55.080 --> 0:17:57.640 Yeah you can, you can ask, uh, you know what 0:17:57.720 --> 0:18:00.239 Hugh Grant was doing last week? Right? Yeah, you can 0:18:00.280 --> 0:18:04.480 get the most popular or the highest ranking search results, 0:18:04.520 --> 0:18:07.760 which could give you at least some useful information. But again, 0:18:07.800 --> 0:18:11.879 if you want to do a wide spread study on 0:18:12.119 --> 0:18:16.560 a specific thing, unless someone's already done it, in which 0:18:16.600 --> 0:18:19.399 case you may just need to replicate their their study 0:18:19.440 --> 0:18:22.600 to make sure that it was correct. Um, you're you're 0:18:22.680 --> 0:18:26.080 kind of out of luck. So where can someone turn. 0:18:26.280 --> 0:18:30.760 Let's say that it's a researcher who's working on something 0:18:30.840 --> 0:18:33.400 and they don't work for one of these big companies. 0:18:33.480 --> 0:18:37.560 Where can they turn to leverage the incredible asset that 0:18:37.760 --> 0:18:41.080 is the World Wide Web? One? Gil Lbez started up 0:18:41.119 --> 0:18:46.320 a nonprofit corporation called the Common Crawl Foundation and it 0:18:46.480 --> 0:18:51.440 has been since then working on providing public, publicly accessible, 0:18:51.960 --> 0:18:56.160 free crawls to anyone who wants to use them. And 0:18:56.720 --> 0:18:59.320 uh ls is a really interesting dude. A little bit 0:18:59.320 --> 0:19:01.959 of background on him. Um. He co founded a company 0:19:02.000 --> 0:19:05.879 back in the nineties called Applied Semantics, which created software 0:19:05.880 --> 0:19:10.480 that matched ads to web pages like contextually and automatically. 0:19:10.760 --> 0:19:13.960 Oh we know a little bit about that. Yeah, yeah, 0:19:14.000 --> 0:19:16.879 And this prompted Google to acquire them in two thousand 0:19:16.960 --> 0:19:20.800 three for like a hundred two million bucks, So not 0:19:20.920 --> 0:19:24.040 doing too bad for himself. Also, that's that's essentially the 0:19:24.040 --> 0:19:27.600 reason why Google AdSense exists. That the programming that led 0:19:27.640 --> 0:19:32.760 to Google AdSense so very very practical application of that 0:19:32.840 --> 0:19:37.679 contextual understanding, right right. Um. In two thousand eight, interestingly, 0:19:37.720 --> 0:19:39.439 and kind of a side note, he founded a company 0:19:39.440 --> 0:19:44.600 called Factual, which seeks to gather and analyze global location 0:19:44.840 --> 0:19:49.320 data in order to create a repository of really high quality, 0:19:49.440 --> 0:19:56.239 easily accessible location data that's uh factual um and and 0:19:56.520 --> 0:19:59.760 companies like Bang and Samsung and Yelp all use factuals 0:20:00.119 --> 0:20:05.000 to construct local maps and personalized advertising for mobile consumers. 0:20:05.600 --> 0:20:08.800 So uh so pretty nifty stuff. And what I am 0:20:08.800 --> 0:20:12.439 saying is that elbas is passionate about and experienced with 0:20:12.560 --> 0:20:15.919 big data, right, and we've talked about it before on 0:20:15.960 --> 0:20:20.399 this podcast. That big data is. You know, it sounds 0:20:20.400 --> 0:20:22.600 like one of those just buzz industry terms, but it 0:20:22.720 --> 0:20:25.919 really is one of those things that holds a huge 0:20:25.920 --> 0:20:30.119 amount of potential to affect our lives in different ways, 0:20:30.160 --> 0:20:34.639 assuming that we've developed the right means to analyze that 0:20:34.880 --> 0:20:37.520 massive amount of information that's had there to collect it 0:20:37.520 --> 0:20:40.040 in the first place. Sure, and once you have a 0:20:40.119 --> 0:20:43.320 way of processing, of collecting and being able to access 0:20:43.359 --> 0:20:46.399 and process vast amounts of data, you can do a 0:20:46.400 --> 0:20:50.920 lot of amazing things like big data. And the ability 0:20:51.000 --> 0:20:53.960 to process it might be the key to say, for example, 0:20:54.440 --> 0:21:00.000 computational modeling that predicts complex social phenomenon by analyzing big 0:21:00.119 --> 0:21:03.480 data coming from social media and from news and from 0:21:03.520 --> 0:21:06.359 weather and from all kinds of sources, it can really 0:21:06.359 --> 0:21:09.840 mean that we are able to actually see elements of 0:21:10.000 --> 0:21:13.760 order and what previously appeared to be a truly chaotic system, 0:21:13.880 --> 0:21:17.320 which is kind of exciting. Sure. And then on the 0:21:17.359 --> 0:21:19.679 other hand, a lot of people think that big data 0:21:19.760 --> 0:21:23.040 could be one of the ways that we finally achieve 0:21:23.240 --> 0:21:27.560 that next level of artificial intelligence by having machines sort 0:21:27.600 --> 0:21:31.720 of plumb the depths of this data with self teaching 0:21:31.800 --> 0:21:35.399 and self learning mechanisms. Right, well, let's get back to 0:21:35.560 --> 0:21:39.640 the common crawl okay, Right, So the Foundation began compiling 0:21:39.800 --> 0:21:42.840 crawls back in two thousand eight. The most recent one 0:21:42.880 --> 0:21:45.159 that they released as of this podcast at the end 0:21:45.200 --> 0:21:50.840 of May, was from April. It was some a hundred 0:21:50.840 --> 0:21:56.600 and sixty eight terabits in size bytes in size huge. 0:21:56.920 --> 0:22:01.920 That's big, uh, and contains some two point one billion 0:22:02.160 --> 0:22:08.399 web pages. That's not that many, really, I wrote I 0:22:08.440 --> 0:22:13.480 wrote forty seven million web pages before breakfast. No I 0:22:13.560 --> 0:22:19.360 did not. I'm just kidding. No, that's a lot. Yeah, yeah, yeah, 0:22:19.359 --> 0:22:22.480 it's it's a it's a bunch um uh. But so 0:22:22.720 --> 0:22:26.840 they're they're continuously indexing and releasing new crawls, right, It's 0:22:26.880 --> 0:22:31.240 not like it's here's the Internet and now we're done. Yeah. Yeah, 0:22:31.280 --> 0:22:33.960 they've been releasing a new crawl every month since July. 0:22:35.200 --> 0:22:37.600 That's I mean, that's incredible you think about the amount 0:22:37.600 --> 0:22:39.399 of work that does. It also means that you have 0:22:39.560 --> 0:22:44.000 like a a time a timestamp, like photograph of what 0:22:44.119 --> 0:22:48.520 the web was at that moment from these crawls. Yeah. Yeah, 0:22:48.560 --> 0:22:50.520 I hadn't thought about it quite that way before, but yeah, 0:22:50.560 --> 0:22:53.879 that's it's kind of you know, things that may not 0:22:54.040 --> 0:22:56.640 exist from one month to the next, you could actually 0:22:57.320 --> 0:23:01.040 see and watch those trends. Yeah, it's fascinating. Yeah. I'd 0:23:01.040 --> 0:23:03.560 say one of the main ways that I often encounter 0:23:04.600 --> 0:23:08.520 web archives is when I'm trying to find evidence of 0:23:08.640 --> 0:23:14.240 something somebody did in the past that they wanted expunge. Right, 0:23:14.640 --> 0:23:16.840 This makes sound like I'm some kind of detective, not 0:23:16.960 --> 0:23:19.000 like I'm trying to find but you know what I'm 0:23:19.000 --> 0:23:21.320 talking about. No, I know, I will post something and 0:23:21.359 --> 0:23:23.879 then they'll be like, oh wait a minute, No, that 0:23:23.960 --> 0:23:25.639 was a bad idea. I know if you try to 0:23:25.680 --> 0:23:27.600 delete it. I know, if you use archive dot org, 0:23:27.680 --> 0:23:30.159 you can find one of the web pages I built 0:23:30.240 --> 0:23:35.080 way back when, and I never want anyone to ever 0:23:35.080 --> 0:23:37.960 see it because it was that bad. But they will 0:23:38.080 --> 0:23:40.639