Speaker 1: Brought to you by Toyota. Let's go places. Welcome to
Speaker 1: Forward Thinking. Hey, they're in Loving and Forward Thinking, the
Speaker 1: podcast that looks at the future and says, Kitty McGee's
Speaker 1: in Dublin Town upon the Crawl. I'm Jonathan Strickland and
Speaker 1: I'm Joe McCormick, and today we're gonna be talking about
Speaker 1: the crawl. We are talking about the crawl, not a
Speaker 1: pub crawl. No, sadly, not a pub crawl, which is
Speaker 1: what I was referring to in the lyric. But that's
Speaker 1: not what we're talking about today. The krawl. What is that?
Speaker 1: Is that the name of a movie that was like
Speaker 1: a like a fantasy movie from the eighties, or it
Speaker 1: sounds like a I'm thinking krawl. Oh yeah, that's a
Speaker 1: science fiction fantasy film with a phenomenal one I might
Speaker 1: add phenomenal science fiction fantasy film. Okay, So why would
Speaker 1: we be talking about a crawl that's not a pub
Speaker 1: crawl and not a sci fi fantasy movie. And it's
Speaker 1: not the future of baby these crawling right. No, it
Speaker 1: has something to do with the Internet. Yes, it has
Speaker 1: everything to do with the Internet Web in particular. Actually, uh,
Speaker 1: and here's a funny little little tidbit of information that
Speaker 1: you probably already knew, but you might be a little
Speaker 1: bit fuzzy on. Wait, what's the difference between the Web
Speaker 1: and the Internet. Because when I say the Internet, most
Speaker 1: of the time, what I'm talking about is the place
Speaker 1: where people leave comments and argue about things, which would
Speaker 1: be the Web mostly. Right. So Internet is the network
Speaker 1: of networks of computers. Right. So You've got all these
Speaker 1: different computer networks that then connect to a larger backbone
Speaker 1: that allow all these various networks to interact and communicate
Speaker 1: with one another. That is the Internet. The Worldwide Web
Speaker 1: is one thing that sits on top of this network
Speaker 1: of networks, other things being email and FTP servers and
Speaker 1: other stuff that uses the Internet as its method of
Speaker 1: transmitting data to and from different computers. But the Old
Speaker 1: Wide Web is often what we think of with the
Speaker 1: Internet because it is a very forward facing part of
Speaker 1: the Web, or the Internet rather, Right. One way to
Speaker 1: think about the Web is that it's a gigantic collection
Speaker 1: of interactive documents. Yeah, exactly. Yeah. Some of those documents
Speaker 1: are very static and they don't change frequently or at all.
Speaker 1: Some of them are more like programs. Yeah, yeah, some
Speaker 1: of them are more like like white boards, where you
Speaker 1: know stuff is being put up and taking down and
Speaker 1: put up and taking down constantly. So some of them
Speaker 1: linked to lots of other documents, some do not, yep,
Speaker 1: so some are applications. Right, So you've you've got this
Speaker 1: massive number of documents. And when we say massive, uh,
Speaker 1: it's hard to put it all into context. First of all,
Speaker 1: if you talk about all the information that we have created,
Speaker 1: not us, but humanity, humanity itself, you have the three
Speaker 1: of us have done our share, but no humanity overall.
Speaker 1: All the information that has been created, well back in
Speaker 1: two thousand and twelve, that was estimated to be at
Speaker 1: two point eight zetta bytes two point eight trillion gigabytes,
Speaker 1: trillion gigabytes. It's bigger than my hard drive significantly. So yeah,
Speaker 1: if you're hard drive can hold two point eight zetta bytes,
Speaker 1: I need to see your gaming rig sir. I think
Speaker 1: I have downloaded two point eight zeta bytes of Pirated
Speaker 1: anime before I was gonna say I have two point
Speaker 1: eight zeta bytes of Skyrim mods. But so no, not
Speaker 1: all of this data is necessarily available for access on
Speaker 1: the web, right, This is just data that we have created.
Speaker 1: So let's let's narrow it down and look at the
Speaker 1: information that's actually on the web. So the web has
Speaker 1: between ten billion and one trillion documents on it. Now
Speaker 1: that's a huge range, but it tells you that it's
Speaker 1: hard to make an estimate about something that one is
Speaker 1: so big and two is rapidly evolving. Right, there are
Speaker 1: always things being added to it and deleted from it. Yeah,
Speaker 1: you have servers that go offline from the Internet. If
Speaker 1: those servers had web pages on them, those, unless they've
Speaker 1: been mirrored onto other servers, are no longer accessible. They
Speaker 1: have they have left the web. Other people deleting their
Speaker 1: MySpace accounts. Why would you do that? Look, I have
Speaker 1: so few friends there, but so many awesome bands. Uh yeah,
Speaker 1: so seriously true moment. Does my Space still exist? Yes, yes,
Speaker 1: it's largely How recently did you check? Probably? Probably then
Speaker 1: eight months ago. Let's look it up. Yeah, because it's
Speaker 1: a it's a music discovery site more than anything else. Now,
Speaker 1: Oh yeah, here we go MySpace dot com. Oh oh,
Speaker 1: it's it's breaking my browser home. I was about to say,
Speaker 1: why did you go to that? You realize that my
Speaker 1: Space is like the home of the auto play music file. Right, No,
Speaker 1: we just talked about how that's like my least favorite thing. Well,
Speaker 1: let's not. Let's not invoke the auto playing music gods. Yeah. So,
Speaker 1: so the reason why we're even talking about how much
Speaker 1: information is on the web and how many documents there
Speaker 1: are out there is that the web. You can think
Speaker 1: of the web as representing the world's largest database of information,
Speaker 1: and that information spans every topic imaginable. Yeah, and there's
Speaker 1: lots of great stuff out there that might be really
Speaker 1: relevant to you, might have answers to questions that you have,
Speaker 1: or it might just be very interesting to you. But
Speaker 1: a strange question that you may never have considered is
Speaker 1: how do I get the stuff that I want to
Speaker 1: get from the web? I mean, you know, how you
Speaker 1: get it in practical terms while you go you sit
Speaker 1: down at Google and you type in terms or Google yeah,
Speaker 1: or you or you maybe have some kind of aggregator,
Speaker 1: like a friend on social media or some kind of
Speaker 1: things content writer, or perhaps you have received the direct
Speaker 1: to your l of a website that you wish to visit. Ye,
Speaker 1: might you might have one in particular in mind that
Speaker 1: you go too frequently, and so like dinosaur Comics Dot
Speaker 1: com awesome should example. Yeah, so they're all all these
Speaker 1: different ways. But let's say that you want to use
Speaker 1: the web to do something dion just visiting a particular
Speaker 1: web page if you know the U r L, that's
Speaker 1: pretty simple. But what if you're you're just trying to
Speaker 1: find something. Yeah, maybe that you don't even know what
Speaker 1: that thing is, or you know what that thing is,
Speaker 1: but nobody has gathered that and placed it into an
Speaker 1: easily digestible piece of information. So, in other words, let's
Speaker 1: say that you're looking at some sort of statistical uh
Speaker 1: result that you want to know. You want to know
Speaker 1: the percentage of people who drove red cars in two
Speaker 1: thousand and twelve who ended up getting speeding tickets, and
Speaker 1: you know this this sort of thing like, there may
Speaker 1: be a web page out there that has that specific
Speaker 1: answer on it, but there may not be. However, there
Speaker 1: may be the data out there that exists across multiple
Speaker 1: web pages and multiple places that could answer that question
Speaker 1: for you, but there's no easy way for the average
Speaker 1: person to be able to collect and coalate all that data,
Speaker 1: analyze it and get to a meaningful answer, especially not quickly,
Speaker 1: because if you wanted to go through the entire Internet
Speaker 1: to try to find that information, it would take you
Speaker 1: a minute. Yeah, it would take quite some time. So
Speaker 1: what we wanted to do is tough. Sometimes well, again,
Speaker 1: depending upon what it is you're looking for, right, because
Speaker 1: in some cases you may have very little information and
Speaker 1: it may take you some time just to make sure
Speaker 1: that the information that you do have is worthy of consideration.
Speaker 1: Or you may have the opposite problem. Let's say you
Speaker 1: want to look at anything that has to do with
Speaker 1: about cats, good grief, You're gonna have so much information
Speaker 1: on the inner and on the web that relates to
Speaker 1: cats that finding the you know, separating that the signal
Speaker 1: from the noise would take you a really long time.
Speaker 1: So and uh, this problem is already has a solution,
Speaker 1: and that is why we are today talking about web crawlers. Yeah.
Speaker 1: And web crawlers are something that have been around for
Speaker 1: about as long as the web has been around, because
Speaker 1: people realized early on that in order to make the
Speaker 1: web really user friendly, especially once it grew beyond a
Speaker 1: collection of you know, three computers, right, Yeah, three computers
Speaker 1: with twelve web pages altogether, Once you get past all
Speaker 1: of that and you get to a point where it
Speaker 1: really is growing rapidly. You need a way to navigate
Speaker 1: through the web and find the stuff you're interested in.
Speaker 1: You need an index. Yeah, you have to have that
Speaker 1: index because otherwise the only other option you have is
Speaker 1: to know the address of a particular web page and
Speaker 1: then to just follow whatever links that web page have
Speaker 1: is to have, and then once you hit a dead end,
Speaker 1: you've got to backtrack. And you know, it's kind of
Speaker 1: like a choose your own adventure book, And it's a
Speaker 1: choose your own adventure book that's that isn't even connected
Speaker 1: to all the pages that you need. Right, So indexing
Speaker 1: is a way of creating a means to find web
Speaker 1: pages about any given keyword. Right, And again, this is
Speaker 1: a big, big job. You can't expect this to be
Speaker 1: something that only humans are doing under human power. It
Speaker 1: would take way too long and it would be exhausting.
Speaker 1: So there have to be automated ways to index web pages.
Speaker 1: Well yeah, I mean, just consider the ridiculousness of the alternative.
Speaker 1: So let's say you are searching for a term and
Speaker 1: that term is, I don't know, lobster baseball. Somewhere out there,
Speaker 1: there might be a page about lobster baseball, but it
Speaker 1: would not be a good way to find it. To say, well,
Speaker 1: I'm going to ping every web server in the world
Speaker 1: and see if it's offering any public pages that say
Speaker 1: lobster baseball on them. Yeah, that would not especially you know,
Speaker 1: as the Web grows and gets larger and larger and larger,
Speaker 1: that task becomes impossible. It would just it would take
Speaker 1: your computer longer than your lifespan to complete the job,
Speaker 1: especially considering that, as we mentioned before, the web is
Speaker 1: constantly changing, so we would have new web servers joining
Speaker 1: while you're still doing this pinging operation, which means you
Speaker 1: just have you know, you've added more that you have
Speaker 1: to ping before you're done. You never finish. So what's
Speaker 1: the solution, Well, web crawlers would be would be the solution, Joe.
Speaker 1: Web crawlers and search engines are our favorite things here
Speaker 1: at Health Tough Works. I mean, if if it weren't
Speaker 1: for them, our jobs would be significantly more difficult. So, uh,
Speaker 1: let's say that you've got all right, So to break
Speaker 1: it down, we've got web servers that have web pages
Speaker 1: on them, right, we have it's a computer somewhere out there.
Speaker 1: It's got a public facing document that it will show
Speaker 1: you if you ask for right, and your browser is
Speaker 1: the way that you ask for it right, So your
Speaker 1: browser is your conduit to getting the information that's stored
Speaker 1: on other computers that maybe on completely different networks, on
Speaker 1: another on another part of the world even and the
Speaker 1: fact that you have a browser that is what allows
Speaker 1: you to have the access to that document that exists
Speaker 1: on that other page. But those servers can have really
Speaker 1: funky names. Um, the web pages may not have a
Speaker 1: title that is is identical to what it is you're
Speaker 1: looking for, but the information may be in that page.
Speaker 1: Sure for to to use my prior example, dinosaur comics
Speaker 1: dot com used to be known only as quantz dot com.
Speaker 1: Perfect with the QW the way that you sometimes spell
Speaker 1: words much yes, exactly the way the way words are
Speaker 1: never spelled in English. Uh yeah, I I My example
Speaker 1: of my notes I wrote is that let's say that
Speaker 1: you're looking for funny cat memes. The funniest memes happen
Speaker 1: to be on a page that has the title things.
Speaker 1: FDR definitely didn't say, Well, the title of the page
Speaker 1: wouldn't tell you that there are cat memes on there.
Speaker 1: You would need something to have searched that page to
Speaker 1: understand what actually appears on that page, the context within
Speaker 1: which it appears, and to be able to serve that
Speaker 1: up to you. And that's really where the crawlers come in.
Speaker 1: They they build out these indexes of words and where
Speaker 1: to find those words on the web like uh, they
Speaker 1: use lots of They use well, actually pretty simple software.
Speaker 1: They are often referred to as either robots or spiders,
Speaker 1: and they're called spiders because they crawl the web. That
Speaker 1: is good. Yeah, alright, So here's where we mentioned that
Speaker 1: most of these terms. So wait, are we the flies? Good?
Speaker 1: Good question? I mean, I think the uh cams, I'm
Speaker 1: not so. So here's where we mentioned that a lot
Speaker 1: of these terms were all invented around the same time.
Speaker 1: And boy, when we when we go with a metaphor,
Speaker 1: we just go whole spider. So um so all right,
Speaker 1: So spiders typically start by traveling to web servers that
Speaker 1: have lots of traffic, the ones that are the most popular,
Speaker 1: and they explore the most popular web pages and start
Speaker 1: to build up the index of words of those web pages.
Speaker 1: Then all the links that are on those popular web pages,
Speaker 1: the spiders start to follow those links and index those
Speaker 1: pages in turn, and then do the same thing over
Speaker 1: and over and over again, so they just you know,
Speaker 1: it's it is like a spider web or a crack
Speaker 1: in the glass where you see its splintering over and
Speaker 1: over while the glass shatters. The same sort of thing.
Speaker 1: It's following all those potential pathways, and they can hold
Speaker 1: hundreds of pages open at a time. We're talking like
Speaker 1: three hundred pages a second. So yes, more than Google
Speaker 1: Chrome will allow me to have opened before my computer
Speaker 1: says listen, I give up. Uh. So, depending on the
Speaker 1: crawler of the spiders will index these pages based upon
Speaker 1: which words appear in the page and where those words
Speaker 1: actually appear in that page, like in what context. So
Speaker 1: you may remember in the early days of the web,
Speaker 1: before web search engines got really sophisticated, that some people
Speaker 1: would make a web page and then just litter the
Speaker 1: bottom of the page with tons of random words that
Speaker 1: we're doing really well in search, mostly because they had
Speaker 1: ads served on the page of the type that they
Speaker 1: got money from per page view. So I might have
Speaker 1: a talk Brittney Spear, right, Yeah, it would often be
Speaker 1: celebrity rumors and gossip that kind of stuff, and just
Speaker 1: random recipe. Yeah, yeah, it'd be weird stuff, like totally
Speaker 1: some of it would be disturbing to read. You're like, wow,
Speaker 1: I can't believe that that. To know that this particular
Speaker 1: turn m is a very popular search term is disturbing
Speaker 1: others would Yeah. Yeah, I'm more of an A C,
Speaker 1: D C kind of guy myself, so I'm I'm with
Speaker 1: you there. So anyway, Uh, you know, this was a
Speaker 1: way of fooling search engines into into indexing that page
Speaker 1: on multiple indexes so that it would appear no matter
Speaker 1: what search you put in, your page would pop up.
Speaker 1: You as a as saying, if you're assuming you are
Speaker 1: the one who are administering this web page, you have
Speaker 1: no ethics, Like, you don't care if people come to
Speaker 1: your page and are completely disappointed because it has nothing
Speaker 1: to do with the search term they put in there.
Speaker 1: You just want to get those sweet sweet clicks. You
Speaker 1: just need the page views because you need to pay
Speaker 1: the bills, right, So, uh, search engines and spiders got
Speaker 1: more sophisticated, so they were able to look for the
Speaker 1: placement of words where it fell in the page, whether
Speaker 1: or not it appeared more than once within a page,
Speaker 1: to understand if a page really was about that particular
Speaker 1: search term, or if it was just one of those
Speaker 1: things where the word happened to appear once, it may
Speaker 1: be a saying or a quote that has very little
Speaker 1: to do with the actual substance of the rest of
Speaker 1: the page. You know, this would help the search engine
Speaker 1: rank the page in search right. Right, So, the final
Speaker 1: product of of these spiders doing this indexing is called
Speaker 1: a crawl and and it's essentially a lightweight copy of
Speaker 1: the Worldwide Web that's built to be much more easily
Speaker 1: searched than the whole web itself. Uh and and a
Speaker 1: crawl usually consists, therefore, of this huge cash of data
Speaker 1: about the web, including like the text of each page
Speaker 1: it's spiders encountered, the code that constructed those pages h T, M,
Speaker 1: L or et cetera, uh, and a certain amount of
Speaker 1: metadata um you know, certainly the pages r L and
Speaker 1: maybe the tags. As we discussed, that's not always as
Speaker 1: useful as it used to be due to uh scammy stuff.
Speaker 1: But yeah, so so creating a crawl is a huge
Speaker 1: project in terms of time and computer equipment and drive
Speaker 1: space and spider programming and just sheer Internet bandwidth. Right. So,
Speaker 1: for the longest time, this is something that was really
Speaker 1: only accessible by big corporations like Google or Microsoft, Yahoo that. Yeah,
Speaker 1: we're talking huge companies that have the computer power and
Speaker 1: the bandwidth to pull this sort of stuff off on
Speaker 1: a on a regular basis. And while those are incredibly
Speaker 1: useful for us as consumers, if we are looking for
Speaker 1: a specific piece of information that happens to live somewhere
Speaker 1: on the web page, if we want to do more
Speaker 1: of a big data analysis something where we need to
Speaker 1: collate the information across multiple, perhaps hundreds or thousands of
Speaker 1: web pages, it's not easy, right. We don't have those
Speaker 1: tools for the most part, right right, Because when you
Speaker 1: go to Google, you can't access that level of information.
Speaker 1: Yeah you can, you can ask, uh, you know what
Speaker 1: Hugh Grant was doing last week? Right? Yeah, you can
Speaker 1: get the most popular or the highest ranking search results,
Speaker 1: which could give you at least some useful information. But again,
Speaker 1: if you want to do a wide spread study on
Speaker 1: a specific thing, unless someone's already done it, in which
Speaker 1: case you may just need to replicate their their study
Speaker 1: to make sure that it was correct. Um, you're you're
Speaker 1: kind of out of luck. So where can someone turn.
Speaker 1: Let's say that it's a researcher who's working on something
Speaker 1: and they don't work for one of these big companies.
Speaker 1: Where can they turn to leverage the incredible asset that
Speaker 1: is the World Wide Web? One? Gil Lbez started up
Speaker 1: a nonprofit corporation called the Common Crawl Foundation and it
Speaker 1: has been since then working on providing public, publicly accessible,
Speaker 1: free crawls to anyone who wants to use them. And
Speaker 1: uh ls is a really interesting dude. A little bit
Speaker 1: of background on him. Um. He co founded a company
Speaker 1: back in the nineties called Applied Semantics, which created software
Speaker 1: that matched ads to web pages like contextually and automatically.
Speaker 1: Oh we know a little bit about that. Yeah, yeah,
Speaker 1: And this prompted Google to acquire them in two thousand
Speaker 1: three for like a hundred two million bucks, So not
Speaker 1: doing too bad for himself. Also, that's that's essentially the
Speaker 1: reason why Google AdSense exists. That the programming that led
Speaker 1: to Google AdSense so very very practical application of that
Speaker 1: contextual understanding, right right. Um. In two thousand eight, interestingly,
Speaker 1: and kind of a side note, he founded a company
Speaker 1: called Factual, which seeks to gather and analyze global location
Speaker 1: data in order to create a repository of really high quality,
Speaker 1: easily accessible location data that's uh factual um and and
Speaker 1: companies like Bang and Samsung and Yelp all use factuals
Speaker 1: to construct local maps and personalized advertising for mobile consumers.
Speaker 1: So uh so pretty nifty stuff. And what I am
Speaker 1: saying is that elbas is passionate about and experienced with
Speaker 1: big data, right, and we've talked about it before on
Speaker 1: this podcast. That big data is. You know, it sounds
Speaker 1: like one of those just buzz industry terms, but it
Speaker 1: really is one of those things that holds a huge
Speaker 1: amount of potential to affect our lives in different ways,
Speaker 1: assuming that we've developed the right means to analyze that
Speaker 1: massive amount of information that's had there to collect it
Speaker 1: in the first place. Sure, and once you have a
Speaker 1: way of processing, of collecting and being able to access
Speaker 1: and process vast amounts of data, you can do a
Speaker 1: lot of amazing things like big data. And the ability
Speaker 1: to process it might be the key to say, for example,
Speaker 1: computational modeling that predicts complex social phenomenon by analyzing big
Speaker 1: data coming from social media and from news and from
Speaker 1: weather and from all kinds of sources, it can really
Speaker 1: mean that we are able to actually see elements of
Speaker 1: order and what previously appeared to be a truly chaotic system,
Speaker 1: which is kind of exciting. Sure. And then on the
Speaker 1: other hand, a lot of people think that big data
Speaker 1: could be one of the ways that we finally achieve
Speaker 1: that next level of artificial intelligence by having machines sort
Speaker 1: of plumb the depths of this data with self teaching
Speaker 1: and self learning mechanisms. Right, well, let's get back to
Speaker 1: the common crawl okay, Right, So the Foundation began compiling
Speaker 1: crawls back in two thousand eight. The most recent one
Speaker 1: that they released as of this podcast at the end
Speaker 1: of May, was from April. It was some a hundred
Speaker 1: and sixty eight terabits in size bytes in size huge.
Speaker 1: That's big, uh, and contains some two point one billion
Speaker 1: web pages. That's not that many, really, I wrote I
Speaker 1: wrote forty seven million web pages before breakfast. No I
Speaker 1: did not. I'm just kidding. No, that's a lot. Yeah, yeah, yeah,
Speaker 1: it's it's a it's a bunch um uh. But so
Speaker 1: they're they're continuously indexing and releasing new crawls, right, It's
Speaker 1: not like it's here's the Internet and now we're done. Yeah. Yeah,
Speaker 1: they've been releasing a new crawl every month since July.
Speaker 1: That's I mean, that's incredible you think about the amount
Speaker 1: of work that does. It also means that you have
Speaker 1: like a a time a timestamp, like photograph of what
Speaker 1: the web was at that moment from these crawls. Yeah. Yeah,
Speaker 1: I hadn't thought about it quite that way before, but yeah,
Speaker 1: that's it's kind of you know, things that may not
Speaker 1: exist from one month to the next, you could actually
Speaker 1: see and watch those trends. Yeah, it's fascinating. Yeah. I'd
Speaker 1: say one of the main ways that I often encounter
Speaker 1: web archives is when I'm trying to find evidence of
Speaker 1: something somebody did in the past that they wanted expunge. Right,
Speaker 1: This makes sound like I'm some kind of detective, not
Speaker 1: like I'm trying to find but you know what I'm
Speaker 1: talking about. No, I know, I will post something and
Speaker 1: then they'll be like, oh wait a minute, No, that
Speaker 1: was a bad idea. I know if you try to
Speaker 1: delete it. I know, if you use archive dot org,
Speaker 1: you can find one of the web pages I built
Speaker 1: way back when, and I never want anyone to ever
Speaker 1: see it because it was that bad. But they will
Speaker 1: forever be able to Yeah, you shouldn't talk about it
Speaker 1: on podcasts. Pretty sure they're not going to be able
Speaker 1: to find it. Tens of thousands of people. Let me
Speaker 1: guess you had some You had a bunch of rage
Speaker 1: against the machine lyrics, and it auto played midias. They're
Speaker 1: so close closest, No, you're really far away. I wrote,
Speaker 1: I made some web pages for I particular company I
Speaker 1: worked for, which is unless you know the company I
Speaker 1: worked for that I'm specifically referring to. That's why you're
Speaker 1: never going to find that particular web page, and you shouldn't.
Speaker 1: It was terrible. It was. It was about, we're going
Speaker 1: to find these Go look on his LinkedIn profile. We
Speaker 1: can figure out what company was. I already found some
Speaker 1: lobster baseball stuff, so yeah we can. That comes from
Speaker 1: you almost Got a spit take on you almost got
Speaker 1: and uh yeah, that comes from a pre episode conversation. No,
Speaker 1: that was actually in the episode, wasn't it. I can't
Speaker 1: keep track anymore, all this dungeons and dragons saving throw
Speaker 1: talk we had. OK, hold on, I've got a question,
Speaker 1: So hold on. If the Common Crawl is trying to
Speaker 1: preserve and make accessible continuously updated snapshots of the web
Speaker 1: weird on Earth? Are they're going to like store that
Speaker 1: and make it available, right, because it's not going to
Speaker 1: fit on like a thumb dry So where I think
Speaker 1: it's also funny that that sort of becomes part of
Speaker 1: the web, Like the web now incorporates a snapshot of
Speaker 1: the previous web, and so it just gets that much larger.
Speaker 1: So yeah, where's the stuff living? It is all living
Speaker 1: on Amazon's web services. Uh, specifically, it's it's stored in
Speaker 1: Amazon Simple Storage Service or S three as it is
Speaker 1: sometimes known, and you can analyze it via Amazon's a
Speaker 1: Last Compute Cloud or e C two. And this is
Speaker 1: so cool guys, because because okay, given Amazon's web service scope,
Speaker 1: it means that practically anyone in the whole world can
Speaker 1: download entire crawls for free, or can if they don't
Speaker 1: want to, you know, use hundred sixty terabytes of space.
Speaker 1: They can they can just use e C two to
Speaker 1: really easily run simple data crunches for like an hourly charge,
Speaker 1: like anywhere from a few bucks to maybe fifty dollars
Speaker 1: for pretty simple computations. So and of course, more savvy
Speaker 1: users can write their own code to investigate stuff with.
Speaker 1: But but but yeah, for for the common user. This
Speaker 1: is revolutionary. Yeah. To me, this is uh. I mean,
Speaker 1: obviously I would I would recommend doing the approach where
Speaker 1: you're you're searching on Amazon stuff. I can't imagine the
Speaker 1: phone call you would get from your internet service provider.
Speaker 1: I see you're trying to download a hundred and sixty
Speaker 1: eight terrified it's worth of data over our lines. You
Speaker 1: have gone significantly over your bandwidth cav. Yeah. And they
Speaker 1: can do all of this because Amazon has specifically chosen
Speaker 1: to wave their storage fees for for this and a
Speaker 1: handful of other things that they consider to be of
Speaker 1: wide public interest, like like like weather in census data.
Speaker 1: And it's that's incredible, right, I mean, because you are
Speaker 1: talking about a significantly huge amount of data, So to say,
Speaker 1: you know this, this is so important and so potentially
Speaker 1: beneficial to mankind that we're not going to end up,
Speaker 1: you know, charging these the storage fees for it. That's
Speaker 1: that's great, Yeah, very very encouraging. So when we get
Speaker 1: down to all, right, well, what can you actually use
Speaker 1: all that data for? Well, just think about the stuff
Speaker 1: that's on the web and pretty much anything you could
Speaker 1: think of that you know, you have a question you
Speaker 1: might have that could not be answered through a simple
Speaker 1: search query, you could potentially answer by leveraging this information. So, uh,
Speaker 1: you know, we're talking about everything from lots of stuff
Speaker 1: that deals with AI. Actually, like you were mentioning earlier, Joe,
Speaker 1: stuff like developing better natural language algorithms so that the
Speaker 1: the machines of the future can understand a wider variety
Speaker 1: of inputs and make meaningful connections between that input and
Speaker 1: the desired output. So in other words, I could talk
Speaker 1: to my computer or my phone as if it were
Speaker 1: a person, and no matter how I might word things
Speaker 1: in my own quirky way, the machine understands what I mean.
Speaker 1: So it's not it's not responding to what I say
Speaker 1: versus it's more respond to what I mean, which would
Speaker 1: be awesome. Uh. Also stuff like speech recognition, UM emerging
Speaker 1: global trends. We mentioned that. You know, let's say that
Speaker 1: you wanted to track the outbreak of a disease and
Speaker 1: to try and get at you know, where did this start?
Speaker 1: How can we prevent this from happening again? That kind
Speaker 1: of thing that can be really useful. And sometimes you're
Speaker 1: tracking this through uh, not like official documents, but through
Speaker 1: you know, people on Twitter saying, oh, I have the
Speaker 1: flu right, Yeah, it might be it might be social media,
Speaker 1: it could be news reports. I mean, it could be
Speaker 1: all these different, completely separate pieces that would be way
Speaker 1: too hard for you to put together on your own. Sure,
Speaker 1: we did a whole episode once about financial purposes for
Speaker 1: big data, so play in the stock markets and all that. Yeah, yeah, exactly.
Speaker 1: So this is this is really important stuff, and we're
Speaker 1: gonna see a lot more of examples of people leveraging
Speaker 1: big data, especially now that it's outside the realm of
Speaker 1: just mega corporations right where we can see people are
Speaker 1: researchers who have interest in all sorts of different fields
Speaker 1: taking advantage of this massive amount of information that we
Speaker 1: continue to accumulate day after day. And uh, that's really exciting.
Speaker 1: It makes me think of having access to the best
Speaker 1: research librarians in the world, all boiled into the largest
Speaker 1: library you can imagine. That's essentially what we're talking about here.
Speaker 1: So very exciting, and the common crawl is pretty inspirational.
Speaker 1: It's one of those things where you realize it took
Speaker 1: a lot of determination to make that become a reality
Speaker 1: and the potential benefit, and it also is incredibly forward
Speaker 1: thinking to be all the way back in two thousand
Speaker 1: eight and putting the US together, And that was before
Speaker 1: we had really developed sophisticated tools that could leverage it properly.
Speaker 1: Now we're getting to see that as big data has
Speaker 1: become an industry unto itself. I mean, it's really exciting now.
Speaker 1: So thank goodness that the idea was was implemented long enough,
Speaker 1: long ago enough for it to actually have uh you know,
Speaker 1: uh an established presence and now we can really see
Speaker 1: how we can take advantage of it. So Las is
Speaker 1: such a such a fascinating dude. I have found so
Speaker 1: many interesting interviews and stuff with him. I think that
Speaker 1: we should maybe maybe not on the show, but maybe
Speaker 1: if you'd want to do a text text episode kind
Speaker 1: of focus on them, that would be kind of cool. Yeah.
Speaker 1: I love to do episodes where we are able to
Speaker 1: look at influential figures in technology on that show, So
Speaker 1: that would be fantastic. I will add that to the list.
Speaker 1: Uh So, the Common Crawl is really an interesting project.
Speaker 1: If you have not looked into it, go check it out, um,
Speaker 1: you know, because it may be one of those things
Speaker 1: that could come in handy if you're working on a
Speaker 1: research project. If you are just curious about how big
Speaker 1: data is gonna continue to have a huge impact in
Speaker 1: our lives, go go seek that out. Yeah, it's a
Speaker 1: you can find them at a common crawl dot org.
Speaker 1: And if you if you're into making donations to nonprofit
Speaker 1: organizations that are tax deductible, you can do that thing too.
Speaker 1: That's pretty cool, all right, guys. That wraps up this discussion.
Speaker 1: If you have any suggestions for future topics for forward Thinking,
Speaker 1: you should let us know. Send us an email that
Speaker 1: addresses FW thinking at how stuff Works dot com, or
Speaker 1: drop us a line on Facebook, Twitter or Google Plus.
Speaker 1: At Twitter and Google Plus, we are f w thinking.
Speaker 1: Just search fw thinking in Facebook and we will pop
Speaker 1: right up. Leave us a message. We read all of them.
Speaker 1: We look forward to hearing from you, and you'll hear
Speaker 1: from us again really soon. For more on this topic
Speaker 1: and the future of technology, visit forward thinking dot com,
Speaker 1: brought to you by Toyota. Let's go places,