Speaker 1: Welcome to tech Stuff, a production from iHeartRadio. Hey there,
Speaker 1: and welcome to tech Stuff. I'm your host, Jonathan Strickland.
Speaker 1: I'm an executive producer with iHeart Podcasts and How the
Speaker 1: tech are You. So early on in the days of
Speaker 1: tech stuff, back when I was still a staff writer
Speaker 1: for a little website called HowStuffWorks dot com, my boss
Speaker 1: Connell Burn, who is now a big shot over here
Speaker 1: at iHeart, he came over to me with an assignment.
Speaker 1: He wanted me to do some articles in some episodes
Speaker 1: about this buzzword concept called big data. And I had
Speaker 1: heard the term big data, and obviously there's a pretty
Speaker 1: darn good hint I want. Big data is all about
Speaker 1: just in the nature of the name itself, but beyond that,
Speaker 1: I didn't really know much, so I jumped to it.
Speaker 1: And the interesting thing is that since that time, the
Speaker 1: discipline of big data has evolved significantly. When I was
Speaker 1: first working on my articles and episodes, we were mostly
Speaker 1: talking about how technological tools made it easier to collect
Speaker 1: vast amounts of information very quickly and to store it.
Speaker 1: But we didn't necessarily have equally sufficient tools to do
Speaker 1: anything useful with all that information. We have or at
Speaker 1: least those tools weren't widely known and understood beyond a
Speaker 1: certain circle of computer scientists. Flash forward a few years,
Speaker 1: and we'd see companies developing new methods to analyze large
Speaker 1: chunks of data. Oh, by the way, I do the
Speaker 1: weird data data thing, and there's no rhyme or reason
Speaker 1: to it. I don't even know which one I'm going
Speaker 1: to say before I say it, so I apologize because
Speaker 1: I know it's irritating. It irritates me too, Anyway. Other
Speaker 1: companies sprung up with products that were meant to help
Speaker 1: with data analysis, and it seemed like we were going
Speaker 1: from an era of well, now I have all this information,
Speaker 1: what do I do now, to an era of I
Speaker 1: have discovered cryptic secrets that were hiding in plain sight
Speaker 1: thanks to data analysis, and that somehow it all happened overnight.
Speaker 1: So today I thought we would actually look back over
Speaker 1: the history of the big data concept, how various systems
Speaker 1: have made it possible to sift through seemingly meaningless information
Speaker 1: in order to find nuggets of wisdom, and why we
Speaker 1: might not always be able to trust the answers that
Speaker 1: we discover. So the history of big data starts in
Speaker 1: the twenty tens, or maybe it starts in two thousand
Speaker 1: and five, or maybe in nineteen ninety, or maybe the
Speaker 1: sixteen hundreds, or maybe nearly twenty thousand years ago. You
Speaker 1: might have already picked up on the fact that folks
Speaker 1: don't quite agree on where we should start when talking
Speaker 1: about big data. But that makes sense. Ever since humans
Speaker 1: have started to write stuff down, we've been pretty darn
Speaker 1: invested in the collection and then the classification of information.
Speaker 1: Whether it's to figure out the best time to sew
Speaker 1: or harvest crops, or keep track of how much we've
Speaker 1: traded with that other band of neair dwells who live
Speaker 1: on the other side of the holler, or we just
Speaker 1: want to make a record of how great it was
Speaker 1: that we kicked the butt of that mastodon, real good.
Speaker 1: We've been really obsessed with data and collection and retrieval. Now,
Speaker 1: this obsession also means that we had to come up
Speaker 1: with various ways to store and analyze this information. Raw
Speaker 1: information doesn't do anyone much good, and so throughout antiquity
Speaker 1: we came up with means of recording and storing and
Speaker 1: making use of information. Not only did hardworking humans create
Speaker 1: libraries where we could gather all this knowledge and then
Speaker 1: lose some of those libraries along the way due to
Speaker 1: the fact that we humans also are pretty stupid and
Speaker 1: we end up having disputes that involve burning each other's
Speaker 1: stuff to the ground. Yeah, I'm still bitter about certain
Speaker 1: libraries being destroyed over in antiquity, but it means that
Speaker 1: we also had to come up with methodologies to categorize
Speaker 1: and classify information. Otherwise you may as well just have
Speaker 1: a big old pile of scrolls or books or whatever,
Speaker 1: and then people just you know, have to sort through
Speaker 1: them and see if they can find anything, which actually
Speaker 1: sparks two different memories in my head. One is that
Speaker 1: there used to be a used bookstore I would go
Speaker 1: to here in Atlanta, and often the used bookstore was
Speaker 1: completely unorganized, right, Like you literally could go through a
Speaker 1: bookshelf and it's just going to be books that are
Speaker 1: more or less the same size, but otherwise there's no
Speaker 1: rhyme or reason as to why they were put there,
Speaker 1: and it was like you were on a treasure hunt.
Speaker 1: And then I'm also reminded of a naval museum in Appalachicola, Florida,
Speaker 1: which is on the Panhandle. I went to this little,
Speaker 1: you know, naval museum, like a ship museum, and I
Speaker 1: reminded that all the exhibits were kind of in a
Speaker 1: pile on the floor, and you would literally pick things
Speaker 1: up and look at them. And that's kind of what
Speaker 1: it would be like if we didn't have these means
Speaker 1: of classification. Once you get to a certain size, like
Speaker 1: that little museum in Appalachic Cola wasn't so big as
Speaker 1: to be a problem. But if you're talking about a
Speaker 1: big library, obviously, if you want anything useful, you got
Speaker 1: to come up with a way of classifying all this.
Speaker 1: To that end, ancient folks began to develop a science
Speaker 1: called taxonomy. And this isn't when you stuff dead animals
Speaker 1: so that they look like they might still sort of
Speaker 1: be alive. That's taxon dermy. No. Taxonomy is the science
Speaker 1: of classification, and it's perhaps best known in the field
Speaker 1: of biology, thanks in large part to a Swedish scientist
Speaker 1: from the eighteenth century named Carl Linnaeus. But there are
Speaker 1: many applications of taxonomy that extend beyond biology. It's just
Speaker 1: the biological taxonomy is the one that I think most
Speaker 1: of us are familiar with because most of us were
Speaker 1: taught it when we were going through basic biology science.
Speaker 1: But the ancient Greeks made some early progress on developing
Speaker 1: systems of classification, and obviously, within modern library science, taxonomy
Speaker 1: is an important discipline, though oddly enough, you could say
Speaker 1: taxonomy in library science is distinct from classification. When I
Speaker 1: was looking this up, I found resources for library science
Speaker 1: that made these two distinct disciplines. Classification was one in
Speaker 1: taxonomy was another. Now. This is because there are various
Speaker 1: methods of classification in library science. The one that I
Speaker 1: was most familiar with when I was growing up was
Speaker 1: the Dewey decimal system, which I don't even think is
Speaker 1: the dominant form now, but it was when I was
Speaker 1: growing up. And it's meant to connect a specific work
Speaker 1: to a specific physical location in a library for the
Speaker 1: purposes of, you know, checking down the book, right. But
Speaker 1: taxonomy in library science tends to be more towards metadata
Speaker 1: or data about data. In fact, metadata plays a huge
Speaker 1: part in big data. Oh man, I did it both
Speaker 1: ways in one sentence. I feel awful. Anyway, the information
Speaker 1: about information can be as useful as the information itself.
Speaker 1: In some cases. I have often talked about this with
Speaker 1: personal information about how info about info can give you
Speaker 1: a lot of insight into a person. Maybe you don't
Speaker 1: have a person's name, but you have a couple of
Speaker 1: different data points about that person. In some cases, you
Speaker 1: can actually narrow down the identity of the person you're
Speaker 1: thinking of just by looking at this metadata. You don't
Speaker 1: even have to see the information about them, which shows
Speaker 1: you how powerful metadata can be. So you start to
Speaker 1: see a cascading effect here where you slowly realize that
Speaker 1: you actually have access to even more information than you
Speaker 1: first anticipated because you also have information about that information.
Speaker 1: It gets pretty wild. Another important development in the history
Speaker 1: of big data is the creation of statistics. So let's
Speaker 1: give the Merriam Webster definition of statistics. Shall we just
Speaker 1: have a baseline. It is quote a branch of mathematics
Speaker 1: dealing with the collection, analysis, interpretation, and presentation of masses
Speaker 1: of numerical data. End quote. Now. One famous early example
Speaker 1: of statistics comes to us courtesy of a fellow named
Speaker 1: John Grant Graunt. He was looking at mortality rates in London,
Speaker 1: and that gave him a lot more information and helped
Speaker 1: him analyze the course of the plague. For example, he
Speaker 1: could see when the plague was spiking or receding. Pretty
Speaker 1: cheerful stuff, right, But he also used this information, the
Speaker 1: mortality information, to start drawing some conclusions about the population
Speaker 1: of London as a whole, So counting up everybody, like
Speaker 1: figuring out who lives in London. That would have been
Speaker 1: challenging at the time, to say the least. But Grant
Speaker 1: took information like the number of funerals and then he
Speaker 1: compared it to things like the average family size in
Speaker 1: London to try and make an estimate of London's population.
Speaker 1: So it gave him kind of a working figure that
Speaker 1: was useful for certain applications, specifically government ones. Statistics as
Speaker 1: a branch of mathematics would mature over the following centuries.
Speaker 1: Often it would be the tool that allowed social scientists
Speaker 1: to draw broad conclusions about large populations, but others found
Speaker 1: plenty of alternative applications of statistics. Anyway, the age of
Speaker 1: data analysis was well and truly in swing at this
Speaker 1: point in the late nineteenth century. The United States was
Speaker 1: getting in a bit of a pickle. And I know
Speaker 1: we're making jumps of centuries here, but we need to
Speaker 1: We can't go through every single evolution of data collection
Speaker 1: and data analysis that would be a podcast series all
Speaker 1: in itself. So we're in the late eighteen hundreds and
Speaker 1: the USA US isn't a bit of a problem. The
Speaker 1: country holds a census every ten years, where they're essentially
Speaker 1: gathering information about all the citizens in the United States.
Speaker 1: This is required by the US Constitution, and there are
Speaker 1: several reasons why the Census Bureau holds a census every
Speaker 1: ten years. But one of those reasons is that the
Speaker 1: US House of Representatives its membership depends upon population. So
Speaker 1: the more populous a state is, the more representatives that
Speaker 1: state has in the House of Representatives. So if your
Speaker 1: state has a big population, there are more representatives that
Speaker 1: go to the House. If you have a relatively small population,
Speaker 1: then you have fewer House representatives, right, That's how that works.
Speaker 1: So by eighteen eighty things were getting to a really
Speaker 1: difficult situation. The process of collecting and then analyzing all
Speaker 1: the information was so cumbersome that it would take nearly
Speaker 1: the whole decade just to get to a result, and
Speaker 1: that means by the time you're drawing conclusions, it's actually
Speaker 1: time for you to administer the next census. In fact,
Speaker 1: they projected that in eighteen ninety working on the same
Speaker 1: process that they were dependent upon previously. It would take
Speaker 1: a whole decade, so literally you'd be holding your next
Speaker 1: census while you were just getting your information from the
Speaker 1: last one. So the Census Bureau needed a way to
Speaker 1: collect and analyze this information in a much more efficient process.
Speaker 1: They tapped a man named Herman Holleeth to accomplish this.
Speaker 1: So Holloweth took a punch card system that had been
Speaker 1: used in weaving, weaving with mechanical looms. I've talked about
Speaker 1: this in the past with the history of punch cards.
Speaker 1: In fact, this also gets into perhaps a somewhat apocryphal
Speaker 1: story of where the word sabotage comes from, but that's
Speaker 1: for another time. So he took this punch card system
Speaker 1: that had been used to set weaving patterns with mechanical looms,
Speaker 1: and then he adapted that to serve as a way
Speaker 1: to record information so that you could feed the card
Speaker 1: to a tabulation machine which then could actually tabulate the results.
Speaker 1: And his invention meant that ten years of labor done
Speaker 1: by clerks who are working at desks would actually boil
Speaker 1: down to about three months of labor using the tabulation machine. Obviously,
Speaker 1: that was a huge improvement. Hollerith formed a company that
Speaker 1: over time would evolve into one of the most famous
Speaker 1: companies in all the world, Kentucky Fried Chicken. I'm just kidding.
Speaker 1: It wasn't KFC. Instead, it was IBM. That's the company
Speaker 1: that would grow out of Hollowarith's company that he founded
Speaker 1: in the nineteenth century. Anyway, we're not going to spend
Speaker 1: too much time in all these centuries gone by. We're
Speaker 1: actually going to speed things up and get up to
Speaker 1: the twentieth century. But before we do that, let's take
Speaker 1: a quick break to thank our sponsor. We're back, okay,
Speaker 1: So the actual term big data is still waiting for us.
Speaker 1: We're not going to really get to that until we
Speaker 1: hit the late nineteen nineties or so. But there are
Speaker 1: a few things to point out before we get up
Speaker 1: to there. Folks were starting to notice that we were generating, collecting,
Speaker 1: and storing an awful lot of information in the twentieth century,
Speaker 1: and that the rate of data generation was on the rise.
Speaker 1: Not only were we generating a whole bunch of information,
Speaker 1: we were doing it in larger amounts year over year.
Speaker 1: In fact, it was rising much faster than our rate
Speaker 1: of consumption of information, meaning that we were making way
Speaker 1: more data than we were actually able to use. And
Speaker 1: a big thanks goes out to Forbes for an article
Speaker 1: that's titled A very Short History of Big Data by
Speaker 1: Gil Press. A lot of the information that I'm drawing
Speaker 1: upon came from that article. It is fantastic if you
Speaker 1: want to learn more about this. I'm not going to
Speaker 1: cover every element that they do. I mean, that would
Speaker 1: just be me regurgitating their article. You should check it
Speaker 1: out if you're interested in the history of big data.
Speaker 1: We're going to touch on a few of the important points,
Speaker 1: or what I think of as the important points. So
Speaker 1: one of the earliest ones we're going to talk about
Speaker 1: is in nineteen forty four, a librarian named Fremont Writer,
Speaker 1: which is a fantastic name, wrote a work titled The
Speaker 1: Scholar and the Future of the Research Library. So Writer
Speaker 1: made an observation that reminds me a lot of Gordon
Speaker 1: Moore's famous Moore's law, except this involves not silicon chips
Speaker 1: but physical libraries. So Writer said that your typical library
Speaker 1: in your typical American university was doubling in size every
Speaker 1: sixteen years. He projected that this would mean that by
Speaker 1: the year twenty forty, the library at Yale University would
Speaker 1: be so large as to require a staff of more
Speaker 1: than six thousand people to manage it. Of course, this
Speaker 1: was before we had digital storage and digital filing systems
Speaker 1: that has largely mitigated this particular requirement. We don't need
Speaker 1: the physical space necessarily that we would if everything were
Speaker 1: still in hard copy. But the observation showed that data
Speaker 1: accumulation really had a steep trajectory even back in the
Speaker 1: nineteen forties. Similarly, in the early nineteen sixties, a guy
Speaker 1: named Derek Price published a piece explaining that the number
Speaker 1: of scientific journals and papers was on a path of
Speaker 1: exponential growth. It was doubling every fifteen years, so similar
Speaker 1: to the rate at which university libraries were doubling in
Speaker 1: size now. Part of the reason for this, he said,
Speaker 1: was that scientific discoveries inevitably fuel further discoveries. So you
Speaker 1: find out something new, this inspires other scientists to look
Speaker 1: further into it, they find other new things, and so on.
Speaker 1: In nineteen sixty five, the United States government needed to
Speaker 1: build a place that would store records, including things like
Speaker 1: tax returns and fingerprint sets, and so the plan was
Speaker 1: to take the paper records and then transfer them to
Speaker 1: magnetic tape, and then to store that magnetic tape in
Speaker 1: this so called data center. This project fell through, however,
Speaker 1: because the public got nervous. They felt squiky about this
Speaker 1: idea of the government hoarding vast amounts of information about
Speaker 1: its citizens. They did not fully trust the government. So
Speaker 1: you understand like they're thinking, I don't really feel comfortable
Speaker 1: with you just gathering all this information about us. It
Speaker 1: feels kind of oppressive. Now, what's funny to me is
Speaker 1: that today the average person is more than willing to
Speaker 1: let companies do this to them without even protesting it.
Speaker 1: Because that's how all the online social network companies work, right,
Speaker 1: They work on the basis of gathering information about us
Speaker 1: and then peddling that or or hoarding it, however you
Speaker 1: might think of it. And it's very similar to what
Speaker 1: was happening in the nineteen sixties. And back then we
Speaker 1: were like, no, that's not cool, and now we're like,
Speaker 1: that's just how it works. It's wild to me. Anyway,
Speaker 1: I'm going to skip ahead a little bit to the
Speaker 1: nineteen eighties. There was a lecturer, I a Tjomslend, and
Speaker 1: I know I butchered his name. I apologize anyway. He
Speaker 1: gave a lecture at the IE or IE Symposium in
Speaker 1: which he posits that one reason all this information is
Speaker 1: piling up is that we don't really have a good
Speaker 1: way to determine which information is relevant and which information
Speaker 1: is not. And we can make that determination, but it
Speaker 1: requires work, and meanwhile, we're still accumulating more information. So
Speaker 1: it's the kind of work where you're never done, and
Speaker 1: it feels like you're never making any progress. So most
Speaker 1: of us never bother to do it at all. And
Speaker 1: if our ability to store data is sufficient, in other words,
Speaker 1: if we have ways of storing the information, then we
Speaker 1: have even less incentive to make any determination about the data. Right, Like,
Speaker 1: if we've got plenty of storage, well, let's just go
Speaker 1: ahead and keep the information. There's no reason to have
Speaker 1: to worry about it whether it's useful or not. We
Speaker 1: should keep it because it's better for us to keep
Speaker 1: useless information without needing it, rather than accidentally deleting something
Speaker 1: that turned out to be important. Right, And this kind
Speaker 1: of makes sense. I mean, I'm sure a lot of
Speaker 1: you out there can apply that to your lives. I
Speaker 1: certainly can apply it to my life, right Like, I
Speaker 1: have file folders that are full of stuff that I'm
Speaker 1: never going to touch again, but I still feel reluctant
Speaker 1: to delete it just in case I do need to
Speaker 1: touch it again sometime in the future, even though the
Speaker 1: likelihood of that is very low. So that's anecdotal. I
Speaker 1: can't really call that evidence to prove the point, but
Speaker 1: it feels like the point is relevant. So this is
Speaker 1: also how I play a lot of those big open
Speaker 1: world computer RPGs, by the way, things like Skyrim or whatever,
Speaker 1: because I'll just hoarde potions and scrolls and I never
Speaker 1: use them because what if I need it more in
Speaker 1: the future. Balder's Gate three has really done a number
Speaker 1: on me with this. I got a real problem with
Speaker 1: that anyway. The Forbes article details several more entries indicating
Speaker 1: how very smart people were taking note regarding the accumulation
Speaker 1: of information, as well as methods to store the information,
Speaker 1: and increasingly, as time went on, how we can do
Speaker 1: useful things with all this information. So I recommend you
Speaker 1: check out that Forbes article if you want to learn more.
Speaker 1: I think it goes up to about twenty twelve at
Speaker 1: this point, it has been updated numerous times, but obviously
Speaker 1: twenty twelve was quite a long time ago, so it's
Speaker 1: it's not exactly up to present day. But it's still
Speaker 1: a really interesting article that gives lots more details about this.
Speaker 1: But I don't want to just regurgitate the article, so
Speaker 1: we're going to hop on ahead. Now, Folks, in general,
Speaker 1: we're becoming more aware of this information challenge that was growing.
Speaker 1: But where did the term big data actually come from? Well,
Speaker 1: chances are it's sort of rose organically in conversations within
Speaker 1: the computer sector. As you know, hackers and computer scientists
Speaker 1: and programmers and researchers were all wrestling with ways to
Speaker 1: deal with data. Now, by this time, folks had adapted
Speaker 1: an observation made by Cyril Northcote Parkinson to apply to
Speaker 1: computer systems and to information. So Parkinson's original observation was
Speaker 1: that generally speaking, in public administration offices, you know, like
Speaker 1: government offices, work expands to fill the time that was
Speaker 1: allowed for that work. So if you have a project
Speaker 1: that's going to be due in three weeks, but really,
Speaker 1: if you were to be brutally honest, there's only a
Speaker 1: week's worth of work to do for that project. Well,
Speaker 1: that work will almost magically expand so that it actually
Speaker 1: takes three weeks to complete. This gets more nuanced and
Speaker 1: it brings into account elements like bureaucracy. But you get
Speaker 1: the point right that somehow it doesn't matter, you know what,
Speaker 1: who is working the job. It doesn't matter the nature
Speaker 1: of the work. The work will expand to fill the
Speaker 1: amount of time it requires to do that work, which
Speaker 1: meant that if you had said it would take two weeks,
Speaker 1: it would have just expanded to two weeks, not three.
Speaker 1: It's very weird, right Anyway, Folks in the computer biz
Speaker 1: adapted this to say that data will expand to fill
Speaker 1: whatever space you have available for that data. So, in
Speaker 1: other words, you make a bigger storage unit, you're going
Speaker 1: to fill it like that data will just expand to
Speaker 1: fill that even though you thought, oh, I'm future proofing this,
Speaker 1: and again anecdotally, I have observed this in my personal life.
Speaker 1: I remember when hard disk drives first became a thing
Speaker 1: in personal computers, like they were already existed, but personal
Speaker 1: computers didn't have them when they first it came out, right,
Speaker 1: you were using external drives like floppy disks and stuff,
Speaker 1: and I remember whenever there would be a dramatic expansion
Speaker 1: of storage space, and it always seemed to be dramatic, right,
Speaker 1: it always seemed like it had doubled since last time.
Speaker 1: And typically that's how it worked. Anyway, I would walk
Speaker 1: away thinking, Wow, I'm never gonna fill all this space.
Speaker 1: I mean, who even needs that much space? Two hundred
Speaker 1: and fifty six megabytes? Who the heck needs that much space?
Speaker 1: That's way too much. I mean, I'll never fill it up.
Speaker 1: But of course I would prove myself wrong, typically in
Speaker 1: record time. But beyond anecdotes, which again don't really count
Speaker 1: as evidence, the observation really pointed out that we will
Speaker 1: eagerly fill up whatever space we're given. You could argue
Speaker 1: this goes back to our tendency to avoid deleting material
Speaker 1: out of concern that it might one day become useful. Anyway,
Speaker 1: By the mid nineteen nineties, there was a computer scientist
Speaker 1: named John Mashie, and he was giving presentations that related
Speaker 1: to this concept of big data. Now, Mashie has dismissed
Speaker 1: the idea that he personally coined the phrase. At most,
Speaker 1: he says that he popularized the term big data in
Speaker 1: his talks but his point was that he used the
Speaker 1: phrase big data because it was a shit shorthand way
Speaker 1: to give a nod to several related challenges, ranging from
Speaker 1: storage to analysis. So one could argue that Mashie's use
Speaker 1: of the term approached what we mean by big data today,
Speaker 1: but it wasn't one hundred percent the same thing. And
Speaker 1: the earliest use I've seen cited happened sometime around nineteen
Speaker 1: ninety eight. So we know Mashie didn't invent the phrase,
Speaker 1: and we know that partly because researchers found an instance
Speaker 1: that predates his talks by nearly a decade. Steve Lohr
Speaker 1: wrote a piece for The New York Times titled the
Speaker 1: Origins of Big Data, An etymological detective Story. A great,
Speaker 1: great article. By the way, Lore spoke with an associate
Speaker 1: librarian in Yale Law School named Fred Shapiro, and Fred
Speaker 1: Shapiro did some research and uncovered an instance of the
Speaker 1: phrase big data in a nineteen eighty nine article in
Speaker 1: Harper's magazine. The author of that piece was Eric Larson,
Speaker 1: who said, quote, the keepers of big data say they
Speaker 1: do it for the consumer's benefit, but data have a
Speaker 1: way of being used for purposes other than originally intended quote,
Speaker 1: and boy howdy, we have seen that observation play out
Speaker 1: again and again, haven't we. It's remarkable because nineteen eighty
Speaker 1: nine predates the World Wide Web, certainly predates all the
Speaker 1: social networks that we talk about. But Eric Larson's observation
Speaker 1: is just as relevant, if not more relevant, today than
Speaker 1: it was in nineteen eighty nine. Also, incidentally, Eric Larson
Speaker 1: wrote one of my favorite books of all time. It's
Speaker 1: titled The Devil in the White City. Famous book. I'm
Speaker 1: sure a lot of you have already read it, but
Speaker 1: for those who haven't, it's a book that tells two
Speaker 1: somewhat intertwined stories, the eighteen ninety three World's Columbian Exposition
Speaker 1: in Chicago and the tale behind HH Holmes, credited as
Speaker 1: one of America's first serial killers. Now, I originally bought
Speaker 1: the book because I was interested in Holmes's story, but
Speaker 1: I got to be honest, I actually found the chapters
Speaker 1: about the exposition to be far more captivating, and it
Speaker 1: ties up into a lot of the stuff we talk
Speaker 1: about on tech stuff. So it's a great book if
Speaker 1: you're looking for something to read. But now let's get
Speaker 1: back to Big data. So things continue on their inevitable
Speaker 1: path through time. As it goes, time marches on, we
Speaker 1: get up to the two thousands. By now the Internet
Speaker 1: has greatly exacerbated our data creation and accumulation problem. In
Speaker 1: two thousand, Francis Diebold wrote, quote big data refers to
Speaker 1: the explosion in the quantity and sometimes quality of available
Speaker 1: and potentially relevant data, largely the result of recent and
Speaker 1: unprecedented advancements in data recording and storage technology end quote.
Speaker 1: So we're really starting to close in at this point
Speaker 1: on the concept of big data as we understand it today.
Speaker 1: Then we get up to two thousand and five, and
Speaker 1: a couple actually several important things happened that year in
Speaker 1: the realm of big data. We get Tim O'Reilly and
Speaker 1: his media company, fittingly enough called O'Reilly Media, and this
Speaker 1: is the year that he would publish an article titled
Speaker 1: what is web two point zho, a famous or perhaps
Speaker 1: infamous article in tech circles. So the dot com bubble
Speaker 1: had burst several years earlier, around two thousand and two
Speaker 1: thousand and one, and O'Reilly was making observations about the
Speaker 1: qualities that helped the companies that survived that crash versus
Speaker 1: the companies that went under, like what set them apart?
Speaker 1: What are some of the qualities that we can say
Speaker 1: are really valuable on the web. And part of that
Speaker 1: involved how successful web ventures were handling data. Now. That
Speaker 1: same year, he had a guy named Roger Mugalus or
Speaker 1: Mugalas actually I don't know how to say his last name,
Speaker 1: but he was also with O'Reilly, and he argued that
Speaker 1: big data refers to how we now had the capacity
Speaker 1: and the capability to gather and store data sets that
Speaker 1: are so large that our traditional business tools are incapable
Speaker 1: of doing anything useful with that information. It's makes me
Speaker 1: think of the joker in the Dark Knight film where
Speaker 1: he says, as a dog chasing a car, he wouldn't
Speaker 1: know what to do if he caught it. That kind
Speaker 1: of thing. Yeah, we've got all this information, but the
Speaker 1: tools we have aren't sufficient to do anything meaningful with it.
Speaker 1: We were overwhelmed with information. But that same year, because
Speaker 1: an awful lot happened in two thousand and five in
Speaker 1: the big data space, Doug Cutting and Mike Cafferella released
Speaker 1: a tool that would really change things. I'll explain more,
Speaker 1: but first we're going to take another quick break to
Speaker 1: thank our sponsors. Okay, before the break, I teased that
Speaker 1: we were going to talk about a tool made by
Speaker 1: Doug Cutting and Mike Cafferella that would actually change our
Speaker 1: approach to big data and make it possible to do
Speaker 1: meaningful things with it. So these two had read papers
Speaker 1: about Google's file system as well as a tool that
Speaker 1: Go was using called map reduce. Now, the purpose of
Speaker 1: map reduce is to take large clusters of data and
Speaker 1: essentially break them down into more manageable chunks, and then
Speaker 1: analyze these chunks in parallel, and this makes the process
Speaker 1: of data analysis faster. It's really just another form of
Speaker 1: parallel processing when you really think about it. Anyway, Cutting
Speaker 1: and Cafarella were inspired to make their own tool that
Speaker 1: could do similar work, but you know, they can make
Speaker 1: it for everybody, and so they created a project called
Speaker 1: hadoop hadop, and the first version of hadoop would come
Speaker 1: out in two thousand and six, and it's an open
Speaker 1: source project. It's still around today with thousands of contributors.
Speaker 1: But the important bit is that we were now starting
Speaker 1: to develop new business tools that actually could handle the
Speaker 1: massive amounts of information that we were accumulating. But let's
Speaker 1: take a quick step back. Let's also consider what's going
Speaker 1: on around this same time, the mid to late two thousands,
Speaker 1: and by that I mean the first decade of the
Speaker 1: two thousands. So for the first several years in the
Speaker 1: computer age, it was really computer systems themselves that were
Speaker 1: seen as the genesis of data creation, right like it's
Speaker 1: the computers are the things making all this info. But
Speaker 1: other elements were starting to come into play by this point.
Speaker 1: So when we get up to two thousand and seven,
Speaker 1: we're into the consumer smartphone era, because that was the
Speaker 1: introduction of the Apple iPhone. These consumer smartphones can generate
Speaker 1: enormous amounts of information. You can perform all sorts of
Speaker 1: computational tasks on them. They can track your location, you
Speaker 1: can connect the internet, et cetera. We also were getting
Speaker 1: into the age of the Internet of Things, so we
Speaker 1: were starting to create millions of these tiny devices, usually
Speaker 1: designed to collect specific bits of information and then zip
Speaker 1: that info off to somewhere else. So it might be
Speaker 1: a speed sensor along a road. It might be a
Speaker 1: thermometer at a weather data collection site. It might be
Speaker 1: a thermostat in your own home. It could be anything.
Speaker 1: Could be a smart speaker. All of these individual little
Speaker 1: components would add to the amount of information we were
Speaker 1: gathering and storing and creating, all in the hopes of
Speaker 1: being able to do something useful with that info. And
Speaker 1: we also had another buzz term that was starting to
Speaker 1: gain traction, just as big data was really beginning to
Speaker 1: transition from a topic that was talked about in a
Speaker 1: relatively small subculture of computer scientists and such into a
Speaker 1: topic that the general public had actually heard about. You know,
Speaker 1: usually we're a few years behind whatever group is really
Speaker 1: focused on the subject matter. So this other buzz term
Speaker 1: was cloud computing, which I also got an assignment to
Speaker 1: work on right around the same time as big data. Now,
Speaker 1: the simplest way to describe cloud computing is that it's
Speaker 1: when you use someone else's computer to do your computational
Speaker 1: tasks because you log in through your computer, but it's
Speaker 1: this other computer that's actually doing the work, or it
Speaker 1: might be a net work of other computers doing that work.
Speaker 1: That work could be that you're storing photos or ekitty
Speaker 1: cats on a drive on some cloud storage, or it
Speaker 1: might be that you're using cloud computing to help you
Speaker 1: crunch really big numbers that your computer could not handle
Speaker 1: and you're peeling back the mysteries of quantum mechanics or something.
Speaker 1: So cloud computing would rise at the same time as
Speaker 1: big data and cloud computing and big data are very
Speaker 1: closely related. They're enablers of one another in a way.
Speaker 1: Organizations and companies feel the need to engage with cloud
Speaker 1: computing services because their data tasks are growing increasingly complex
Speaker 1: and voluminous, and it gets harder and harder to handle
Speaker 1: all of that on your own. Right, Like most businesses
Speaker 1: these days are not using exclusively on premises computing systems
Speaker 1: to do all their computation and all their storage. It
Speaker 1: just is not practical, right. You would have to continuously
Speaker 1: buy or lease more space just to hold all the
Speaker 1: systems you would need. So instead they engage with cloud
Speaker 1: computing companies that will provide those services for them, and
Speaker 1: then the cloud computing companies will go out and build
Speaker 1: a warehouse and fill it full of computers. Big data
Speaker 1: leans on cloud computing to make it practical to even
Speaker 1: accumulate all that data in the first place, let alone
Speaker 1: analyze it. Now, the lure a big data the reason
Speaker 1: why we're concerned with it. I mentioned this in the
Speaker 1: very beginning of this episode. The lure is that there
Speaker 1: are nuggets of truth hiding inside vast amounts of possibly
Speaker 1: useless information. There is signal, but there's also an enormous
Speaker 1: amount of noise. If we can identify those little nuggets
Speaker 1: of truth, then we can potentially benefit from them. But
Speaker 1: these huge piles of information are just so vast that
Speaker 1: our ability to zero in on the important stuff is
Speaker 1: just not up to snuff. It is the proverbial needle
Speaker 1: in a hay haystack problem. So the promise of big
Speaker 1: data in our current age is that when we use
Speaker 1: the right tools, we can sift through the haystack and
Speaker 1: we can find all the needles, which is a really
Speaker 1: tempting concept, because who knows what you might find when
Speaker 1: you analyze large amounts of information. Maybe you identify patterns
Speaker 1: that you can then use to lead you to change
Speaker 1: things so that you can save huge sums of money
Speaker 1: in the way you do business. Or maybe you identify
Speaker 1: a previously unknown opportunity. Or maybe you can spot connections
Speaker 1: between data points that you didn't see before and you
Speaker 1: start to see correlation. Maybe you even determine causation. Maybe
Speaker 1: this leads you to make some incredible scientific progress, and
Speaker 1: it might be on anything from medicine to astronomy. It
Speaker 1: all depends on the type of data. Obviously, However, there's
Speaker 1: a big caveat that goes along with this sort of
Speaker 1: beautiful concept, and it's possible that the tools we use
Speaker 1: will make mistakes that they're going to spot patterns or
Speaker 1: meaning when in reality there isn't anything there. They mistake
Speaker 1: something to be meaningful when in fact it's not. This
Speaker 1: is kind of like when you look up at the
Speaker 1: clouds and you see a pattern that makes you think
Speaker 1: of a specific shape, very like a whale, As Hamlet
Speaker 1: and Polonius would say, so, the shape of the cloud
Speaker 1: might remind you of a whale or a dog, or
Speaker 1: a hand or whatever, but you probably are aware that
Speaker 1: the cloud isn't actually a whale or whatever. In fact,
Speaker 1: you might even realize that your point of view is
Speaker 1: part of what is shaping your perception. It's part of
Speaker 1: the reason why it looks like a whale. Maybe if
Speaker 1: you were a mile away to the east or something
Speaker 1: and you were to look at that same cloud, the
Speaker 1: angle you would be at would mean that the cloud
Speaker 1: wouldn't look anything like a whale. Maybe it would look
Speaker 1: like something entirely different, or maybe it wouldn't remind you
Speaker 1: of anything at all. So, from one perspective, the cloud
Speaker 1: shape appears to have some meaning. From other perspectives, it doesn't.
Speaker 1: So it would be a mistake to draw any conclusions
Speaker 1: based on that one perception, because it would just be
Speaker 1: the illusion of meaning, not actual meaning. And that can
Speaker 1: happen when you're looking at huge data sets too. You
Speaker 1: might see something that looks like it's meaningful, that it
Speaker 1: represents a pattern or a connection, when in fact it doesn't.
Speaker 1: That can lead you on a wild goose chase, and
Speaker 1: in a worst case scenario, you might dedicate a lot
Speaker 1: of time and effort and money toward pursuing this perceived
Speaker 1: meaning and you only find out much later that there
Speaker 1: was nothing there at all. Now that's not to say
Speaker 1: that we can't trust the outcomes of big data analysis,
Speaker 1: but it does mean that we have to make sure
Speaker 1: that we have tests to ensure the validity of those analyzes.
Speaker 1: We need to take a scientific approach toward big data,
Speaker 1: or else we run the risk of chasing a dream
Speaker 1: rather than and learning more about reality. And anytime there's
Speaker 1: any uncertainty, there will be people who move in to
Speaker 1: exploit that uncertainty, hucksters, scam artists, snake oil salesman. So
Speaker 1: as an example of this, I would point to the
Speaker 1: explosion we are seeing in artificial intelligence right now. AHI
Speaker 1: has tons of applications, including in the analysis of big data,
Speaker 1: and that means that there is also opportunity there to
Speaker 1: take advantage of people. So it doesn't take much imagination
Speaker 1: to think of a company that actually uses a cheap
Speaker 1: human labor and to pass it off as a truly
Speaker 1: AI company and to market that company's services to big
Speaker 1: businesses that may not know any better. And really you're
Speaker 1: just exploiting people in poorer countries and passing it off
Speaker 1: as being this really high tech business. As it stands,
Speaker 1: even if you're not doing that, human labor is already
Speaker 1: the backbone of the AI industry, like it or not.
Speaker 1: People in countries that have low wages and have very
Speaker 1: little protect in place for working citizens, they're spending countless
Speaker 1: hours tagging data so that AI can actually make use
Speaker 1: of it. So as we marvel at how clever AI
Speaker 1: tools appear to be, there are folks out there on
Speaker 1: the margins who are the ones labeling images and applying
Speaker 1: metadata to text so that the AI can grab the
Speaker 1: right stuff based upon a query. Anyway, I think it's
Speaker 1: important to remember that big data can, with the right tools,
Speaker 1: provide us insights that we might not otherwise make because
Speaker 1: the amount of information is just too large for us
Speaker 1: to handle. Those insights might mean we can do things
Speaker 1: like streamline supply chains, or identify a market for specific product,
Speaker 1: or find a new way to treat an illness. Big
Speaker 1: data can also lead us to some darker outcomes. Companies
Speaker 1: will scrape as much of your personal information as they
Speaker 1: possibly can. They will sell it to other companies. These
Speaker 1: other companies will market you to yet more companies on
Speaker 1: an effort to serve you ads or to lure you
Speaker 1: into doing something foolish like downloading malware or consuming misinformation.
Speaker 1: Because behind every silver lining is a big, scary cloud.
Speaker 1: Maybe it's in the shape of a whale. That is
Speaker 1: a brief history of big data. It is a history
Speaker 1: that is ongoing. I'm sure we're going to see some incredible,
Speaker 1: incredible discoveries thanks to analysis of big data. I'm sure
Speaker 1: we're also going to see some pretty scary stuff as
Speaker 1: a result of it as well, such is life, but
Speaker 1: it is fascinating to see how we have arrived at
Speaker 1: this point, like first from the point of how do
Speaker 1: we collect all this information? And then what do we
Speaker 1: do with it? I hope you enjoyed this episode. I
Speaker 1: hope you are all well, and I will talk to
Speaker 1: you again really soon. Tech Stuff is an iHeartRadio production.
Speaker 1: For more podcasts from iheartradioit the iHeartRadio app, Apple Podcasts,
Speaker 1: or wherever you listen to your favorite shows.