WEBVTT - A Small Episode About Big Data 0:00:04.440 --> 0:00:12.360 Welcome to tech Stuff, a production from iHeartRadio. Hey there, 0:00:12.400 --> 0:00:16.239 and welcome to tech Stuff. I'm your host, Jonathan Strickland. 0:00:16.320 --> 0:00:19.959 I'm an executive producer with iHeart Podcasts and How the 0:00:20.160 --> 0:00:24.360 tech are You. So early on in the days of 0:00:24.360 --> 0:00:26.439 tech stuff, back when I was still a staff writer 0:00:26.560 --> 0:00:30.400 for a little website called HowStuffWorks dot com, my boss 0:00:31.000 --> 0:00:34.840 Connell Burn, who is now a big shot over here 0:00:34.840 --> 0:00:37.560 at iHeart, he came over to me with an assignment. 0:00:37.640 --> 0:00:40.360 He wanted me to do some articles in some episodes 0:00:40.520 --> 0:00:46.680 about this buzzword concept called big data. And I had 0:00:46.760 --> 0:00:50.040 heard the term big data, and obviously there's a pretty 0:00:50.080 --> 0:00:52.839 darn good hint I want. Big data is all about 0:00:52.920 --> 0:00:55.720 just in the nature of the name itself, but beyond that, 0:00:55.800 --> 0:00:58.680 I didn't really know much, so I jumped to it. 0:00:59.080 --> 0:01:01.960 And the interesting thing is that since that time, the 0:01:02.040 --> 0:01:05.959 discipline of big data has evolved significantly. When I was 0:01:06.000 --> 0:01:08.760 first working on my articles and episodes, we were mostly 0:01:08.800 --> 0:01:13.119 talking about how technological tools made it easier to collect 0:01:13.760 --> 0:01:18.280 vast amounts of information very quickly and to store it. 0:01:18.720 --> 0:01:22.160 But we didn't necessarily have equally sufficient tools to do 0:01:22.200 --> 0:01:25.360 anything useful with all that information. We have or at 0:01:25.440 --> 0:01:29.680 least those tools weren't widely known and understood beyond a 0:01:29.720 --> 0:01:34.759 certain circle of computer scientists. Flash forward a few years, 0:01:35.080 --> 0:01:39.320 and we'd see companies developing new methods to analyze large 0:01:39.440 --> 0:01:42.160 chunks of data. Oh, by the way, I do the 0:01:42.200 --> 0:01:45.360 weird data data thing, and there's no rhyme or reason 0:01:45.360 --> 0:01:46.680 to it. I don't even know which one I'm going 0:01:46.720 --> 0:01:49.360 to say before I say it, so I apologize because 0:01:49.400 --> 0:01:52.960 I know it's irritating. It irritates me too, Anyway. Other 0:01:53.000 --> 0:01:57.480 companies sprung up with products that were meant to help 0:01:57.720 --> 0:02:01.040 with data analysis, and it seemed like we were going 0:02:01.080 --> 0:02:03.800 from an era of well, now I have all this information, 0:02:03.880 --> 0:02:06.200 what do I do now, to an era of I 0:02:06.280 --> 0:02:09.440 have discovered cryptic secrets that were hiding in plain sight 0:02:09.680 --> 0:02:13.679 thanks to data analysis, and that somehow it all happened overnight. 0:02:13.960 --> 0:02:16.639 So today I thought we would actually look back over 0:02:16.680 --> 0:02:21.000 the history of the big data concept, how various systems 0:02:21.000 --> 0:02:25.520 have made it possible to sift through seemingly meaningless information 0:02:25.880 --> 0:02:28.480 in order to find nuggets of wisdom, and why we 0:02:28.560 --> 0:02:31.960 might not always be able to trust the answers that 0:02:32.000 --> 0:02:35.960 we discover. So the history of big data starts in 0:02:36.000 --> 0:02:38.680 the twenty tens, or maybe it starts in two thousand 0:02:38.720 --> 0:02:42.240 and five, or maybe in nineteen ninety, or maybe the 0:02:42.280 --> 0:02:46.239 sixteen hundreds, or maybe nearly twenty thousand years ago. You 0:02:46.520 --> 0:02:48.280 might have already picked up on the fact that folks 0:02:48.280 --> 0:02:51.120 don't quite agree on where we should start when talking 0:02:51.160 --> 0:02:54.640 about big data. But that makes sense. Ever since humans 0:02:54.720 --> 0:02:58.800 have started to write stuff down, we've been pretty darn 0:02:58.880 --> 0:03:03.360 invested in the collection and then the classification of information. 0:03:03.840 --> 0:03:07.079 Whether it's to figure out the best time to sew 0:03:07.320 --> 0:03:10.480 or harvest crops, or keep track of how much we've 0:03:10.520 --> 0:03:13.120 traded with that other band of neair dwells who live 0:03:13.160 --> 0:03:15.440 on the other side of the holler, or we just 0:03:15.520 --> 0:03:18.160 want to make a record of how great it was 0:03:18.200 --> 0:03:20.640 that we kicked the butt of that mastodon, real good. 0:03:20.840 --> 0:03:26.440 We've been really obsessed with data and collection and retrieval. Now, 0:03:27.120 --> 0:03:30.240 this obsession also means that we had to come up 0:03:30.280 --> 0:03:34.000 with various ways to store and analyze this information. Raw 0:03:34.000 --> 0:03:37.720 information doesn't do anyone much good, and so throughout antiquity 0:03:37.960 --> 0:03:41.520 we came up with means of recording and storing and 0:03:41.560 --> 0:03:45.960 making use of information. Not only did hardworking humans create 0:03:46.120 --> 0:03:49.760 libraries where we could gather all this knowledge and then 0:03:50.000 --> 0:03:52.840 lose some of those libraries along the way due to 0:03:52.880 --> 0:03:55.640 the fact that we humans also are pretty stupid and 0:03:55.680 --> 0:03:59.480 we end up having disputes that involve burning each other's 0:03:59.520 --> 0:04:03.200 stuff to the ground. Yeah, I'm still bitter about certain 0:04:03.240 --> 0:04:07.080 libraries being destroyed over in antiquity, but it means that 0:04:07.120 --> 0:04:10.120 we also had to come up with methodologies to categorize 0:04:10.280 --> 0:04:13.600 and classify information. Otherwise you may as well just have 0:04:13.680 --> 0:04:17.240 a big old pile of scrolls or books or whatever, 0:04:17.600 --> 0:04:20.200 and then people just you know, have to sort through 0:04:20.240 --> 0:04:23.159 them and see if they can find anything, which actually 0:04:23.240 --> 0:04:25.640 sparks two different memories in my head. One is that 0:04:25.839 --> 0:04:28.640 there used to be a used bookstore I would go 0:04:28.680 --> 0:04:33.120 to here in Atlanta, and often the used bookstore was 0:04:33.560 --> 0:04:37.919 completely unorganized, right, Like you literally could go through a 0:04:37.920 --> 0:04:40.240 bookshelf and it's just going to be books that are 0:04:40.720 --> 0:04:43.440 more or less the same size, but otherwise there's no 0:04:43.680 --> 0:04:45.679 rhyme or reason as to why they were put there, 0:04:46.040 --> 0:04:47.800 and it was like you were on a treasure hunt. 0:04:48.000 --> 0:04:55.000 And then I'm also reminded of a naval museum in Appalachicola, Florida, 0:04:55.000 --> 0:04:57.480 which is on the Panhandle. I went to this little, 0:04:57.880 --> 0:05:02.159 you know, naval museum, like a ship museum, and I 0:05:02.360 --> 0:05:05.480 reminded that all the exhibits were kind of in a 0:05:05.520 --> 0:05:08.839 pile on the floor, and you would literally pick things 0:05:08.920 --> 0:05:12.560 up and look at them. And that's kind of what 0:05:12.880 --> 0:05:15.159 it would be like if we didn't have these means 0:05:15.200 --> 0:05:17.880 of classification. Once you get to a certain size, like 0:05:17.920 --> 0:05:21.719 that little museum in Appalachic Cola wasn't so big as 0:05:21.760 --> 0:05:24.000 to be a problem. But if you're talking about a 0:05:24.040 --> 0:05:27.240 big library, obviously, if you want anything useful, you got 0:05:27.279 --> 0:05:29.120 to come up with a way of classifying all this. 0:05:29.680 --> 0:05:32.760 To that end, ancient folks began to develop a science 0:05:33.160 --> 0:05:37.640 called taxonomy. And this isn't when you stuff dead animals 0:05:37.720 --> 0:05:39.800 so that they look like they might still sort of 0:05:39.839 --> 0:05:44.320 be alive. That's taxon dermy. No. Taxonomy is the science 0:05:44.320 --> 0:05:47.600 of classification, and it's perhaps best known in the field 0:05:47.640 --> 0:05:51.440 of biology, thanks in large part to a Swedish scientist 0:05:51.440 --> 0:05:55.280 from the eighteenth century named Carl Linnaeus. But there are 0:05:55.480 --> 0:05:59.159 many applications of taxonomy that extend beyond biology. It's just 0:05:59.160 --> 0:06:02.080 the biological taxonomy is the one that I think most 0:06:02.080 --> 0:06:04.400 of us are familiar with because most of us were 0:06:04.440 --> 0:06:08.320 taught it when we were going through basic biology science. 0:06:08.760 --> 0:06:11.840 But the ancient Greeks made some early progress on developing 0:06:11.880 --> 0:06:17.000 systems of classification, and obviously, within modern library science, taxonomy 0:06:17.120 --> 0:06:20.640 is an important discipline, though oddly enough, you could say 0:06:20.680 --> 0:06:24.960 taxonomy in library science is distinct from classification. When I 0:06:25.000 --> 0:06:29.039 was looking this up, I found resources for library science 0:06:29.279 --> 0:06:32.920 that made these two distinct disciplines. Classification was one in 0:06:32.920 --> 0:06:36.080 taxonomy was another. Now. This is because there are various 0:06:36.400 --> 0:06:39.800 methods of classification in library science. The one that I 0:06:39.960 --> 0:06:41.880 was most familiar with when I was growing up was 0:06:41.920 --> 0:06:45.320 the Dewey decimal system, which I don't even think is 0:06:45.680 --> 0:06:48.760 the dominant form now, but it was when I was 0:06:48.800 --> 0:06:51.680 growing up. And it's meant to connect a specific work 0:06:51.760 --> 0:06:54.919 to a specific physical location in a library for the 0:06:54.920 --> 0:06:57.880 purposes of, you know, checking down the book, right. But 0:06:58.040 --> 0:07:01.760 taxonomy in library science tends to be more towards metadata 0:07:01.920 --> 0:07:06.839 or data about data. In fact, metadata plays a huge 0:07:06.880 --> 0:07:11.160 part in big data. Oh man, I did it both 0:07:11.200 --> 0:07:14.160 ways in one sentence. I feel awful. Anyway, the information 0:07:14.240 --> 0:07:17.840 about information can be as useful as the information itself. 0:07:17.840 --> 0:07:20.280 In some cases. I have often talked about this with 0:07:20.720 --> 0:07:24.720 personal information about how info about info can give you 0:07:24.760 --> 0:07:28.160 a lot of insight into a person. Maybe you don't 0:07:28.200 --> 0:07:30.560 have a person's name, but you have a couple of 0:07:30.560 --> 0:07:34.200 different data points about that person. In some cases, you 0:07:34.240 --> 0:07:37.920 can actually narrow down the identity of the person you're 0:07:37.960 --> 0:07:41.040 thinking of just by looking at this metadata. You don't 0:07:41.040 --> 0:07:43.240 even have to see the information about them, which shows 0:07:43.240 --> 0:07:46.120 you how powerful metadata can be. So you start to 0:07:46.120 --> 0:07:48.680 see a cascading effect here where you slowly realize that 0:07:48.720 --> 0:07:51.400 you actually have access to even more information than you 0:07:51.440 --> 0:07:54.800 first anticipated because you also have information about that information. 0:07:54.880 --> 0:07:58.320 It gets pretty wild. Another important development in the history 0:07:58.360 --> 0:08:01.840 of big data is the creation of statistics. So let's 0:08:01.840 --> 0:08:05.360 give the Merriam Webster definition of statistics. Shall we just 0:08:05.800 --> 0:08:10.520 have a baseline. It is quote a branch of mathematics 0:08:10.920 --> 0:08:16.440 dealing with the collection, analysis, interpretation, and presentation of masses 0:08:16.480 --> 0:08:21.200 of numerical data. End quote. Now. One famous early example 0:08:21.240 --> 0:08:24.160 of statistics comes to us courtesy of a fellow named 0:08:24.360 --> 0:08:31.080 John Grant Graunt. He was looking at mortality rates in London, 0:08:31.440 --> 0:08:34.160 and that gave him a lot more information and helped 0:08:34.200 --> 0:08:38.000 him analyze the course of the plague. For example, he 0:08:38.040 --> 0:08:41.240 could see when the plague was spiking or receding. Pretty 0:08:41.280 --> 0:08:45.080 cheerful stuff, right, But he also used this information, the 0:08:45.120 --> 0:08:49.520 mortality information, to start drawing some conclusions about the population 0:08:49.600 --> 0:08:52.840 of London as a whole, So counting up everybody, like 0:08:52.920 --> 0:08:56.320 figuring out who lives in London. That would have been 0:08:56.400 --> 0:08:59.600 challenging at the time, to say the least. But Grant 0:08:59.720 --> 0:09:02.680 took information like the number of funerals and then he 0:09:02.760 --> 0:09:06.240 compared it to things like the average family size in 0:09:06.320 --> 0:09:09.559 London to try and make an estimate of London's population. 0:09:09.679 --> 0:09:11.440 So it gave him kind of a working figure that 0:09:11.960 --> 0:09:16.880 was useful for certain applications, specifically government ones. Statistics as 0:09:17.080 --> 0:09:21.439 a branch of mathematics would mature over the following centuries. 0:09:21.920 --> 0:09:25.520 Often it would be the tool that allowed social scientists 0:09:25.600 --> 0:09:31.080 to draw broad conclusions about large populations, but others found 0:09:31.120 --> 0:09:35.800 plenty of alternative applications of statistics. Anyway, the age of 0:09:35.920 --> 0:09:39.320 data analysis was well and truly in swing at this 0:09:39.440 --> 0:09:42.600 point in the late nineteenth century. The United States was 0:09:42.679 --> 0:09:45.040 getting in a bit of a pickle. And I know 0:09:45.280 --> 0:09:48.080 we're making jumps of centuries here, but we need to 0:09:48.240 --> 0:09:52.920 We can't go through every single evolution of data collection 0:09:53.160 --> 0:09:56.600 and data analysis that would be a podcast series all 0:09:56.640 --> 0:09:59.599 in itself. So we're in the late eighteen hundreds and 0:09:59.640 --> 0:10:02.120 the USA US isn't a bit of a problem. The 0:10:02.160 --> 0:10:06.000 country holds a census every ten years, where they're essentially 0:10:06.080 --> 0:10:08.920 gathering information about all the citizens in the United States. 0:10:09.040 --> 0:10:12.199 This is required by the US Constitution, and there are 0:10:12.240 --> 0:10:15.960 several reasons why the Census Bureau holds a census every 0:10:15.960 --> 0:10:19.280 ten years. But one of those reasons is that the 0:10:19.400 --> 0:10:24.800 US House of Representatives its membership depends upon population. So 0:10:25.679 --> 0:10:29.280 the more populous a state is, the more representatives that 0:10:29.320 --> 0:10:31.920 state has in the House of Representatives. So if your 0:10:31.960 --> 0:10:35.280 state has a big population, there are more representatives that 0:10:35.400 --> 0:10:39.200 go to the House. If you have a relatively small population, 0:10:39.280 --> 0:10:43.000 then you have fewer House representatives, right, That's how that works. 0:10:43.240 --> 0:10:48.480 So by eighteen eighty things were getting to a really 0:10:48.840 --> 0:10:53.400 difficult situation. The process of collecting and then analyzing all 0:10:53.440 --> 0:10:58.240 the information was so cumbersome that it would take nearly 0:10:58.760 --> 0:11:02.040 the whole decade just to get to a result, and 0:11:02.120 --> 0:11:05.880 that means by the time you're drawing conclusions, it's actually 0:11:05.920 --> 0:11:08.120 time for you to administer the next census. In fact, 0:11:08.360 --> 0:11:11.680 they projected that in eighteen ninety working on the same 0:11:11.720 --> 0:11:15.160 process that they were dependent upon previously. It would take 0:11:15.200 --> 0:11:18.480 a whole decade, so literally you'd be holding your next 0:11:18.480 --> 0:11:21.000 census while you were just getting your information from the 0:11:21.080 --> 0:11:23.440 last one. So the Census Bureau needed a way to 0:11:23.440 --> 0:11:26.840 collect and analyze this information in a much more efficient process. 0:11:27.280 --> 0:11:32.360 They tapped a man named Herman Holleeth to accomplish this. 0:11:32.920 --> 0:11:36.720 So Holloweth took a punch card system that had been 0:11:36.800 --> 0:11:40.840 used in weaving, weaving with mechanical looms. I've talked about 0:11:40.840 --> 0:11:43.160 this in the past with the history of punch cards. 0:11:43.400 --> 0:11:47.840 In fact, this also gets into perhaps a somewhat apocryphal 0:11:47.880 --> 0:11:51.080 story of where the word sabotage comes from, but that's 0:11:51.280 --> 0:11:54.360 for another time. So he took this punch card system 0:11:54.400 --> 0:11:58.160 that had been used to set weaving patterns with mechanical looms, 0:11:58.360 --> 0:12:00.760 and then he adapted that to serve as a way 0:12:01.120 --> 0:12:05.040 to record information so that you could feed the card 0:12:05.080 --> 0:12:09.840 to a tabulation machine which then could actually tabulate the results. 0:12:10.080 --> 0:12:13.600 And his invention meant that ten years of labor done 0:12:13.600 --> 0:12:16.320 by clerks who are working at desks would actually boil 0:12:16.400 --> 0:12:21.080 down to about three months of labor using the tabulation machine. Obviously, 0:12:21.640 --> 0:12:24.920 that was a huge improvement. Hollerith formed a company that 0:12:25.080 --> 0:12:28.040 over time would evolve into one of the most famous 0:12:28.040 --> 0:12:32.120 companies in all the world, Kentucky Fried Chicken. I'm just kidding. 0:12:32.200 --> 0:12:36.760 It wasn't KFC. Instead, it was IBM. That's the company 0:12:37.040 --> 0:12:40.200 that would grow out of Hollowarith's company that he founded 0:12:40.600 --> 0:12:44.360 in the nineteenth century. Anyway, we're not going to spend 0:12:44.800 --> 0:12:47.520 too much time in all these centuries gone by. We're 0:12:47.520 --> 0:12:50.560 actually going to speed things up and get up to 0:12:50.600 --> 0:12:54.000 the twentieth century. But before we do that, let's take 0:12:54.240 --> 0:13:07.480 a quick break to thank our sponsor. We're back, okay, 0:13:08.200 --> 0:13:12.439 So the actual term big data is still waiting for us. 0:13:12.480 --> 0:13:14.280 We're not going to really get to that until we 0:13:14.360 --> 0:13:17.200 hit the late nineteen nineties or so. But there are 0:13:17.200 --> 0:13:20.120 a few things to point out before we get up 0:13:20.160 --> 0:13:24.760 to there. Folks were starting to notice that we were generating, collecting, 0:13:24.880 --> 0:13:28.920 and storing an awful lot of information in the twentieth century, 0:13:29.200 --> 0:13:33.280 and that the rate of data generation was on the rise. 0:13:33.440 --> 0:13:35.960 Not only were we generating a whole bunch of information, 0:13:36.360 --> 0:13:39.680 we were doing it in larger amounts year over year. 0:13:39.880 --> 0:13:42.520 In fact, it was rising much faster than our rate 0:13:42.559 --> 0:13:47.040 of consumption of information, meaning that we were making way 0:13:47.120 --> 0:13:50.320 more data than we were actually able to use. And 0:13:50.360 --> 0:13:53.560 a big thanks goes out to Forbes for an article 0:13:53.600 --> 0:13:57.080 that's titled A very Short History of Big Data by 0:13:57.200 --> 0:14:00.560 Gil Press. A lot of the information that I'm drawing 0:14:00.640 --> 0:14:03.280 upon came from that article. It is fantastic if you 0:14:03.320 --> 0:14:05.320 want to learn more about this. I'm not going to 0:14:05.360 --> 0:14:07.960 cover every element that they do. I mean, that would 0:14:07.960 --> 0:14:10.560 just be me regurgitating their article. You should check it 0:14:10.559 --> 0:14:13.080 out if you're interested in the history of big data. 0:14:13.240 --> 0:14:15.440 We're going to touch on a few of the important points, 0:14:15.760 --> 0:14:18.000 or what I think of as the important points. So 0:14:18.679 --> 0:14:20.400 one of the earliest ones we're going to talk about 0:14:20.440 --> 0:14:24.400 is in nineteen forty four, a librarian named Fremont Writer, 0:14:24.600 --> 0:14:28.640 which is a fantastic name, wrote a work titled The 0:14:28.680 --> 0:14:32.040 Scholar and the Future of the Research Library. So Writer 0:14:32.240 --> 0:14:34.840 made an observation that reminds me a lot of Gordon 0:14:34.920 --> 0:14:39.800 Moore's famous Moore's law, except this involves not silicon chips 0:14:40.280 --> 0:14:44.880 but physical libraries. So Writer said that your typical library 0:14:45.120 --> 0:14:49.240 in your typical American university was doubling in size every 0:14:49.320 --> 0:14:52.720 sixteen years. He projected that this would mean that by 0:14:52.760 --> 0:14:56.440 the year twenty forty, the library at Yale University would 0:14:56.480 --> 0:14:59.280 be so large as to require a staff of more 0:14:59.320 --> 0:15:04.480 than six thousand people to manage it. Of course, this 0:15:04.640 --> 0:15:08.280 was before we had digital storage and digital filing systems 0:15:08.560 --> 0:15:12.600 that has largely mitigated this particular requirement. We don't need 0:15:13.040 --> 0:15:16.960 the physical space necessarily that we would if everything were 0:15:17.000 --> 0:15:20.640 still in hard copy. But the observation showed that data 0:15:20.680 --> 0:15:24.280 accumulation really had a steep trajectory even back in the 0:15:24.360 --> 0:15:28.800 nineteen forties. Similarly, in the early nineteen sixties, a guy 0:15:28.880 --> 0:15:32.080 named Derek Price published a piece explaining that the number 0:15:32.080 --> 0:15:35.400 of scientific journals and papers was on a path of 0:15:35.600 --> 0:15:39.520 exponential growth. It was doubling every fifteen years, so similar 0:15:39.760 --> 0:15:42.880 to the rate at which university libraries were doubling in 0:15:42.960 --> 0:15:45.480 size now. Part of the reason for this, he said, 0:15:45.840 --> 0:15:50.400 was that scientific discoveries inevitably fuel further discoveries. So you 0:15:50.480 --> 0:15:53.320 find out something new, this inspires other scientists to look 0:15:53.360 --> 0:15:56.160 further into it, they find other new things, and so on. 0:15:56.520 --> 0:15:59.480 In nineteen sixty five, the United States government needed to 0:15:59.480 --> 0:16:02.200 build a place that would store records, including things like 0:16:02.320 --> 0:16:06.600 tax returns and fingerprint sets, and so the plan was 0:16:06.680 --> 0:16:09.800 to take the paper records and then transfer them to 0:16:10.000 --> 0:16:13.360 magnetic tape, and then to store that magnetic tape in 0:16:13.440 --> 0:16:17.800 this so called data center. This project fell through, however, 0:16:18.040 --> 0:16:21.920 because the public got nervous. They felt squiky about this 0:16:22.000 --> 0:16:25.080 idea of the government hoarding vast amounts of information about 0:16:25.080 --> 0:16:28.200 its citizens. They did not fully trust the government. So 0:16:28.240 --> 0:16:30.680 you understand like they're thinking, I don't really feel comfortable 0:16:30.720 --> 0:16:33.520 with you just gathering all this information about us. It 0:16:33.560 --> 0:16:37.240 feels kind of oppressive. Now, what's funny to me is 0:16:37.280 --> 0:16:41.000 that today the average person is more than willing to 0:16:41.080 --> 0:16:44.320 let companies do this to them without even protesting it. 0:16:44.560 --> 0:16:48.840 Because that's how all the online social network companies work, right, 0:16:48.880 --> 0:16:51.440 They work on the basis of gathering information about us 0:16:51.760 --> 0:16:55.480 and then peddling that or or hoarding it, however you 0:16:55.560 --> 0:16:58.320 might think of it. And it's very similar to what 0:16:58.360 --> 0:17:00.360 was happening in the nineteen sixties. And back then we 0:17:00.360 --> 0:17:02.480 were like, no, that's not cool, and now we're like, 0:17:02.520 --> 0:17:05.719 that's just how it works. It's wild to me. Anyway, 0:17:05.720 --> 0:17:07.359 I'm going to skip ahead a little bit to the 0:17:07.440 --> 0:17:14.160 nineteen eighties. There was a lecturer, I a Tjomslend, and 0:17:14.200 --> 0:17:18.600 I know I butchered his name. I apologize anyway. He 0:17:18.680 --> 0:17:24.159 gave a lecture at the IE or IE Symposium in 0:17:24.200 --> 0:17:27.600 which he posits that one reason all this information is 0:17:27.640 --> 0:17:30.560 piling up is that we don't really have a good 0:17:30.560 --> 0:17:34.520 way to determine which information is relevant and which information 0:17:35.119 --> 0:17:39.000 is not. And we can make that determination, but it 0:17:39.040 --> 0:17:42.919 requires work, and meanwhile, we're still accumulating more information. So 0:17:43.280 --> 0:17:45.480 it's the kind of work where you're never done, and 0:17:45.560 --> 0:17:47.840 it feels like you're never making any progress. So most 0:17:47.840 --> 0:17:50.280 of us never bother to do it at all. And 0:17:50.720 --> 0:17:54.200 if our ability to store data is sufficient, in other words, 0:17:54.200 --> 0:17:57.280 if we have ways of storing the information, then we 0:17:57.359 --> 0:18:01.119 have even less incentive to make any determination about the data. Right, Like, 0:18:01.160 --> 0:18:03.679 if we've got plenty of storage, well, let's just go 0:18:03.720 --> 0:18:06.639 ahead and keep the information. There's no reason to have 0:18:06.720 --> 0:18:09.080 to worry about it whether it's useful or not. We 0:18:09.119 --> 0:18:12.200 should keep it because it's better for us to keep 0:18:12.840 --> 0:18:17.960 useless information without needing it, rather than accidentally deleting something 0:18:18.160 --> 0:18:21.520 that turned out to be important. Right, And this kind 0:18:21.520 --> 0:18:23.840 of makes sense. I mean, I'm sure a lot of 0:18:23.880 --> 0:18:26.119 you out there can apply that to your lives. I 0:18:26.160 --> 0:18:28.359 certainly can apply it to my life, right Like, I 0:18:28.480 --> 0:18:31.600 have file folders that are full of stuff that I'm 0:18:31.640 --> 0:18:35.159 never going to touch again, but I still feel reluctant 0:18:35.200 --> 0:18:37.600 to delete it just in case I do need to 0:18:37.600 --> 0:18:39.679 touch it again sometime in the future, even though the 0:18:39.760 --> 0:18:43.399 likelihood of that is very low. So that's anecdotal. I 0:18:43.440 --> 0:18:46.639 can't really call that evidence to prove the point, but 0:18:46.720 --> 0:18:50.240 it feels like the point is relevant. So this is 0:18:50.280 --> 0:18:52.639 also how I play a lot of those big open 0:18:52.640 --> 0:18:56.080 world computer RPGs, by the way, things like Skyrim or whatever, 0:18:56.160 --> 0:18:59.800 because I'll just hoarde potions and scrolls and I never 0:19:00.160 --> 0:19:02.679 use them because what if I need it more in 0:19:02.760 --> 0:19:05.600 the future. Balder's Gate three has really done a number 0:19:05.640 --> 0:19:07.600 on me with this. I got a real problem with 0:19:07.640 --> 0:19:12.120 that anyway. The Forbes article details several more entries indicating 0:19:12.160 --> 0:19:15.520 how very smart people were taking note regarding the accumulation 0:19:15.600 --> 0:19:19.320 of information, as well as methods to store the information, 0:19:19.480 --> 0:19:22.520 and increasingly, as time went on, how we can do 0:19:22.640 --> 0:19:25.480 useful things with all this information. So I recommend you 0:19:25.560 --> 0:19:27.480 check out that Forbes article if you want to learn more. 0:19:27.920 --> 0:19:31.240 I think it goes up to about twenty twelve at 0:19:31.240 --> 0:19:34.080 this point, it has been updated numerous times, but obviously 0:19:34.400 --> 0:19:38.760 twenty twelve was quite a long time ago, so it's 0:19:38.840 --> 0:19:41.680 it's not exactly up to present day. But it's still 0:19:41.720 --> 0:19:45.440 a really interesting article that gives lots more details about this. 0:19:46.040 --> 0:19:47.960 But I don't want to just regurgitate the article, so 0:19:47.960 --> 0:19:50.840 we're going to hop on ahead. Now, Folks, in general, 0:19:51.000 --> 0:19:54.840 we're becoming more aware of this information challenge that was growing. 0:19:55.119 --> 0:19:58.840 But where did the term big data actually come from? Well, 0:19:58.920 --> 0:20:02.760 chances are it's sort of rose organically in conversations within 0:20:02.840 --> 0:20:07.160 the computer sector. As you know, hackers and computer scientists 0:20:07.240 --> 0:20:11.080 and programmers and researchers were all wrestling with ways to 0:20:11.200 --> 0:20:14.840 deal with data. Now, by this time, folks had adapted 0:20:14.880 --> 0:20:19.119 an observation made by Cyril Northcote Parkinson to apply to 0:20:19.240 --> 0:20:23.679 computer systems and to information. So Parkinson's original observation was 0:20:23.680 --> 0:20:27.560 that generally speaking, in public administration offices, you know, like 0:20:27.680 --> 0:20:31.960 government offices, work expands to fill the time that was 0:20:32.000 --> 0:20:35.000 allowed for that work. So if you have a project 0:20:35.040 --> 0:20:38.120 that's going to be due in three weeks, but really, 0:20:38.160 --> 0:20:40.760 if you were to be brutally honest, there's only a 0:20:40.800 --> 0:20:44.479 week's worth of work to do for that project. Well, 0:20:44.800 --> 0:20:48.360 that work will almost magically expand so that it actually 0:20:48.400 --> 0:20:51.720 takes three weeks to complete. This gets more nuanced and 0:20:51.760 --> 0:20:54.880 it brings into account elements like bureaucracy. But you get 0:20:54.920 --> 0:20:59.760 the point right that somehow it doesn't matter, you know what, 0:21:00.119 --> 0:21:02.480 who is working the job. It doesn't matter the nature 0:21:02.520 --> 0:21:05.159 of the work. The work will expand to fill the 0:21:05.200 --> 0:21:07.920 amount of time it requires to do that work, which 0:21:07.960 --> 0:21:10.439 meant that if you had said it would take two weeks, 0:21:10.760 --> 0:21:12.879 it would have just expanded to two weeks, not three. 0:21:13.000 --> 0:21:15.960 It's very weird, right Anyway, Folks in the computer biz 0:21:16.000 --> 0:21:19.679 adapted this to say that data will expand to fill 0:21:19.840 --> 0:21:23.199 whatever space you have available for that data. So, in 0:21:23.240 --> 0:21:27.560 other words, you make a bigger storage unit, you're going 0:21:27.640 --> 0:21:30.240 to fill it like that data will just expand to 0:21:30.280 --> 0:21:34.119 fill that even though you thought, oh, I'm future proofing this, 0:21:34.640 --> 0:21:38.640 and again anecdotally, I have observed this in my personal life. 0:21:38.640 --> 0:21:41.320 I remember when hard disk drives first became a thing 0:21:41.400 --> 0:21:44.879 in personal computers, like they were already existed, but personal 0:21:44.920 --> 0:21:47.920 computers didn't have them when they first it came out, right, 0:21:47.960 --> 0:21:51.080 you were using external drives like floppy disks and stuff, 0:21:51.320 --> 0:21:54.440 and I remember whenever there would be a dramatic expansion 0:21:54.440 --> 0:21:57.399 of storage space, and it always seemed to be dramatic, right, 0:21:57.480 --> 0:22:00.600 it always seemed like it had doubled since last time. 0:22:00.640 --> 0:22:03.240 And typically that's how it worked. Anyway, I would walk 0:22:03.280 --> 0:22:05.880 away thinking, Wow, I'm never gonna fill all this space. 0:22:06.040 --> 0:22:08.840 I mean, who even needs that much space? Two hundred 0:22:08.840 --> 0:22:12.359 and fifty six megabytes? Who the heck needs that much space? 0:22:12.400 --> 0:22:14.880 That's way too much. I mean, I'll never fill it up. 0:22:15.520 --> 0:22:17.800 But of course I would prove myself wrong, typically in 0:22:17.880 --> 0:22:22.280 record time. But beyond anecdotes, which again don't really count 0:22:22.320 --> 0:22:25.480 as evidence, the observation really pointed out that we will 0:22:25.560 --> 0:22:28.919 eagerly fill up whatever space we're given. You could argue 0:22:29.160 --> 0:22:31.920 this goes back to our tendency to avoid deleting material 0:22:32.000 --> 0:22:35.960 out of concern that it might one day become useful. Anyway, 0:22:36.280 --> 0:22:39.160 By the mid nineteen nineties, there was a computer scientist 0:22:39.240 --> 0:22:43.720 named John Mashie, and he was giving presentations that related 0:22:43.840 --> 0:22:48.120 to this concept of big data. Now, Mashie has dismissed 0:22:48.160 --> 0:22:51.800 the idea that he personally coined the phrase. At most, 0:22:52.160 --> 0:22:55.919 he says that he popularized the term big data in 0:22:55.920 --> 0:22:58.520 his talks but his point was that he used the 0:22:58.520 --> 0:23:00.720 phrase big data because it was a shit shorthand way 0:23:00.760 --> 0:23:04.360 to give a nod to several related challenges, ranging from 0:23:04.480 --> 0:23:07.919 storage to analysis. So one could argue that Mashie's use 0:23:07.960 --> 0:23:11.159 of the term approached what we mean by big data today, 0:23:11.320 --> 0:23:13.680 but it wasn't one hundred percent the same thing. And 0:23:14.119 --> 0:23:18.640 the earliest use I've seen cited happened sometime around nineteen 0:23:18.760 --> 0:23:22.679 ninety eight. So we know Mashie didn't invent the phrase, 0:23:23.359 --> 0:23:26.600 and we know that partly because researchers found an instance 0:23:26.680 --> 0:23:29.920 that predates his talks by nearly a decade. Steve Lohr 0:23:30.000 --> 0:23:32.480 wrote a piece for The New York Times titled the 0:23:32.520 --> 0:23:37.119 Origins of Big Data, An etymological detective Story. A great, 0:23:37.240 --> 0:23:40.639 great article. By the way, Lore spoke with an associate 0:23:40.840 --> 0:23:44.640 librarian in Yale Law School named Fred Shapiro, and Fred 0:23:44.680 --> 0:23:47.600