WEBVTT - Drowning in Data 0:00:00.160 --> 0:00:07.400 Brought to you by Toyota. Let's go places. Welcome to 0:00:07.560 --> 0:00:15.280 Forward Thinking, either everyone, and welcome to Forward Thinking, the 0:00:15.440 --> 0:00:18.240 podcast that looks at the future and says, what's the story, 0:00:18.320 --> 0:00:21.880 Morning Glory, what's the word hummingbird? I'm Jonathan Strickland, I'm 0:00:21.920 --> 0:00:25.000 laurenco and I'm Joe McCormick, and we have jazz hands 0:00:25.760 --> 0:00:29.640 and today you know it totally does not translate to radio. 0:00:29.760 --> 0:00:34.040 But anyway, we're doing Yeah, and we were. Actually I wasn't, 0:00:34.120 --> 0:00:37.400 but you two were honesty and podcasting. That's true. So 0:00:37.440 --> 0:00:40.840 today we wanted to talk about the concept of big 0:00:40.920 --> 0:00:44.479 data or big data, which is not a gigantic character 0:00:44.520 --> 0:00:47.840 from Star Trek the Next Generation. Unfortunately, No, that would 0:00:47.840 --> 0:00:52.320 have been both interesting and terrifying. But that is not 0:00:52.360 --> 0:00:54.520 the case. Actually, I guess you could still argue that 0:00:54.560 --> 0:00:58.080 big data is both interesting and to some people terrifying. Yeah, 0:00:58.400 --> 0:01:01.040 it's it's so what is data? I've got a couple 0:01:01.040 --> 0:01:04.320 of official definitions, and then Joe, I think you have 0:01:04.400 --> 0:01:06.440 your own definition, So let me let me go through 0:01:06.480 --> 0:01:10.280 these quote unquote official definitions. These are from IBM, and 0:01:10.400 --> 0:01:12.800 IBM is one of those companies that has a lot 0:01:12.959 --> 0:01:16.520 invested in big data in general and big data management 0:01:16.920 --> 0:01:20.959 and so in a paper called Demystifying Big Data. Here's 0:01:21.000 --> 0:01:22.640 one of the definitions, which is big data is a 0:01:22.680 --> 0:01:26.199 phenomenon defined by the rapid acceleration and the expanding volume 0:01:26.240 --> 0:01:29.360 of high velocity, complex, and diverse types of data. Big 0:01:29.440 --> 0:01:34.360 data is often defined along three dimensions, volume, velocity, and variety. 0:01:34.480 --> 0:01:36.959 And then the other definition is big data as a 0:01:37.080 --> 0:01:40.480 term that describes large volumes of high velocity, complex and 0:01:40.520 --> 0:01:44.160 variable data that require advanced techniques and technologies to enable 0:01:44.200 --> 0:01:49.000 to capture, storage, distribution, management, and analysis of the information. So, Joe, 0:01:49.000 --> 0:01:51.480 what's your what's your definition? That was a lot of words. Yeah, 0:01:51.480 --> 0:01:54.880 it was, uh, well it just seems to me and 0:01:55.320 --> 0:01:58.880 I'm no expert here, but big what's the difference between 0:01:58.920 --> 0:02:01.680 just a lot of data to and then big data. 0:02:01.960 --> 0:02:04.920 That's a good question because we've had a lot of 0:02:05.000 --> 0:02:08.120 data before, but suddenly there's sort of this new paradigm 0:02:08.160 --> 0:02:10.400 where we have to think about, oh, it's big data. 0:02:10.520 --> 0:02:13.440 It's a separate thing. It's not just a matter of degrees. 0:02:13.800 --> 0:02:17.200 And I think it's the point at which our intuition 0:02:17.360 --> 0:02:20.800 kicks in and tells us something weird is going on, right, 0:02:20.840 --> 0:02:23.320 there's so much here to handle that I can't even 0:02:23.400 --> 0:02:28.040 imagine handling it. Yeah, it's it's it's when you suddenly realize, oh, 0:02:28.280 --> 0:02:32.240 hold onto your butts. The phrase was actually entered into 0:02:32.280 --> 0:02:35.600 the Oxford English Dictionary just this quarter, like this month. 0:02:35.639 --> 0:02:40.720 As of June, they entered it into their quarterly online 0:02:40.800 --> 0:02:43.880 update and and there there there's a definition that is 0:02:44.000 --> 0:02:46.000 very much like the one that Jonathan read off, but 0:02:46.120 --> 0:02:50.040 slightly more succinct. Um, well, it's the O E. D Well. 0:02:50.080 --> 0:02:53.200 They also include several several examples of it of its 0:02:53.320 --> 0:02:58.480 use um the original going back to when a social 0:02:58.520 --> 0:03:02.760 historian um by name of C. Tilly referenced it. But 0:03:02.840 --> 0:03:06.680 in terms of meaning being obscured when you are in 0:03:06.720 --> 0:03:12.200 the presence of an increasingly complex system of of of information. Right, 0:03:12.400 --> 0:03:14.680 so it has something to do with it's hard for 0:03:14.760 --> 0:03:19.040 us to grasp intuitively. Right. Well, well, yeah, it's kind 0:03:19.040 --> 0:03:21.080 of like, if you want to boil it down to 0:03:21.160 --> 0:03:24.160 a cliche, it's the whole not being able to see 0:03:24.160 --> 0:03:27.040 the forest for the trees. Like you're able to see 0:03:27.720 --> 0:03:30.639 the stuff that's immediately around you, but when you're trying 0:03:30.639 --> 0:03:33.320 to get a bigger picture, your perspective is blocked by 0:03:33.320 --> 0:03:36.120 the fact that there's just so much there. So if 0:03:36.160 --> 0:03:38.240 you can't get a good grasp on the big picture, 0:03:38.480 --> 0:03:40.560 so exactly how many trees are we talking about here, Well, 0:03:40.840 --> 0:03:43.240 let's let's boil that down. Let's boil that down to 0:03:43.360 --> 0:03:46.480 talking about how we measure data in the computer world. 0:03:46.480 --> 0:03:50.560 And for anyone who has any background in computers at all, 0:03:50.800 --> 0:03:52.840 this is probably going to seem super basic to you. 0:03:52.880 --> 0:03:54.840 But it's important to have the building blocks there for 0:03:54.920 --> 0:03:58.280 us to understand the enormity of big data. Right, if 0:03:58.280 --> 0:03:59.840 you go look on the internet, it seems like a 0:03:59.840 --> 0:04:03.760 lot of times people confuse terms like data and facts 0:04:04.000 --> 0:04:07.440 and information like they just use them interchangeably. Data and 0:04:07.480 --> 0:04:10.640 information are you know, of course that they are synonyms. 0:04:10.640 --> 0:04:14.040 But when we're talking about data in the terms of computation, 0:04:14.480 --> 0:04:17.480 and then we're talking about bites, right, So a bite 0:04:17.680 --> 0:04:22.080 is eight bits, and one bite can represent one character. 0:04:22.440 --> 0:04:24.840 So when I'm talking about character, I'm talking about like 0:04:24.880 --> 0:04:26.680 a letter or a number or a symbol. I'm not 0:04:26.720 --> 0:04:30.240 talking about Jean Valjean from La Misseraba. That's totally different 0:04:30.320 --> 0:04:33.559 kind of character. So eight bits can be one character, 0:04:33.600 --> 0:04:37.359 So it takes about ten or bytes, So eighty bits 0:04:37.400 --> 0:04:40.080 total ten bytes to make up about you know, one 0:04:40.160 --> 0:04:45.240 word or so. Uh So that's your basic unit of data. 0:04:45.640 --> 0:04:48.080 So if we look at kill a bite, that's we 0:04:48.080 --> 0:04:51.880 we say it's a thousand bytes. Technically it's one thousand, 0:04:51.960 --> 0:04:56.080 twenty four bytes. And in counting, yeah, this this gets 0:04:56.080 --> 0:04:59.480 a little, this gets a little complex as we go, 0:04:59.760 --> 0:05:03.240 high are up, so I will be rounding down to 0:05:03.320 --> 0:05:10.280 the nearest. Uh yeah, because otherwise, uh we'll be spending 0:05:10.279 --> 0:05:12.360 the entire podcast just listening to me read out an 0:05:12.360 --> 0:05:14.920 incredibly long number. I'll give an example of that when 0:05:14.920 --> 0:05:16.800 I get to it, but I'll skip over most of 0:05:16.800 --> 0:05:20.320 them anyway. So a kilobyte will rough says roughly a 0:05:20.360 --> 0:05:22.600 thousand bites. So if you were to type out a 0:05:22.600 --> 0:05:25.800 page of text, that would be about two kilobytes of information. 0:05:25.800 --> 0:05:28.640 Now that's if you're just typing text, not like images 0:05:28.720 --> 0:05:31.480 or anything else. But that would be about two kilobytes. Now, 0:05:31.480 --> 0:05:35.800 if you had a low res photo, that's probably around 0:05:35.839 --> 0:05:39.440 a hundred kilobytes, maybe fewer, depending upon the resolution. I 0:05:39.440 --> 0:05:41.360 mean that you know some things that if you save 0:05:41.400 --> 0:05:43.960 it for the web, it can be between like thirty 0:05:44.000 --> 0:05:47.080 and a hundred kilobytes or so. Uh, the next step 0:05:47.160 --> 0:05:51.839 up is megabyte. So that's technically one million, forty eight thousand, 0:05:51.920 --> 0:05:54.000 five D seventy six bites, but we'll usually just say 0:05:54.000 --> 0:05:55.960 one million. And that's the last I'm going to do 0:05:56.040 --> 0:05:59.560 of the specific numbers. Uh, so high res photo could 0:05:59.600 --> 0:06:02.800 be at east two megabytes. Five megabytes is enough to 0:06:02.839 --> 0:06:06.719 hold the complete works of Shakespeare. So now wait a minute, 0:06:06.760 --> 0:06:11.240 would that be plain text or formatted documents? It would 0:06:11.240 --> 0:06:14.000 be essentially plain text? Yeah, plain text would be about 0:06:14.040 --> 0:06:18.960 five megabytes Shakespeare. Yeah, now if you want to if 0:06:18.960 --> 0:06:23.520 you want there's there's all these bears pursuing you in 0:06:24.080 --> 0:06:27.040 Winter's Tale, so you got to fill that out. Uh. 0:06:27.240 --> 0:06:31.039 The a CD ROM, do you guys remember those? There's 0:06:31.040 --> 0:06:32.960 still they still exist. I have one in my computer 0:06:33.040 --> 0:06:36.280 in front of us, an optical drive. They round, they 0:06:36.320 --> 0:06:39.320 were uh you know, along one side, but if you 0:06:39.360 --> 0:06:42.120 looked at them in profile, they were flat. That's what 0:06:42.160 --> 0:06:45.200 you used to play missed right, Yes it was in fact, 0:06:45.480 --> 0:06:48.200 but a CD ROM could hold between about six fifty 0:06:48.279 --> 0:06:51.599 and nine hundred megabytes. And we're talking about the twelve 0:06:51.600 --> 0:06:55.040 centimeter discs, not the eight centimeter discs, because they had 0:06:55.040 --> 0:06:57.600 many discs as well. It wasn't as popular here in 0:06:57.600 --> 0:07:00.599 the United States, but in Asia they were very popular. 0:07:01.240 --> 0:07:05.000 Then you have gigabyte that ends up being the billion 0:07:05.040 --> 0:07:09.320 mark of bytes. Again, I'm just simplifying here about one 0:07:09.360 --> 0:07:13.200 gigabyte can hold a broadcast quality movie. By the way, 0:07:13.320 --> 0:07:15.920 I remember when I had a computer that had a 0:07:16.160 --> 0:07:20.560 I think three gigabyte hard drive, and that was so huge. Yeah. 0:07:20.560 --> 0:07:22.520 I remember when I got a two and fifty six 0:07:22.640 --> 0:07:24.920 megabyte hard drive and I thought that there's no way 0:07:24.960 --> 0:07:27.400 anyone could fill up that budget space. And look at 0:07:27.480 --> 0:07:31.480 what I know. A twenty gigabyte drive could hold the 0:07:31.560 --> 0:07:35.640 high fidelity recordings of the entire works of Beethoven. A 0:07:35.760 --> 0:07:38.800 fifty gigabyte hard drive is equivalent to a floor of 0:07:38.880 --> 0:07:42.760 books in a typical library. The next step up is terabyte, 0:07:42.960 --> 0:07:46.040 which is one trillion bytes. That two terabytes would be 0:07:46.080 --> 0:07:49.960 equivalent to an academic research library, and ten terabytes would 0:07:49.960 --> 0:07:53.600 be equivalent to all of the printed collection at the 0:07:53.720 --> 0:07:57.000 United States Library of Congress, just the printed materials. It's 0:07:57.040 --> 0:08:00.520 amazing how much cheaper the storage has been come over 0:08:00.560 --> 0:08:02.880 the years. What does it cost to go buy like 0:08:02.960 --> 0:08:06.240 a one tero byte external hard drate? It all depends 0:08:06.240 --> 0:08:08.800 on where you go, but you're talking around a hundred 0:08:09.080 --> 0:08:11.880 couple hundred dollars at most for most places. And and 0:08:11.920 --> 0:08:14.200 the thing is that you know, the the technology has 0:08:14.240 --> 0:08:18.080 improved over time and the manufacturing processes have improved over time, 0:08:18.240 --> 0:08:21.240 which has brought the price down over time. But we're 0:08:21.240 --> 0:08:24.080 not done yet. We gotta go back. So if you 0:08:24.520 --> 0:08:26.360 that was one trillion bites, let's say you one of 0:08:26.400 --> 0:08:29.600 the levels, let's say you want to quadrup Well, the 0:08:29.680 --> 0:08:32.280 numbers we're going to be talking about are way bigger 0:08:32.280 --> 0:08:36.000 than well, they don't know, we haven't said it yet. 0:08:36.040 --> 0:08:39.040 So next is A is one quadrillion bites. That's a 0:08:39.240 --> 0:08:42.360 that's a pedo byte and uh, to pedal bytes would 0:08:42.400 --> 0:08:46.880 be all US academic research libraries combined. Two hundred peda 0:08:46.920 --> 0:08:51.840 bytes would be all printed material everywhere. Uh. If you 0:08:51.880 --> 0:08:57.199 went up to one quintillion bites, I don't know how 0:08:57.200 --> 0:09:00.720 many zeros that is, it's a lot. Uh, that's exabyte 0:09:01.200 --> 0:09:04.360 and five exabytes would be enough to contain all the 0:09:04.400 --> 0:09:07.839 words ever spoken by human beings. If you were to 0:09:07.880 --> 0:09:11.120 break down all the words we've ever spoken into bites, 0:09:11.640 --> 0:09:14.559 it would fit within five exabytes. Did we say whose 0:09:14.640 --> 0:09:17.839 estimates these are? Oh? Well, these are actually estimates that 0:09:17.880 --> 0:09:21.840 are up all over the place there. You know. IBM 0:09:21.880 --> 0:09:27.920 actually cites these as their their benchmarks as well. Yeah, 0:09:28.000 --> 0:09:30.400 and then the next two levels up you want to 0:09:30.400 --> 0:09:33.240 go up. So we've done quintillions, So sextillion would be 0:09:33.360 --> 0:09:37.040 zeta bite and I guess septilian would be YadA bite. 0:09:37.640 --> 0:09:43.800 So anyway, or yakta YadA, not yoda, not yoda, not 0:09:44.000 --> 0:09:50.520 yakta not yakta YadA. So so anyway, those those are 0:09:50.520 --> 0:09:52.760 the scales exabite. We get to the point where it's 0:09:52.760 --> 0:09:55.480 all the words we've ever spoken, right, So that gives 0:09:55.520 --> 0:09:59.000 you the idea of of the basics of bites and 0:09:59.120 --> 0:10:03.560 how much information is equivalent to you know, various real 0:10:03.600 --> 0:10:06.680 world examples. So let's talk about the amount of information 0:10:07.120 --> 0:10:11.600 that we create on a daily basis. Soil for example, 0:10:11.600 --> 0:10:18.200 of corey DIABM, we create about twelve terabytes of tweets 0:10:18.240 --> 0:10:22.840 in one day. That's twelve trillion bytes of messages at 0:10:22.840 --> 0:10:26.040 a hundred forty characters or fewer tweets alone. And Twitter 0:10:26.120 --> 0:10:29.680 isn't even by far the most popular social networking No 0:10:30.160 --> 0:10:32.640 is not is not the most popular at all. And 0:10:32.720 --> 0:10:35.760 yet that's not even rich text isn't. That's just plain text. 0:10:35.800 --> 0:10:38.760 It's plain texts short messaging service really is what it is. 0:10:38.920 --> 0:10:42.080 So so remember too, terabytes is equivalent to a single 0:10:42.160 --> 0:10:45.240 academic research library. So you've got the equivalent of six 0:10:45.360 --> 0:10:50.480 research libraries academic research libraries, just in tweets alone. Now, 0:10:50.520 --> 0:10:52.440 I'm not saying that you're going to be able to 0:10:52.760 --> 0:10:58.320 research your next uh thesis only using Twitter. It's not 0:10:58.400 --> 0:11:01.439 necessarily useful data that's out there, but that's how much, 0:11:01.720 --> 0:11:04.080 you know, once you put it together. Lots of people 0:11:04.120 --> 0:11:08.280 are are tracking trends and keywords and and you know, 0:11:08.600 --> 0:11:11.960 sure all kinds of things. There are lots of useful 0:11:12.480 --> 0:11:15.600 Twitter to predict the stock market and earthquakes, and they're 0:11:15.679 --> 0:11:19.160 using Twitter for all sorts of stuff. Yeah, consumer behavior, 0:11:19.840 --> 0:11:22.480 Not that we're endorsing those ideas, by the way, but 0:11:24.000 --> 0:11:26.079 going back to the kind of data that we actually 0:11:26.080 --> 0:11:28.520 are producing on a daily basis. Back in two thousand twelve, 0:11:28.600 --> 0:11:32.880 when Facebook still had just under one billion registered users, 0:11:33.520 --> 0:11:36.360 they were collecting According to Facebook, they did an earnings 0:11:36.400 --> 0:11:39.319 call where they said they were collecting about five hundred 0:11:39.480 --> 0:11:44.120 terabytes per day from users. So that's all the stuff 0:11:44.160 --> 0:11:47.200 that everyone is doing on Facebook, whether they are posting 0:11:47.200 --> 0:11:50.680 a status update or liking a page or sharing a link. 0:11:51.200 --> 0:11:54.600 All of that was folded into this number. But five 0:11:54.840 --> 0:12:00.640 hundred terabytes every day. So then you've got YouTube. You guys, 0:12:00.720 --> 0:12:04.000 of course have heard the famous UH stat Now it 0:12:04.120 --> 0:12:07.640 is one hundred hours of video that's uploaded every minute 0:12:09.880 --> 0:12:12.080 between the time we shot that video and by the 0:12:12.080 --> 0:12:14.319 time we but between the time we wrote it and 0:12:14.480 --> 0:12:17.400 the time we shot, really time we shot in the 0:12:17.440 --> 0:12:20.079 time it published exactly. Yeah, it went from seventy two. 0:12:21.240 --> 0:12:23.760 Technically it had been growing all that time, but Google 0:12:23.880 --> 0:12:26.679 gave the official announcement. Yeah, so one hundred hours of 0:12:26.720 --> 0:12:29.440 YouTube footage is upload every minute. So that means that 0:12:29.920 --> 0:12:31.679 in a single day you get about a hundred and 0:12:31.760 --> 0:12:37.960 forty four thousand hours of video on YouTube added every day. 0:12:38.000 --> 0:12:44.000 That's sick. It's a little it's a little stomach churning actually. Um. 0:12:44.080 --> 0:12:45.880 And so if you were to look at all the 0:12:45.960 --> 0:12:49.760 data that we are creating, and this is beyond social media, 0:12:49.800 --> 0:12:52.560 we're talking about all the information being created not just 0:12:52.600 --> 0:12:55.280 by human beings, but by things like sensors that are 0:12:55.600 --> 0:12:58.079 connected to computer systems and are sending that data in 0:12:58.200 --> 0:13:03.000 So things like whether uh sensors or traffic sensors, the 0:13:03.080 --> 0:13:05.080 cameras we've talked about in the past. All of this 0:13:05.160 --> 0:13:11.080 stuff combined generates about two point five quintillion bytes of 0:13:11.200 --> 0:13:15.920 data also known as XA bytes every single day. Well, 0:13:15.960 --> 0:13:19.760 that makes me curious, does more data come from human 0:13:19.880 --> 0:13:24.120 entry or from other machines? Right now, humans are actually 0:13:24.120 --> 0:13:27.960 generating most of that data. Eight percent of it, in fact, 0:13:28.160 --> 0:13:33.960 is coming from unstructured information, which includes things like email, video, blogs, uh, 0:13:34.080 --> 0:13:38.439 social media, call center conversations. All of this goes into 0:13:38.480 --> 0:13:43.400 that that data that's going to change. Probably if you 0:13:43.440 --> 0:13:45.640 remember our Internet of Things episode, we talked a lot 0:13:45.640 --> 0:13:47.559 about how we're going to be living in this world 0:13:47.640 --> 0:13:50.920 where our environments are going to be constantly collecting Yeah, 0:13:50.960 --> 0:13:53.880 the more devices are collecting information about us and yeah, 0:13:53.679 --> 0:13:55.520 yeah it Well, it'll it'll get to a point where 0:13:55.559 --> 0:13:59.520 we'll start to see that number probably uh fluctuate quite 0:13:59.559 --> 0:14:03.280 a bit. We'll see that percentage drop for human produced 0:14:03.400 --> 0:14:08.720 data versus you know, automated sensors. So two point five 0:14:08.800 --> 0:14:11.920 quintillion bytes of data produced every day, that means every 0:14:11.960 --> 0:14:15.440 two days we produce as much information as all the 0:14:15.480 --> 0:14:19.160 words we've ever spoken. So that's kind of incredible and also, 0:14:19.280 --> 0:14:23.080 according to ibm UH, in the last two years, we 0:14:23.200 --> 0:14:27.800 have produced of all the world's data, meaning that everything 0:14:27.960 --> 0:14:33.280 prior to those two years represents of all the data. Ever, 0:14:33.480 --> 0:14:36.680 it's a it's a pretty crazy, crazily curved graph as 0:14:36.720 --> 0:14:40.000 of So, so, you know, years ago at this point, 0:14:40.480 --> 0:14:42.360 Eric Schmidt of Google said that we were creating as 0:14:42.440 --> 0:14:45.160 much info every two days as as we had from 0:14:45.200 --> 0:14:49.320 the dawn of human kind up through two thousand three. Yeah, 0:14:49.760 --> 0:14:52.840 so in one point we created one point eight zeta 0:14:52.920 --> 0:14:57.280 bytes of information globally UH in that year, and that 0:14:57.320 --> 0:15:02.080 amount is expected to double every year. So that means 0:15:02.720 --> 0:15:05.120 one point a zeta bites. If you want to know, like, okay, 0:15:05.120 --> 0:15:06.680 well what does that mean to me? Like, how how 0:15:06.720 --> 0:15:12.600 can I conceptualize this amount of information? That's equivalent to 0:15:13.800 --> 0:15:19.920 two hundred billion two our HD movies? And if you 0:15:19.920 --> 0:15:23.520 wanted to watch those two hundred billion HD movies and 0:15:23.600 --> 0:15:26.040 just sit down and have a marathon, it would take 0:15:26.080 --> 0:15:33.320 you forty seven million years to do it, no bathroom breaks. Yeah, 0:15:33.480 --> 0:15:36.040 that that those these these numbers don't even make any 0:15:36.200 --> 0:15:39.040 sense to me at that point. It's just you know, 0:15:39.200 --> 0:15:41.800 and that's and that's the that's the key, right that 0:15:41.800 --> 0:15:44.160 that's why we call it big data, because they are 0:15:44.400 --> 0:15:47.160 such huge numbers that when you think about you're like, 0:15:48.120 --> 0:15:51.160 what what can I even do with all this information? 0:15:51.200 --> 0:15:53.560 How can I make use of it? But that's exactly 0:15:53.600 --> 0:15:55.960 the thing, right, We're going to a point where it's 0:15:56.000 --> 0:15:58.240 not so much that you're going to do something with it, 0:15:58.560 --> 0:16:01.240 but the machines are going to do something right And 0:16:01.240 --> 0:16:04.760 and there's some very creative processes that people have come 0:16:04.840 --> 0:16:08.440 up with that break this down into more manageable problems 0:16:08.520 --> 0:16:11.200 that machines can handle. Well, I shouldn't say that you're 0:16:11.240 --> 0:16:12.960 not going to use it for sure. I mean, this 0:16:13.040 --> 0:16:15.440 kind of thing is probably really useful to people who 0:16:15.440 --> 0:16:18.520 are involved in, say marketing. Well, it's useful for marketing, 0:16:18.560 --> 0:16:21.800 but it's also and we'll do yeah, well, we'll do 0:16:21.840 --> 0:16:26.680 a full episode on some of the applications of using data. Yeah, 0:16:27.000 --> 0:16:28.560 that will be our next episode that we recorded. Me 0:16:30.000 --> 0:16:33.680 But but yeah, there here's an example that all of 0:16:33.760 --> 0:16:37.160 us in this room could use UH with big data. 0:16:37.200 --> 0:16:39.840 Now we would not be actually doing the analyzing ourselves, 0:16:40.160 --> 0:16:43.560 but we would benefit from the UH, the the the 0:16:43.600 --> 0:16:47.400 actual work, and that would be traffic, real time traffic 0:16:47.440 --> 0:16:51.600 reports So if you are using some sort of GPS 0:16:51.640 --> 0:16:56.480 that allows you to get incoming traffic information that's being 0:16:56.520 --> 0:16:59.960 gathered by various means, and there are different ways of 0:17:00.040 --> 0:17:02.960 doing it depending upon what system you're using, then you 0:17:03.000 --> 0:17:07.800 are essentially benefiting from the from the analysis of big 0:17:07.920 --> 0:17:11.840 data because it's taking all this information about the actual 0:17:11.920 --> 0:17:15.440 environment around you that's gathered from multiple sources and helping 0:17:15.480 --> 0:17:18.760 you route the most efficient way. Right, does it dynamic routing? 0:17:18.800 --> 0:17:22.640 Which is really a cool thing and and and obvious benefit. 0:17:22.880 --> 0:17:27.879 But that's just one application. So how do you end 0:17:27.960 --> 0:17:31.879 up navigating this much information? Like what what's the what's 0:17:31.920 --> 0:17:34.600 the magic key to it? And there are actually a 0:17:34.640 --> 0:17:36.600 couple different ways. I don't want to I don't want 0:17:36.600 --> 0:17:38.480 to throw my co host under the bus because I 0:17:38.520 --> 0:17:41.280 specifically look this stuff up. And so I ask the question. 0:17:41.280 --> 0:17:45.000 They're like, I, uh, it's a lot throw a dart. 0:17:45.160 --> 0:17:53.119 I don't know. Um, you use computers, that's good, Joe, supercomputers. 0:17:53.200 --> 0:17:56.359 Perhaps you could use supercomputers. You could also use grid computing, 0:17:56.359 --> 0:17:58.520 which is where you end up using a lot of 0:17:58.560 --> 0:18:02.120 computers to work on a single problem. So you guys 0:18:02.160 --> 0:18:06.440 have probably heard the term parallel processing, parallel parallel processing 0:18:06.520 --> 0:18:08.800 is is an idea where you are able to take 0:18:08.880 --> 0:18:12.400 certain kinds of computer problems where you can divide up 0:18:12.400 --> 0:18:16.879 the problem into different UH sections, maybe even subsections of data, 0:18:17.320 --> 0:18:21.160 and then assign each of those parts to a different processor. 0:18:21.520 --> 0:18:24.280 And this this could happen within a computer, or you 0:18:24.280 --> 0:18:28.080 could be talking about spreading it out lots of different computers. 0:18:28.119 --> 0:18:30.080 So if you have a computer that has multiple cores, 0:18:30.200 --> 0:18:33.359 for example a multi core processor, each core could be 0:18:33.400 --> 0:18:36.680 taking part of this problem and working on it separately. 0:18:37.200 --> 0:18:39.520 And really what you have is you have you essentially 0:18:39.600 --> 0:18:44.160 have one unit acting as the director, and the director's 0:18:44.240 --> 0:18:46.560 job is to take the problem and to divide it 0:18:46.680 --> 0:18:50.000 up into manageable chunks, and then to parcel that out 0:18:50.080 --> 0:18:53.120 to all the other elements of the system, whether it's 0:18:53.119 --> 0:18:56.840 other computers or other processors or other cores. Their job 0:18:56.920 --> 0:18:59.400 is to work on that particular part of the problem 0:18:59.680 --> 0:19:03.320 and then send the results back to the master. The 0:19:03.359 --> 0:19:06.000 master then takes all the results and has a collective 0:19:06.000 --> 0:19:10.080 result as as the final product, and that's where you 0:19:10.119 --> 0:19:13.639 get the answer that you're looking for UH. It's called 0:19:14.119 --> 0:19:17.119 often this this approach is called a map reduce framework. 0:19:17.200 --> 0:19:19.720 You're mapping out the problem and then routing it out 0:19:20.200 --> 0:19:23.240 and then you reduce the problem that way, when all 0:19:23.280 --> 0:19:25.840 the uh and when all the answers are sent back 0:19:25.880 --> 0:19:28.200 to you, you you reduce all those answers into one answer. 0:19:28.600 --> 0:19:32.200 So that's the whole process for taking generally a huge 0:19:32.240 --> 0:19:35.400 problem and making it manageable. Now, the key to all 0:19:35.440 --> 0:19:39.000 of this, and a lot of people and companies that 0:19:39.040 --> 0:19:42.439 specialize in big data will tell everyone this is that 0:19:42.480 --> 0:19:45.199 you can't just do anything with this information. What you 0:19:45.320 --> 0:19:48.120 have to do is decide what is it that you 0:19:48.200 --> 0:19:50.760 want to do with the information, specific thing are we 0:19:50.880 --> 0:19:54.120 looking for, right, and then you build a system that 0:19:54.200 --> 0:19:57.520 lets you get that from derive that from all the 0:19:57.600 --> 0:20:00.520 data you have. So it's not like you just look 0:20:00.560 --> 0:20:02.920 at it. Yeah, you can't just look at this big 0:20:02.960 --> 0:20:06.640 ball of zeros and wines and then just magically draw 0:20:06.720 --> 0:20:09.320 out the information you need. What you do and or 0:20:09.359 --> 0:20:11.160 you just sit there and you look at and say, 0:20:11.400 --> 0:20:13.840 you know, what can we learn from this? If you 0:20:13.880 --> 0:20:15.720 look for a specific kind of pattern, like if you're 0:20:15.760 --> 0:20:18.000 looking for a needle in a haystack, you're you're looking 0:20:18.000 --> 0:20:21.199 for something shiny, for example, and and you know, if 0:20:21.240 --> 0:20:23.520 you if you just go like, yeah, if you just say, well, 0:20:23.520 --> 0:20:26.000 I'm looking for something kind of pointy and short. Then 0:20:26.040 --> 0:20:29.080 that's not What is interesting is that when you get 0:20:29.160 --> 0:20:32.199 information on this scale, this huge amount of information, you 0:20:32.200 --> 0:20:35.080 can actually start to recognize patterns that otherwise would have 0:20:35.119 --> 0:20:38.720 been completely Yeah, you would never have been able to Again, 0:20:38.720 --> 0:20:40.920 it's the forest for the trees thing. You would never 0:20:40.920 --> 0:20:43.119 have been able to have seen the forest because you 0:20:43.160 --> 0:20:45.119 were right there in the middle of all those trees. 0:20:45.560 --> 0:20:47.440 So the same sort of thing, you'd be able to 0:20:47.440 --> 0:20:50.440 see these big patterns that happen. And that's where especially 0:20:50.520 --> 0:20:53.520 things like marketing ends up being a big deal, because 0:20:53.920 --> 0:20:57.920 you can see things like tendencies for customers to behave 0:20:57.960 --> 0:20:59.679 in a certain way, and if you want them to 0:20:59.720 --> 0:21:01.959 behave of a particular way, you can start to focus 0:21:02.000 --> 0:21:04.000 on things that kind of guide them in that direction. 0:21:04.040 --> 0:21:07.639 But I've even seen bizarre representations a thing online that 0:21:07.800 --> 0:21:10.560 was the strange corn ucopia shape and it was just 0:21:10.720 --> 0:21:14.680 labeled like the geometry of big data. Yeah, it's it's 0:21:14.680 --> 0:21:17.760 a little like again, when we're talking about something so 0:21:17.880 --> 0:21:20.479 huge that it's difficult for us to get a mental 0:21:20.560 --> 0:21:25.080 grasp on it, trying to find a representation of that 0:21:25.080 --> 0:21:28.800 that makes sense. To us is something of an uphill 0:21:28.840 --> 0:21:31.919 battle it. You know, it's a lot of people have tried, 0:21:32.400 --> 0:21:36.639 but it's really difficult to make this and make it 0:21:36.760 --> 0:21:39.280 understandable in a way that doesn't just blow out the 0:21:39.320 --> 0:21:43.160 scale immediately, where you know, you have the manageable amount 0:21:43.200 --> 0:21:46.280 of data and then the spike just goes all the 0:21:46.280 --> 0:21:48.199 way up through the top of the graph and you 0:21:48.240 --> 0:21:52.280 can't see the topic. That's so it's really big. Other 0:21:52.359 --> 0:21:55.240 ways to handle all that data, there's also uh the 0:21:55.280 --> 0:21:58.200 approach of doing just real time analytics and streaming of data. 0:21:58.640 --> 0:22:00.480 In this case, this would be kind of like traffic 0:22:00.520 --> 0:22:04.240 example I gave earlier. So with traffic, you have all 0:22:04.240 --> 0:22:07.480 these sensors gathering data, and then you have uh that 0:22:07.880 --> 0:22:11.360 analysis and streaming of the data happening immediately, and then 0:22:11.400 --> 0:22:14.800 you get the results. UH. In this case, you don't 0:22:14.840 --> 0:22:17.639 have to worry so much about storing lots of data 0:22:17.760 --> 0:22:21.960 because it doesn't really matter if there was a slow 0:22:22.080 --> 0:22:25.600 spot on the highway two hours ago. What matters is 0:22:25.600 --> 0:22:28.119 what's going on right now, So you don't have to 0:22:28.160 --> 0:22:31.800 worry about building these huge data centers to store all 0:22:31.800 --> 0:22:34.240 that information. You just have to build a system that's 0:22:34.280 --> 0:22:38.200 large enough to handle incoming information and give outgoing information. 0:22:38.200 --> 0:22:41.320 So you have to have a good input output basis. Uh, 0:22:41.440 --> 0:22:43.960 that's what's important in those types of systems. Now, in 0:22:44.040 --> 0:22:46.119 the other types of systems where you are collecting and 0:22:46.160 --> 0:22:48.879 analyzing enormous amounts of information, you have to have a 0:22:48.880 --> 0:22:51.360 place for that information to live, and that's where storage 0:22:51.359 --> 0:22:54.560 comes into play. And that's where we see these enormous 0:22:54.720 --> 0:22:58.680 data centers things that buildings are specifically made to house 0:22:59.240 --> 0:23:02.200