WEBVTT - Magic Leap and the Big Fall 0:00:04.400 --> 0:00:07.800 Welcome to text Stuff, a production from I Heart Radio. 0:00:12.200 --> 0:00:14.960 Hey there, and welcome to tech Stuff. I'm your host, 0:00:15.200 --> 0:00:18.600 Jonathan Strickland. I'm an executive producer with I Heart Radio 0:00:18.720 --> 0:00:22.200 and I love all things tech and Back in the 0:00:22.320 --> 0:00:26.720 nineties nineties, when Nirvana was teaching us how to have 0:00:26.880 --> 0:00:32.320 all the fields, virtual reality appeared ready to transform the world. 0:00:32.640 --> 0:00:36.000 The hype for the young technology was reaching peak levels. 0:00:36.320 --> 0:00:40.240 Hollywood made movies that incorporated the idea, suggesting that we 0:00:40.280 --> 0:00:43.599 would soon be able to jump into computer generated worlds 0:00:43.600 --> 0:00:47.520 and do everything from navigate file systems in three dimensions 0:00:48.120 --> 0:00:52.440 exciting I know, to taking off like Superman. But as 0:00:52.479 --> 0:00:55.880 people got a chance to actually try out virtual reality 0:00:55.920 --> 0:00:59.760 set ups in the nineteen nineties, they became disillusioned. The 0:00:59.800 --> 0:01:02.600 tech chnology just couldn't live up to the hype. You 0:01:02.680 --> 0:01:06.760 could get an immersive experience, but the graphics were primitive. 0:01:07.000 --> 0:01:09.280 The headsets were typically so heavy they had to be 0:01:09.319 --> 0:01:13.759 suspended from the ceiling with cables. Your movement was limited, 0:01:14.040 --> 0:01:16.560 and system latency caused a lot of folks to get 0:01:16.600 --> 0:01:21.120 motion sickness while trying one out. Interest in VR plummeted, 0:01:21.400 --> 0:01:25.199 and the industry found itself the proverbial redheaded step child 0:01:25.360 --> 0:01:28.360 of the tech world as well as the investment world. 0:01:28.840 --> 0:01:31.679 A lot of people continued to work on VR hardware 0:01:31.720 --> 0:01:34.959 and software, but it was much more challenging to get funding. 0:01:35.240 --> 0:01:38.560 Progress was slow. It typically meant that you were appropriating 0:01:38.640 --> 0:01:42.080 stuff from other industries like the video game world or 0:01:42.200 --> 0:01:46.520 smartphones in order to get your stuff done. Thirty years later, 0:01:46.680 --> 0:01:51.840 we're seeing something similar with dedicated augmented reality headsets. Magic 0:01:51.920 --> 0:01:54.800 Leap a company that at one point was so exciting 0:01:55.120 --> 0:02:00.960 that it raised around three billion dollars that's billion with 0:02:01.240 --> 0:02:04.640 a B. Much of that money, by the way, was 0:02:04.760 --> 0:02:08.320 raised years before the company had anything significant to show off. 0:02:08.919 --> 0:02:11.120 It seemed to be on a pretty similar path as 0:02:11.200 --> 0:02:14.360 VR back in the nineties, and sure enough now in 0:02:14.400 --> 0:02:18.079 twenty twenty, the company scrambled to secure a three fifty 0:02:18.160 --> 0:02:22.920 million dollar lifeline investment after laying off about one thousand 0:02:22.919 --> 0:02:27.200 workers plus the founder of the company has stepped down. 0:02:27.560 --> 0:02:31.160 Today we're going to explore Magic Leaps story, and while 0:02:31.720 --> 0:02:34.600 like a Leap, there's a rise and a fall, this 0:02:34.680 --> 0:02:38.639 story is not yet over. Magic Leap could potentially recover, 0:02:39.040 --> 0:02:41.960 or they might fade into obscurity, or it may just 0:02:42.080 --> 0:02:45.520 become a company lesser than what it was intended to be. 0:02:46.120 --> 0:02:48.520 But before we get into the history of the company, 0:02:48.560 --> 0:02:51.320 it is good for us to define terms like virtual 0:02:51.440 --> 0:02:55.000 reality and augmented reality, both of which fall into a 0:02:55.080 --> 0:03:00.000 spectrum of technologies that a lot of folks call mixed reality. 0:02:59.840 --> 0:03:01.960 And it all has to do with a combination of 0:03:02.120 --> 0:03:07.480 real world experiences and computer generated experiences and meshing them 0:03:07.520 --> 0:03:10.919 together in some way. So if you think of mixed 0:03:10.960 --> 0:03:14.920 reality as a spectrum, on one extreme end, you've got 0:03:15.000 --> 0:03:19.600 pure virtual reality. In VR, computer is providing the sites 0:03:19.960 --> 0:03:23.960 and sounds, and theoretically it could provide other sensations as well, 0:03:24.160 --> 0:03:29.239 including touch through haptic feedback peripherals, or scent through various 0:03:29.320 --> 0:03:33.560 perfumes or oils and dispensers, though not many VR systems 0:03:33.560 --> 0:03:36.920 go as far as replicating smells, and very few, as 0:03:36.920 --> 0:03:39.840 far as I know, replicate taste. But the idea is 0:03:39.840 --> 0:03:44.640 that the computer generated sensory input is substituting for what 0:03:44.720 --> 0:03:47.800 we would see in our real world. So instead of 0:03:47.800 --> 0:03:51.520 seeing a boring office or a retail space in a 0:03:51.600 --> 0:03:54.560 shopping mall, as was the case when I first experienced 0:03:54.600 --> 0:03:58.200 VR back in the ES, we would be transported into 0:03:58.280 --> 0:04:03.160 say a rainforest, or a pirate ship, or a fantasy realm, or, 0:04:03.440 --> 0:04:06.800 as was my case, a bunch of vector graphics of 0:04:07.240 --> 0:04:11.680 really blocky tero dactyls flying around. Using a control system, 0:04:11.880 --> 0:04:15.360 we can navigate these virtual environments and we can interact 0:04:15.480 --> 0:04:19.600 with them. A true VR experience should be both interactive 0:04:19.920 --> 0:04:23.400 and immersive, and not all VR is created equal. Some 0:04:23.520 --> 0:04:27.000 VR experiences are fairly light touch, with only a bit 0:04:27.040 --> 0:04:31.360 of interactivity and primitive graphics. Typically, the hardware for VR 0:04:31.440 --> 0:04:35.000 includes a head mounted display so that your vision focuses 0:04:35.080 --> 0:04:38.280 only on the world that's created by the computer, and 0:04:38.360 --> 0:04:41.840 some headphones, plus some sort of control system. There may 0:04:41.880 --> 0:04:45.280 also be external cameras to help track your motion and 0:04:45.360 --> 0:04:48.839 translate that into what you experience in the virtual environment, 0:04:48.920 --> 0:04:51.760 giving you a little more nuance to your actions. But 0:04:51.839 --> 0:04:54.280 in the end, the thing to remember is that the 0:04:54.320 --> 0:04:57.960 stuff you are experiencing through your senses is in large 0:04:58.000 --> 0:05:03.119 part computer generated. Augmented reality is closer to the other 0:05:03.360 --> 0:05:07.200 end of the spectrum of mixed reality. With a R, 0:05:07.760 --> 0:05:10.599 you have a view of the real world through some 0:05:10.640 --> 0:05:13.600 sort of display. It could be a head mounted display 0:05:13.680 --> 0:05:15.920 like VR, or it could be something as simple as 0:05:15.960 --> 0:05:19.680 a smartphone or handheld game system. But whatever it is, 0:05:19.880 --> 0:05:22.719 it has to be able to overlay digital information on 0:05:22.839 --> 0:05:26.479 top of your view of the real world. So let's 0:05:26.520 --> 0:05:29.839 imagine a pair of glasses that can display information on 0:05:29.920 --> 0:05:32.640 the inside of the lenses so that you can see it. 0:05:33.000 --> 0:05:36.040 The lenses themselves are transparent, so you are looking at 0:05:36.040 --> 0:05:38.240 the world as if you were just wearing a normal 0:05:38.279 --> 0:05:40.960 pair of glasses. And maybe you put on a pair 0:05:41.040 --> 0:05:44.400 while you're walking in an unfamiliar city, and the glasses 0:05:44.400 --> 0:05:47.960 start to display information on the inside of those lenses 0:05:48.360 --> 0:05:51.520 about the area around you. Maybe they provide you directions 0:05:51.640 --> 0:05:55.240 so you can get to a specific destination. With augmented reality, 0:05:55.560 --> 0:05:58.760 the whole idea is that a computer system isn't replacing 0:05:58.960 --> 0:06:02.760 your real world ex experience, but instead it's enhancing or 0:06:02.920 --> 0:06:07.680 augmenting your experience in the real world. Between true VR 0:06:07.839 --> 0:06:11.920 and a R is a whole band of mixed reality technologies. 0:06:12.360 --> 0:06:15.960 Some are headsets allow wearers to have experiences in which 0:06:16.040 --> 0:06:20.640 the display they use inserts visual elements into their physical space. 0:06:21.080 --> 0:06:24.240 So you could wear a mixed reality headset, look at 0:06:24.240 --> 0:06:26.720 the room around you and watch as a video game 0:06:26.800 --> 0:06:30.440 character like I don't know, like Mario or Master chief 0:06:30.520 --> 0:06:34.320 or crash Bandicoot just walks right into the room. A 0:06:34.400 --> 0:06:38.560 tabletop might become a virtual battleground for a little toy armies. 0:06:38.839 --> 0:06:41.640 A wall could become a window into a fantasy realm. 0:06:41.680 --> 0:06:46.520 The possibilities are limited only by computer processors, the software, 0:06:46.640 --> 0:06:50.880 and the imagination of developers. VR and a R share 0:06:51.000 --> 0:06:53.920 a lot of history, and a lot of tech engineers 0:06:53.920 --> 0:06:57.200 have repurposed technology intended for one and used it for 0:06:57.240 --> 0:06:59.799 the other. And I think it's safe to say that most, 0:07:00.080 --> 0:07:04.000 but not all VR implementations have a head mounted display 0:07:04.040 --> 0:07:06.960 that can track where someone is looking and then adjust 0:07:07.000 --> 0:07:09.720 the point of view in the virtual environment accordingly. So 0:07:09.760 --> 0:07:13.120 that way, if you turn your head left, your virtual 0:07:13.160 --> 0:07:16.680 point of view also turns left. And that's incredibly important 0:07:16.720 --> 0:07:20.040 if you want an immersive experience. Just imagine how disorienting 0:07:20.040 --> 0:07:22.240 it would be if you put on a headset and 0:07:22.320 --> 0:07:24.640 you're looking at a video game world and you turn 0:07:24.720 --> 0:07:27.120 your head to look up at the sky, but your 0:07:27.200 --> 0:07:30.520 view looks down at your feet. That would really throw 0:07:30.560 --> 0:07:34.760 you off. Likewise, not all a OUR implementations have a 0:07:34.880 --> 0:07:38.920 head mounted display, but many do. These displays have to 0:07:38.920 --> 0:07:41.440 do something a little different from VR. They have to 0:07:41.480 --> 0:07:44.640 present a view of the real world. Now, they either 0:07:44.760 --> 0:07:48.520 do that through a transparent view screen or through a 0:07:48.640 --> 0:07:52.320 video screen that's got a video feed of the real world. 0:07:52.400 --> 0:07:54.320 So in other words, that screen has to be paired 0:07:54.360 --> 0:07:58.080 with a camera that's feeding a live image to the screen. 0:07:58.600 --> 0:08:01.560 To do that, thee has to be able to analyze 0:08:01.920 --> 0:08:06.520 what someone is looking at in order to overlay relevant information. Right, 0:08:06.640 --> 0:08:09.440 it doesn't do any good if you just get random information. 0:08:09.760 --> 0:08:11.520 I mean, if you were to wear an a R 0:08:11.640 --> 0:08:14.920 headset and you're walking through Paris, but then the headset 0:08:14.960 --> 0:08:17.320 starts to giving you directions as if you're on a 0:08:17.400 --> 0:08:21.960 walk in Moscow, that would be less than useless. So 0:08:22.200 --> 0:08:26.240 the tech needs sophisticated code to interpret information. That could 0:08:26.280 --> 0:08:32.280 include things like image recognition, compass directions, GPS, coordinates, your orientation, 0:08:32.400 --> 0:08:36.560 and more. All of that is super complicated, and it 0:08:36.559 --> 0:08:39.240 goes a bit beyond our story, but it's a good 0:08:39.280 --> 0:08:42.680 thing to remember that to pull off either convincing VR 0:08:42.880 --> 0:08:45.960 or a really useful a R experience, you need a 0:08:46.000 --> 0:08:49.559 lot of sophisticated code, and typically you need a fairly 0:08:49.640 --> 0:08:53.559 powerful computer system running in the background, whether it's on 0:08:53.640 --> 0:08:57.200 board in some sort of device you're wearing or tethered 0:08:57.440 --> 0:09:01.720 to a smartphone or tethered to a PC. VR and 0:09:01.800 --> 0:09:05.360 a R are not interchangeable, but they are close cousins. 0:09:05.600 --> 0:09:09.120 Augmented reality is probably the more productive member of the family. 0:09:09.760 --> 0:09:13.560 Many but not all, of virtual realities applications relate to 0:09:13.880 --> 0:09:17.760 entertainment in some way, and many but not all of 0:09:17.800 --> 0:09:22.319 augmented realities applications relate to productivity. Okay, that's a good 0:09:22.440 --> 0:09:26.680 overview of two of the big subcategories of mixed reality. 0:09:26.720 --> 0:09:29.360 Though I should point out there are many other aspects 0:09:29.360 --> 0:09:32.120 of mixed reality that I've not covered, but that could 0:09:32.120 --> 0:09:35.520 be its own episode. Now let's move on to Magic Leap. 0:09:35.920 --> 0:09:39.880 The company's founder is Ronnie Abovitz. Abovitz was born in Ohio. 0:09:40.240 --> 0:09:42.400 His family moved to Florida when he was a kid, 0:09:42.440 --> 0:09:45.240 and he went to the University of Miami, where he 0:09:45.400 --> 0:09:49.520 ultimately earned a master's degree in biomedical engineering. So I 0:09:49.559 --> 0:09:53.240 think it's safe to say he is pretty darn sharp. 0:09:53.760 --> 0:09:57.480 He earned his master's in nine and the following year 0:09:57.520 --> 0:10:01.280 he would co found his first company, e CAT that's 0:10:01.320 --> 0:10:05.440 a Z hyphen k a t Z. CAT was a 0:10:05.480 --> 0:10:08.839 medical technology company. The focus of the company was to 0:10:08.880 --> 0:10:14.280 develop technologies to assist in minimally invasive surgical procedures, which 0:10:14.559 --> 0:10:18.600 have numerous benefits including a reduced risk of infection and 0:10:18.720 --> 0:10:23.360 faster recovery times. The company developed robotic tools that surgeons 0:10:23.360 --> 0:10:28.400 could control to make precise, delicate incisions. The control systems 0:10:28.400 --> 0:10:32.040 could translate surgeon commands into steady and small motions with 0:10:32.080 --> 0:10:35.760 the robotic arm removing jitter and other muscular movements that 0:10:35.800 --> 0:10:38.600 would make these kind of procedures difficult or impossible with 0:10:38.640 --> 0:10:41.880 that level of precision. In two thousand four, the company 0:10:41.960 --> 0:10:44.960 chose to spin off its robotics division into a new 0:10:45.120 --> 0:10:49.679 entity called Mako Surgical Corporation. And the logo for Mako 0:10:49.800 --> 0:10:52.920 was super cool too, reminiscent of a shark's fin makos 0:10:52.960 --> 0:10:56.079 a type of shark, and as a side note, the 0:10:56.160 --> 0:11:00.240 Magic Leap logo is reminiscent of a leaping whale, So 0:11:00.360 --> 0:11:03.680 there's a very marine theme going with Abovitz's work. So 0:11:03.800 --> 0:11:06.760 Abavits served not just as a co founder of Mako, 0:11:06.800 --> 0:11:10.880 but the chief technology officer and chief visionary officer for 0:11:10.920 --> 0:11:14.720 the company. Like z Cat, Mako would develop technologies to 0:11:14.720 --> 0:11:17.760 help surgeons as well as implants for stuff like knee 0:11:17.800 --> 0:11:21.880 replacement surgery. The MAKO systems gave surgeons the information they 0:11:21.880 --> 0:11:25.359 needed to make real time decisions during a surgical procedure, 0:11:25.600 --> 0:11:29.120 including feedback systems to alert surgeons when they were getting 0:11:29.280 --> 0:11:31.959 close to the border of a predefined safety zone for 0:11:32.000 --> 0:11:35.000 a particular procedure. It's all super cool stuff that helps 0:11:35.040 --> 0:11:38.679 minimize any impact to quality of life during the recovery period. 0:11:39.320 --> 0:11:45.079 Sometime around twenty Orlen Abovitz began to think about another project. 0:11:45.360 --> 0:11:49.680 The surgical assistance technologies rely partly on augmented reality tech. 0:11:50.280 --> 0:11:54.520 Digital information would augment the surgeons operating room, but A 0:11:54.640 --> 0:11:58.840 Pivitz saw possible applications for this kind of technology that 0:11:58.920 --> 0:12:02.440 extended not used into the medical field, but beyond. He 0:12:02.600 --> 0:12:05.960 called the project Magic Leap, and he would create a 0:12:06.000 --> 0:12:10.239 corporation under that name INN. But as for the details 0:12:10.320 --> 0:12:14.520 of what Magic Leap was all about, that remained a mystery. 0:12:14.720 --> 0:12:19.080 Things didn't exactly coalesce the following year when Abavits put 0:12:19.120 --> 0:12:22.320 together a presentation for the two thousand twelve ted X 0:12:22.360 --> 0:12:26.840 event in Saratoga, Florida. The presentation is on YouTube and 0:12:26.960 --> 0:12:30.720 it's really more performance art than anything else. It actually 0:12:30.760 --> 0:12:34.160 reminds me a lot of the interactive art experiences from 0:12:34.280 --> 0:12:37.199 meal Wolf, a collection of artists out of Santa Fe, 0:12:37.320 --> 0:12:42.040 New Mexico, and perhaps by no coincidence, Meal Wolf is 0:12:42.559 --> 0:12:48.079 listed as a potential artistic partner with Magic Leap. By 0:12:48.080 --> 0:12:52.240 the way, if you're interested in art, performance, tech, and absurdism, 0:12:52.280 --> 0:12:55.320 I highly recommend you check out Meal Wolf. Their work 0:12:55.520 --> 0:12:59.640 ranges from whimsical to unsettling. Anyway, I don't think I 0:12:59.679 --> 0:13:03.800 could describe the ted X presentation from two thousand twelve effectively. 0:13:04.520 --> 0:13:07.839 It's just kind of weird. If you want to check 0:13:07.880 --> 0:13:11.000 it out. It is on YouTube and it's called the 0:13:11.040 --> 0:13:15.360 Synthesis of Imagination. And I'll just say this, there is 0:13:15.480 --> 0:13:18.560 no hint at all what the heck Magic Leap is 0:13:18.600 --> 0:13:22.920 about in that video. Now, that was sort of okay. 0:13:22.960 --> 0:13:26.600 At that point, the company name wasn't widely known, and 0:13:26.640 --> 0:13:29.400 you would be forgiven if you were walking away thinking 0:13:29.480 --> 0:13:33.040 magically was some sort of art collective that was particularly 0:13:33.120 --> 0:13:36.640 obsessed with fudge. Seriously, you need to watch the ted 0:13:36.800 --> 0:13:39.320 X talk from two thousand twelve to understand that reference. 0:13:39.720 --> 0:13:42.840 The team began working on a longer term goal of 0:13:42.880 --> 0:13:47.439 creating a mixed reality system capable of generating really compelling, 0:13:47.600 --> 0:13:51.720 impressive digital overlays on top of our real world, not 0:13:51.840 --> 0:13:56.160 even just on top of incorporated into our real world. 0:13:56.720 --> 0:13:59.800 In two thousand fourteen, more people would begin to take 0:13:59.800 --> 0:14:02.400 note of Magic Leap as it held a round of 0:14:02.440 --> 0:14:06.559 investment funding and brought in fifty million dollars in the 0:14:06.600 --> 0:14:09.920 first quarter of the year, But Magic Leap had yet 0:14:09.960 --> 0:14:13.360 to get truly widespread fame at that point. That did 0:14:13.440 --> 0:14:16.280 begin to change once people heard who some of these 0:14:16.320 --> 0:14:20.280 investors were. One of them was Richard Taylor, the Oscar 0:14:20.320 --> 0:14:23.720 winning special effects guru who worked on movies like Lord 0:14:23.720 --> 0:14:27.200 of the Rings and Peter Jackson's King Kong. Other prominent 0:14:27.240 --> 0:14:31.280 figures in fields like video games and tech companies also 0:14:31.440 --> 0:14:34.280 were among the number of investors, and in October two 0:14:34.320 --> 0:14:39.840 thousand fourteen, Magic Leap received five hundred forty two million 0:14:40.160 --> 0:14:45.520 dollars from numerous investors, including Google. It also didn't hurt 0:14:45.920 --> 0:14:50.520 that Abavits had recently seen his company Mako get acquired 0:14:50.920 --> 0:14:54.680 for a significant sum of money, more than a billion dollars, 0:14:54.680 --> 0:14:59.040 so things were going pretty fast and furious at this point. Now, 0:14:59.080 --> 0:15:03.040 clearly something special had to be happening, right, But what 0:15:03.080 --> 0:15:06.480 was it? As we'll see the mystery would be part 0:15:06.600 --> 0:15:10.520 of the selling point and the problem for Magic Leap. 0:15:11.040 --> 0:15:14.200 But before we get to that, let's take a quick break. 0:15:21.440 --> 0:15:24.320 Abevids and crew made a choice to be secretive and 0:15:24.440 --> 0:15:28.360 mysterious with the goal of the company, which did drum 0:15:28.440 --> 0:15:31.360 up a lot of speculation and buzz about Magic Leap. 0:15:31.960 --> 0:15:35.120 But that can be a double edged sword, because while 0:15:35.120 --> 0:15:39.360 people were starting to talk about this enigmatic company, it 0:15:39.400 --> 0:15:41.240 also meant that they were left to fill in the 0:15:41.280 --> 0:15:44.880 gaps of their knowledge by themselves. And here's a little 0:15:44.920 --> 0:15:47.680 secret for all of you out there. The odds are 0:15:47.720 --> 0:15:51.480 that whatever thing you're really working on is unlikely to 0:15:51.520 --> 0:15:56.040 match up to unfettered imagination. And this is why horror 0:15:56.080 --> 0:15:58.760 movies that leave stuff up to the imagination of the 0:15:58.800 --> 0:16:02.200 audience can have a much longer lasting effect than a 0:16:02.240 --> 0:16:05.600 movie that just shows you everything. The same is true 0:16:05.640 --> 0:16:09.400 for technology. If you don't lay out your vision, people 0:16:09.800 --> 0:16:12.680 kind of do that for you. And if that vision 0:16:12.800 --> 0:16:16.880 doesn't line up with reality, you're setting yourself up for failure. 0:16:17.040 --> 0:16:20.000 As you know, you're bound to disappoint people in the end. 0:16:20.560 --> 0:16:24.320 But maybe I'm jumping ahead too much here. The company 0:16:24.400 --> 0:16:27.000 did give a few members of the media some early 0:16:27.040 --> 0:16:31.200 glimpses into technology it was working on. One of those 0:16:31.440 --> 0:16:35.400 journalists was a New York Times reporter named John Markoff. 0:16:35.680 --> 0:16:39.520 Magically demonstrated some early technology to mark Off, not a 0:16:39.520 --> 0:16:42.320 production model, just something that was sort of in the 0:16:42.440 --> 0:16:46.600 R and D department. This implementation wasn't a wearable headset. 0:16:46.680 --> 0:16:50.680 Markoff described it as looking like quote, something from an 0:16:50.720 --> 0:16:54.880 optometrist's office end quote, And it was projecting what is 0:16:54.920 --> 0:16:59.360 called a digital light field into his eyes. So what 0:16:59.440 --> 0:17:02.000 the heck does that mean and why is it necessary 0:17:02.040 --> 0:17:04.840 at all? Well, Markoff goes into the fact that one 0:17:05.000 --> 0:17:07.960 of the challenges with both VR and a R has 0:17:08.040 --> 0:17:11.000 to do with depth and the way we perceive depth. 0:17:11.400 --> 0:17:15.040 So let's say you're looking down the street and you're 0:17:15.040 --> 0:17:19.080 watching a someone. Let's say it's Josh Clark, of stuff, 0:17:19.119 --> 0:17:22.440 you should know someone this Josh Clark. He's skipping merrily 0:17:22.520 --> 0:17:25.880 down the street towards you. And actually, yeah, that's that's 0:17:25.880 --> 0:17:29.000 actually how he walks, and no, I don't know why 0:17:29.040 --> 0:17:32.560 he does that. Anyway, you would see Josh Clark is 0:17:32.640 --> 0:17:34.960 starting off pretty far away from you. You'd focus on 0:17:35.040 --> 0:17:38.719 him in the distance, but then he's gradually getting closer 0:17:38.760 --> 0:17:43.680 and closer, skipping, perhaps somewhat menacingly toward you. He would 0:17:43.680 --> 0:17:46.760 appear to get larger because he's getting closer, until at 0:17:46.800 --> 0:17:49.320 some point he would block out your vision due to 0:17:49.359 --> 0:17:51.480 the fact that he has also and he does not 0:17:51.800 --> 0:17:55.600 mention this very much. He's also a giant. Okay, but 0:17:56.000 --> 0:17:59.359 let's say you're sitting with a VR headset on instead, 0:17:59.800 --> 0:18:04.080 and this is simulating you looking down the street. Now 0:18:04.320 --> 0:18:08.840 you see a virtual representation of Josh Clark skipping toward you, 0:18:09.119 --> 0:18:11.840 but you're not actually looking at someone who is really 0:18:11.920 --> 0:18:15.520 far away. Right, the virtual Josh Clark is an image 0:18:15.600 --> 0:18:18.200 on a screen, and that screen is just an inch 0:18:18.320 --> 0:18:21.719 or two away from your eyes. Josh isn't actually getting 0:18:21.760 --> 0:18:24.800 closer to you. Instead, the image is getting bigger. To 0:18:24.960 --> 0:18:30.280 simulate that sensation, you're getting visual cues that he's getting closer, 0:18:30.320 --> 0:18:32.760 but the entire time you're focusing on something that's actually 0:18:33.119 --> 0:18:36.200 right in front of you that screen. Now, it's hard 0:18:36.240 --> 0:18:39.639 to create a real sense of depth on a screen. 0:18:39.840 --> 0:18:43.960 You can create visual cues to simulate depth, mimicking what 0:18:44.000 --> 0:18:46.480 we perceive in the real world, but then we have 0:18:46.520 --> 0:18:49.120 a situation in which our eyes are trying to focus 0:18:49.160 --> 0:18:52.120 on something that in our brains appears to be very 0:18:52.119 --> 0:18:54.560 far away from us, but in fact it's on a 0:18:54.640 --> 0:18:57.359 screen that's right in front of us. This can create 0:18:57.440 --> 0:19:00.240 fatigue and discomfort for the viewer, and it's called old 0:19:00.480 --> 0:19:04.560 virgins accommodation conflict. And if we move this over to 0:19:04.640 --> 0:19:08.600 augmented reality, we can really see a disconnect. The physical 0:19:08.600 --> 0:19:12.000 world around us has actual depth. If we want a 0:19:12.080 --> 0:19:16.040 compelling a our experience that is more than just projecting 0:19:16.119 --> 0:19:18.959 text in front of our eyes, we need a system 0:19:19.040 --> 0:19:22.800 that can also incorporate depth in our virtual images so 0:19:22.840 --> 0:19:26.320 that they fit into the real world around us. Otherwise 0:19:26.600 --> 0:19:29.040 we'll be trying to focus on stuff that's on a screen, 0:19:29.400 --> 0:19:32.120 even if it's a transparent one that's right in front 0:19:32.119 --> 0:19:35.080 of us, while also trying to focus beyond in the 0:19:35.119 --> 0:19:39.600 real space around us. Enter the light field. Now, way 0:19:39.640 --> 0:19:42.720 back in the day, I talked about light field technology 0:19:42.800 --> 0:19:46.560 with the lightro camera. This digital camera could do something 0:19:46.600 --> 0:19:49.760 super cool. You could take a digital photo with it 0:19:49.960 --> 0:19:53.800 and then after the fact, you could change the focus 0:19:53.840 --> 0:19:56.920 of that image. Now, to understand why that's cool, let's 0:19:56.960 --> 0:20:01.680 think about standard photography. A camera, whether film or digital, 0:20:02.000 --> 0:20:06.000 has a lens. Typically there are cameras that don't have lenses, 0:20:06.040 --> 0:20:09.440 but let's just talk about the lensed ones. The purpose 0:20:09.480 --> 0:20:12.600 of the lens is to direct light. It bends the 0:20:12.680 --> 0:20:17.320 light toward a photoreactive element, either photoreactive film or a 0:20:17.359 --> 0:20:21.960 digital sensor. The physical curvature of the lens will determine 0:20:22.040 --> 0:20:25.600 how much it will bend that light. By moving the 0:20:25.680 --> 0:20:30.040 lens closer to or further from that photoreactive element, you 0:20:30.080 --> 0:20:34.560 can direct light from different distances toward that element. You 0:20:34.600 --> 0:20:38.240 will get a different point of focus, a different focal point. 0:20:38.560 --> 0:20:41.920 So you might have one setting that you can capture 0:20:42.320 --> 0:20:44.399 so that you get a clear image of a very 0:20:44.520 --> 0:20:47.440 distant object, and then you would have a totally different 0:20:47.520 --> 0:20:49.520 setting if you wanted to get a clear photo of 0:20:49.600 --> 0:20:52.199 something that was much closer to you, or you might 0:20:52.240 --> 0:20:54.760 even swap out lenses if you had to get a 0:20:54.760 --> 0:20:57.280 wider angle, or if you wanted to really zoom into 0:20:57.359 --> 0:21:00.840 something that was really further away. But the nitro camera 0:21:01.080 --> 0:21:04.880 and other light field cameras are different. They capture a 0:21:05.000 --> 0:21:09.359 light field, which describes all the rays of light moving 0:21:09.359 --> 0:21:12.879 in all directions within a given area the frame of 0:21:13.000 --> 0:21:16.199 view of the camera, and it includes not just the 0:21:16.240 --> 0:21:19.320 direction of the light, but the intensity of each ray 0:21:19.359 --> 0:21:23.200 of light. The special images these cameras produce, which require 0:21:23.320 --> 0:21:27.480 a special app to manipulate afterward, represent all the different 0:21:27.560 --> 0:21:30.160 rays of light within a shot, and so you can 0:21:30.240 --> 0:21:34.520 change the focus of that image. You can choose which 0:21:34.720 --> 0:21:38.679 rays of light you want to really focus on. So 0:21:38.720 --> 0:21:40.639 if you've got one object that's close to you and 0:21:40.680 --> 0:21:43.600 another that's in the background, you can actually switch focus 0:21:43.720 --> 0:21:46.840 from background to foreground over and over again, just choosing 0:21:46.840 --> 0:21:49.119 whichever one you want to look at at a given time. 0:21:49.800 --> 0:21:53.520 But how the heck does this technology actually work. It's 0:21:53.560 --> 0:21:56.200 one thing to say that the camera captures the light 0:21:56.359 --> 0:21:59.200 field of a scene, and another to actually make something 0:21:59.240 --> 0:22:02.200 that can do at So, your typical light field camera 0:22:02.440 --> 0:22:05.080 has all the elements of a normal camera. That is, 0:22:05.240 --> 0:22:08.879 has a lens to direct incoming light. There's an aperture 0:22:09.240 --> 0:22:12.199 or a hole that the light must pass through in 0:22:12.320 --> 0:22:15.919 order to hit a sensor for recording the image. But 0:22:16.080 --> 0:22:19.639 the lightro also has what's called a micro lens array, 0:22:20.080 --> 0:22:24.280 which sits between the main lens and the sensor. Now, 0:22:24.320 --> 0:22:28.040 as that name suggests, a micro lens array is a 0:22:28.040 --> 0:22:32.320 collection of very small lenses that can capture light coming 0:22:32.359 --> 0:22:36.720 from different distances from the camera's sensor. If you think 0:22:36.720 --> 0:22:39.880 of a camera as having detachable lenses where you can 0:22:39.880 --> 0:22:42.199 put a different lens on in order to get a 0:22:42.240 --> 0:22:46.680 different focal point. Just imagine a whole sheet of tiny 0:22:46.720 --> 0:22:49.120 little lenses that all can do that, and each lens 0:22:49.160 --> 0:22:51.240 has a slightly different curve to it so it can 0:22:51.280 --> 0:22:54.680 focus at a slightly different distance. You've got an entire 0:22:54.880 --> 0:22:58.160 array of these tiny lenses and they're all directing light 0:22:58.240 --> 0:23:03.000 toward particular pixels the digital cameras sensor. Each micro lens 0:23:03.080 --> 0:23:06.760 is doing the same thing and for its own focal 0:23:06.800 --> 0:23:09.160 point that is. That means you can have different micro 0:23:09.280 --> 0:23:12.600 lenses dedicated to those specific focal points, all capturing all 0:23:12.640 --> 0:23:16.080 the information to that sensor. Using the special app and 0:23:16.160 --> 0:23:19.879 photo file type, you can then select which batch of 0:23:19.960 --> 0:23:22.720 information you want to look at, and in this case, 0:23:22.760 --> 0:23:26.840 the batches of information relate to different focal points. So 0:23:26.880 --> 0:23:28.359 you can say, I want to look at all the 0:23:28.440 --> 0:23:32.040 light that was at this object, and it brings that 0:23:32.119 --> 0:23:35.240 object into focus. Then you click on a different object 0:23:35.240 --> 0:23:36.600 and you say, now I want to look at that one, 0:23:36.760 --> 0:23:39.680 and it brings all that light into focus instead. It's 0:23:39.720 --> 0:23:44.280 just switching lenses virtually in a way. There are companies 0:23:44.320 --> 0:23:47.320 working on technologies that will allow for light field displays 0:23:47.359 --> 0:23:50.959 without a micro lens array, but that topic is pretty 0:23:51.040 --> 0:23:54.640 darn complicated, even more so than just the standard light 0:23:54.720 --> 0:23:58.080 field cameras, So I'm gonna save that for another time. Now. 0:23:58.119 --> 0:24:00.520 The magic Leap concept that Mark of got to see 0:24:00.560 --> 0:24:03.919 also had light field technology, but in this case it 0:24:03.960 --> 0:24:07.359 was a light field display, and the goal was to 0:24:07.440 --> 0:24:11.120 project an image onto the retina of the user's eye, 0:24:11.400 --> 0:24:15.600 rather than to create a special type of digital photo. Ideally, 0:24:15.920 --> 0:24:19.040 the goal would be to create three dimensional virtual objects 0:24:19.160 --> 0:24:22.399 using depth cues and meshing those virtual objects with the 0:24:22.440 --> 0:24:25.239 real environment, so that to your eye it would be 0:24:25.280 --> 0:24:28.600 no different than if the virtual object really were there 0:24:28.880 --> 0:24:31.320 in the same space as you. So if you had 0:24:31.359 --> 0:24:35.399