WEBVTT - Using Satellites to Prevent Famines 0:00:15.396 --> 0:00:15.826 Pushkin. 0:00:31.606 --> 0:00:33.946 I'm Jacob Goldstein, and this is What's Your Problem? My 0:00:33.986 --> 0:00:38.346 guest today is Catherine Nakalembe. Catherine is an assistant professor 0:00:38.426 --> 0:00:41.396 at the University of Maryland. And she runs the Africa 0:00:41.446 --> 0:00:45.006 program at NASA Harvest, which is a project that uses 0:00:45.106 --> 0:00:49.766 satellite images to track agriculture around the world. And it 0:00:49.786 --> 0:00:53.076 wouldn't be wrong to say that Catherine's problem is using 0:00:53.196 --> 0:00:57.716 satellite imagery and GoPro cameras and AI to predict seasonal 0:00:57.756 --> 0:01:03.046 food shortages in sub-Saharan Africa. But in fact, her work, 0:01:03.346 --> 0:01:06.786 the problem she's trying to solve, is problematic. ultimately more 0:01:06.886 --> 0:01:10.646 human than that, ultimately more interesting, I would say, because 0:01:10.706 --> 0:01:13.446 the problem she's really trying to solve is, how do 0:01:13.466 --> 0:01:17.516 you use evidence to convince leaders to act to prevent 0:01:17.596 --> 0:01:21.996 people from starving? Catherine grew up in Uganda. She went 0:01:22.016 --> 0:01:24.976 to the University of Maryland to start a PhD, and 0:01:25.016 --> 0:01:27.936 then she went back to Uganda to do her graduate research. 0:01:28.276 --> 0:01:32.546 And the project was combining satellite data and on-the-ground research 0:01:32.566 --> 0:01:35.116 to to predict when the crops were likely to fail, 0:01:35.456 --> 0:01:38.796 leading to widespread hunger. And she was working on this 0:01:38.816 --> 0:01:41.356 for a few years. And then in the summer of 2015, 0:01:41.356 --> 0:01:45.056 when she was still a graduate student, there was a 0:01:45.116 --> 0:01:48.716 particularly bad drought in the region she studied. And she knew, 0:01:48.916 --> 0:01:51.196 she knew from her work that the people there would 0:01:51.256 --> 0:01:53.286 not have enough to eat in a few months. And 0:01:53.316 --> 0:01:56.066 she decided she had to get that information to someone 0:01:56.086 --> 0:01:57.756 with the power to do something about it. 0:01:58.226 --> 0:02:03.306 I realized really quickly that I knew more about the severity, 0:02:03.786 --> 0:02:07.846 basically the distribution of drought across this region more than 0:02:07.926 --> 0:02:10.306 anybody because I was able to look at it from space. 0:02:11.006 --> 0:02:13.326 And one year I finished my work and I was like, 0:02:13.366 --> 0:02:15.546 I can't just go back with this information because it 0:02:15.566 --> 0:02:17.246 had been an extreme drought year. I went to the 0:02:17.286 --> 0:02:20.146 office of the prime minister to share my My data 0:02:20.186 --> 0:02:23.046 and evidence in that year had taken lots of photographs 0:02:23.326 --> 0:02:25.926 and videos, et cetera. So I brought that. And then 0:02:25.946 --> 0:02:28.546 they had a meeting with the prime minister that led to— Wait. 0:02:28.586 --> 0:02:32.106 Can I just pause? So you're, what, a grad student 0:02:32.186 --> 0:02:33.606 or a young professor. 0:02:33.146 --> 0:02:34.266 At this point? No, I'm a grad student. 0:02:35.056 --> 0:02:38.416 And so it's an unusual thing for a grad student 0:02:38.456 --> 0:02:40.536 to be like, oh, my God, I have to tell 0:02:40.596 --> 0:02:44.896 the prime minister about this. What happened? How did that happen? 0:02:45.416 --> 0:02:49.126 So I've been coming for the last three years. People 0:02:49.146 --> 0:02:51.206 are trying to do the same things and crops fail. 0:02:52.186 --> 0:02:54.826 And I'd already started to kind of figure out how to, 0:02:55.106 --> 0:02:57.226 you know, predict what's going to happen, you know, early 0:02:57.266 --> 0:03:00.176 in the season. And there's always news around this issue, 0:03:00.316 --> 0:03:03.456 you know, Karamoja drought again, etc. 0:03:03.676 --> 0:03:07.516 So Karamoja is the region in, I guess, northern Uganda 0:03:07.576 --> 0:03:08.896 where you were studying. 0:03:09.176 --> 0:03:13.496 Yeah. And so it's always contentious. Is it happening? Are they, 0:03:13.556 --> 0:03:16.996 you know, just pretending? And this year, I had so 0:03:17.026 --> 0:03:20.756 much evidence, you know. First, because I'd taken so many photographs, 0:03:20.866 --> 0:03:24.386 I'd recorded video. And I wanted this to be sort 0:03:24.406 --> 0:03:27.726 of complementary to my PhD research. So I went and 0:03:27.746 --> 0:03:31.586 I shared it to the commissioner. And then he asked me, 0:03:32.326 --> 0:03:35.566 can you stay an extra few days? Because we have 0:03:35.606 --> 0:03:39.116 this food security meeting on Thursday with the prime minister 0:03:39.276 --> 0:03:43.006 and his steering committee, which are the other ministers. And 0:03:43.066 --> 0:03:45.226 I was like, of course. So I show up and 0:03:45.306 --> 0:03:47.846 I share the evidence and they asked me to say 0:03:47.906 --> 0:03:50.036 a few words. And I was like, you see this photograph? 0:03:50.756 --> 0:03:52.796 You see all of this thing that's dead here? This 0:03:52.876 --> 0:03:57.716 map basically represents the same thing everywhere. And it's eminent 0:03:57.916 --> 0:04:00.716 and people will be hungry pretty soon. And three days 0:04:00.756 --> 0:04:03.966 later or two days later, they sent food trucks. I 0:04:03.986 --> 0:04:06.446 have this photo that was sent to me from the 0:04:06.486 --> 0:04:10.306 guy who was the secretary to the prime minister or 0:04:10.386 --> 0:04:15.706 personal assistant. He was like, the trucks left this morning. 0:04:15.726 --> 0:04:19.496 I have this email, and I was amazed that I 0:04:20.116 --> 0:04:23.356 did that, and it went that far, basically. 0:04:23.496 --> 0:04:26.216 And that fast. So when you say people will be 0:04:26.256 --> 0:04:30.796 hungry pretty soon, that is an understated way, it seems, 0:04:30.876 --> 0:04:33.336 of saying what's going on. What was going on? 0:04:34.056 --> 0:04:38.156 You have subsistence farming where you produce what you eat. 0:04:38.216 --> 0:04:41.836 Very little is left over. And at the end of 0:04:41.896 --> 0:04:44.836 the season, you're supposed to sort of start picking up 0:04:45.196 --> 0:04:47.996 or eating what will be available to you through to 0:04:48.056 --> 0:04:50.956 the end of the next season. So this is typically 0:04:50.976 --> 0:04:56.916 around September. And so by August, everything had failed. And 0:04:56.976 --> 0:05:01.146 so there wasn't going to be anything harvested. But if 0:05:01.246 --> 0:05:05.856 you've kind of been out in the field a collecting data, 0:05:06.016 --> 0:05:09.356 and you run into kids eating sorghum shoots, for example, 0:05:09.396 --> 0:05:10.186 you know, the middle part of a. 0:05:10.196 --> 0:05:14.556 It's kind of like eating grass. 0:05:15.106 --> 0:05:17.586 Exactly. So like the middle part of a corn stem, 0:05:18.386 --> 0:05:22.426 it's sugary. And so, you know, you find kids eating 0:05:22.466 --> 0:05:26.526 that and they don't have anything else. They don't have livestock. 0:05:26.586 --> 0:05:28.726 Their family does not have any other source of income. 0:05:29.696 --> 0:05:32.546 It's pretty dire. So I think the other thing that 0:05:32.586 --> 0:05:35.286 might have made a huge difference in this particular instance 0:05:35.306 --> 0:05:39.686 is the photo evidence that I had. I take thousands 0:05:39.726 --> 0:05:41.666 of photos. I think I have over 300,000 photos. 0:05:41.866 --> 0:05:47.576 That's such a low-tech thing. Do you know what I mean? 0:05:47.616 --> 0:05:52.436 It's not like you were doing some wild satellite AI something. 0:05:52.476 --> 0:05:54.396 You were just walking around and taking pictures. 0:05:54.416 --> 0:05:58.846 Mm-hmm. And I had my map, my drought analysis map, 0:05:59.046 --> 0:06:02.346 that was very simple. And then we called it NDVI, 0:06:02.446 --> 0:06:05.476 Normalized Difference Vegetation Index, the anomaly of it. 0:06:06.296 --> 0:06:09.016 Basically, how much drier than usual. Exactly. 0:06:09.276 --> 0:06:09.656 Yeah. 0:06:10.836 --> 0:06:13.276 So, okay, so you do this, and the result in 0:06:13.316 --> 0:06:15.636 that first year is that they get food aid to 0:06:15.676 --> 0:06:16.976 people sooner. 0:06:16.996 --> 0:06:23.116 Right. Almost immediately. I have this text message from the commissioner. 0:06:23.156 --> 0:06:26.336 He said, this is the first time that nobody argued 0:06:26.376 --> 0:06:27.156 with the evidence. 0:06:28.346 --> 0:06:32.746 Did they then integrate your work in an ongoing way? 0:06:32.766 --> 0:06:35.686 How did that play out in the years after that? 0:06:36.166 --> 0:06:40.586 So the project that's called Disaster Risk Financing was basically 0:06:40.626 --> 0:06:43.566 being designed the following week. So I stayed that extra 0:06:43.606 --> 0:06:46.836 week and was like, I can not only develop the methods, 0:06:46.876 --> 0:06:49.476 but I can train the office how to utilize the 0:06:49.496 --> 0:06:51.536 data that's available in order for them to do that 0:06:51.616 --> 0:06:55.336 early warning, as well as write summaries, reports, et cetera. 0:06:56.576 --> 0:06:59.816 Take pictures. It sounds like one big lesson is take pictures. 0:06:59.956 --> 0:07:03.676 It's a validation for sure, but they'd already understood. So 0:07:03.756 --> 0:07:06.756 I kind of created this method to understand, compared to 0:07:06.816 --> 0:07:10.116 the history, if we hit this threshold, things are going 0:07:10.136 --> 0:07:13.506 to fail. Considering what has happened in the last 30 0:07:13.506 --> 0:07:16.676 years or last 20 years, If we hit this threshold, 0:07:16.756 --> 0:07:19.096 vegetation won't recover and we need to start planning. 0:07:19.256 --> 0:07:22.966 Basically a threshold of drought. If rain is below some level. 0:07:23.326 --> 0:07:27.286 So initially it was like if vegetation conditions drop below 0:07:27.806 --> 0:07:32.386 this level within this period, a few months, months within 0:07:32.426 --> 0:07:34.526 the growing season, then we need to do something. 0:07:35.106 --> 0:07:38.946 So it's interesting. I mean, I might naively have thought 0:07:38.996 --> 0:07:43.316 that the constraint, the fundamental constraint was just money. 0:07:43.726 --> 0:07:43.886 Right. 0:07:43.906 --> 0:07:45.626 That there's just not the money to get the food 0:07:45.646 --> 0:07:49.496 to the people or infrastructure or something. But it's information 0:07:49.656 --> 0:07:51.716 at some level, like reliable evidence. 0:07:52.676 --> 0:07:58.136 Reliable evidence that is accessible. But I think money and 0:07:58.176 --> 0:08:03.646 the willingness to use that money is also really important. 0:08:03.746 --> 0:08:07.406 So that region where I was studying, even though it 0:08:07.466 --> 0:08:11.606 had the lowest economic indicators, you know, household income was 0:08:11.646 --> 0:08:16.786 very low. Infrastructure is very poor. The cost of food 0:08:16.846 --> 0:08:20.326 was so much higher than it is in the capital. Yes. 0:08:20.746 --> 0:08:23.486 Well, that goes with poor infrastructure, right? Exactly. When it 0:08:23.566 --> 0:08:27.596 costs a lot to move food, the food costs a lot. 0:08:27.916 --> 0:08:30.996 Exactly. Now, add the fact that there is a drought 0:08:31.236 --> 0:08:34.176 and traders can do whatever they want. And so it 0:08:34.216 --> 0:08:37.196 becomes much more expensive. And when the government is trying 0:08:37.256 --> 0:08:40.626 to acquire this water and food, et cetera, they end 0:08:40.666 --> 0:08:43.966 up paying a lot more. Uh-huh. And so if you're 0:08:44.006 --> 0:08:47.146 doing it in anticipation, it means that then you can 0:08:47.206 --> 0:08:49.366 acquire things at a much lower cost. 0:08:49.466 --> 0:08:51.946 Oh, so if they know in August that there is 0:08:51.986 --> 0:08:54.976 going to be great need in September, they can buy 0:08:55.166 --> 0:08:56.196 before there's a run. 0:08:56.436 --> 0:09:00.976 Exactly. And the other thing was, because it was in anticipation, 0:09:00.996 --> 0:09:06.176 what ended up happening is they would have projects that 0:09:06.236 --> 0:09:10.036 could hire people to work on that were community development projects. 0:09:10.076 --> 0:09:13.496 It could be tree planting. It could be road construction, 0:09:13.516 --> 0:09:16.796 it could be improving wells, and then they will be 0:09:16.856 --> 0:09:19.816 paid for those jobs. Yeah, that then they could use 0:09:19.836 --> 0:09:21.856 that money to buy food and other things. 0:09:22.336 --> 0:09:24.256 So it's interesting. I thought we were going to be 0:09:24.326 --> 0:09:27.526 talking about satellites and AI, and I guess we will be. 0:09:28.006 --> 0:09:31.006 But it's interesting that this conversation starts in such a 0:09:32.086 --> 0:09:35.026 low-tech way. I mean, obviously, digital cameras were not quite 0:09:35.066 --> 0:09:37.756 as ubiquitous as they are today. But this is 2016. 0:09:37.756 --> 0:09:40.916 It's not 2000 or something. Lots of people already had 0:09:40.936 --> 0:09:41.816 digital cameras. 0:09:44.336 --> 0:09:50.256 Why? Yeah, why? I'd say people believe what they see. 0:09:50.566 --> 0:09:57.706 And the NASA administrator in 2019, Jim Bernstein, he used 0:09:57.746 --> 0:10:02.406 to share my photos in his presentations. He'd talk about 0:10:02.866 --> 0:10:06.246 from NASA to the moon and beyond. But he would 0:10:06.286 --> 0:10:09.086 still use the photos and not the maps or the images. 0:10:09.196 --> 0:10:11.996 the numbers because it tells it you know we can 0:10:12.116 --> 0:10:14.516 see it from above and this is what it means 0:10:14.676 --> 0:10:17.836 on the ground this is the reality of what it 0:10:17.936 --> 0:10:19.876 is on the ground and that's i think much more 0:10:19.896 --> 0:10:22.536 accessible and understandable. 0:10:21.716 --> 0:10:25.586 Um so I feel a little weird now after you're 0:10:25.646 --> 0:10:28.486 talking compellingly about how important photos are. But I do 0:10:28.546 --> 0:10:31.146 want to talk to you about satellite imagery and AI. 0:10:31.166 --> 0:10:32.306 But they're photos. 0:10:32.606 --> 0:10:36.016 They're photos. Very good. They're photos. I find them moving 0:10:36.056 --> 0:10:40.096 in a different kind of way. I mean, you know, 0:10:40.196 --> 0:10:43.306 I've read this sort of big hand-wavy thing, which is, 0:10:44.256 --> 0:10:46.276 You know, there's lots of satellite imagery of the Earth 0:10:46.336 --> 0:10:49.226 now and lots of AI models designed to analyze crops, 0:10:49.326 --> 0:10:54.826 but those are largely designed for kind of big monocultural agribusiness, 0:10:54.906 --> 0:10:57.366 and they don't apply so well, certainly, to the kind 0:10:57.386 --> 0:11:01.546 of smallholder farms in the places you have worked. Like, 0:11:01.586 --> 0:11:05.056 I get that that's kind of the big picture. But 0:11:05.116 --> 0:11:08.216 more narrowly, like, tell me about your work to deal 0:11:08.236 --> 0:11:09.736 with that. Like, first of all, what is it that 0:11:09.776 --> 0:11:13.116 you want to know from satellite imagery? 0:11:13.766 --> 0:11:18.016 We want to understand what's growing where and how it's doing. 0:11:19.796 --> 0:11:23.176 The other is the complexity of the agriculture system. So 0:11:23.196 --> 0:11:28.206 when we're looking for crop fields, we're looking for shapes 0:11:28.926 --> 0:11:31.886 and then we're looking for some sort of seasonal pattern. 0:11:32.586 --> 0:11:36.066 So if you had 10 images over a growing season, 0:11:36.766 --> 0:11:41.596 you should see something like it's cleared now. it browns, 0:11:41.676 --> 0:11:45.256 it's like brown and wet, and then something green pops up, 0:11:45.856 --> 0:11:48.536 it matures, and then it's harvested. So it sort of 0:11:48.576 --> 0:11:51.476 like becomes, you know, dry. And so we should be 0:11:51.516 --> 0:11:56.736 able to see that signal. However, how low the brownness 0:11:56.896 --> 0:12:01.066 might be, like how dry something might be, is contextual. 0:12:02.006 --> 0:12:07.166 And while a reduction might seem like minor in some places, 0:12:07.246 --> 0:12:11.656 it's actually a very serious problem. And so the first 0:12:11.676 --> 0:12:15.756 thing was create a cropland map, basically exclude everything that's 0:12:15.796 --> 0:12:21.496 not cropland, estimate, understand vegetation conditions historically, like what happens 0:12:21.636 --> 0:12:25.216 every season, what months matter the most in terms of vegetation, 0:12:25.576 --> 0:12:28.756 in terms of crop production, and then try to understand 0:12:28.876 --> 0:12:31.996 at what point recovery doesn't happen. So to be able 0:12:32.016 --> 0:12:35.276 to do that at a really large scale, satellites are 0:12:35.296 --> 0:12:39.036 a phenomenal tool for this. At the time, even today, 0:12:39.536 --> 0:12:41.746 the only satellite that allows us to be able to 0:12:41.826 --> 0:12:45.146 do this at a much higher frequency is MODIS. And 0:12:45.206 --> 0:12:49.726 MODIS is a 250-meter resolution. That could be multiple crop 0:12:49.746 --> 0:12:51.726 fields in a single pixel. 0:12:52.386 --> 0:12:56.136 So the farms are too small or too varied. 0:12:56.436 --> 0:12:57.396 Too varied, yep. 0:12:58.206 --> 0:13:01.856 Tell me, what is a typical farm in this region like? 0:13:02.096 --> 0:13:07.216 So in Karamoja, people grow sorghum, millet, a little bit 0:13:07.256 --> 0:13:09.316 of maize. There's a lot of sunflower, too. 0:13:10.236 --> 0:13:13.346 And will they grow all that in one place, more 0:13:13.366 --> 0:13:13.526 or less? 0:13:13.546 --> 0:13:17.566 So you will find sunflower mixed in with sorghum, for example. 0:13:17.966 --> 0:13:19.886 And how big is a typical farm? 0:13:20.146 --> 0:13:26.476 So farm plot... On average, less than 2.5 hectares. So 0:13:26.756 --> 0:13:29.726 a football field, a soccer field is about an acre. 0:13:31.546 --> 0:13:37.076 That's the average over most of sub-Saharan Africa, but they're 0:13:37.116 --> 0:13:39.636 much smaller. So like in a single soccer field, you 0:13:39.656 --> 0:13:43.076 might find maybe four plots as an example. 0:13:43.756 --> 0:13:45.316 Like four different people's little farms? 0:13:45.336 --> 0:13:49.346 Four different people's plots with hedges, and they're not like 0:13:49.406 --> 0:13:51.386 distinct perfect shapes like you would see. 0:13:51.606 --> 0:13:56.326 Quite small. Hard to see from a satellite, presumably, especially 0:13:56.366 --> 0:13:58.106 if there's a bunch of different crops there. 0:13:58.466 --> 0:14:01.736 So hard to see their boundaries. So a single modus 0:14:01.796 --> 0:14:04.196 pixel might include four of those fields that might have 0:14:04.256 --> 0:14:05.736 different things, for example. 0:14:06.076 --> 0:14:12.076 Yeah. Let's talk about the GoPros. So fun. Tell me 0:14:12.116 --> 0:14:13.206 about your work with GoPros. 0:14:13.906 --> 0:14:17.456 So one of the issues is in order to get 0:14:17.596 --> 0:14:20.556 from cropland, where all the crops are growing, to go 0:14:20.596 --> 0:14:23.916 to crop type. So the USDA, for example, produces a 0:14:23.996 --> 0:14:27.576 cropland data layer, which if you take a look at it, 0:14:27.656 --> 0:14:30.206 you can see where corn, wheat, soy is growing. And 0:14:30.236 --> 0:14:32.426 now that's been changing over the last couple of years. 0:14:32.506 --> 0:14:35.686 In the US. In the US, yes. This also exists 0:14:35.806 --> 0:14:40.166 for Europe. They have something similar. There's, of course, some simplicity. 0:14:40.246 --> 0:14:44.576 There's a single crop. The fields are homogeneous. They're kind 0:14:44.596 --> 0:14:47.706 of easy to map. In East Africa, in addition to 0:14:47.746 --> 0:14:51.646 fields being small, I already mentioned the intercropping, mixed cropping, 0:14:51.866 --> 0:14:55.336 and all the heterogeneity, how people plant at different days. 0:14:56.176 --> 0:14:58.036 To be able to get a crop type map, you 0:14:58.096 --> 0:15:00.996 need a lot of training data. So, you know, one 0:15:01.016 --> 0:15:04.076 of the biggest advances, I think, you know, the reason 0:15:04.096 --> 0:15:07.076 why we have huge leaps in terms of AI is 0:15:07.096 --> 0:15:10.516 the availability of training data. You know, same as face detection, 0:15:10.676 --> 0:15:13.856 all people's faces on Facebook, et cetera, makes that technology 0:15:13.876 --> 0:15:16.856 a lot easier to develop. And so this is true 0:15:16.956 --> 0:15:19.616 also for crop type mapping. So it's like face detection, 0:15:19.676 --> 0:15:20.316 but for crops. 0:15:20.616 --> 0:15:22.376 Why do you want a crop type map? 0:15:22.776 --> 0:15:26.466 There's so many benefits to knowing exactly what crop is 0:15:26.506 --> 0:15:29.706 growing where. You want this map in order to understand, 0:15:29.786 --> 0:15:32.966 for example, what kind of fertilizer will be relevant in 0:15:33.006 --> 0:15:37.586 a particular region. So if you know there's places dominated 0:15:37.636 --> 0:15:40.886 by rice, just like in Ghana, a particular place in Ghana, 0:15:41.366 --> 0:15:45.266 where could irrigation infrastructure make a difference? The other and 0:15:45.366 --> 0:15:48.506 most important, most basic thing is, if you want to 0:15:48.666 --> 0:15:51.236 understand how much will be produced at the end of 0:15:51.296 --> 0:15:54.016 the season, you need to know the total area by 0:15:54.076 --> 0:15:57.216 specific crop. So if we have 25,000 hectares of corn planted... 0:16:00.286 --> 0:16:03.906 And the average condition or the average yield is 2.5 0:16:03.906 --> 0:16:07.146 tons per hectare. You multiply that and you have your production, 0:16:07.236 --> 0:16:10.496 which will allow you to understand how much corn is available. 0:16:10.616 --> 0:16:13.276 If it's a critical crop in a particular area, you 0:16:13.316 --> 0:16:16.856 would know that, you know, food security would be okay. 0:16:17.296 --> 0:16:21.476 So we had a big issue in Illinois with corn, 0:16:21.796 --> 0:16:25.256 et cetera. If Germany had a big issue with wheat production, 0:16:25.376 --> 0:16:26.796 bread being so important. 0:16:27.176 --> 0:16:27.376 Yeah. 0:16:27.716 --> 0:16:30.336 They would have to plan accordingly. So you need to 0:16:30.376 --> 0:16:32.896 figure out, what do I need to import to have 0:16:32.946 --> 0:16:34.036 some sort of stability? 0:16:34.276 --> 0:16:38.326 And so for Sub-Saharan Africa in general, or the regions 0:16:38.366 --> 0:16:41.426 you study, are there crop-type maps now? 0:16:42.086 --> 0:16:47.306 There are some crop-type maps, yes. Some are done on 0:16:47.366 --> 0:16:50.736 a very small scale, regional scale. But what this project 0:16:50.776 --> 0:16:53.866 was trying to do was try to, one, just figure 0:16:54.066 --> 0:16:57.006 out the fastest possible way we can collect data over 0:16:57.126 --> 0:17:00.316 really large areas. One, because in some countries there are 0:17:00.336 --> 0:17:03.816 three seasons. So Rwanda has three growing seasons. So you're 0:17:03.836 --> 0:17:07.356 looking at three crops. So to cover the whole country 0:17:07.496 --> 0:17:13.646 and get an understanding of did they grow corn more generally, 0:17:13.786 --> 0:17:16.886 bananas more generally? No. You need to be able to 0:17:16.926 --> 0:17:18.226 do the survey continuously. 0:17:18.806 --> 0:17:22.016 And the satellites don't have the resolution, essentially, to do it. 0:17:22.016 --> 0:17:22.956 To be able to do it, yeah. 0:17:23.176 --> 0:17:24.516 You can't tell from satellite data. 0:17:24.556 --> 0:17:27.376 So we need to train a model to tell the model, 0:17:27.456 --> 0:17:31.776 this pixel, this particular spot is banana. Find me banana 0:17:31.876 --> 0:17:33.536 everywhere else in the image, yeah. 0:17:34.296 --> 0:17:39.016 So you have this idea of like, okay, we need 0:17:39.076 --> 0:17:42.436 more on-the-ground data. So what do you do? What is 0:17:42.496 --> 0:17:43.416 the GoPro project? 0:17:44.796 --> 0:17:49.606 GoPros are perfect because they have a GPS. So when 0:17:49.626 --> 0:17:52.146 we take a photo, the photo has a location. So 0:17:52.466 --> 0:17:56.096 typically when I'm collecting field data, I need a photo, evidence, 0:17:56.576 --> 0:17:59.476 but also I need to be clear when somebody labels 0:17:59.736 --> 0:18:03.586 in the form that maize is in this growing stage 0:18:03.866 --> 0:18:05.486 and you have to stand in the middle of the field. 0:18:06.006 --> 0:18:08.126 Because I want to use that location to train my 0:18:08.166 --> 0:18:10.176 model to predict where else maze might be. 0:18:10.326 --> 0:18:13.216 This is the old school, slow, never going to scale 0:18:14.316 --> 0:18:17.956 technique that you're trying to kind of quasi-automate here. 0:18:17.976 --> 0:18:21.816 Yes. At this point, even if all I had were 0:18:21.976 --> 0:18:27.166 images that had a longitude and latitude, and I know 0:18:27.186 --> 0:18:30.346 where the camera is facing, I could go through all 0:18:30.386 --> 0:18:34.146 these individual photos in Google Earth Pro and add the 0:18:34.186 --> 0:18:34.986 label myself. 0:18:36.216 --> 0:18:38.276 So at least you would have a bunch of images 0:18:38.876 --> 0:18:43.656 with a very specific geolocation on latitude and longitude. 0:18:43.876 --> 0:18:46.176 And I know what the crops look like. I know 0:18:46.216 --> 0:18:47.056 what the crops are. 0:18:47.536 --> 0:18:50.196 So this is phase one. But you don't want to 0:18:50.216 --> 0:18:54.236 have to label a thousand things corn, whatever. 0:18:54.256 --> 0:18:58.866 Yeah, I would have been okay with that too, though. Considering, 0:18:58.946 --> 0:19:01.506 so I did this in Mali. I did it in Senegal. 0:19:02.336 --> 0:19:02.976 In Tanzania. 0:19:02.996 --> 0:19:07.376 Wait, this meaning driving around with the GoPro? No, the phone. Oh, 0:19:07.396 --> 0:19:08.136 the walking around. 0:19:08.196 --> 0:19:09.236 Walking around, yeah. 0:19:09.336 --> 0:19:11.996 Yeah, okay. I mean, it sounds kind of charming in 0:19:12.036 --> 0:19:12.516 a way. 0:19:12.536 --> 0:19:16.556 I learn a lot from doing field work. You learn 0:19:16.856 --> 0:19:19.116 what it means for someone's field to be washed out 0:19:19.136 --> 0:19:22.276 when you go. But if the idea is to try 0:19:22.336 --> 0:19:25.226 and create a map that will be useful for this task, 0:19:26.606 --> 0:19:29.386 there's only so much you can do when you cover 0:19:29.406 --> 0:19:32.886 a very small area. Yeah. So 2020, I call this 0:19:32.926 --> 0:19:36.236 project a COVID-safe project. 2020 was like, oh, well, now 0:19:36.246 --> 0:19:38.616 we have this funding. We have to figure this out. 0:19:38.696 --> 0:19:41.596 I go buy a GoPro at a store, a fashion 0:19:43.656 --> 0:19:47.056 sort of amount with my dad because I couldn't order 0:19:47.096 --> 0:19:51.206 it from Amazon. And then we make this magnetic thing 0:19:51.356 --> 0:19:53.846 on the car and just drive. I go to, there's 0:19:53.866 --> 0:19:57.546 an agricultural research station. And the director was like, I 0:19:57.566 --> 0:19:59.106 described to him what I'm trying to do. He was like, oh, 0:19:59.166 --> 0:20:02.086 course you can drive around here, whatever. So we try 0:20:02.106 --> 0:20:03.996 it out. So the first thing was, if we get 0:20:04.036 --> 0:20:07.256 the images, are they useful? Is the GPS good enough? 0:20:07.836 --> 0:20:08.396 And it was. 0:20:09.116 --> 0:20:12.436 And so then we were like, oh, this could actually work. 0:20:17.256 --> 0:20:29.206 We'll be back in just a minute. That's the end 0:20:29.226 --> 0:20:31.046 of the ads. We're going back to the show. And 0:20:31.266 --> 0:20:33.786 as you probably recall, when we left off before the break, 0:20:34.186 --> 0:20:37.546 Catherine had just tested out a GoPro herself and decided 0:20:37.586 --> 0:20:40.066 that GoPros could be a good way to gather crop 0:20:40.086 --> 0:20:41.506 data from small farms. 0:20:42.106 --> 0:20:46.146 So I have a friend in Kenya. I asked him 0:20:46.266 --> 0:20:48.456 if he can buy, if he can order things on 0:20:48.496 --> 0:20:53.336 Amazon and to work with a team there to see 0:20:53.356 --> 0:20:56.896 if they can collect data in Kenya. So we get 0:20:56.936 --> 0:21:00.016 a team of two. So basically four. So it's a 0:21:00.036 --> 0:21:03.706 team of two people. One team drives south, the other 0:21:03.726 --> 0:21:08.006 drives north. And we cover all of Western Kenya in 0:21:08.046 --> 0:21:08.406 a week. 0:21:09.186 --> 0:21:12.206 Wow. And they're just driving everywhere they can drive, in 0:21:12.226 --> 0:21:13.486 a car, on a motorcycle? 0:21:13.846 --> 0:21:15.846 So you can see from some of the original data 0:21:15.886 --> 0:21:18.326 that we have from Kenya that we're kind of covering 0:21:18.346 --> 0:21:22.326 these big... There are lots of fields, but they're proximal 0:21:22.376 --> 0:21:27.716 to these big roads, which... It's great, but not good enough. 0:21:28.436 --> 0:21:31.596 And so it became pretty quickly obvious that a motorcycle 0:21:31.616 --> 0:21:33.836 might be better because a motorcycle can go to a 0:21:33.896 --> 0:21:37.556 level two, level three, you know, smaller paths and go through. 0:21:37.596 --> 0:21:39.446 And in Tanzania, they went a little bit crazy because 0:21:39.526 --> 0:21:42.206 I feel like they're driving into rice paddies and going 0:21:42.246 --> 0:21:42.626 like this. 0:21:42.666 --> 0:21:45.366 Well, people live far off of roads that you could 0:21:45.386 --> 0:21:48.006 drive a car on. Exactly. The kinds of farms you're 0:21:48.046 --> 0:21:50.826 talking about exist far from drivable roads. 0:21:50.926 --> 0:21:55.626 Exactly. So... But just from the Kenya team, we had over, 0:21:55.646 --> 0:22:01.166 I think the first iteration might have been maybe 300,000 0:22:01.166 --> 0:22:06.086 images or more. Then the team of two in Uganda, 0:22:06.106 --> 0:22:11.266 I kind of equipped them. And during COVID, there's nothing 0:22:11.286 --> 0:22:12.866 else you could do and asked them to kind of 0:22:12.906 --> 0:22:16.266 just go drive around. And they went crazy because then 0:22:16.306 --> 0:22:19.116 we had over 2 million images from them. Okay. And 0:22:19.176 --> 0:22:22.066 so then it's just too many images. Even I got 0:22:22.186 --> 0:22:24.706 tired of just like trying to figure it out. But 0:22:24.726 --> 0:22:27.466 I had an undergraduate computer science student who worked over 0:22:27.506 --> 0:22:31.526 this and developed Street to Sat, the first iteration of it. 0:22:32.676 --> 0:22:38.136 And remove clouds, adjust images automatically so that they're kind 0:22:38.176 --> 0:22:44.776 of like straight. And then remove faces and ensure that 0:22:44.836 --> 0:22:47.816 they're actually crops in the image. 0:22:48.376 --> 0:22:50.896 Crop and not crop, you called it in the paper. 0:22:50.916 --> 0:22:52.806 It reminded me, I don't know if you watched the 0:22:52.846 --> 0:22:56.866 show Silicon Valley, but there's a guy in that show 0:22:57.006 --> 0:22:57.866 who invents AI. 0:22:59.596 --> 0:23:02.536 And they show it a hot dog and it says 0:23:02.596 --> 0:23:04.556 hot dog. And everybody's like, wow, it can detect food. 0:23:04.576 --> 0:23:05.816