Speaker 1: Pushkin. Hey, y'all, this is justin Richmond. I hope you're
Speaker 1: enjoying a nice and safe holiday season. We're looking forward
Speaker 1: to coming back to you with new episodes of Broken
Speaker 1: Record in twenty twenty one, starting on January twelve. But
Speaker 1: in the meantime, we wanted to gift you an interview
Speaker 1: Malcolm Gladwell did for another podcast called Solvable. Solvable is
Speaker 1: hosted by Pushkin CEO Jacob Weisberg and occasionally Malcolm as well.
Speaker 1: In this episode, Malcolm speaks with a software engineer who
Speaker 1: solved awkward video conference calls by reducing the delay time
Speaker 1: that occurs enough that musicians can now jam over Zoom
Speaker 1: in real time. Enjoy this episode. Here's Malcolm Gladwell. Was Solvable.
Speaker 1: This is Solvable. I'm Malcolm Gladwell. Occasionally we would get
Speaker 1: to go on break out rooms on Zoom and just
Speaker 1: discuss our feelings about what's happening at the time, and
Speaker 1: when the lockdown began, we were all devastated because singing
Speaker 1: was our life. Sam is twelve years old and like
Speaker 1: many kids, he's struggling with the restrictions COVID nineteen has
Speaker 1: imposed on his life. There's a lot of talking to
Speaker 1: news about the impact of COVID on classroom education. But
Speaker 1: for Sam and his friends, what they're missing is after school.
Speaker 1: Ragotsi was like a sanctuary to go to at the
Speaker 1: end of the school day. Rogatsi is a boy's choir
Speaker 1: based in California. Sam's been singing in the group for
Speaker 1: seven years. In pre COVID times, he would rehearse twice
Speaker 1: a week for several hours each day, but all of
Speaker 1: that came to a halt when COVID arrived in the US.
Speaker 1: The choir was basically stuck. If you've ever tried to
Speaker 1: sing something like Happy Birthday during a zoom meeting, you
Speaker 1: know what I'm talking about. Making music sync up with
Speaker 1: other people in real time over the Internet is really tough.
Speaker 1: The connect he delays, what they call latency, can be
Speaker 1: pretty awkward. Give us the technical explanation of the problem
Speaker 1: your son was facing. It's rooted in latency. Fundamentally, the
Speaker 1: time that it takes the sound you make to reach
Speaker 1: another musician that you're performing with. You need to have
Speaker 1: a latency of about less than twenty five milliseconds. Otherwise,
Speaker 1: the asense of the pulse or the beat of the
Speaker 1: music is just lost across those musicians. Mike Dickie is
Speaker 1: a software engineer, and after months of seeing his son
Speaker 1: Sam bummed out about not singing, he realized he could
Speaker 1: do something about it now for far less than the
Speaker 1: cost of a year of private music lessons. Dickie created
Speaker 1: a tool that allows groups of musicians to make music online.
Speaker 1: And it might be just the beginning. So we could
Speaker 1: have real time cheering and fan reaction in a basketball game. Yeah, absolutely,
Speaker 1: Performing in rehearsing with others in real time is really
Speaker 1: difficult or even impossible for many people. But this problem
Speaker 1: is solvable. I talked with Mike Dickie about Virtual Studio.
Speaker 1: It's a low cost device that works with the open
Speaker 1: source tool called jack Trip to make remote music collaboration
Speaker 1: possible in real time. Here is our conversation. Mike tell
Speaker 1: me about the first time this that Rugazzi rehearse using
Speaker 1: virtual Studio. What was that like? At first? It was
Speaker 1: a bit terrifying progress out. Let's progress down, okay and
Speaker 1: keep go. I thought of all the things that could
Speaker 1: go wrong, and I knew that this was all very
Speaker 1: new and untested, and I was also worried just of
Speaker 1: what people would think of me and my son if
Speaker 1: this was a huge disaster. But when we when we
Speaker 1: started getting people online and it's started to work, and
Speaker 1: the boys started to sing right. It was the first
Speaker 1: time they were singing together in a long time, and
Speaker 1: I could see through the zoom the faces on the
Speaker 1: different boys start to light up. I could see the smiles,
Speaker 1: I could see it in their eyes, and I could
Speaker 1: see it in the director's eyes, thinking, hey, this actually
Speaker 1: is working. That experience of seeing people and having them
Speaker 1: so happy and having their experience with others impact their lives,
Speaker 1: that's really what led me to put more and more
Speaker 1: time into this and ultimately to devote all my time
Speaker 1: towards it. Prior to the lockdown, people must have thought
Speaker 1: about this problem and tried to solve it. Were there
Speaker 1: existing technical solutions to the problem of how to sing remotely?
Speaker 1: There were, thankfully. I looked at several of them when
Speaker 1: I started out, and I came across this small project
Speaker 1: from out of Stanford University called jack Trip, which is
Speaker 1: an open source technology that was developed over a year
Speaker 1: ago by the Computer Music department. It seemed very promising.
Speaker 1: It had a high quality, lossless audio. It focused on
Speaker 1: low latency and over the Internet, and there were a
Speaker 1: lot of people already using it. The challenge that I
Speaker 1: found with that is that it was a command line tool.
Speaker 1: You had to open a terminal on your computer. In
Speaker 1: many cases, you had to build it from the source code,
Speaker 1: the C plus plus source code, and then you had
Speaker 1: to know the command lines and figure out the right
Speaker 1: command line options in order to use it with other people.
Speaker 1: That's great for somebody who's very technically oriented, but in
Speaker 1: my son's chorus, I was trying to figure out how
Speaker 1: how can we roll this out to a large number
Speaker 1: of young boys that don't have this advanced technical expertise.
Speaker 1: And it was actually a challenge that I thought I
Speaker 1: was well suited for because I understood the technical side
Speaker 1: and I also had the product background in order to
Speaker 1: try to make something that was fairly difficult to use
Speaker 1: fairly easy for most of the people that were out there.
Speaker 1: So we're used to is it a normal sonic collaboration,
Speaker 1: a conversation up with people singing, We're already used to
Speaker 1: a degree of latency. We've incorporated that into our definition
Speaker 1: of normal. You don't have to you don't have to
Speaker 1: take latency to zero to make it sound normal. In
Speaker 1: other words, Yes, it's all about tricking your brain. Sound
Speaker 1: travels at roughly one foot per millisecond, and so if
Speaker 1: you're ten feet away from someone, it takes you ten
Speaker 1: milliseconds to actually hear something that they say or a
Speaker 1: note that they play on their instrument, and ours have
Speaker 1: evolved to adapt to that. And when we play music
Speaker 1: with a group, our brains are used to having people
Speaker 1: that are a certain distance away. Once they get further
Speaker 1: than twenty five feet away from you, it becomes really
Speaker 1: hard for you to stay in sync with them. So
Speaker 1: the trick here with these software solutions is simply to
Speaker 1: get under that twenty five threshold with a way that
Speaker 1: the sound is being transmitted exactly. And there's many steps
Speaker 1: along that path when you think about it. There's there's
Speaker 1: the path from let's say you're singing. There's the path
Speaker 1: from your voice your vocal chords to the microphone. Then
Speaker 1: there's the path from the microphone to your computer or
Speaker 1: whatever audio device you're using. There's the process of digitizing
Speaker 1: that audio, turning it from an analog sound into digital
Speaker 1: samples that can be buffered and transmitted electronically. And then
Speaker 1: there's the process of sending all that data from your
Speaker 1: computer to somebody else's computer and then essentially reversing the
Speaker 1: process in order to get into the headphones that someone's
Speaker 1: listening to you on. And each of those steps adds latency,
Speaker 1: and the key is just minimizing the latency of as
Speaker 1: many of those steps as possible along the way. Can
Speaker 1: you give a layperson's explanation of what we mean when
Speaker 1: we talk about buffer size? Sure? Buffer size is similar
Speaker 1: to if you're a shipping company and you're trying to
Speaker 1: send something from China to the United States. You have
Speaker 1: to load all of these containers onto a ship and
Speaker 1: then send that ship across the ocean to its destination port.
Speaker 1: All those containers then get unloaded and delivered to someone else.
Speaker 1: So each of those containers could be considered a buffer.
Speaker 1: The ship itself could also be considered a buffer. So
Speaker 1: you're kind of grouping things into these larger buckets and
Speaker 1: then transmitting them from one location to another. Describe virtual
Speaker 1: studio to make what is this thing that you have created?
Speaker 1: So the device it's basically using a Raspberry pie microcomputer
Speaker 1: or a single board computer that fits in the palm
Speaker 1: of your hand, and that this board only costs about
Speaker 1: thirty five dollars, and so I combine that with a
Speaker 1: sound card from a company called Hi Fi Berry, which
Speaker 1: is extremely low latency, and it plugs directly into this
Speaker 1: single board computer, so you don't have to go through
Speaker 1: a USB bus, which by itself adds latency by the way,
Speaker 1: it just talks directly to the CPU and other components
Speaker 1: on that board, and that allows it to achieve a
Speaker 1: latency as low as about one millisecond. But when I
Speaker 1: realized that I could do that, and I tested it
Speaker 1: and measured it and everything else, then I realized, Hey,
Speaker 1: this is really possible. We can really do something with this.
Speaker 1: Wait back up for a moment. So there is this
Speaker 1: thing Jack Trip, which comes out of Stanford. How do they,
Speaker 1: without without getting two swallowed up in the technical but
Speaker 1: explain to a computer dummy like me, how they propose
Speaker 1: to solve this latency problem. So you have this lag
Speaker 1: how does Jack Trip solve that problem? There's really no
Speaker 1: rocket science to it. What Jack Trip does is it
Speaker 1: just tries to take the audio packets, to extract them
Speaker 1: from the chip sets that are in your sound card
Speaker 1: and transmit them over the internet as quickly and as
Speaker 1: efficiently as possible. A large part of that is just
Speaker 1: having small buffer sizes, so jack Trip effectively just tries
Speaker 1: to minimize the overhead of transmitting this data over the networks.
Speaker 1: How long did it take you to build the first
Speaker 1: kind of prototype of this? Building the raspberry pie itself
Speaker 1: was fairly easy. It just took a few days to
Speaker 1: convert that over. And I had a few people helping
Speaker 1: test things with me. And I remember the first time
Speaker 1: I tested it myself, just as a loop back to
Speaker 1: the server and back to hear myself, and I was
Speaker 1: certain I did something wrong because it was instantaneous, or
Speaker 1: at least to me, it sounded instantaneous, and I thought, now,
Speaker 1: this can't possibly be sending. But yeah, the more I
Speaker 1: dug into it, the more I convinced myself that this
Speaker 1: was really happening. Were you Do you remember what? Were
Speaker 1: you singing or speaking? Do you remember why? I can't
Speaker 1: say you don't want to hear anything? No, I was
Speaker 1: just I was whistling a lot. For usually I whistled
Speaker 1: from my test. Sometimes I do try to sing if
Speaker 1: I know nobody's listening or in the room. If you
Speaker 1: do try to sing, what do you try to sing,
Speaker 1: I sing about my dog. This is like didn't what
Speaker 1: is it that Alexander Graham Bell said to his assistant
Speaker 1: on the first telephone transmission. Was it like, you know,
Speaker 1: doctor Watson, come quick or something like that? This is
Speaker 1: like up there with that? And when was that when
Speaker 1: you did that first test and you couldn't believe it?
Speaker 1: When was that? That was probably in May? Oh wow,
Speaker 1: this whole thing went really quickly. Yeah. Yeah. We shut
Speaker 1: things down here in California in mid March, late March.
Speaker 1: And I started out just you know, some evenings I
Speaker 1: would look at what's out there and spend a little
Speaker 1: time here and there. And the amount of time I
Speaker 1: spent grew more and more every week, until finally I
Speaker 1: started really putting things together and building things and testing things.
Speaker 1: By may, So, walk me through how Rigazzi does their rehearsals. Now,
Speaker 1: how do you put this into practical service? Well, interestingly,
Speaker 1: Rigassi still starts all of their rehearsals with Zoom, and
Speaker 1: we still use that as a tool to carry the
Speaker 1: video and connect everyone visually with one another. And then
Speaker 1: they in parallel to running Zoom, they run jack Trip
Speaker 1: and they use jack Trip for the audio and that
Speaker 1: means that the video is very heavily delayed compared to
Speaker 1: the audio. It takes a lot longer for you to
Speaker 1: see something versus when you hear it. And one of
Speaker 1: the downsides of that is that conducting, we've learned, is
Speaker 1: just not possible. You can't conduct visually, in particular with
Speaker 1: the gesticulations that you would normally see a conductor doing
Speaker 1: throughout a performance. So what the conductors have done and
Speaker 1: learned over time is that they have started conducting through audio.
Speaker 1: They give audio cues throughout a performance to either become
Speaker 1: a little bit louder, a little bit softer, a little
Speaker 1: bit faster, and when to cut off. All these things
Speaker 1: ended up moving from the visual side to the audio.
Speaker 1: So we still we still use the video just to
Speaker 1: help everybody seeing one another, helps give them a little
Speaker 1: bit more sense of presence. So everyone does. Every boy
Speaker 1: in the choir have to have his own little virtual studio. Yes,
Speaker 1: early on a major donor that allowed us to distribute
Speaker 1: this out to the entire chorus. It's about one hundred
Speaker 1: and fifty dollars for the device, and that doesn't include
Speaker 1: the accessories, which would be the microphone and some cables.
Speaker 1: So altogether it's about two hundred dollars. Did you when
Speaker 1: you set out to solve this particular rob you quit
Speaker 1: your old job to focus on this? Was that was? That?
Speaker 1: Was that a big step? Did that seem like a
Speaker 1: rash step? It still seems like a rash step. It was.
Speaker 1: It was certainly a monumental and life changing step for me.
Speaker 1: Um I also realized that this isn't something that's unique
Speaker 1: to Ragatzi. This is a problem that's really general across
Speaker 1: the entire world. And to me that was that was
Speaker 1: one of those moments where I realized that I could
Speaker 1: do something to help, and I may not ever have
Speaker 1: an opportunity to have an impact like I could at
Speaker 1: that moment. How do people I mean, so you have
Speaker 1: this thing that's now being used by Rigazzi, how do
Speaker 1: you get the Rigazzi cam't be the only choir out there?
Speaker 1: Or people who might I mean, have I come from
Speaker 1: a family where three members of my family are in
Speaker 1: church choirs and they've been grounded since March. I don't
Speaker 1: think they think it's even possible to be able to
Speaker 1: sing together. How have you gotten the word out about
Speaker 1: virtual studio? Different mechanisms I'm certainly in the Within the
Speaker 1: Bay area itself, we had a number of connections to
Speaker 1: other choruses and performing arts organizations. We've also been working
Speaker 1: a lot with Chorus America, which is a great organization
Speaker 1: of many choruses across the country. So we're slowly getting
Speaker 1: the information and the word out that this is possible
Speaker 1: through through not just our mechanism, but different mechanisms. So
Speaker 1: it'll be interesting to see as this progresses over the
Speaker 1: next several months. But wait, I would go even further.
Speaker 1: There's a whole category of live experiences which could be
Speaker 1: recreated using this. So I'm thinking, I mean, I don't
Speaker 1: know whether you're a basketball fan, but imagine an NBA
Speaker 1: game where everyone who's watching, every fan who's watching the game,
Speaker 1: has one of these devices in their house, So we
Speaker 1: could have real time cheering and fan reaction in a
Speaker 1: basketball game. We can recreate a reel. Right now, they're
Speaker 1: just doing fake crowd noise. Right it's a total phony experience.
Speaker 1: But if I have if I elimit, eliminate the latency
Speaker 1: for fans to react to a sporting event, I've I've
Speaker 1: got a virtual I've got a realistic virtual fan experience.
Speaker 1: Do I not. Yeah, absolutely, Honestly, I haven't even thought
Speaker 1: about sports. But I think you're totally correct, Mike, Mike,
Speaker 1: this is this is you can this is the future
Speaker 1: of this, This is That's the first thing I thought
Speaker 1: about when when I was thinking of because this is
Speaker 1: this huge because every single I don't know why I'm
Speaker 1: going on like this, but every single sporting event right
Speaker 1: now and that goes on television is suffering huge ratings declines.
Speaker 1: And that is because the experience of watching sports doesn't
Speaker 1: seem real. And if you're telling me, if it's this easy,
Speaker 1: I just have to I gotta buy it. I gotta
Speaker 1: buy a device if I want to play. But people
Speaker 1: are they were they were paying one hundred and fifty
Speaker 1: bucks for their ticket to the to one game before.
Speaker 1: I mean, it's not like people aren't used to investing
Speaker 1: in a fan experience. Yeah, it's a it's a great point.
Speaker 1: I think what one challenge we have is just with
Speaker 1: the video side of it is we can make the
Speaker 1: audio very quick, but you're still going to see things
Speaker 1: delayed on the video side. Potentially, if the delay of
Speaker 1: the video is the same for everyone who's watching, that
Speaker 1: may not be an issue, like they'll still hear each
Speaker 1: other in sync with what's happening. Yeah. Yeah. What's fascinating
Speaker 1: to me is how you start with a very very
Speaker 1: specific and very very personal problem, which is your son
Speaker 1: is bombed out that he can't sing with his chorus,
Speaker 1: And now you know, we're six months later and you're
Speaker 1: thinking about recreating the way in which we experience live music,
Speaker 1: and I'm going on and on about rescuing the NBA,
Speaker 1: And is it I mean, it's this sort of funny
Speaker 1: and fascinating picture of how innovation works. It is. It's
Speaker 1: It's amazing, isn't it. Yeah, that whole process of innovation
Speaker 1: being this this thing that builds and builds and changes
Speaker 1: over time and ways you don't even expect it to. Yeah.
Speaker 1: That's That's what I've always loved about technology. It's always
Speaker 1: drawn me into the world of entrepreneurism and creating products.
Speaker 1: And now I have the opportunity to do that in
Speaker 1: a way that involves something I really love, which is music,
Speaker 1: but also is helping people. Where could you go to
Speaker 1: listen in on something that's been performance has been conducted
Speaker 1: or recorded over virtual studio. The Rigassi Voye Chorus has
Speaker 1: recorded and published a number of our rehearsals online on YouTube,
Speaker 1: so the best place to see a few examples is
Speaker 1: probably at Rigazzi's YouTube channel. One last thing, can you
Speaker 1: give us a little sense of what listeners can do
Speaker 1: if they want to make use of this new idea
Speaker 1: or virtual studio or they want to spread the word.
Speaker 1: Do you have any practical some practical words of advice
Speaker 1: to our listeners. Yes, I'd suggest going to our website
Speaker 1: at jacktrip dot org, where we have a lot of
Speaker 1: different resources and people bowl who are volunteers available to
Speaker 1: help answer questions. Mike Dickie is co founder and chairman
Speaker 1: of the Jack Tripp Foundation and the creator of Virtual Studio.
Speaker 1: And we're hearing the Ragatzi Boys Choir from California rehearsing
Speaker 1: from over thirty locations at once using jack Trip and
Speaker 1: Virtual Studio. Be sure to check out our show notes
Speaker 1: to learn more about online music collaboration and ways you
Speaker 1: can get involved. Next week, Unsolvable, just ahead of Thanksgiving,
Speaker 1: retired NASA astronaut Scott Kelly. We'll talk about how social
Speaker 1: isolation is solvable even when feels like you're trapped alone
Speaker 1: so many miles and miles away from those who love most.
Speaker 1: I hope you'll join us. Solvable is brought to you
Speaker 1: by Pushkin Industries. Our show is produced by Camille Baptista,
Speaker 1: Senior producer jos and Frank. Catherine Gerardo is our managing
Speaker 1: producer and our Executive producer is Miilabell. Special thanks to
Speaker 1: Kobe Gilford, Heather Fane, Eric Sandler, Carle Migliori and Kadisha Holland.
Speaker 1: I'm Malcolm Bobber