Speaker 1: Bushkim, this is solvable. I'm Malcolm Glabo. Occasionally we would
Speaker 1: get to go and break out rooms on Zoom and
Speaker 1: just discuss our feelings about what's happening at the time,
Speaker 1: and when the lockdown began, we were all devastated because
Speaker 1: singing was our life. Sam is twelve years old and
Speaker 1: like many kids, he's struggling with the restrictions COVID nineteen
Speaker 1: has imposed on his life. There's a lot of talking
Speaker 1: to news about the impact of COVID on classroom education,
Speaker 1: but for Sam and his friends, what they're missing is
Speaker 1: after school. Ragotzi was like a sanctuary to go to
Speaker 1: at the end of the school day. Rogatsi is a
Speaker 1: boy's choir based in California. Sam's been singing in the
Speaker 1: group for seven years. In pre COVID times, he would
Speaker 1: rehearse twice a week for several hours each day, but
Speaker 1: all of that came to a halt when COVID arrived
Speaker 1: in the US. The choir was basically stuck. If you've
Speaker 1: ever tried to sing something like happy Birthday during a
Speaker 1: zoom meeting, you know what I'm talking about. Making music
Speaker 1: sync up with other people in real time over the Internet.
Speaker 1: Is really tough. The connectivity delays, what they call latency,
Speaker 1: can be pretty awkward. Give us the technical explanation of
Speaker 1: the problem your son was facing. It's rooted in latency. Fundamentally,
Speaker 1: the time that it takes the sound you make to
Speaker 1: reach another musician that you're performing with, you need to
Speaker 1: have a latency of about less than twenty five milliseconds.
Speaker 1: Otherwise the sense of the pulse or the beat of
Speaker 1: the music is just lost across those musicians. Mike Dickie
Speaker 1: is a software engineer, and after months of seeing his
Speaker 1: son Sam bummed out about not singing, he realized he
Speaker 1: could do something about it now for far less than
Speaker 1: the cost of a year of private music lessons. Dickie
Speaker 1: created a tool that allows groups of musicians to make
Speaker 1: music online. And it might be just the beginning. So
Speaker 1: we could have real time cheering and fan reaction in
Speaker 1: a basketball game. Yeah. Absolutely, Performing in rehearsing with others
Speaker 1: in real time is really difficult or even impossible for
Speaker 1: many people. But this problem is solvable. I talked with
Speaker 1: Mike Dickie about Virtual Studio. It's a low cost device
Speaker 1: that works with the open source tool called jack Trip
Speaker 1: to make remote music collaboration possible in real time. Here
Speaker 1: is our conversation. Mike tell me about the first time
Speaker 1: this that Rigazzi rehearse using virtual studio. What was that like?
Speaker 1: At first? It was a bit terrifying progress out let's
Speaker 1: progress down, okay, And I thought of all the things
Speaker 1: that could go wrong, And I knew that this was
Speaker 1: all very new and untested, And I was also worried
Speaker 1: just of what people would think of me and my
Speaker 1: son if this was a huge disaster. But when we
Speaker 1: when we started getting people online and it started to work,
Speaker 1: and the boys started to sing right. It was the
Speaker 1: first time they were singing together in a long time,
Speaker 1: and I could see through the zoom the faces on
Speaker 1: the different boys start to light up. I could see
Speaker 1: the smiles, I could see it in their eyes, and
Speaker 1: I could see it in the director's eyes, thinking, Hey,
Speaker 1: this actually is working. That experience of of seeing people
Speaker 1: and having them so happy and having their experience with
Speaker 1: others impact their lives, that's really what led me to
Speaker 1: put more and more time into this and ultimately to
Speaker 1: devote all my time towards it. Prior to the lockdown,
Speaker 1: people must have thought about this problem and tried to
Speaker 1: solve it. Were there existing technical solutions to the problem
Speaker 1: of how to sing remotely? There were, thankfully. I looked
Speaker 1: at several of them when I started out, and I
Speaker 1: came across this small project from out of Stanford University
Speaker 1: called jack Trip, which is an open source technology that
Speaker 1: was developed over a year ago by the Computer Music Department.
Speaker 1: It seemed very promising. It had a high quality, lossless audio,
Speaker 1: it focused on low latency and over the Internet, and
Speaker 1: there were a lot of people already using it. The
Speaker 1: challenge that I found with that is that it was
Speaker 1: a command line tool. You had to open a terminal
Speaker 1: on your computer. In many cases, you had to build
Speaker 1: it from the source code, the C plus plus source code,
Speaker 1: and then you had to know the command lines and
Speaker 1: figure out the right command line options in order to
Speaker 1: use it with other people. That's great for somebody who's
Speaker 1: very technically oriented, but in my son's chorus, I was
Speaker 1: trying to figure out, how can we roll this out
Speaker 1: to a large number of young boys that don't have
Speaker 1: this advanced technical expertise. And it was actually a challenge
Speaker 1: that I thought I was well suited for because I
Speaker 1: understood the technical side, and I also had the product
Speaker 1: background in order to try to make something that was
Speaker 1: fairly difficult to use fairly easy for most of the
Speaker 1: people that were out there. So we're used to is
Speaker 1: it an in normal sonic collaboration, a conversation with people singing,
Speaker 1: We're already used to a degree of latency. We've incorporated
Speaker 1: that into our definition of normal. You don't have to
Speaker 1: you don't have to take latency to zero to make
Speaker 1: it sound normal. In other words, yes, it's all about
Speaker 1: tricking your brain. Sound travels at roughly one foot per millisecond,
Speaker 1: and so if you're ten feet away from someone, it
Speaker 1: takes you ten milliseconds to actually hear something that they
Speaker 1: say or a note that they play on their instrument.
Speaker 1: And our brains have evolved to adapt to that. And
Speaker 1: when we play music with a group, our brains are
Speaker 1: are used to having people that are a certain distance away.
Speaker 1: Once they get further than twenty five feet away from you,
Speaker 1: it becomes really hard for you to stay in sync
Speaker 1: with them. So the trick here with these software solutions
Speaker 1: is simply to get under that twenty five threshold with
Speaker 1: the way that the sound is being transmitted exactly, and
Speaker 1: there's many steps along that path. When you think about it,
Speaker 1: there's there's the path from let's say you're singing. There's
Speaker 1: the path from your voice your vocal chords to the microphone.
Speaker 1: Then there's the path from the microphone to your computer
Speaker 1: or whatever audio device you're using. There's the process of
Speaker 1: digitizing that audio, turning it from an analog sound into
Speaker 1: digital samples that can be buffered and transmitted electronically. And
Speaker 1: then there's the process of sending all that data from
Speaker 1: your computer to somebody else's computer and then essentially reversing
Speaker 1: the process in order to get into the headphones that
Speaker 1: someone's listening to you on. And each of those steps
Speaker 1: adds latency, and the key is just minimizing the latency
Speaker 1: of as many of those steps as possible along the way.
Speaker 1: Can you give a layperson's explanation of what we mean
Speaker 1: when we talk about buffer size? Sure? Buffer size is
Speaker 1: similar to if you're a shipping company and you're trying
Speaker 1: to send something from China to the United States. You
Speaker 1: have to load all of these containers onto a ship
Speaker 1: and then send that ship across the ocean to its
Speaker 1: destination port. All those containers then get unloaded and delivered
Speaker 1: to someone else. So each of those containers could be
Speaker 1: considered a buffer. The ship itself could also be considered buffer.
Speaker 1: So you're kind of grouping things into these larger buckets
Speaker 1: and then transmitting them from one location to another. Describe
Speaker 1: virtual studio to make what is this thing that you
Speaker 1: have created? So the device, it's basically using a Raspberry
Speaker 1: Pie microcomputer or a single board computer that fits in
Speaker 1: the palm of your hand, and that this board only
Speaker 1: costs about thirty five dollars. And so I combine that
Speaker 1: with a sound card from a company called hi Fiberry,
Speaker 1: which is extremely low latency, and it plugs directly into
Speaker 1: this single board computer, so you don't have to go
Speaker 1: through a USB bus, which by itself adds latency. By
Speaker 1: the way, it just talks directly to the CPU and
Speaker 1: other components on that board, and that allows it to
Speaker 1: achieve a latency as low as about one millisecond. But
Speaker 1: when I realized that I could do that, and I
Speaker 1: tested it and measured it and everything else, then I realized, hey,
Speaker 1: this is really possible. We can really do something with this.
Speaker 1: Wait back up for a moment. So there is this
Speaker 1: thing Jack Trip, which comes out of Stanford. How do
Speaker 1: they without without getting too swallowed up in the technical
Speaker 1: but explain to a computer dummy like me, how they
Speaker 1: propose to solve this latency problem. So you have this,
Speaker 1: you have this lag How does jack trip solve that problem?
Speaker 1: There's really no rocket science to it. What jack trip
Speaker 1: does is it just tries to take the audio packets,
Speaker 1: to extract them from the chip sets that are in
Speaker 1: your sound card, and transmit them over the internet as
Speaker 1: quickly and as efficiently as possible. A large part of
Speaker 1: that is just having small buffer sizes, so jack Trip
Speaker 1: effectively just tries to minimize the overhead of transmitting this
Speaker 1: data over the networks. How long did it take you
Speaker 1: to build the first kind of prototype of this? Building
Speaker 1: the Raspberry Pie itself was fairly easy. It just took
Speaker 1: a few days to convert that over and I had
Speaker 1: I had a few people helping test things with me,
Speaker 1: And I remember the first time I tested it myself,
Speaker 1: just as a loop back to the server and back
Speaker 1: to hear myself, and I was certain I did something
Speaker 1: wrong because it was instantaneous, or at least to me
Speaker 1: it sounded instantaneous, and I thought, now this can't possibly
Speaker 1: be sending. But Yeah. The more I dug into it,
Speaker 1: the more I convinced myself that this was really happening.
Speaker 1: Were you do you remember what were you singing or speaking?
Speaker 1: Do you remember why I can't sing? You don't want
Speaker 1: to hear me sing? No, I was just I was
Speaker 1: whistling a lot. For usually I whistled from my test.
Speaker 1: Sometimes I do try to sing if I know nobody's
Speaker 1: listening or in the room. If you do try to sing,
Speaker 1: what do you try to sing? I sing about my dog?
Speaker 1: This is like didn't what is it that Alexander Graham
Speaker 1: Bell said to his assistant on the first telephone transmission.
Speaker 1: Was it like, you know, doctor Watson, come quick or
Speaker 1: something like that? This is like up there with that?
Speaker 1: And when was that when you did that first test
Speaker 1: and you couldn't believe it? When was that? That was
Speaker 1: probably in May? Oh wow, this whole thing went really quickly. Yeah. Yeah.
Speaker 1: We shut things down here in California in mid March,
Speaker 1: late March, and I started out just, you know, some evenings,
Speaker 1: I would look at what's out there and spend a
Speaker 1: little time here and there, And the amount of time
Speaker 1: I spent grew more and more every week, until finally
Speaker 1: I started really putting things together and building things and
Speaker 1: testing things by may so walk me through how Rigazzi
Speaker 1: does their rehearsals. Now, how do you put this into
Speaker 1: practical service? Well, interestingly, Rigassi still starts all of their
Speaker 1: rehearsals with Zoom, and we still use that as a
Speaker 1: tool to carry the video and connect everyone visually with
Speaker 1: one another. And then they in parallel to running Zoom,
Speaker 1: they run Jack and they use jack trip for the audio.
Speaker 1: And that means that the video is very heavily delayed
Speaker 1: compared to the audio. Takes a lot longer for you
Speaker 1: to see something versus when you hear it. And one
Speaker 1: of the downsides of that is that conducting, we've learned,
Speaker 1: is just not possible. You can't conduct visually, in particular
Speaker 1: with the gesticulations that you would normally see a conductor
Speaker 1: doing throughout a performance. So what the conductors have done
Speaker 1: and learned over time is that they have started conducting
Speaker 1: through audio. They give audio cues throughout a performance to
Speaker 1: either become a little bit louder, a little bit softer,
Speaker 1: a little bit faster, and when to cut off. All
Speaker 1: these things ended up moving from the visual side to
Speaker 1: the audio. So we still we still use the video
Speaker 1: just to help everybody seeing one another helps give them
Speaker 1: a little bit more sense of presence. So everyone does.
Speaker 1: Every boy in the choir have to have his own
Speaker 1: little virtual studio. Yes, early on a major donor that
Speaker 1: allowed us to distribute this out to the entire chorus.
Speaker 1: It's about one hundred and fifty dollars for the device,
Speaker 1: and that doesn't include the accessories, which it would be
Speaker 1: the microphone and some cables, so altogether it's about two
Speaker 1: hundred dollars. Did you when you set out to solve
Speaker 1: this particular problem you quit your old job to focus
Speaker 1: on this? Was that? Was? That? Was that a big step?
Speaker 1: Did that seem like a rash step? It still seems
Speaker 1: like a rash step? It was. It was certainly a
Speaker 1: monumental and life changing step for me. I also realized
Speaker 1: that this isn't something that's unique to Ragatzi. This is
Speaker 1: a problem that's really general across the entire world. And
Speaker 1: to me that was that was one of those moments
Speaker 1: where I realized that I could do something to help,
Speaker 1: and I may not ever have an opportunity to have
Speaker 1: an impact like I could at that moment. How do
Speaker 1: people I mean, so you have this thing that's now
Speaker 1: being used by Rigazzi. How do you get the Ragazzi
Speaker 1: can't be the only choir out there or people who
Speaker 1: might I mean, have I come from a family where
Speaker 1: three members of my family are in church choirs and
Speaker 1: they've been grounded since March. I don't think they think
Speaker 1: it's even possible to be able to sing together. How
Speaker 1: have you gotten the word out about virtual studio through
Speaker 1: different mechanisms? I'm certainly in Within the Bay Area itself,
Speaker 1: we had a number of connections to other choruses and
Speaker 1: performing arts organizations. We've also been working a lot with
Speaker 1: Chorus America, which is a great organization of many choruses
Speaker 1: across the country. So we're slowly getting the information and
Speaker 1: the word out that this is possible through through not
Speaker 1: just our mechanism, but different mechanisms. So it'll be interesting
Speaker 1: to see this as this progresses over the next several months.
Speaker 1: But wait, I would go even further. There's a whole
Speaker 1: category of live experiences which could be recreated using this.
Speaker 1: So I'm thinking, I mean, I don't know whether you're
Speaker 1: a basketball fan, but imagine an NBA game where everyone
Speaker 1: who's watching every fan who's watching the game has one
Speaker 1: of these devices in their house, So we could have
Speaker 1: real time cheering and fan reaction in a basketball game.
Speaker 1: We can recreate a reel. Right now, they're just doing
Speaker 1: fake crowd noise. Right it's a total phony experience. But
Speaker 1: if I have, if I eliminate, eliminate the latency for
Speaker 1: fans to react to a sporting event, I've I've got
Speaker 1: a virtual I've got a realistic virtual fan experience. Do
Speaker 1: I not? Yeah? Absolutely, honestly, I haven't even thought about sports.
Speaker 1: But you think you're totally correct, Mike, Mike, this is
Speaker 1: this is you can this is the future of this
Speaker 1: is That's the first thing I thought about when when
Speaker 1: I was thinking about it, because this is this huge
Speaker 1: because every single I don't know why I'm going on
Speaker 1: like this, but every single sporting event right now and
Speaker 1: that goes on television is suffering huge ratings to climb.
Speaker 1: And that is because the experience of watching sports doesn't
Speaker 1: seem real. And if you're telling me, if it's this easy,
Speaker 1: I just have to I gotta buy it. I gotta
Speaker 1: buy a device if I want to play. But people
Speaker 1: are there, they were paying one hundred and fifty bucks
Speaker 1: for their ticket to one game before. I mean, it's
Speaker 1: not like people aren't used to investing in a fan experience. Yeah,
Speaker 1: it's a it's a great point. I think what one
Speaker 1: challenge we have is just with the video side of it,
Speaker 1: as we can make the audio very quick, but you're
Speaker 1: still going to see things delayed on the video side. Potentially,
Speaker 1: if the delay of the video is the same for
Speaker 1: everyone who's watching, that may not be an issue, Like
Speaker 1: they'll still hear each other yea, in sync with what's happening. Yeah. Yeah.
Speaker 1: What's fascinating to me is how you start with a
Speaker 1: very very specific and very very personal problem, which is
Speaker 1: your son is bombed out that he can't sing with
Speaker 1: his chorus. And now you know, we're six months later
Speaker 1: and you're thinking about recreating the way in which we
Speaker 1: experienced live music and I'm going on and on about
Speaker 1: rescuing the NBA, And is it I mean, it's this
Speaker 1: sort of funny and fascinating picture of how innovation works.
Speaker 1: It is. It's it's amazing, isn't it. Yeah? That whole
Speaker 1: process of innovation being this this thing that that builds
Speaker 1: and builds and changes over time, and ways you don't
Speaker 1: even expect it to. That's That's what I've always loved
Speaker 1: about technology. It's always drawn me into the world of
Speaker 1: entrepreneurism and creating products and and now I have the
Speaker 1: opportunity to do that in a way that involves something
Speaker 1: I really love, which is music, but also is helping people.
Speaker 1: Where could you go to listen in on something that's
Speaker 1: been performance has been conducted or recorded over virtual studio.
Speaker 1: The Rigassi Boys Chorus has recorded and published a number
Speaker 1: of our rehearsals online on YouTube, so the best place
Speaker 1: to see a few examples is probably at Rigazzi's YouTube channel.
Speaker 1: One lasting m Can you give us a little sense
Speaker 1: of what listeners can do if they want to make
Speaker 1: use of this new idea or virtual studio or they
Speaker 1: want to spread the word. Do you have any practical
Speaker 1: some practical words of advice to our listeners. Yes, I'd
Speaker 1: suggest going to our website at Jacktrip dot org, where
Speaker 1: we have a lot of different resources and people who
Speaker 1: are volunteers available to help answer questions. Mike Dickie is
Speaker 1: co founder and chairman of the Jack Trip Foundation and
Speaker 1: the creator of Virtual Studio. And we're hearing the Ragatzi
Speaker 1: Boys choir from California, rehearsing from over thirty locations at
Speaker 1: once using jack Trip and Virtual Studio. Be sure to
Speaker 1: check out our show notes to learn more about online
Speaker 1: music collaboration and ways you can get involved. Next week, Unsolvable,
Speaker 1: just ahead of Thanksgiving, retired NASA astronaut Scott Kelly. We'll
Speaker 1: talk about how social isolation is solvable even when it
Speaker 1: feels like you're trapped alone, so many miles and miles
Speaker 1: away from those who love most. I hope you'll join us.
Speaker 1: Solvable is brought to you by Pushkin Industries. Our show
Speaker 1: is produced by Camille Baptista, Senior Producer jos and Frank.
Speaker 1: Katherine Girardoe is our managing producer and our executive producer
Speaker 1: is Miilabell. Special thanks to Kobe Gilford, Head of Fane,
Speaker 1: Eric Sandler, Carly Migliori, and Kadisha Holland. I'm Malcolm Bomba
Speaker 1: Zero