Speaker 1: Welcome to Tech Stuff, a production from iHeartRadio. Today, we
Speaker 1: are witnessed to one of those rare moments in history,
Speaker 1: the rise of an innovative technology with the potential to
Speaker 1: radically transform business in society forever. That technology, of course,
Speaker 1: is artificial intelligence, and it's the central focus for this
Speaker 1: new season of Smart Talks with IBM. Join hosts from
Speaker 1: your favorite Pushkin podcasts as they talk with industry experts
Speaker 1: and leaders to explore how businesses can integrate AI into
Speaker 1: their workflows and help drive real change in this new
Speaker 1: era of AI, and of course, host Malcolm Gladwell will
Speaker 1: be there to guide you through the season and throw
Speaker 1: in his two cents as well. Look out for new
Speaker 1: episodes of Smart Talks with IBM every other week on
Speaker 1: the iHeartRadio app, Apple Podcasts, wherever you get your podcasts,
Speaker 1: and learn more at IBM dot com, slash smart Talks.
Speaker 2: Pushkin Hello, Hello, Welcome to Smart Talks with IBM, a
Speaker 2: podcast from Pushkin Industries, iHeartRadio and IBM. I'm Malcolm Gladwell.
Speaker 2: This season, we're diving back into the world of artificial intelligence,
Speaker 2: but with a focus on the powerful concept of open
Speaker 2: its possibilities, implications, and misconceptions. We'll look at openness from
Speaker 2: a variety of angles and explore how the concept is
Speaker 2: already reshaping industries, ways of doing business, and a very
Speaker 2: notion of what's possible. In today's episode, I sat down
Speaker 2: with Mo Duffy, software engineering manager at red Hat, who
Speaker 2: works on instruct Lab, a project co developed by red
Speaker 2: Hat and IBM. Most shared with me how this a
Speaker 2: new initiative, is revolutionizing AI training, making it not only
Speaker 2: more accessible, but also more inclusive. This project, unique in
Speaker 2: the industry, allows developers to submit incremental contributions to one
Speaker 2: base AI model, creating a continuous loop of development, much
Speaker 2: like normal open source software. By leveraging community contributions and
Speaker 2: IBM's cutting edge granite models, Mo in the team of
Speaker 2: ibmrs and red hatters are paving the way for a
Speaker 2: future where AI development is a communal endeavor. Our insights
Speaker 2: into open source software extend beyond technical proficiency to the
Speaker 2: profound impact of collaborative effort. At the heart of Moe's
Speaker 2: work is a belief in democratizing technology, ensuring that AI
Speaker 2: becomes a tool accessible to all. So let's explore how Moe,
Speaker 2: red Hat and IBM are empowering individuals and businesses alike
Speaker 2: to reshape the fuel future of technology through collaboration and innovation.
Speaker 2: We thank you for joining me today. Thank you so
Speaker 2: much for I have just about the most Irish name ever.
Speaker 2: I do very proud you weren't born in Ireland my grandparents,
Speaker 2: Oh your grandparents, I see? Where did you grow up?
Speaker 3: New York Queens?
Speaker 2: Oh you're l a see. So tell me a little
Speaker 2: bit about how how you got to red Hat. What
Speaker 2: was your path?
Speaker 3: When I was in high school? It was a chatty girl,
Speaker 3: teenage girl on the phone. We had one phone line.
Speaker 3: My older brother was studying at the local state college
Speaker 3: computer science, and he had to tell that end to
Speaker 3: compile his homework one phone line, and I'm on it
Speaker 3: all the time. He got very frustrated and he needed
Speaker 3: a compiler to do his homework. So he bought red
Speaker 3: Hat Linux from a CompUSA, brought it home and that
Speaker 3: was on the family computer. So I learned Linux and
Speaker 3: I started playing around with it. I really liked it
Speaker 3: because you could customize everything, like the entire user interface.
Speaker 3: You could actually modify the code of the programs you
Speaker 3: were using to do what you wanted. And for me,
Speaker 3: It was really cool because especially when you're a kid
Speaker 3: and like people tell you this is the way things
Speaker 3: are and you just have to deal with it. It's
Speaker 3: nice to be like I'm going to make things the
Speaker 3: way I want, modify the code and playing. Yeah, it
Speaker 3: was amazing and it was just such a time and
Speaker 3: like before it was cool, I was doing it and
Speaker 3: what I saw on that is sort of the potential
Speaker 3: like number one of like a community of people working together.
Speaker 3: And like the Internet existed, it was slow, it involved modems,
Speaker 3: but there were people that you could talk to who
Speaker 3: would give you tips and you'd share information, and this
Speaker 3: collaborative building something together is really something special. Right. I
Speaker 3: could file a complaint to whatever large software company made
Speaker 3: whatever software I was into, or I could go to
Speaker 3: an open source software community and be like, hey, guys,
Speaker 3: I think we should do this, and like, yeah, okay,
Speaker 3: I'll help, I'll pitch in. So you don't feel powerless,
Speaker 3: you feel like you can have an impact, and that
Speaker 3: was really exciting to me. However, open source software has
Speaker 3: a reputation for not having the best user interface, not
Speaker 3: the best user experience. So I ended up studying computer
Speaker 3: science and Electronic Media dual major, and then I did
Speaker 3: Human computeraction as my master's and my thought was, wouldn't
Speaker 3: it be nice if this free software accessible to anybody,
Speaker 3: if it was easier to use, some more people could
Speaker 3: use it and take advantage of it. And so, long
Speaker 3: story short, I ended up going to red Hat saying, Hey,
Speaker 3: I want to learn how you guys work. Let me
Speaker 3: embed in your team draft out of my graduate program.
Speaker 3: And I'm like, I want to do this for a living.
Speaker 3: This is cooler. So I thought this is the way
Speaker 3: to go, and I've been there ever since. They haven't
Speaker 3: been able to get rid of me.
Speaker 2: To backtrack this a little bit, you were talking about
Speaker 2: the sense of community that surrounds this way of thinking
Speaker 2: about software. Talk a little bit more about what that
Speaker 2: community is like, the benefits of that community, why it
Speaker 2: appeals to you.
Speaker 3: Sure, well, you know part of the reason I actually
Speaker 3: ended up going going to the graduate school track. Suddenly
Speaker 3: you're a peer of your professors and you're working side
Speaker 3: by side with them. At some point they retire and
Speaker 3: you're in the next generation. So it's sharing information, building
Speaker 3: on the work of others in sort of this cycle
Speaker 3: that extends past the human lifespan and in the same way,
Speaker 3: Like the open source model is very similar, but you're
Speaker 3: actually you're building something, and it's something in me. I'm
Speaker 3: just really attracted. Like I don't like talking about stuff.
Speaker 3: I like doing stuff with open source software. The software
Speaker 3: doesn't cost anything, the code is out there, generally uses
Speaker 3: open standards for the file formats. I can open up
Speaker 3: files that I created and open source tools as a
Speaker 3: high school student today because they were using open formats
Speaker 3: and that software still exists. I can still compile the
Speaker 3: code and it's an active community project. Like these things
Speaker 3: can outlast any single company in the same way that
Speaker 3: the academic community has been going on for so many
Speaker 3: years and hopefully we'll continue moving on. So it's sort
Speaker 3: of like not just the community around it, but just
Speaker 3: the knowledge sharing and also bringing up the next generation
Speaker 3: as well. Like all of that stuff really appealed to me.
Speaker 3: And also at the center of it, the fact that
Speaker 3: we could democratize it by following this open source process
Speaker 3: and feel like we have some control, We're not at
Speaker 3: the mercy of some faceless corporation making changes and we
Speaker 3: have no impact. Like that really appealed to me too.
Speaker 2: Yeah, for those of us who are not software phishonados,
Speaker 2: take a step backwards and give me a kind of
Speaker 2: description of terms. What's the opposite of open source proprietary?
Speaker 3: Proprietary is what we say, So.
Speaker 2: Specifically and practically, the difference would be what between something
Speaker 2: that was opens us in something that was proprietary.
Speaker 3: Sure, so there's a lot of difference. So with open
Speaker 3: source software you get these rights when you're given the software,
Speaker 3: you get the right to be able to share it.
Speaker 3: And depending on the lot, different licenses that are considered
Speaker 3: open source have different little things that you have to
Speaker 3: be aware of. With proprietary code, it's one copyright the company.
Speaker 3: Even a lot of times, when you sign your employment
Speaker 3: contract for a software company and you write code for them,
Speaker 3: you don't own it. You sign over your rights to
Speaker 3: the company. So if you leave the company, the code
Speaker 3: doesn't go with you. It stays in the ownership of
Speaker 3: that company. So then one like one company buys out
Speaker 3: another and kills a product, that code's gone.
Speaker 2: It's gone. For a business, why would a business want
Speaker 2: to be have open source code as opposed to proprietary.
Speaker 3: Well, for the same reasons, Like, say you're a business,
Speaker 3: You've invested all this money into this software platform right,
Speaker 3: and you've upskilled your employees on it and it's a
Speaker 3: core part of your business, and then a few years
Speaker 3: later that company goes out of business or something happens,
Speaker 3: or even something less drastic. You really need this future,
Speaker 3: but for the company that makes the software, it's not
Speaker 3: in their best interests. It's not worth the investment. They're
Speaker 3: not going to do it. How do you get that future?
Speaker 3: You either have to completely migrate to another solution, and
Speaker 3: this is something that's core at your business that's going
Speaker 3: to be a big deal to But if it's open source,
Speaker 3: you could either hire a team of experts. You could
Speaker 3: hire software engineers who are able to go do this
Speaker 3: for you. Go in the upstream software community, implement the
Speaker 3: feature that you want, and it'll be rolled into the
Speaker 3: next version of that company software. So even if that
Speaker 3: company didn't want to implement the future, if they did
Speaker 3: it open source, they would inherit that feature from the
Speaker 3: upstream community, is what we call it, So you have
Speaker 3: some control over the situation. If it's open source, you
Speaker 3: have an opportunity to actually affect change in the product,
Speaker 3: and you could then pick it up or pay somebody
Speaker 3: else to pick it up, or another company could form
Speaker 3: and pick it up and keep it going, So there's
Speaker 3: more possibilities. If it's open source, it's more like it's
Speaker 3: like an insurance policy almost.
Speaker 2: So innovation from the standpoint of the customer, innovation is
Speaker 2: a lot easier when you're working in an open source environment.
Speaker 3: Absolutely.
Speaker 2: Yeah. So now at RedHat, you're working with something called
Speaker 2: instruct lab. Tell us a little bit about what that is.
Speaker 3: So the thing that really excites to me about getting
Speaker 3: to work on this project is AI is sort of
Speaker 3: that has been this scary thing for me because it's
Speaker 3: one of those things like in order to be able
Speaker 3: to pre train a model, you have to have unobtainium GPUs,
Speaker 3: you have to have rich resources. It takes months, it
Speaker 3: takes expertise. There's a small handful of companies that can
Speaker 3: build a model from pre train to something usable, and
Speaker 3: it kind of feels like those early days when I
Speaker 3: was kind of delving in software and the same way.
Speaker 3: I think if more people could contribute to AI models,
Speaker 3: then it wouldn't be just influenced by whichever company had
Speaker 3: the resources to build it. And there's been a lot
Speaker 3: of emphasis on pre training models, so taking massive terabytes
Speaker 3: data sets, throwing them through masses of GPUs over months
Speaker 3: of time, spending hundreds of millions of dollars to build
Speaker 3: a base model. But when instruct lab does is say, okay,
Speaker 3: you have a base model. We're going to fine tune in.
Speaker 3: On the other end, it takes less compute resources. The
Speaker 3: way we've built in struck lab, you can play around
Speaker 3: with the technology and learn it on it off the
Speaker 3: shelf laptop that you can actually buy. So in this
Speaker 3: way we're enabling a much broader set of people to
Speaker 3: play with AI, to contribute it, to modify it. And
Speaker 3: I'll tell you one story from red Hat. Succi, who
Speaker 3: is our chief diversity officer, very interested in inclusive language
Speaker 3: and open source software, doesn't have any experience with AI.
Speaker 3: We have a community model that we have an upstream
Speaker 3: project around for people to contribute knowledge and skills to
Speaker 3: the model. She's like, I want to teach the model
Speaker 3: how to use inclusive language, like replace this word with
Speaker 3: this word or this word with this word. Oh my,
Speaker 3: oh that's so cool. So she paired up with Nicholas
Speaker 3: who is a technical guy at red Hat, and they
Speaker 3: built and submitted a skill to the model that you
Speaker 3: can just tell the model, can you please take this
Speaker 3: document and translate this language to more inclusive language, and
Speaker 3: it will do it. And they submitted it to the community.
Speaker 3: They were so proud. It was like, that's the kind
Speaker 3: of thing that, like, you know, maybe a company would
Speaker 3: be incentivized to do that, but if you have some
Speaker 3: tooling that's open source and something that anybody could access,
Speaker 3: then those communities could actually get together and build that
Speaker 3: knowledge into AI models.
Speaker 2: Just so understand, what you guys have is the structure
Speaker 2: for an AI system, and in other cases, individual companies
Speaker 2: own and train their own AI systems. It takes enormous
Speaker 2: amount of resources. They hoover up all kinds of information,
Speaker 2: train it according to their own hidden set of rules,
Speaker 2: and then a customer might use that for some price.
Speaker 2: What you're saying is, in the same way that we
Speaker 2: democratize the writing of software before, let's democratize the training
Speaker 2: of an AI system. So anyone can contribute here and
Speaker 2: teach the model the things that they're interested in teaching
Speaker 2: the model. I'm guessing correct me. On the one hand,
Speaker 2: this model, at least in the beginning, is going to
Speaker 2: have a lot fewer resources available to it. But on
Speaker 2: the other hand, it's going to have a much more
Speaker 2: diverse set of inputs.
Speaker 3: That's right. And the other thing is that IBM, basically
Speaker 3: is part of this project, has something called the Granite
Speaker 3: Model family, and they've donated some granite models. So these
Speaker 3: are the ones that take the months and terabytes of
Speaker 3: data and all the GPUs to train. So IBM has
Speaker 3: created one of those, and they have listed out and
Speaker 3: linked to the data sets that they used, and they
Speaker 3: talk about the relative proportions they used when pre training,
Speaker 3: so it's not just a black box. You know where
Speaker 3: the data came from, which is a pretty open position
Speaker 3: to take. That is what we recommend as the base.
Speaker 3: So you use the instruct lab tuning. You take this
Speaker 3: base granite model that IBM has provided, and you use
Speaker 3: the instruct lab tooling that red Hat works on, and
Speaker 3: you use that to fine tune the model to make
Speaker 3: it whatever you want.
Speaker 2: I want to go back to the partnership between IBM
Speaker 2: and red Hat here with them providing the granite model
Speaker 2: to your instruct lab. Is this the first ti I'm
Speaker 2: red Hat and IBM have collaborated like this.
Speaker 3: I think it's something that's been going on, Like another
Speaker 3: a product within the red Hat family would be open
Speaker 3: Shift AI, where they collaborate a lot with IBM Research team,
Speaker 3: like BLM is one of the components of that product
Speaker 3: that there's a nice kind of exchange and collaboration between
Speaker 3: the two companies.
Speaker 2: How large is the potential community of people who might
Speaker 2: contribute to instruct lab It could.
Speaker 3: Be thousands of people. I mean, we'll see, it's early days.
Speaker 3: This is early technology that was invented at IBM Research
Speaker 3: that they partnered with us at red Hat to kind
Speaker 3: of build the software around it. There's still more to go,
Speaker 3: Like right now, we have a team in the community
Speaker 3: that's actually trying to build a web interface to make
Speaker 3: it easier for anybody to contribute. So we have a
Speaker 3: lot of those sort of user experience for the contributor
Speaker 3: to the model stuff to work out that we're still
Speaker 3: actively building on. But like my vision for it even
Speaker 3: is I like going back to that academic model of
Speaker 3: learning from what others and building upon it over time.
Speaker 3: It would be very good for us to sort of
Speaker 3: go out and try to collaborate with academics of all fields, like, hey,
Speaker 3: you know, the model doesn't know about your field, would
Speaker 3: you like to put something into the model about your
Speaker 3: field so it knows about it, or even you know,
Speaker 3: talk to the model it got it wrong, let's correct it.
Speaker 3: Can we lean on your expertise to correct it and
Speaker 3: make sure it gets it right and sort of use
Speaker 3: that community model as a way for everybody to collaborate
Speaker 3: because before instruct Lab, my understanding is if you wanted
Speaker 3: to take a model that's open source license and play
Speaker 3: with it, you could do that. You could take a
Speaker 3: model kind of off the shelf from Hugging Face and
Speaker 3: fine tune it yourself. But it's a bit of a
Speaker 3: dead end because you made your contributions, but there's no
Speaker 3: way for other people to collaborate with you. So the
Speaker 3: way that we've built this is based on how the
Speaker 3: technology works. Everybody can contribute to it. This is something
Speaker 3: that you can keep growing and growing and growing over time.
Speaker 2: Yeah. Yeah, what's the level of expert te is necessary
Speaker 2: to be a contributor.
Speaker 3: You don't need to be a data scientist, and you
Speaker 3: don't need to have exotic hardware. Honestly, if you don't
Speaker 3: even have laptop hardware that meets SUSPEC for doing instruct
Speaker 3: Labs laptop version, you can submit it to the community
Speaker 3: and then we'll actually build it for you. We have
Speaker 3: bots and stuff that do that, and we're hoping over
Speaker 3: time to make that more accessible, first by having a
Speaker 3: user interface and then maybe later on having a web service.
Speaker 2: Yeah, so give me an example of how a business
Speaker 2: might make use of instruct lab.
Speaker 3: One of the things that businesses are doing with AI
Speaker 3: right now is using hosted API services. They're quite expensive,
Speaker 3: but they're finding value, but it's hard given the amount
Speaker 3: of money they're spending. And one of the things that's
Speaker 3: a little scary about it too, is like you have
Speaker 3: very sensitive internal documents and you have employees maybe not
Speaker 3: understanding what they're actually doing because you know, how would
Speaker 3: you if you're not technical enough when you're asking said
Speaker 3: public web service AI model information about out you're copy
Speaker 3: pasting internal company documents. It's going across the Internet into
Speaker 3: another company's hands, and that company probably shouldn't have access
Speaker 3: to that. So what both RedHat and IBM and the
Speaker 3: space are looking at, like the instruct lab model is
Speaker 3: very modest. It's seven billion parameter model, very small. It's
Speaker 3: very cheap to serve inference on a seven billion parameter model.
Speaker 3: It's competing with trillion parameter models that are hosted. You
Speaker 3: take this small model that is cheap to run inference on,
Speaker 3: you train it with your own company's proprietary data inside
Speaker 3: the walls of your company, on your own hardware. You
Speaker 3: can do all sorts of actual data analysis on your
Speaker 3: most sensitive data and have the confidence that has not
Speaker 3: left the premises.
Speaker 2: In that use case, you're not actually training the model
Speaker 2: for everyone. You're just taking it and doing some private
Speaker 2: stuff on it. Exactly doesn't leave the building. But that's
Speaker 2: separate from an interaction where you're doing something that contributes overall.
Speaker 3: Right, That's something maybe that I should be more clear
Speaker 3: about is there's sort of two tracks here, and this
Speaker 3: is very red hat classic. You have your upstream community
Speaker 3: track and you have your business product track. So the
Speaker 3: upstream community track is just enabling anybody to contribute to
Speaker 3: a model in a collaborative way and play with it.
Speaker 3: The downstream product business oriented track is now take that
Speaker 3: tech that we've honed and developed in the open community
Speaker 3: and apply it to your business knowledge and skills.
Speaker 2: This community driven approach marks a pivotal shift towards more
Speaker 2: accessible AI solutions. The contrast between externally hosted AI services,
Speaker 2: and the open model enhanced by instruct lab underscores the
Speaker 2: potential for broader adoption of AI in diverse business contexts.
Speaker 2: She envisions a future in which technological innovation is more
Speaker 2: tailored to individual business needs, guided by principles of openness
Speaker 2: and security. Seer imaginary case study. Sure, I'm a law firm,
Speaker 2: I'm an entertainment law I have one hundred clients who
Speaker 2: are big stars. They all have incredibly complicated contracts. I
Speaker 2: feed a thousand of my company's contracts from the last
Speaker 2: ten years into the model, and then every time I
Speaker 2: have a new contract, I ask the model, am I
Speaker 2: missing something? Can you go back and look through all
Speaker 2: our own contracts and show me a contract that is
Speaker 2: missing key components or exposes us to some liability. In
Speaker 2: that case, the model would know my law firm contracts really,
Speaker 2: really well. It's as if they've been working out my
Speaker 2: law firm. They're not distracted by other people's particular styles
Speaker 2: or a bunch of contracts from the utility industry, or
Speaker 2: they know entertainment law contracts exactly.
Speaker 3: Yeah, And you can train it in your own image,
Speaker 3: your style of doing things. It's something that your company
Speaker 3: can that is uniquely helpful to you. No third party
Speaker 3: could do that because no third party understands how you
Speaker 3: do business and understands your history and your documents. So
Speaker 3: it's sort of a way of getting value out of
Speaker 3: the stuff you already have sitting in a file cabinet somewhere.
Speaker 3: It's it's very cool.
Speaker 2: Yeah, give me a sort of a real world case
Speaker 2: study where you think the business use case would be
Speaker 2: really powerful. What's a business that really could see an
Speaker 2: advantage to using instruct lab in its way.
Speaker 3: The demo that I've given a couple of times at
Speaker 3: different events used an imaginary insurance company. So you say,
Speaker 3: you have this company, you have to recommend repairs for
Speaker 3: various types of claims. You've been doing this for years,
Speaker 3: you know. If you know the windshield's broken and you've
Speaker 3: gotten this type of accident and it's this model car,
Speaker 3: these are the kinds of things you want to look at.
Speaker 3: So you could talk to any insurance agent in the
Speaker 3: field and be like, oh, you know, it's a Tesla.
Speaker 3: You might want to look at the or something. They'll
Speaker 3: have some latent knowledge just so you can take that
Speaker 3: and train it into a model. Honestly, I think these
Speaker 3: kind of new technologies are better when they're less visible.
Speaker 3: So say you have the claims agents in the field
Speaker 3: and they have this tool and they're kind of entering
Speaker 3: the claim data. They're on the scene at the car,
Speaker 3: and it might say, oh, look, I see this is
Speaker 3: a Ford fiesta. These are things you want to look
Speaker 3: at for this type of accident. As you're entering the data,
Speaker 3: it could be going through the knowledge you had loaded
Speaker 3: into the model and be making these suggestions based on
Speaker 3: your company's background, and hey, you know, let's not make
Speaker 3: the same mistake twice. Let's make new mistakes, and let's
Speaker 3: learn from the stuff we already did. So that's one example,
Speaker 3: but there's so many different industries in ways that this
Speaker 3: could help, and it could make those agents in the
Speaker 3: field more efficient.
Speaker 2: Have you had anyone talk to you about using instruct
Speaker 2: lab in a way that surprised you.
Speaker 3: I mean, some people have done funky things, but sort
Speaker 3: of playing with the skills stuff, that's where I see
Speaker 3: a lot of creativity. The difference between knowledge and skills
Speaker 3: is that knowledge is pretty pretty understandable, right, like, oh,
Speaker 3: historical insurance claims or you know, legal contracts. Skills are
Speaker 3: a little different so whenever somebody submits a skill, sometimes
Speaker 3: it tends to be really creative because it's not something
Speaker 3: that's super intuitive. Somebody submitted a skill. I don't know
Speaker 3: how well it worked, but it was like making ASKI art,
Speaker 3: like draw me a I don't know, draw me a
Speaker 3: dog I would do like an ASKI art dog. I mean,
Speaker 3: it's stuff that you can do programmatically. One that was
Speaker 3: actually very very helpful was you know, take this table
Speaker 3: of data and convert it to this format. Like, oh,
Speaker 3: that's nice. That actually saves me time.
Speaker 2: How far away are we from the day when I
Speaker 2: Malcolm Globwell technology ignore Amus can go home and easily
Speaker 2: interact with instruct lab Maybe a few months, a few months,
Speaker 2: you're gonna say a few years.
Speaker 3: No, I think it could be a few months.
Speaker 2: Wow, I hope.
Speaker 3: Open source innovation.
Speaker 2: Yeah, oh that's really interesting. Yeah. I'm always take it
Speaker 2: by surprise. I'm still thinking in twentieth century terms about
Speaker 2: how long things take, and you're in the twenty second
Speaker 2: century as well as.
Speaker 3: I can tell. The instruct lab core invention was invented
Speaker 3: in a hotel room at an AI conference in December
Speaker 3: with an amazing group of IBM research guys December of
Speaker 3: twenty twenty three.
Speaker 2: Wait back up, you have to tell the story.
Speaker 3: This group of guys we've been working with, they were
Speaker 3: at this conference together and it's a really funny story
Speaker 3: because you know, it's hard to get access to GPUs
Speaker 3: and like even you know, you're at IBM and it's
Speaker 3: hard to get access because everybody wants access. They did
Speaker 3: it over Christmas break because nobody was using the cluster
Speaker 3: at the time, and they ran all of these experiments
Speaker 3: and I'm like, whoa, this is really cool.
Speaker 2: And their idea was we can do a stripped down
Speaker 2: AI model, and was the idea and even back then
Speaker 2: combine it with granted, what was the original idea?
Speaker 3: The original idea, Yeah, it's sort of multi there's like
Speaker 3: multiple aspects to it. So like one of the aspects
Speaker 3: it actually came on later, but it starts at the
Speaker 3: beginning of the workflow. Is you're using a taxonomy to
Speaker 3: organize how you're fine tuning the model. So the old
Speaker 3: approach they call it the blender approach, to just take
Speaker 3: a bunch of data of roughly the type of data
Speaker 3: that you'd like and you kind of throw it in
Speaker 3: and then see what comes out, don't like it, Okay,
Speaker 3: throw in more, try again, see what comes out. They
Speaker 3: had used this taxonomy technique, so you actually build like
Speaker 3: a taxonomy of like categories and subfolders of like this
Speaker 3: is the knowledge and skills that we want to train
Speaker 3: into the model. And that way you're sort of systematic
Speaker 3: about what you're adding, and you can also identify gaps
Speaker 3: pretty easily. Oh, I don't have a category for that.
Speaker 3: Let me add that. So that's like one of the
Speaker 3: parts of the invention here.
Speaker 2: Point number one is let's be intentional and deliberate in
Speaker 2: how we build and train this thing.
Speaker 3: Yeah, and then the next component would be okay, so
Speaker 3: is actually quite expensive. Part of the expense of like
Speaker 3: tuning models and just training models in general is coming
Speaker 3: up with the data. And what they wanted to do
Speaker 3: is have a technique where you could have just a
Speaker 3: little bit of data and expand it with something they're
Speaker 3: calling synthetic data generation. And this is where it's sort
Speaker 3: of like you have this student and teacher workflow. So
Speaker 3: you have your taxonomy. The taxonomy has sort of the
Speaker 3: knowledge like a business's knowledge documents, their insurance claims, and
Speaker 3: it has these quizzes that you write, and that's to
Speaker 3: teach the model. So I'm writing a quiz based just
Speaker 3: like you do in school. You read the chapter all
Speaker 3: in the American Revolution, and then you answer a ten
Speaker 3: question quiz where you're giving the model quiz. You need
Speaker 3: at least five questions and answers, and the answers need
Speaker 3: to be taken from the context of the document, and
Speaker 3: then you run it through a process called synthetic data generation,
Speaker 3: and it looks at the documents or look at the
Speaker 3: history chapter. It'll look at the questions and answers, and
Speaker 3: then it'll look to that original document and come up
Speaker 3: with more questions and answers based on the format of
Speaker 3: the questions and answers you made. So you can take
Speaker 3: five questions of answers amplify them into one hundred questions
Speaker 3: and answers, two hundred questions and answers, and it's a
Speaker 3: second model that is making the questions and answers. So
Speaker 3: it's synthetic data generation using an AI model to make
Speaker 3: the questions. We use an open source model to do that.
Speaker 3: So that's the second part, and then the third part
Speaker 3: is we have a multi phase tuning technique to actually
Speaker 3: take the synthetic data and then basically bake it into
Speaker 3: the model. So sort of that's the approach. A general
Speaker 3: philosophy of the approach is using grantede because we know
Speaker 3: where the data came from. Another approach is the fact
Speaker 3: that we're using small models that are cheap to run
Speaker 3: inference on. They're small enough that you can tune them
Speaker 3: on laptop hardware. You don't need all the fancy expensive
Speaker 3: GPU mania you're good. So sort of like a whole system,
Speaker 3: it's like not any one component, but it's sort of
Speaker 3: the approach they took with somewhat novel, and they were
Speaker 3: very excited when they saw the experimental results. There was
Speaker 3: a meeting between red hat and IBM. It was actually
Speaker 3: an IBM research meeting that red Hatters were invited to,
Speaker 3: and I think the red Hatters involves sort of saw
Speaker 3: the potential, WHOA, we can make models open source finally,
Speaker 3: rather than them just being these endless dead forks, we
Speaker 3: could make it so people could contribute back and collaborate
Speaker 3: around it. So that's when red hat became interested in
Speaker 3: it and we sort of worked together and the research
Speaker 3: engineers from IBM Research who came up with the technique
Speaker 3: and then my team, the software engineers who know how
Speaker 3: to take research code and productize it into actually runnable,
Speaker 3: supportable software, kind of got together. We've been hanging out
Speaker 3: in the Boston office at red Hat and building it out.
Speaker 3: April eighteenth was when we went open source and we
Speaker 3: made all of our repositories with all of the code public,
Speaker 3: and right now we're working towards a product release, so
Speaker 3: a supported product.
Speaker 2: How long did it take you to be convinced of
Speaker 2: the value of this idea? I mean, so people get
Speaker 2: together in this hotel room. They're running these experiments over Christmas.
Speaker 2: Are you aware of the experiments as they're running them?
Speaker 3: I didn't find out till February.
Speaker 2: So they come to you in February and they say, mo,
Speaker 2: can you recreate that conversation?
Speaker 3: Well, our CEO, Matt Hicks, and then Jeremy Eater, who's
Speaker 3: one of our distinguished engineers, and Steve Watt, who's a VP,
Speaker 3: were present I think at that meeting. So they kind
Speaker 3: of brought it back to us and said, listen, we've
Speaker 3: invited these IBM research folks to come visit in Boston,
Speaker 3: you know, work with them, like, see, does this have
Speaker 3: any merit could we build something from it, and so
Speaker 3: they gave us some presentations. We were very excited when
Speaker 3: they came to us. It only had support for Mac laptops.
Speaker 3: Of course, you know Red Hat were Linux people, So
Speaker 3: we're like, all right, we've got to fix that. So
Speaker 3: a bunch of the junior engineers around the office kind
Speaker 3: of came in and they're like, okay, we're going to
Speaker 3: build Linux support. And they had it within like a
Speaker 3: couple of days. It was crazy because this was just
Speaker 3: meant to be. Hey, guys, you know now what these
Speaker 3: are invited gas visiting our office, see what happens. And
Speaker 3: we end up doing like weeks of hackfests and late
Speaker 3: night pizzas in the conference room and like playing around
Speaker 3: with it and learning, and it was It was very fun.
Speaker 3: It's very cool.
Speaker 2: Anyone else do anything like this.
Speaker 3: Is not my understanding that anybody else is doing it yet,
Speaker 3: maybe others will try. A lot of the focus has
Speaker 3: been on that pre training phase, but for us, again
Speaker 3: that fine tuning. It's more accessible because you don't need
Speaker 3: all the exotic hardware, it doesn't take months. You can
Speaker 3: do it on a laptop. You can do a smoke
Speaker 3: test version of it in less than an hour.
Speaker 2: What does the word smoke test.
Speaker 3: Smoke test means you're not doing a full fine tuning
Speaker 3: on the model. It's a different tuning process. It's like
Speaker 3: kind of lower quality so to run on lower grade
Speaker 3: hardware so you can kind of see them didn't move
Speaker 3: the model or not, but it's not gonna give you
Speaker 3: like the full picture. You need higher end hardware to
Speaker 3: actually do the full thing. So that's what the product
Speaker 3: will enable you to do once it's launched, is you're
Speaker 3: gonna need GPUs, but when you have them, will help
Speaker 3: you make the best usage of them.
Speaker 2: Yeah. Yeah, And there's a little detail. I want to
Speaker 2: go back to sure in order to run the tests
Speaker 2: on this idea way back when they needed time on
Speaker 2: the GPUs. So this this will be the in house
Speaker 2: IBM and they were quiet at Christmas, So how much
Speaker 2: time would you need on the GPUs to kind of
Speaker 2: get proof of concept?
Speaker 3: Well what happens is and it's sort of like a
Speaker 3: lot of trial and error, right, And there's a lot
Speaker 3: about this stuff that like you come up with a hypothesis,
Speaker 3: you test it out, did it work or not? Okay,
Speaker 3: it's just like you know, in the lab, you know,
Speaker 3: buns and burners and beakers and whatever. So it really depends.
Speaker 3: But it can be hours, it can be days. It
Speaker 3: really depends on what they're trying to do. And then
Speaker 3: sometimes they can cut the time down, you know, with
Speaker 3: the number of GPUs you have, So like I have
Speaker 3: a cluster of agpus, Okay, it might take a day,
Speaker 3: but then if I can get thirty two, I can
Speaker 3: pipeline it and make it go faster and get it
Speaker 3: down a few hours. So it really depends, you know.
Speaker 3: But it's like everybody's home for the holidays. It's a
Speaker 3: lovely playground to kind of get that stuff going fast.
Speaker 2: Let's jump forward one year. Tell me the status of
Speaker 2: this project, Tell me who's using it, tell me how
Speaker 2: big is it. Give me your optimistic but plausible prediction
Speaker 2: about what instruct Lab looks like a year from now.
Speaker 3: A year from now, I would like to see kind
Speaker 3: of a vibrant community around not just building knowledge and
Speaker 3: skills into a model, but coming up with better techniques
Speaker 3: and innovation around how we do it. So I'd like
Speaker 3: to see like the contributor experience as we grow more
Speaker 3: and more contributors to be refined, So like a year
Speaker 3: from now, Malcolm Gladwell could come impart some of his
Speaker 3: wisdom into the model and it wouldn't be difficult, it
Speaker 3: wouldn't be a big lift. I would love to see
Speaker 3: the user interface tooling for doing that to be more soificated.
Speaker 3: I would love to see more people taking this and
Speaker 3: even using it. Maybe you're not sharing it with the community,
Speaker 3: but you're using it for some private usage. Like I'll
Speaker 3: give you an example. I'm in contact with a fellow
Speaker 3: who is doing AI research and he's working with doctors.
Speaker 3: They're GPS in an area of Canada where there's not
Speaker 3: enough GPS for the number of patients, So you know,
Speaker 3: anything you can do to save doctors time to get
Speaker 3: to the next patient. It's like one of the things
Speaker 3: that he has been doing experiments with is can we
Speaker 3: use an open source, licensed model that the doctor can
Speaker 3: run on their laptop so they don't have to worry
Speaker 3: about all of the different privacy rules, Like it's privates
Speaker 3: on the laptop right there, take his live transcription of
Speaker 3: his conversation with the patient and then convert it automatically
Speaker 3: to a SOAP format that can be entered in the database.
Speaker 3: Typically this will take a doctor fifteen to twenty minutes
Speaker 3: of paperwork. Why not save them the paperwork at least
Speaker 3: have the model take a stab.
Speaker 2: Does the model then hold on to that information? He
Speaker 2: interacts with the model again when well, that's the.
Speaker 3: Thing not withinstruct lab. Maybe that could be a future development.
Speaker 3: It doesn't once you're doing inference, it's not ingesting that
Speaker 3: what you're saying to it back in. It's only the
Speaker 3: fine tuning phase. So the idea would be the doctor
Speaker 3: could maybe load in past patient data as knowledge, and
Speaker 3: then when he's trying to diagnose, maybe you know what
Speaker 3: I'm saying. But the main idea is somebody might have
Speaker 3: some private usage. I would love to see more usage
Speaker 3: of this tool to enable people who otherwise never would
Speaker 3: have had access to this type of technology, who never, like,
Speaker 3: you know, a small country GP doctor is it doesn't
Speaker 3: have GPUs. They're not going to hire some company to
Speaker 3: custom build them a model. But maybe on the weekend,
Speaker 3: if he's a techie guy, he could say with this.
Speaker 2: Well, I mean, the more you talk, the more I'm
Speaker 2: realizing that the simplicity of this model is the killer
Speaker 2: app here. Once you know you can run it on
Speaker 2: a laptop. You have democratized use in a way that's
Speaker 2: inconceivable with some of these other much more complex. But
Speaker 2: that's interesting because one would have thought intuitively that at
Speaker 2: the beginning that the winner is going to be the
Speaker 2: one with the biggest, most complex version. And you're saying, actually, no,
Speaker 2: there's a whole series of uses where being lean and
Speaker 2: focused focused is actually you know, it enables a whole
Speaker 2: class of uses. Maybe another way of saying this is
Speaker 2: who wouldn't be a potential instruct lab customer.
Speaker 3: We don't know yet. It's so new, like we haven't
Speaker 3: really had enough people experimenting and playing with it and
Speaker 3: finding out all the things yet. But that's the thing
Speaker 3: that's so exciting about it. It's like, I can't wait
Speaker 3: to see what people do.
Speaker 2: Is this the most exciting thing you've worked on in
Speaker 2: your career?
Speaker 3: I think so.
Speaker 2: I think so. Yeah, Well, we are reaching the end
Speaker 2: of our time, but before we finished, we can do
Speaker 2: a little speed round. Sure, all right, complete the following sentence.
Speaker 2: In five years, AI will.
Speaker 3: Be boring, it will be integrated, It'll just work, and
Speaker 3: there will be no now with AI thing. It'll just
Speaker 3: be normal.
Speaker 2: What's the number one thing? That people misunderstand about AI.
Speaker 3: It's just matrix algebra. It's just numbers. It's not sentient.
Speaker 3: It's not coming to take us over. It's just numbers.
Speaker 2: You're on this side of you're on the team humanity. Yeah,
Speaker 2: you're good. What advice would you give yourself ten years
Speaker 2: ago to better prepare for today?
Speaker 3: Learn Python for real. It's a programming language that's extensively
Speaker 3: used in the community. I've always dabbled in it, but
Speaker 3: I wish I had taken it more seriously.
Speaker 2: Yeah, did you say, who had a daughter?
Speaker 3: I have three daughters?
Speaker 2: You have three daughters. I have two. You're if you
Speaker 2: got three year you're you're on your own. What are
Speaker 2: you making them study Python?
Speaker 3: I am actually trying to do that. We're using a
Speaker 3: microbit micro controller tool to do like a custom video
Speaker 3: game controller. They prefer because it's a visual programming language,
Speaker 3: but it has a Python interface too, and I'm like
Speaker 3: pushing them towards Python.
Speaker 2: Good. Chatbox and image generators are the biggest things in
Speaker 2: consumer AI right now. What do you think is the
Speaker 2: next big business application?
Speaker 3: Private models? Small models fine tuned on your company's data
Speaker 3: for you to use exclusively.
Speaker 2: Are you using AI in your own personal life these days.
Speaker 3: Honestly, I think a lot of us are using it
Speaker 3: and we don't even realize it. Yeah, I mean, I'm
Speaker 3: a ficiano of foreign languages. There's translation programs that are
Speaker 3: built using machine learning underneath. One of the things I've
Speaker 3: been dabbling with lately is using tech summarizations because I
Speaker 3: tend to be very loquacious in my note taking and
Speaker 3: that is not so useful for other people who would
Speaker 3: just like a paragraph. So that's something I've been experimenting
Speaker 3: with myself just to help my everyday work.
Speaker 2: Yeah. We hear many definitions of open related to technology.
Speaker 2: What's your definition of open and how does it help
Speaker 2: you innovate?
Speaker 3: My definition of open is basically sharing and being vulnerable,
Speaker 3: like not just sharing in a have a cookie way,
Speaker 3: but in a you know what, I don't actually know
Speaker 3: how this works? Could you help me? And being open
Speaker 3: to being wrong, being open to somebody helping you and
Speaker 3: making that collaboration work. So it's not just about like
Speaker 3: the artifact you're opening, it's your approach, like how you
Speaker 3: do things being open?
Speaker 2: Yeah, yeah, well I think that wraps us up. How
Speaker 2: can listeners follow your work and learn more about Granted
Speaker 2: and instruct lab.
Speaker 3: Sure you can visit our project web page at instruct
Speaker 3: lab dot ai, or you can visit our GitHub at
Speaker 3: GitHub dot com slash instruct lab. We have lots of
Speaker 3: instructions on how to get involved in an instruct lab wonderful.
Speaker 2: Thank you so much, Thank you Malcolm. A big thank
Speaker 2: you to Mo for the engaging discussion on the groundbreaking
Speaker 2: possibilities of instruct lab Lord how this platform has the
Speaker 2: potential to revolutionize industries from insurance to entertainment law by
Speaker 2: using an open source community approach that makes it easier
Speaker 2: for more people from all backgrounds to fine tune models
Speaker 2: for specific purposes, ultimately making AI more accessible and impactful
Speaker 2: than ever. Looking ahead, the future of AI isn't just
Speaker 2: about technological efficiency. It's about enhancing our everyday experiences in
Speaker 2: ways that were never possible before, like streamlining work for
Speaker 2: doctors to improve the patient experience or assisting insurance agents
Speaker 2: to improve the claims experience. Instruct Lab is paving the
Speaker 2: way for more open, accessible AI future, one that's built
Speaker 2: on collaboration and humanity. Smart Talks with IBM is produced
Speaker 2: by Matt Romano, Joey Fishground and Jacob Goldstein were edited
Speaker 2: by Lydia gy Caught. Our engineers are Sarah Bruguerer and
Speaker 2: Ben Toliday. Theme song by Gramscow. Special thanks to the
Speaker 2: eight Bar and IBM teams, as well as the Pushkin
Speaker 2: marketing team. Smart Talks with IBM is a production of
Speaker 2: Pushkin Industries and Ruby Studio at iHeartMedia. To find more
Speaker 2: Pushkin podcasts, listen on the iHeartRadio app, Apple Podcasts, or
Speaker 2: wherever you listen to podcasts. I'm Malcolm Gladwell. This is
Speaker 2: a paid advertisement from IBM. The conversations on this podcast
Speaker 2: don't necessarily represent IBM's positions, strategies, or opinions.