Speaker 1: Hello, Hello, Welcome to Smart Talks with IBM, a podcast
Speaker 1: from Pushkin Industries, iHeartRadio and IBM. I'm Malcolm Gladwell. This season,
Speaker 1: we're diving back into the world of artificial intelligence, but
Speaker 1: with a focus on the powerful concept of open its possibilities, implications,
Speaker 1: and misconceptions. We'll look at openness from a variety of
Speaker 1: angles and explore how the concept is already reshaping industries,
Speaker 1: ways of doing business and a very notion of what's possible.
Speaker 1: In today's episode, I sat down with mo Duffy, software
Speaker 1: engineering manager at red Hat, who works on instruct Lab,
Speaker 1: a project co developed by red Hat and IBM. Most
Speaker 1: shared with me how this new initiative is revolutionizing AI training,
Speaker 1: making it not only more accessible, but also more inclusive.
Speaker 1: This project, unique in the industry, allows developers to submit
Speaker 1: incremental contributions to one base AI model, creating a continuous
Speaker 1: loop of development, much like normal open source software. By
Speaker 1: leveraging community contributions and IBM's cutting edge granite models, Mo
Speaker 1: in the team of ibmrs and red hatters are paving
Speaker 1: the way for a future where AI development is a
Speaker 1: communal endeavor. Our insights into open source software extend beyond
Speaker 1: technical proficiency to the profound impact of collaborative effort. At
Speaker 1: the heart of Moe's work is a belief in democratizing technology,
Speaker 1: ensuring that AI becomes a tool accessible to all. So
Speaker 1: let's explore how Mo, red Hat and IBM are empowering
Speaker 1: individuals and businesses alike to reshape the future of technology
Speaker 1: through collaboration and innovation. Mo, thank you for joining me
Speaker 1: today so much, for you have just about the most
Speaker 1: Irish name ever. I do very proud you weren't born
Speaker 1: in Ireland.
Speaker 2: No, my grandparents or your grandparents.
Speaker 1: I see, where did you grow up?
Speaker 2: New York Queens?
Speaker 1: Oh you're a see. So tell me a little bit
Speaker 1: about how how you got to red hat? What was
Speaker 1: your path?
Speaker 2: When I was in high school, it was a chatty girl,
Speaker 2: teenage girl on the phone. We had one phone line.
Speaker 2: My older brother was studying at the local state college
Speaker 2: computer science, and he had to tell that end to
Speaker 2: compile his homework. One phone line and I'm on it
Speaker 2: all the time. He got very frustrated and he needed
Speaker 2: a compiler to do his homework. So he bought red
Speaker 2: hat Linux from a CompUSA, brought it home and that
Speaker 2: was on the family computer. So I learned Linux and
Speaker 2: I started playing around with it. I really liked it
Speaker 2: because you could customize everything, like the entire user interface.
Speaker 2: You could actually modify the code of the programs you
Speaker 2: were using to do what you wanted. And for me,
Speaker 2: it was really cool because, especially when you're a kid
Speaker 2: and like people tell you this is the way things
Speaker 2: are and you just have to deal with it, it's
Speaker 2: nice to be like, I'm going to make things the
Speaker 2: way I want, modify the code and playing. Yeah, it
Speaker 2: was amazing and it was just such a time and
Speaker 2: like before it was cool, I was doing it and
Speaker 2: what I saw on that is sort of the potential
Speaker 2: like number one of like a community of people working together.
Speaker 2: And like the Internet existed, it was slow, it involved modems,
Speaker 2: but there were people that you could talk to who
Speaker 2: would give you tips and you'd share information, and this
Speaker 2: collaborative building something together is really something special. Right. I
Speaker 2: could file a complaint to whatever large software company made
Speaker 2: whatever software I was into, or I could go to
Speaker 2: an open source software community and be like, hey, guys,
Speaker 2: I think we should do this. I'm like, yeah, okay,
Speaker 2: I'll help. I'll pitch in so you don't feel powerless.
Speaker 2: You feel like you can have an impact, and that
Speaker 2: was really exciting to me. However, open source software has
Speaker 2: a reputation for not having the best user interface, not
Speaker 2: the best user experience. So I ended up studying computer
Speaker 2: science and electronic media, and then I did human computeraction
Speaker 2: as my master's And my thought was, wouldn't it be
Speaker 2: nice if this free software accessible to anybody, if it
Speaker 2: was easier to use, some more people could use it
Speaker 2: and take advantage of it. And so, long story short,
Speaker 2: I ended up going to Red Hat saying, Hey, I
Speaker 2: want to learn how you guys work. Let me embed
Speaker 2: in your team draft out of my graduate program, and
Speaker 2: I'm like, I want to do this for a living.
Speaker 2: This is cooler. So I thought this is the way
Speaker 2: to go, and I've been there ever since. They haven't
Speaker 2: been able to get rid of me.
Speaker 1: To backtrack just a little bit, you were talking about
Speaker 1: the sense of community that surrounds this way of thinking
Speaker 1: about software. Talk a little bit more about what that
Speaker 1: community is like, the benefits of that community, why it
Speaker 1: appeals to you.
Speaker 2: Sure, well, you know part of the reason I actually
Speaker 2: ended up going to the graduate school track. Suddenly you're
Speaker 2: a peer of your professors and you're working side by
Speaker 2: side with them. At some point they retire and you're
Speaker 2: in the next generation. So it's sharing information, building on
Speaker 2: the work of others in sort of this cycle that
Speaker 2: extends past human lifespan and in the same way, like
Speaker 2: the open source model is very similar, but you're actually
Speaker 2: you're building something, and it's something in me. I'm just
Speaker 2: really attracted, Like I don't like talking about stuff. I
Speaker 2: like doing stuff with open source software. The software doesn't
Speaker 2: cost anything, the code is out there, generally uses open
Speaker 2: standards for the file formats. I can open up files
Speaker 2: that I created and open source tools as a high
Speaker 2: school student today because they were using open formats and
Speaker 2: that software still exists, I can still compile the code
Speaker 2: and it's an active community project. Like these things can
Speaker 2: outlast any single company in the same way that the
Speaker 2: academic community has been going on for so many years,
Speaker 2: and hopefully we'll continue moving on. So it's sort of
Speaker 2: like not just the community around it, but just the
Speaker 2: knowledge sharing and also bringing up the next generation as well.
Speaker 2: Like all of that stuff really appealed to me. And
Speaker 2: also at the center of it the fact that we
Speaker 2: could democratize it by following this open source process and
Speaker 2: feel like we have some control. We're not at the
Speaker 2: mercy of some faceless corporation making changes and we have
Speaker 2: no impact. Like that really appealed to me too.
Speaker 1: For those of us who are not software phisionados, take
Speaker 1: a step backwards and give me a kind of description
Speaker 1: of terms. What's the opposite of open to proprietary?
Speaker 2: Proprietary is what we say, So.
Speaker 1: Specifically and practically, the difference would be what between something
Speaker 1: that was opened us in something that was proprietary.
Speaker 2: Sure, so there's a lot of difference. So with open
Speaker 2: source software you get these rights. When you're given the software,
Speaker 2: you get the right to be able to share it.
Speaker 2: And depending on the lot, different licenses that are considered
Speaker 2: open source have different little things that you have to
Speaker 2: be aware of. With proprietary code, it's one copyright the company.
Speaker 2: Even a lot of times, when you sign your employment
Speaker 2: contract for a software company and you write code for them,
Speaker 2: you don't own it. You sign over your rights to
Speaker 2: the company, So if you leave the company, the code
Speaker 2: doesn't go with you. It stays in the ownership of
Speaker 2: that company. So then one one company buys out another
Speaker 2: and kills a product that code's gone.
Speaker 1: It's gone. For a business, Why would a business want
Speaker 1: to be have open source code as opposed to proprietary.
Speaker 2: Well, for the same reasons. Like say you're a business.
Speaker 2: You've invested all this money into this software platform, right,
Speaker 2: and you've upskilled your employees on it, and it's a
Speaker 2: core part of your business, and then a few years
Speaker 2: later that company goes out of business or something happens,
Speaker 2: or even something less drastic. You really need this feuture.
Speaker 2: But for the company that makes the software, it's not
Speaker 2: in their best interests. It's not worth the investment. They're
Speaker 2: not going to do it. How do you get that feature?
Speaker 2: You either have to completely migrate to another solution, and
Speaker 2: this is something that's core at your business, that's going
Speaker 2: to be a big deal to migrate. But if it's
Speaker 2: open source, you could either hire a team of experts.
Speaker 2: You could hire software engineers who are able to go
Speaker 2: do this for you. Go in the upstream software community,
Speaker 2: implement the future that you want, and it'll be rolled
Speaker 2: into the next version of that company software. So even
Speaker 2: if that company didn't want to implement the feature, if
Speaker 2: they did it open source, they would inherit that feature
Speaker 2: from the upstream community is what we call it, so
Speaker 2: you have some control over the situation. If it's open source,
Speaker 2: you have an opportunity to actually affect change in the product,
Speaker 2: and you could then pick it up or pay somebody
Speaker 2: else to pick it up, or another company could form
Speaker 2: and pick it up and keep it going. So there's
Speaker 2: more possibilities. If it's open source, it's more like it's
Speaker 2: like an insurance policy almost.
Speaker 1: So innovation from the standpoint of the customer, innovation is
Speaker 1: a lot easier when you're working in an open source environment.
Speaker 2: Absolutely.
Speaker 1: Yeah. So now at RedHat you're working with something called
Speaker 1: instruct lab. Tell us a little bit about what that is.
Speaker 2: So the thing that really excites me about getting to
Speaker 2: work on this project is AI is sort of that
Speaker 2: has been this scary thing for me because it's one
Speaker 2: of those things like in order to be able to
Speaker 2: pre train a model, you have to have unobtainium GPUs,
Speaker 2: you have to have rich resources, It takes months, it
Speaker 2: takes expertise. There's a small handful of companies that can
Speaker 2: build a model from pre train to something usable, and
Speaker 2: it kind of feels like those early days when I
Speaker 2: was kind of delving in software in the same way.
Speaker 2: I think if more people could contribute to AI models,
Speaker 2: then it wouldn't be just influenced by whichever company had
Speaker 2: the resources to build it. And there's been a lot
Speaker 2: of emphasis on pre training models, so taking massive terabytes
Speaker 2: data sets, throwing them through masses of GPUs over months
Speaker 2: of time, spending hundreds of millions of dollars to build
Speaker 2: a base model. But when instruct lab does is say okay,
Speaker 2: you have a base model, we're going to fine tune in.
Speaker 2: On the other end, it takes less compute resources. The
Speaker 2: way we've built in struck lab, you can play around
Speaker 2: with the technology and learn it on it off the
Speaker 2: shelf laptop that you can actually buy. So in this
Speaker 2: way we're enabling a much broader set of people to
Speaker 2: play with AI, to contribute it, to modify it. And
Speaker 2: I'll tell you one story from red Hat Succi, who
Speaker 2: is our chief diversity officer, very interested in inclusive language
Speaker 2: and open source software, doesn't have any experience with AI.
Speaker 2: We have a community model that we have an upstream
Speaker 2: project around for people to contribute knowledge and skills to
Speaker 2: the model. She's like, I want to teach the model
Speaker 2: how to use inclusive language like replace this word with
Speaker 2: this word, or this word with this word. Oh my, oh,
Speaker 2: that's so cool. So she paired up with Nicholas who
Speaker 2: is a technical guy at red Hat, and they built
Speaker 2: and submitted a skill to the model that you can
Speaker 2: just tell the model, can you please take this document
Speaker 2: and translate this language to more inclusive language and it
Speaker 2: will do it. And they submitted it to the community.
Speaker 2: They were so proud. It was like, that's the kind
Speaker 2: of thing that like, you know, maybe a company would
Speaker 2: be incentivized to do that, but if you have some
Speaker 2: tooling that's open source and something that anybody could access,
Speaker 2: then those communities could actually get together and build that
Speaker 2: knowledge into AI models.
Speaker 1: Just so understand, what you guys have is the structure
Speaker 1: for an AI system, and in other cases, individual companies
Speaker 1: own and train their own AI systems. It takes enormous
Speaker 1: amount of resources. They hoover up all kinds of information,
Speaker 1: train it according to their own hidden set of rules,
Speaker 1: and then a customer might use that for some price.
Speaker 1: What you're saying is, in the same way that we
Speaker 1: democratize the writing of software before, let's democratize the training
Speaker 1: of an AI system. So anyone can contribute here and
Speaker 1: teach the model the things that they're interested in teaching
Speaker 1: the model. I'm guessing correct me. On the one hand,
Speaker 1: this model, at least in the beginning, is going to
Speaker 1: have a lot fewer resources available to it. But on
Speaker 1: the other hand, it's going to have a much more
Speaker 1: diverse set of inputs.
Speaker 2: That's right. And the other thing is that IBM, basically
Speaker 2: as part of this project, has something called the granite model,
Speaker 2: and they've donated some granite models. So these are the
Speaker 2: ones that take the months and terabytes of data and
Speaker 2: all the GPUs to train. So IBM has created one
Speaker 2: of those, and they have listed out and linked to
Speaker 2: the data sets that they used, and they talk about
Speaker 2: the relative proportions they used when pre training, so it's
Speaker 2: not just the black box. You know where the data
Speaker 2: came from, which is a pretty open position to take.
Speaker 2: That is what we recommend as the base. So you
Speaker 2: use the instruct lab tuning. You take this base granite
Speaker 2: model that IBM has provided, and you use the instruct
Speaker 2: lab tooling that red Hat works on, and you use
Speaker 2: that to fine tune the model to make it whatever
Speaker 2: you want.
Speaker 1: I want to go back to the partnership between IBM
Speaker 1: and red Hat here with them providing the granite model
Speaker 1: to your instruct lab. Is this the first time red
Speaker 1: Hat and IBM have collaborated like this.
Speaker 2: I think it's something that's been going on. Like another
Speaker 2: a product within the red Hat family would be OpenShift AI,
Speaker 2: where they collaborate a lot with IBM Research team, Like
Speaker 2: bolm is one of the components of that product that
Speaker 2: there's a nice kind of exchange and collaboration between the
Speaker 2: two companies.
Speaker 1: How large is the potential community of people who might
Speaker 1: contribute to instruct lab.
Speaker 2: It could be thousands of people. I mean, we'll see.
Speaker 2: It's early days. This is early technology that was invented
Speaker 2: at IBM Research that they partnered with us at red
Speaker 2: Hat to kind of build the software around it. There's
Speaker 2: still more to go, Like right now, we have a
Speaker 2: team in the community that's actually trying to build a
Speaker 2: web interface to make it easier for anybody to contribute.
Speaker 2: So we have a lot of those sort of user
Speaker 2: experience for the contributor to the model stuff to work
Speaker 2: out that we're still actively building on. But like my
Speaker 2: vision for it even is I like going back to
Speaker 2: that academic model of learning from what others and building
Speaker 2: upon it over time. It would be very good for
Speaker 2: us to sort of go out and try to collaborate
Speaker 2: with academics of all fields, like, hey, you know, the
Speaker 2: model doesn't know about your field. Would you like to
Speaker 2: put something into the model about your field so it
Speaker 2: knows about it? Or even you know, talk to the model.
Speaker 2: It got it wrong, let's correct it. Can we lean
Speaker 2: on your expertise to correct it and make sure it
Speaker 2: gets it right and sort of use that community model
Speaker 2: as a way for everybody to collaborate because before instruct lab,
Speaker 2: my understanding is if you wanted to take a model
Speaker 2: that's open source license and play with it, you could
Speaker 2: do that. You could take a model kind of off
Speaker 2: the shelf from Hugging Face and fine tune it yourself.
Speaker 2: But it's a bit of a dead end because you
Speaker 2: made your contributions, but there's no way for other people
Speaker 2: to collaborate with you. So the way that we've built
Speaker 2: this is based on how the technology works. Everybody can
Speaker 2: contribute to it. This is something that it can keep
Speaker 2: growing and growing and growing over time.
Speaker 1: Yeah. Yeah, what's the level of expertise necessary to be
Speaker 1: a contributor?
Speaker 2: You don't need to be a data scientist, and you
Speaker 2: don't need to have exotic hardware. Honestly, if you don't
Speaker 2: even have laptop hardware that meets the spec for doing
Speaker 2: instruct labs laptop version, you can minutes to the community
Speaker 2: and then we'll actually build it for you. We have
Speaker 2: bots and stuff that do that, and we're hoping over
Speaker 2: time to make that more accessible, first by having a
Speaker 2: user interface and then maybe later on having a web service.
Speaker 1: Yeah, so give me an example of how a business
Speaker 1: might make use of instruct lab.
Speaker 2: One of the things that businesses are doing with AI
Speaker 2: right now is using hosted API services. You're quite expensive,
Speaker 2: but they're finding value. But it's hard given the amount
Speaker 2: of money they're spending. And one of the things that's
Speaker 2: a little scary about it too, is like you have
Speaker 2: very sensitive internal documents and you have employees maybe not
Speaker 2: understanding what they're actually doing because you know, how would
Speaker 2: you if you're not technical enough when you're asking said
Speaker 2: public web service AI model information about your copy pasting
Speaker 2: internal company documents. It's going across the Internet into another
Speaker 2: company's hands, and that company probably shouldn't have access to that.
Speaker 2: So what both RedHat and IBM and the space are
Speaker 2: looking at, like, the instruct lab model is very modest.
Speaker 2: It's seven billion parameter model, very small. It's very cheap
Speaker 2: to serve inference on a seven billion parameter model. It's
Speaker 2: competing with trillion parameter models that are hosted. You take
Speaker 2: this small model that is cheap to run inference on,
Speaker 2: you train it with your own company's proprietary data inside
Speaker 2: the walls of your company, on your own hardware. You
Speaker 2: can do all sorts of actual data analysis on your
Speaker 2: most sensitive data and have the confidence that has not
Speaker 2: left the premises.
Speaker 1: In that use case, you're not actually training the model
Speaker 1: for everyone. You're just taking it and doing some private
Speaker 1: stuff on it exactly, which doesn't leave the building. But
Speaker 1: that's separate from an interaction where you're doing something that
Speaker 1: contributes overall.
Speaker 2: Right, And that's something maybe that I should be more
Speaker 2: clear about. Is there's sort of two tracks here, and
Speaker 2: this is very red hat classic. You have your upstream
Speaker 2: community track and you have your business product track. So
Speaker 2: the upstream community track is just enabling anybody to contribute
Speaker 2: to a model in a collaborative way and play with it.
Speaker 2: The downstream product business oriented track is now take that
Speaker 2: tech that we've honed and developed in the open community,
Speaker 2: and apply it to your business knowledge and skills.
Speaker 1: This community driven approach marks a pivotal shift towards more
Speaker 1: accessible AI solutions. The contrast between externally hosted AI services
Speaker 1: and the open model enhanced by instruct lab underscores the
Speaker 1: potential for broader adoption of AI in diverse business contexts.
Speaker 1: She envisions a future in which technological innovation is more
Speaker 1: tailored to individual business needs, guided by principles of openness
Speaker 1: and security. To an imaginary case study, Sure, I'm a
Speaker 1: law firm. I'm an entertainment law I have one hundred
Speaker 1: clients who are big stars. They all have incredibly complicated contracts.
Speaker 1: I feed a thousand of my company's contracts from the
Speaker 1: last ten years into the model, and then every time
Speaker 1: I have a new contract, I ask the model, am
Speaker 1: I missing something? Can you go back and look through
Speaker 1: all our own contracts and show me a contract that
Speaker 1: is missing key components or exposes us to some liability.
Speaker 1: In that case, the model would know my law firm
Speaker 1: contracts really really well. It's as if they've been working
Speaker 1: out my law firm. They're not distracted by other people's
Speaker 1: particular styles, or a bunch of contracts from the utility industry,
Speaker 1: or they know entertainment law contracts exactly.
Speaker 2: Yeah, you can train it in your own image, your
Speaker 2: style of doing things. It's something that your company can
Speaker 2: produce that is uniquely helpful to you. No third party
Speaker 2: could do that because no third party understands how you
Speaker 2: do business and understands your history and your documents. So
Speaker 2: it's sort of a way of getting value out of
Speaker 2: the stuff if you already have sitting in a file
Speaker 2: cabinet somewhere, it's very cool.
Speaker 1: Yeah, give me a sort of a real world case
Speaker 1: study where you think the business use case would be
Speaker 1: really powerful. What's a business that really could see an
Speaker 1: advantage to using instruct lab in its way.
Speaker 2: The demo that I've given a couple of times at
Speaker 2: different events used an imaginary insurance company. So you say,
Speaker 2: you have this company, you have to recommend repairs for
Speaker 2: various types of claims. You've been doing this for years,
Speaker 2: you know. If you know the windshield's broken and you've
Speaker 2: gotten this type of accident and it's this model car,
Speaker 2: these are the kinds of things you want to look at.
Speaker 2: So you could talk to any insurance agent in the
Speaker 2: field and be like, oh, you know, it's a Tesla.
Speaker 2: You might want to look at the battery or something like.
Speaker 2: They'll have some latent knowledge just so you can take
Speaker 2: that and train it into a model. Honestly, I think
Speaker 2: these kind of new technologies are better when they're less visible.
Speaker 2: So say you have the as agents in the field
Speaker 2: and they have this tool and they're kind of entering
Speaker 2: the claim data. They're on the scene at the car,
Speaker 2: and it might say, oh, look, I see this is
Speaker 2: a Ford Fiesta. These are things you want to look
Speaker 2: at for this type of accident. As you're entering the data,
Speaker 2: it could be going through the knowledge you had loaded
Speaker 2: into the model and be making these suggestions based on
Speaker 2: your company's background, and hey, you know, let's not make
Speaker 2: the same mistake twice. Let's make new mistakes and let's
Speaker 2: learn from the stuff we already did. So that's one example,
Speaker 2: but there's so many different industries in ways that this
Speaker 2: could help, and it could make those agents in the
Speaker 2: field more efficient.
Speaker 1: Have you had anyone talk to you about using instruct
Speaker 1: lab in a way that surprised you.
Speaker 2: I mean, some people have done funky things, but sort
Speaker 2: of playing with the skills stuff. That's where I see
Speaker 2: a lot of creativity. The difference between knowledge and skills
Speaker 2: is that knowledge is pretty pretty understandable, right, like oh,
Speaker 2: historical insurance claims or you know, legal contracts. Skills are
Speaker 2: a little different. So whenever somebody submits a skill, sometimes
Speaker 2: it tends to be really creative because it's not something
Speaker 2: that's super intuitive. Somebody submitted a skill. I don't know
Speaker 2: how well it worked, but it was like making ASKI art,
Speaker 2: like draw me a I don't know, draw me a dog,
Speaker 2: and would do like an ASKI art dog. I mean,
Speaker 2: it's stuff that you can do programmatically. One that was
Speaker 2: actually very very helpful was you know, take this table
Speaker 2: of data and convert it to this format like, ooh,
Speaker 2: that's nice. That actually saves me time.
Speaker 1: How far away are we from the day when I,
Speaker 1: Malcolm Globwell technology ignore Amus can go home and easily
Speaker 1: interact with instruct lab.
Speaker 2: Maybe a few months, a few.
Speaker 1: Months, you're gonna say a few years.
Speaker 2: No, I think it'd be a few months. Wow, I
Speaker 2: hope it's power open source innovation.
Speaker 1: Yeah, oh that's really interesting. Yeah, I'm always taken by surprise.
Speaker 1: I'm still thinking in twentieth century terms about how long
Speaker 1: things take, and you're in the twenty second century as
Speaker 1: well as I could tell.
Speaker 2: The instruct lab core invention was invented in a hotel
Speaker 2: room at an AI conference in December with an amazing
Speaker 2: group of IBM research guys December of twenty twenty three.
Speaker 1: Wait back up, you have to tell the story.
Speaker 2: This group of guys we've been working with, they were
Speaker 2: at this conference together, and it's a really funny story
Speaker 2: because you know, it's hard to get access to GPUs
Speaker 2: and like even you know, you're at IBM and it's
Speaker 2: hard to get access because everybody wants access. They did
Speaker 2: it over Christmas break because nobody was using the cluster
Speaker 2: at the time, and they ran all of these experiments
Speaker 2: and I'm like, whoa, this is really cool.
Speaker 1: And wait. Their idea was we can do a stripped
Speaker 1: down AI model, and was the idea and even back
Speaker 1: then combine it with grantite.
Speaker 2: What was the original the original idea. It's sort of
Speaker 2: multi there's like multiple aspects to it. So like one
Speaker 2: of the aspects it actually came on later, but it
Speaker 2: starts at the beginning of the workflow. Is you're using
Speaker 2: a taxonomy to organize how you're fine too the model.
Speaker 2: So the old approach they call it the blender approach,
Speaker 2: to just take a bunch of data of roughly the
Speaker 2: type of data that you'd like and you kind of
Speaker 2: throw it in and then see what comes out. Don't
Speaker 2: like it, Okay, throw in more, try again, see what
Speaker 2: comes out. They had used this taxonomy technique, so you
Speaker 2: actually build like a taxonomy of like categories and subfolders
Speaker 2: of like this is the knowledge and skills that we
Speaker 2: want to train into the model. And that way you're
Speaker 2: sort of systematic about what you're adding, and you can
Speaker 2: also identify gaps pretty easily. Oh, I don't have a
Speaker 2: category for that, let me add that. So that's like
Speaker 2: one of the parts of the invention here.
Speaker 1: Point number one is let's be intentional and deliberate in
Speaker 1: how we build and train this thing.
Speaker 2: Yeah, and then the next component would be Okay, So
Speaker 2: it is actually quite expensive. Part of the expense of
Speaker 2: like tuning models and just training models in general is
Speaker 2: coming up with the data. And what they wanted to
Speaker 2: do is have a technique where you could have just
Speaker 2: a little bit of data and expand it with something
Speaker 2: they're calling synthetic data generation. And this is where it's
Speaker 2: sort of like you have this student and teacher workflow,
Speaker 2: so you have your taxonomy. The taxonomy has sort of
Speaker 2: the knowledge like a business's knowledge documents, their insurance claims,
Speaker 2: and it has these quizzes that you write and that's
Speaker 2: to teach the model. So I'm writing a quiz based
Speaker 2: just like you do in school. You read the chapter
Speaker 2: on the American Revolution, and then you answer a ten
Speaker 2: question quiz where you're giving the model quiz. You need
Speaker 2: at least five questions and answers, and the answers need
Speaker 2: to be taken from the context of the document, and
Speaker 2: then you run it through a process called synthetic data generation,
Speaker 2: and it looks at the documents, so we'll look at
Speaker 2: the history chapter, it'll look at the questions and answers,
Speaker 2: and then it'll look to that original document and come
Speaker 2: up with more questions and answers based on the format
Speaker 2: of the questions and answers you made. So you can
Speaker 2: take five questions of answers amplify them into one hundred
Speaker 2: questions and answers two hundred questions and answers. And it's
Speaker 2: a second model that is making the questions and answers,
Speaker 2: so it's synthetic data generation using an AI model to
Speaker 2: make the questions. We use an open source model to
Speaker 2: do that. So that's the second part. And then the
Speaker 2: third part is we have a multi phase tuning technique
Speaker 2: to actually take the synthetic data and then basically bake
Speaker 2: it into the model. So sort of that's the approach.
Speaker 2: A general philosophy of the approach is using granite because
Speaker 2: we know where the data came from. Another approach is
Speaker 2: the fact that we're using small models that are cheap
Speaker 2: to run inference on. They're small enough that you can
Speaker 2: tune them on laptop hardware. You don't need all the
Speaker 2: fancy expensive GPU mania you're good. So sort of like
Speaker 2: a whole system, it's like not any one component. But
Speaker 2: it's sort of the approach they took with somewhat novel,
Speaker 2: and they were very excited when they saw the experimental results.
Speaker 2: There was a meeting between red hat and IBM. It
Speaker 2: was actually an IBM research meeting that red hatters were
Speaker 2: invited to, and I think the red Hatter's involves sort
Speaker 2: of saw the potential, WHOA, we can make models open
Speaker 2: source finally, rather than them just being these endless dead forks,
Speaker 2: we could make it so people could contribute back and
Speaker 2: collaborate around it. So that's when red Hat became interested
Speaker 2: in it and we sort of worked together, and the
Speaker 2: research engineers from IBM Research who came up with the technique,
Speaker 2: and then my team, the software engineers who know how
Speaker 2: to take research code and productize it into actually runnable,
Speaker 2: supportable software, kind of got together. We've been hanging out
Speaker 2: in the Boston office at red Hat and building it out.
Speaker 2: April eighteenth was when we went open source and we
Speaker 2: made all of our repositories with all of the code public,
Speaker 2: and right now we're working towards a product release, so
Speaker 2: a supported product.
Speaker 1: How long did it take you to be convinced of
Speaker 1: the value of this idea? I mean, so people get
Speaker 1: together in this hotel room. They're running these experiments over Christmas.
Speaker 1: Are you aware of the experiments as they're running them?
Speaker 2: I didn't find out till February.
Speaker 1: So they come to you in February and they say, MO,
Speaker 1: can you recreate that conversation?
Speaker 2: Well, our CEO, Matt Hicks, and then Jeremy Eater, who's
Speaker 2: one of our distinguished engineers, and Steve Watt, who's a VP,
Speaker 2: were present, I think at that meeting. So they kind
Speaker 2: of brought it back to us and said, listen, we've
Speaker 2: invited these IBM research folks to come visit in Boston,
Speaker 2: you know, work with them, like, see, does this have
Speaker 2: any merit? Could we build something from it? And so
Speaker 2: they gave us some presentations. We were very excited when
Speaker 2: they came to us. It only had support for Mac laptops.
Speaker 2: Of course, you know, Red Hat were Linux people, so
Speaker 2: we're like, all right, we've got to fix that. So
Speaker 2: a bunch of the junior engineers around the office kind
Speaker 2: of came and they're like, okay, we're going to build
Speaker 2: Linux support. And they had it within like a couple
Speaker 2: of days. It was crazy because this was just meant
Speaker 2: to be, Hey, guys, you know what, these are invited
Speaker 2: guests visiting our office. See what happens. And we ended
Speaker 2: up doing like weeks of hack fests and late night
Speaker 2: pizzas in the conference room and like playing around with
Speaker 2: it and learning, and it was it was very fun.
Speaker 2: It's very cool.
Speaker 1: Anyone else do anything like this is.
Speaker 2: Not my understanding that anybody else is doing it. Yet
Speaker 2: maybe others will try. A lot of the focus has
Speaker 2: been on that pre training phase. But for us, again
Speaker 2: that fine tuning, it's more accessible because you don't need
Speaker 2: all the exactic hardware. It doesn't take months. You can
Speaker 2: do it on a laptop. You can do a smoke
Speaker 2: test version of it in less than an hour.
Speaker 1: What is the word smoke test.
Speaker 2: Smoke test means you're not doing a full fine tuning
Speaker 2: on the model. It's a different tuning process. It's like
Speaker 2: kind of lower quality so to run on lower grade
Speaker 2: hardware so you can kind of see them didn't move
Speaker 2: the model or not, but it's not going to give
Speaker 2: you like the full picture. You need higher end hardware
Speaker 2: to actually do the full thing. So that's what the
Speaker 2: product will enable you to do once it's launched. Is
Speaker 2: you're going to need the GPUs, but when you have them,
Speaker 2: will help you make the best usage of them.
Speaker 1: Yeah, yeah, and no, there's all the detail. I want
Speaker 1: to go back to sure in order to run the
Speaker 1: tests on this idea way back when they needed time
Speaker 1: on the GPUs, So this will be the in house
Speaker 1: IBM and they were quiet at Christmas, So how much
Speaker 1: time would you need on the GPUs to kind of
Speaker 1: get proof of concept.
Speaker 2: Well, what happens is and it's sort of like a
Speaker 2: lot of trial and error, right, And there's a lot
Speaker 2: about this stuff that like you come up with the hypothesis,
Speaker 2: you test it out, did it work or not? Okay,
Speaker 2: it's just like you know in the lab, but you know,
Speaker 2: buns and burners and beakers and whatever. So it really depends.
Speaker 2: But it can be hours, it can be days. It
Speaker 2: really depends on what they're trying to do. And then
Speaker 2: sometimes they can cut the time down, you know, with
Speaker 2: the number of GPUs you have. So like I have
Speaker 2: a cluster of agpus, Okay, it might take a day,
Speaker 2: but then if I can get thirty two, I can
Speaker 2: pipeline it and make it go faster and get it
Speaker 2: down to a few hours. So it really depends, you know.
Speaker 2: But it's like everybody's home for the holidays. It's a
Speaker 2: lovely playground to kind of get that stuff going fast.
Speaker 1: Let's jump forward one year. Tell me the status of
Speaker 1: this project, tell me who's using it, tell me how
Speaker 1: big is it. Give me your optimistic but plausible prediction
Speaker 1: about what instruct lab looks like a year from now.
Speaker 2: A year from now, I would like to see kind
Speaker 2: of a vibrant community around not just building knowledge and
Speaker 2: skills into a model, but coming up with better techniques
Speaker 2: and innovation around how we do it. So I'd like
Speaker 2: to see the contributor experience as we grow more and
Speaker 2: more contributors to be refined. So like a year from now,
Speaker 2: Malcolm Gladwell could come impart some of his wisdom into
Speaker 2: the model and it wouldn't be difficult, it wouldn't be
Speaker 2: a big lift. I would love to see the user
Speaker 2: interface tooling for doing that to be more sophisticated. I
Speaker 2: would love to see more people taking this and even
Speaker 2: using it. Maybe you're not sharing it with the community,
Speaker 2: but you're using it for some private usage. Like I'll
Speaker 2: give you an example. I'm in contact with a fellow
Speaker 2: who is doing AI research and he's working with doctors.
Speaker 2: They're GPS in an area of Canada where there's not
Speaker 2: enough GPS for the number of patients. So you know,
Speaker 2: anything you can do to save doctors time to get
Speaker 2: to the next patient. It's like one of the things
Speaker 2: that he has been doing experiments with is can we
Speaker 2: use an open source, licensed model that the doctor can
Speaker 2: run on their laptop so they don't have to worry
Speaker 2: about all of the different privacy rules, Like it's privates
Speaker 2: on the laptop right there, take his live transcription of
Speaker 2: his conversation with the patient and then convert it automatically
Speaker 2: to a soap format that can be entered in the database.
Speaker 2: Typically this will take a doctor fifteen to twenty minutes
Speaker 2: of paperwork. Why not save them the paperwork at least
Speaker 2: have the model take a stab.
Speaker 1: Does the model then hold on to that information and
Speaker 1: he interacts with the model again when well, that's.
Speaker 2: The thing not within struct lab. Maybe that could be
Speaker 2: a future development. It doesn't once you're doing inference, it's
Speaker 2: not ingesting that what you're saying to it back in.
Speaker 2: It's only the fine tuning phase. So the idea would
Speaker 2: be the doctor could maybe load in past patient data
Speaker 2: as knowledge and then when he's trying to diagnose maybe
Speaker 2: you know what I'm saying. Like, But the main idea
Speaker 2: is somebody might have some private users. I would love
Speaker 2: to see more usage of this tool to enable people
Speaker 2: who otherwise never would have had access to this type
Speaker 2: of technology who never like you know, a small country
Speaker 2: GP doctors, it doesn't have GPUs. They're not going to
Speaker 2: hire some company to custom build them a model. But
Speaker 2: maybe on the weekend, if he's a techie guy, he
Speaker 2: could say with that.
Speaker 1: Sim well, I mean the more you talk, the more
Speaker 1: I'm realizing that the simplicity of this model is the
Speaker 1: killer app here. Once you know you can run it
Speaker 1: on a laptop, you have democratized use in a way
Speaker 1: that's inconceivable with some of these other much more complex.
Speaker 1: But that's interesting because one would have thought intuitively that
Speaker 1: at the beginning that the winner is going to be
Speaker 1: the one with the biggest, most complex version, saying actually, no,
Speaker 1: there's a whole series of uses where being lean and focused,
Speaker 1: focused is actually you know, it enables a whole class
Speaker 1: of uses. Maybe another way of saying this is who
Speaker 1: wouldn't be a potential instruct lab customer.
Speaker 2: We don't know yet. It's it's so new, like we
Speaker 2: haven't really had enough people experimenting and playing with it
Speaker 2: and finding out all the things yet. But that's that's
Speaker 2: the thing that's so exciting about it. It's like, I
Speaker 2: can't wait to see what people do.
Speaker 1: Is this the most exciting thing you've worked on in
Speaker 1: your career?
Speaker 2: I think so?
Speaker 1: I think so, Yeah, Well, we are reaching the end
Speaker 1: of our time. But before we finished, we can do
Speaker 1: a little speed round. Sure, all right, complete the following sentence.
Speaker 1: In five years, AI will.
Speaker 2: Be boring, it will be integrated, It'll just work, and
Speaker 2: there will be no now with AI thing. It'll just
Speaker 2: be normal.
Speaker 1: What's the number one thing that people misunderstand about AI?
Speaker 2: It's just matrix algebra. It's just numbers. It's not sentient.
Speaker 2: It's not coming to take us over. It's just numbers.
Speaker 1: You're on this side of you're on the team humanity. Yeah,
Speaker 1: you're good. What advice would you give yourself ten years
Speaker 1: ago to better prepare for today?
Speaker 2: Learn Python for real. It's a programming language that's extensively
Speaker 2: used in the community. I've always dabbled in it, but
Speaker 2: I wish I had taken it more seriously.
Speaker 1: Yeah, did you say, who had a daughter?
Speaker 2: I have three daughters?
Speaker 1: You have three daughters. I have two. You're if you
Speaker 1: got three year you're you're on your own. Are you
Speaker 1: making them study Python?
Speaker 2: I am actually trying to do that. We're using a
Speaker 2: microbit micro controller tool to do like a custom video
Speaker 2: game controller. They prefer Scratch because it's a visual programming language,
Speaker 2: but it has a Python interface too, and I'm like
Speaker 2: pushing them towards Python.
Speaker 1: Good chat box and image generators are the biggest things
Speaker 1: in consumer AI right now. What do you think is
Speaker 1: the next business application?
Speaker 2: Private models, small models, fine tuned on your company's data
Speaker 2: for you to use exclusively.
Speaker 1: Are you using AI in your own personal life these days?
Speaker 2: Honestly, I think a lot of us are using it
Speaker 2: and we don't even realize it.
Speaker 1: Yeah.
Speaker 2: I mean, I'm a ficiano of foreign languages. There's translation
Speaker 2: programs that are built using machine learning underneath. One of
Speaker 2: the things I've been dabbling with lately is using tech
Speaker 2: summarizations because I tend to be very loquacious in my
Speaker 2: note taking and that is not so useful for other
Speaker 2: people who would just like a paragraph. So that's something
Speaker 2: I've been experimenting with myself just to help my everyday work.
Speaker 1: Yeah. We hear many definitions of open related to technology.
Speaker 1: What's your definition of open and how does it help
Speaker 1: you innovate?
Speaker 2: My definition of open is basically sharing and being vulnerable,
Speaker 2: like not just sharing in a have a cookie way,
Speaker 2: but in a you know what, I don't actually know
Speaker 2: how this works? Could you help me? And being open
Speaker 2: to being wrong, being open to somebody helping you and
Speaker 2: making that collaboration work. So it's not just about like
Speaker 2: the artifactor opening, it's your approach, like how you do
Speaker 2: things being open.
Speaker 1: Yeah yeah, well I think that wraps us up. How
Speaker 1: can listeners follow your work and learn more about granted
Speaker 1: and instruct lab.
Speaker 2: Sure, you can visit our project webpage at instruct lab
Speaker 2: dot ai, or you can visit our GitHub at GitHub
Speaker 2: dot com slash instruct lab. We have lots of instructions
Speaker 2: on how to get involved in an instruct lab wonderful.
Speaker 1: Thank you so much, Thank you, Malcolm. A big thank
Speaker 1: you to MO for the engaging discussion on the groundbreaking
Speaker 1: possibilities of instruct lab. We've explored how this platform has
Speaker 1: the potential to revolutionize industries from insurance to entertainment law
Speaker 1: by using an open source community approach that makes it
Speaker 1: easier for more people from all backgrounds to fine tune
Speaker 1: my for specific purposes, ultimately making AI more accessible and
Speaker 1: impactful than ever. Looking ahead, the future of AI isn't
Speaker 1: just about technological efficiency. It's about enhancing our everyday experiences
Speaker 1: in ways that were never possible before, like streamlining work
Speaker 1: for doctors to improve the patient experience or assisting insurance
Speaker 1: agents to improve the claims experience. Instruct Lab is paving
Speaker 1: the way for more open, accessible AI future, one that's
Speaker 1: built on collaboration and humanity. Smart Talks with IBM is
Speaker 1: produced by Matt Romano, Joey Fishground and Jacob Goldstein. We're
Speaker 1: edited by Lydia jen Kott. Our engineers are Sarah Bruger
Speaker 1: and Ben Tolliday. Theme song by Gramscope Special thanks to
Speaker 1: the eight Bar and IBM teams, as well as the
Speaker 1: Pushkin marketing team. Smart Talks with ib VBM is a
Speaker 1: production of Pushkin Industries and Ruby Studio at iHeartMedia. To
Speaker 1: find more Pushkin podcasts, listen on the iHeartRadio app, Apple Podcasts,
Speaker 1: or wherever you listen to podcasts. I'm Malcolm Gladwell. This
Speaker 1: is a paid advertisement from IBM. The conversations on this
Speaker 1: podcast don't necessarily represent IBM's positions, strategies or opinions.