Speaker 1: Hey everyone, it's Robert and Joe here. Today we've got
Speaker 1: something a little bit different to share with you. It
Speaker 1: is a new season of the Smart Talks with IBM
Speaker 1: podcast series.
Speaker 2: Today we are witnessed to one of those rare moments
Speaker 2: in history, the rise of an innovative technology with the
Speaker 2: potential to radically transform business and society forever. The technology,
Speaker 2: of course, is artificial intelligence, and it's the central focus
Speaker 2: for this new season of Smart Talks with IBM.
Speaker 1: Join hosts from your favorite Pushkin podcasts as they talk
Speaker 1: with industry experts and leaders to explore how businesses can
Speaker 1: integrate AI into their workflows and help drive real change
Speaker 1: in this new era of AI. And of course, host
Speaker 1: Malcolm Gladwell will be there to guide you through the
Speaker 1: season and throw in his two cents as well.
Speaker 2: Look out for new episodes of Smart Talks with IBM
Speaker 2: every other week on the iHeartRadio app, Apple Podcasts, or
Speaker 2: wherever you get your podcasts. And learn more at IBM
Speaker 2: dot com, slash smart Talks, Pushkin.
Speaker 3: Hello, Hello, welcome to Smart Talks with IBM, a podcast
Speaker 3: from Pushkin Industries, iHeartRadio and IBM. I'm Malcolm Gladwell. This
Speaker 3: season We're diving back into the world of artificial intelligence,
Speaker 3: but with a focus on the powerful concept of open
Speaker 3: its possibilities, implications, and misconceptions. We'll look at openness from
Speaker 3: a variety of angles and explore how the concept is
Speaker 3: already reshaping industries, ways of doing business, and a very
Speaker 3: notion of what's possible. In today's episode, I sat down
Speaker 3: with mo Duffy, software engineering manager at red Hat, who
Speaker 3: works on instruct Lab, a project co developed by red
Speaker 3: Hat and IBM. Most shared with me how this new
Speaker 3: initiative is revolutionizing AI training, making it not only more accessible,
Speaker 3: but also more inclusive. In this project, unique in the industry,
Speaker 3: allows developers to submit incremental contributions to one base AI model,
Speaker 3: creating a continuous loop of development, much like normal open
Speaker 3: source software. By leveraging community contributions and IBM's cutting edge
Speaker 3: granite models, Mo in the team of ibmrs and red
Speaker 3: hatters are paving the way for a future where AI
Speaker 3: development is a communal endeavor. Our insights into open source
Speaker 3: software extend beyond technical proficiency to the profound impact of
Speaker 3: collaborative effort. At the heart of Moe's work is a
Speaker 3: belief in democratizing technology, ensuring that AI becomes a tool
Speaker 3: accessible to all. So let's explore how MOE, red Hat
Speaker 3: and IBM are empowering individuals and businesses alike to reshape
Speaker 3: the future of technology through collaboration and innovation. MO, thank
Speaker 3: you for joining me today. Thank you so much, for
Speaker 3: I have just about the most Irish name ever. I
Speaker 3: do very proudure you weren't born in Ireland.
Speaker 4: No, my grandparents.
Speaker 3: Oh your grandparents, So I see, where did you grow up?
Speaker 4: New York Queens?
Speaker 3: Oh you're la see. So tell me a little bit
Speaker 3: about how how you got to red hat. What was
Speaker 3: your path?
Speaker 4: When I was in high school, it was a chatty girl,
Speaker 4: teenage girl on the phone. We had one phone line.
Speaker 4: My older brother was studying at the local state college
Speaker 4: computer science, and he had to tell that end to
Speaker 4: compile his homework. One phone line, and I'm on it
Speaker 4: all the time. He got very frustrated and he needed
Speaker 4: a compiler to do his homework. So he bought red
Speaker 4: Hat Linux from a CompUSA, brought it home and that
Speaker 4: was on the family computer. So I learned Linux and
Speaker 4: I started playing around with it. I really liked it
Speaker 4: because you could customize everything, like the entire user interface.
Speaker 4: You could actually modify the code of the programs you
Speaker 4: were using to do what you wanted. And for me,
Speaker 4: it was really cool because especially when you're a kid
Speaker 4: and like people tell you this is the way things
Speaker 4: are and you just have to deal with it, it's
Speaker 4: nice to be like, I'm going to make things the
Speaker 4: way I want, modify the code and playing. Yeah, it
Speaker 4: was amazing and it was just such a time and
Speaker 4: like before it was cool, I was doing it and
Speaker 4: what I saw on that is sort of the potential
Speaker 4: like number one of like a community of people working together.
Speaker 4: And like the Internet existed, it was slow, it involved modems,
Speaker 4: but there were people that you could talk to who
Speaker 4: would give you tips and you'd share information, and this
Speaker 4: collaborative building something together is really something special.
Speaker 2: Right.
Speaker 4: I could file a complaint to whatever large software company
Speaker 4: made whatever software I was into, or I could go
Speaker 4: to an open source software community and be like, hey, guys,
Speaker 4: I think we should do this. I'm like, yeah, okay,
Speaker 4: I'll help. I'll pitch in. So you don't feel powerless,
Speaker 4: you feel like you can have an impact, and that
Speaker 4: was really exciting to me. However, open source software has
Speaker 4: a reputation for not having the best user interface, not
Speaker 4: the best user experience. So I ended up studying Computer
Speaker 4: science and Electronic media dual major, and then I did
Speaker 4: human computeraction as my master's And my thought was, wouldn't
Speaker 4: it be nice if this free software accessible to anybody,
Speaker 4: if it was easier to use, some more people could
Speaker 4: use it and take advantage of it. And so, long
Speaker 4: story short, I ended up going to Red Hat saying, Hey,
Speaker 4: I want to learn how you guys work. Let me
Speaker 4: embed in your team draft out of my graduate program.
Speaker 4: And I'm like, I want to do this for a living.
Speaker 4: This is cooler. So I thought this is the way
Speaker 4: to go, and I've been there ever since. They haven't
Speaker 4: been able to get rid of me.
Speaker 3: To backtrack just a little bit, you were talking about
Speaker 3: the sense of community that surrounds this way of thinking
Speaker 3: about software. Talk a little bit more about what that
Speaker 3: community is like, the benefits of that community, why it
Speaker 3: appeals to you.
Speaker 4: Sure, well, you know part of the reason I actually
Speaker 4: ended up going to the graduate school track. Suddenly you're
Speaker 4: a peer of your professors and you're working side by
Speaker 4: side with them. At some point they retire and you're
Speaker 4: in the next generation. So it's sharing information, building on
Speaker 4: the work of others in sort of this cycle that
Speaker 4: extends past human lifespan and in the same way, like
Speaker 4: the open source model is very similar, but you're actually
Speaker 4: you're building something, and it's something in me. I'm just
Speaker 4: really attracted, Like I don't like talking about stuff. I
Speaker 4: like doing stuff with open source software. The software doesn't
Speaker 4: cost anything, the code is out there, generally uses open
Speaker 4: standards for the file formats. I can open up files
Speaker 4: that I created and open source tools as a high
Speaker 4: school student today because they were using open formats and
Speaker 4: that software still exists. I can still compile the code
Speaker 4: and it's an active community project. Like these things can
Speaker 4: outlast any single company in the same way that the
Speaker 4: academic community has been going on for so many years
Speaker 4: and hopefully we'll continue moving on. So it's sort of
Speaker 4: like not just the community around it, but just the
Speaker 4: knowledge sharing and also bringing up the next generation as well.
Speaker 4: Like all of that stuff really appealed to me. And
Speaker 4: also so at the center of it the fact that
Speaker 4: we could democratize it by following this open source process
Speaker 4: and feel like we have some control. We're not at
Speaker 4: the mercy of some faceless corporation making changes and we
Speaker 4: have no impact. Like that really appealed to me too.
Speaker 3: For those of us who are not software phisionados, take
Speaker 3: a step backwards and give me a kind of description
Speaker 3: of terms. What's the opposite of open source proprietary?
Speaker 4: Proprietary is what we say.
Speaker 3: So specifically and practically, the difference would be what between
Speaker 3: something that was opens us in something that was proprietary.
Speaker 4: Sure, so there's a lot of difference. So with open
Speaker 4: source software you get these rights when you're given the software,
Speaker 4: you get the right to be able to share it.
Speaker 4: And depending on the lot, different licenses that are considered
Speaker 4: open source have different little things that you have to
Speaker 4: be aware of. With proprietary code, it's one copyright the company.
Speaker 4: Even a lot of times, when you sign your employment
Speaker 4: contract for a software company and you write code for them,
Speaker 4: you don't own it. You sign over your rights to
Speaker 4: the company. So if you leave the company, the code
Speaker 4: doesn't go with you. It stays in the ownership of
Speaker 4: that company. So then one like one company buys out
Speaker 4: another and kills a product, that code's gone.
Speaker 3: It's gone for a business, Why would a business want
Speaker 3: to be have open source code as opposed to a proprietary.
Speaker 4: Well for the same reasons. Like, say you're a business.
Speaker 4: You've invested all this money into this software platform, right
Speaker 4: and you've upskilled your employees on it, and it's a
Speaker 4: core part of your business, and then a few years
Speaker 4: later that company goes out of business or something happens,
Speaker 4: or even something less drastic. You really need this future,
Speaker 4: But for the company that makes the software, it's not
Speaker 4: in their best interests. It's not worth the investment. They're
Speaker 4: not going to do it. How do you get that future?
Speaker 4: You either have to completely migrate to another solution, and
Speaker 4: this is something it's core at your business, that's going
Speaker 4: to be a big deal to migrate. But if it's
Speaker 4: open source, you could either hire a team of experts.
Speaker 4: You could hire software engineers who are able to go
Speaker 4: do this for you. Go in the upstream software community,
Speaker 4: implement the feature that you want, and it'll be rolled
Speaker 4: into the next version of that company software. So even
Speaker 4: if that company didn't want to implement the feature, if
Speaker 4: they did it open source, they would inherit that feature
Speaker 4: from the upstream community, is what we call it. So
Speaker 4: you have some control over the situation. If it's open source,
Speaker 4: you have an opportunity to actually affect change in the product,
Speaker 4: and you could then pick it up or pay somebody
Speaker 4: else to pick it up, or another company could form
Speaker 4: and pick it up and keep it going. So there's
Speaker 4: more possibilities. If it's open source, it's more like it's
Speaker 4: like an insurance policy almost.
Speaker 3: So innovation from the standpoint of the customer, innovation is
Speaker 3: a lot easier when you're working in an open source environment.
Speaker 4: Absolutely.
Speaker 3: Yeah. So now at RedHat you're working with something called
Speaker 3: instruct lab. Tell us a little bit about what that is.
Speaker 4: So the thing that really excites me about getting to
Speaker 4: work on this project is AI is sort of that
Speaker 4: has been this scary thing for me because it's one
Speaker 4: of those things like in order to be able to
Speaker 4: pre train a model, you have to have unobtainium GPS,
Speaker 4: you have to have rich resources, It takes months, it
Speaker 4: takes expertise. There's a small handful of companies that can
Speaker 4: build a model from pre train to something usable, and
Speaker 4: it kind of feels like those early days when I
Speaker 4: was kind of delving in software in the same way.
Speaker 4: I think if more people could contribute to AI models,
Speaker 4: then it wouldn't be just influenced by whichever company had
Speaker 4: the resources to build it. And there's been a lot
Speaker 4: of emphasis on pre training models, so taking massive terabytes
Speaker 4: data sets, throwing them through masses of GPS over months
Speaker 4: of time, spending hundreds of millions of dollars to build
Speaker 4: a base model. But when instruct lab does is say okay,
Speaker 4: you have a base model. We're going to fine tune in.
Speaker 4: On the other end, it takes less compute resources. The
Speaker 4: way we've built in struck lab, you can play around
Speaker 4: with the technology and learn it on it off the
Speaker 4: shelf laptop that you can actually buy. So in this
Speaker 4: way we're enabling a much broader set of people to
Speaker 4: play with AI, to contribute it, to modify it. And
Speaker 4: I'll tell you one story from red Hat Succi, who
Speaker 4: is our chief diversity officer, very interested in inclusive language
Speaker 4: and open source software, doesn't have any experience with AI.
Speaker 4: We have a community model that we have an upstream
Speaker 4: project around for people to contribute knowledge and skills to
Speaker 4: the model. She's like, I want to teach the model
Speaker 4: how to use inclusive language, like replace this word with
Speaker 4: this word or this word with this word. OHM Like, oh,
Speaker 4: that's so cool. So She paired up with Nicholas who
Speaker 4: is a technical guy at red Hat, and they built
Speaker 4: and submitted a skill to the model that you can
Speaker 4: just tell the model, can you please take this document
Speaker 4: and translate this language to more inclusive language and it
Speaker 4: will do it. And they submitted it to the community.
Speaker 4: They were so proud. It was like, that's the kind
Speaker 4: of thing that, like, you know, maybe a company would
Speaker 4: be incentivized to do that, but if you have some
Speaker 4: tooling that's open source and something that anybody could access,
Speaker 4: than those communities could actually get together and build that
Speaker 4: knowledge into AI models.
Speaker 3: Just so understand, what you guys have is the structure
Speaker 3: for an AI system, And in other cases, individual companies
Speaker 3: own and train their own AI systems. It takes enormous
Speaker 3: amount of resources. They hoover up all kinds of information,
Speaker 3: train it according to their own hidden set of rules,
Speaker 3: and then a customer might use that for some price.
Speaker 3: What you're saying is, in the same way that we
Speaker 3: democratize the writing of software before, let's democratize the training
Speaker 3: of an AI system. So anyone can contribute here and
Speaker 3: teach the model the things that they're interested in teaching
Speaker 3: the model. I'm guessing correct me. On the one hand,
Speaker 3: this model, at least in the beginning, is going to
Speaker 3: have a lot fewer resources available to it. But on
Speaker 3: the other hand, it's going to have a much more
Speaker 3: diverse set of inputs.
Speaker 4: That's right. And the other thing is that IBM, basically
Speaker 4: is part of this project, has something called the Granite
Speaker 4: Model family, and they've donated some granite models. So these
Speaker 4: are the ones that take the months and terabytes of
Speaker 4: data and all the GPUs to train. So IBM has
Speaker 4: created one of those, and they have listed out and
Speaker 4: linked to the data sets that they used, and they
Speaker 4: talk about the relative proportions they used when pre training,
Speaker 4: so it's not just the black box. You know where
Speaker 4: the data came from, which is a pretty open position
Speaker 4: to take. That is what we recommend as the base.
Speaker 4: So you use the instruct lab tuning. You take this
Speaker 4: base granite model that IBM has provided, and you use
Speaker 4: the instruct lab tooling that red Hat works on, and
Speaker 4: you use that to fine tune the model to make
Speaker 4: it whatever you want.
Speaker 3: I want to go back to the partnership between IBM
Speaker 3: and red Hat here with them providing the granite model
Speaker 3: to your instruct lab Is this the first time red
Speaker 3: hat and IBM have collaborated like this, I think it's.
Speaker 4: Something that's been going on. Like another a product within
Speaker 4: the red hat family would be open Shift AI, where
Speaker 4: they collaborate a lot with IBM Research team, Like BLM
Speaker 4: is one of the components of that product that there's
Speaker 4: a nice kind of exchange and collaboration between the two companies.
Speaker 3: How large is the potential community of people who might
Speaker 3: contribute to instruct lab.
Speaker 4: It could be thousands of people. I mean, we'll see.
Speaker 4: It's early days. This is early technology that was invented
Speaker 4: at IBM Research that they partnered with us at red
Speaker 4: Hat to kind of build the software around it. There's
Speaker 4: still more to go, Like right now, we have a
Speaker 4: team in the community that's actually trying to build a
Speaker 4: web interface to make it easier for anybody to contribute.
Speaker 4: So we have a lot of those sort of user
Speaker 4: experience for the contributor to the model stuff to work
Speaker 4: out that we're still actively building on. But like my
Speaker 4: vision for it even is I like going back to
Speaker 4: that academic model of learning from what others and building
Speaker 4: upon it over time. It would be very good for
Speaker 4: us to sort of go out and try to collaborate
Speaker 4: with academics of all, like, hey, you know, the model
Speaker 4: doesn't know about your field, would you like to put
Speaker 4: something into the model about your field so it knows
Speaker 4: about it, or even you know, talk to the model
Speaker 4: it got it wrong, let's correct it. Can we lean
Speaker 4: on your expertise to correct it and make sure it
Speaker 4: gets it right and sort of use that community model
Speaker 4: as a way for everybody to collaborate because before instruct lab,
Speaker 4: my understanding is if you wanted to take a model
Speaker 4: that's open source license and play with it, you could
Speaker 4: do that. You could take a model kind of off
Speaker 4: the shelf from Hugging Face and fine tune it yourself.
Speaker 4: But it's a bit of a dead end because you
Speaker 4: made your contributions, but there's no way for other people
Speaker 4: to collaborate with you. So the way that we've built
Speaker 4: this is based on how the technology works. Everybody can
Speaker 4: contribute to it. This is something that you can keep
Speaker 4: growing and growing and growing over time.
Speaker 3: Yeah. Yeah, what's the level of expertise necessary to be
Speaker 3: a contributor?
Speaker 4: You don't need to be a data scientist and you
Speaker 4: don't need to have exotic hardware. Honestly, if you don't
Speaker 4: even have laptop hardware that meets SUSPEC for doing instruct
Speaker 4: labs laptop version. You can submit it to the community
Speaker 4: and then we'll actually build it for you. We have
Speaker 4: bots and stuff that do that, and we're hoping over
Speaker 4: time to make that more accessible, first by having a
Speaker 4: user interface and then maybe later on having a web service.
Speaker 3: Yeah, so give me an example of how a business
Speaker 3: might make use of instruct lab.
Speaker 4: One of the things that businesses are doing with AI
Speaker 4: right now is using hosted API services. They're quite expensive,
Speaker 4: but they're finding value, but it's hard given the amount
Speaker 4: of money they're spending. And one of the things that's
Speaker 4: a little scary about it too, is like you have
Speaker 4: very sensitive internal documents and you have employees maybe not
Speaker 4: understanding what they're actually doing because you know, how would
Speaker 4: you if you're not technical enough when you're asking said
Speaker 4: public web service AI model information about your copy pasting
Speaker 4: internal company documents. It's going across the Internet into another
Speaker 4: company's hands, and that company probably shouldn't have access to that.
Speaker 4: So what both RedHat and IBM in the space are
Speaker 4: looking at, like the instruct lab model is very modest.
Speaker 4: It's seven billion parameter model very small. It's very cheap
Speaker 4: to serve inference on a seven billion parameter model. It's
Speaker 4: competing with trillion parameter models that are hosted. You take
Speaker 4: this small model that is cheap to run inference on,
Speaker 4: you train it with your own company's proprietary data inside
Speaker 4: the walls of your company, on your own hardware. You
Speaker 4: can do all sorts of actual data analysis on your
Speaker 4: most sensitive data and have the confidence that has not
Speaker 4: left the premises.
Speaker 3: In that use case, you're not actually training the model
Speaker 3: for everyone. You're just taking it and doing some private
Speaker 3: stuff on it exactly, which doesn't leave the building. But
Speaker 3: that's separate from an interaction where you're doing something that
Speaker 3: contributes overall.
Speaker 4: Right, And that's something maybe that I should be more
Speaker 4: clear about is there's sort of two tracks here, and
Speaker 4: this is very red hat classic. You have your upstream
Speaker 4: community track and you have your business product tract. So
Speaker 4: the upstream community track is just enabling anybody to contribute
Speaker 4: to a model in a collaborative way and play with it.
Speaker 4: The downstream product business oriented track is now take that
Speaker 4: tech that we've honed and developed in the open community
Speaker 4: and apply it to your business knowledge and skills.
Speaker 3: This community driven approach marks a pivotal shift towards more
Speaker 3: accessible AI solutions. The contrast between externally hosted AI services
Speaker 3: and the open model enhanced by instruct lab underscores the
Speaker 3: potential for broader adoption of AI in diverse business contexts.
Speaker 3: She envisions a future in which technological innovation is more
Speaker 3: tailored to individual business needs, guided by principles of openness
Speaker 3: and security. To an imaginary case study, Sure, I'm a
Speaker 3: law firm, I'm an entertainment law I have one hundred
Speaker 3: clients who are big stars. They all have incredibly complicated contracts.
Speaker 3: I feed a thousand of my company's contracts from the
Speaker 3: last ten years into the model, and then every time
Speaker 3: I have a new contract, I ask the model, am
Speaker 3: I missing something? Can you go back and look through
Speaker 3: all our own contracts and show me a contract that
Speaker 3: is missing key components or exposes us to some liability.
Speaker 3: In that case, the model would know my law firm
Speaker 3: contracts really, really well. It's as if they've been working
Speaker 3: out my law firm. They're not distracted by other people's
Speaker 3: particular styles or a bunch of contracts from the utility industry,
Speaker 3: or the They know entertainment law contracts exactly.
Speaker 4: Yeah, and you can train it in your own image,
Speaker 4: your style of doing things. It's something that your company
Speaker 4: can produce that is uniquely helpful to you. No third
Speaker 4: party could do that because no third party understands how
Speaker 4: you do business and understands your his street in your documents.
Speaker 4: So it's sort of a way of getting value out
Speaker 4: of the stuff you already have sitting in a file
Speaker 4: cabinet somewhere. It's very cool.
Speaker 3: Yeah, give me a sort of a real world case
Speaker 3: study where you think the business use case would be
Speaker 3: really powerful. What's a business that really could see an
Speaker 3: advantage to using instruct lab in its way.
Speaker 4: The demo that I've given a couple of times at
Speaker 4: different events used an imaginary insurance company. So you say,
Speaker 4: you have this company, you have to recommend repairs for
Speaker 4: various types of claims. You've been doing this for years,
Speaker 4: you know. If you know the windshield's broken and you've
Speaker 4: gotten this type of accident and it's this model car,
Speaker 4: these are the kinds of things you want to look at.
Speaker 4: So you could talk to any insurance agent in the
Speaker 4: field and be like, oh, you know, it's a Tesla.
Speaker 4: You might want to look at the battery or something like.
Speaker 4: They'll have some latent knowledge just so you can take
Speaker 4: that and train it into a model. Honestly, I think
Speaker 4: these kind of new technologies are better when they're less visible.
Speaker 4: So say you have the claims agents in the field
Speaker 4: and they have this tool and they're kind of entering
Speaker 4: the claim data. They're on the scene at the car,
Speaker 4: and it might say, oh, look, I see this is
Speaker 4: a Ford fiesta. These are things you want to look
Speaker 4: at for this type of accident. As you're entering the data,
Speaker 4: it could be going through the knowledge you had loaded
Speaker 4: into the model and be making these suggestions based on
Speaker 4: your company's background, and hey, you know, let's not make
Speaker 4: the same mistake twice. Let's make new mistakes and let's
Speaker 4: learn from the stuff we already did. So that's one example,
Speaker 4: but there's so many different industries in ways that this
Speaker 4: could help, and it could make those agents in the
Speaker 4: field more efficient.
Speaker 3: Have you had anyone talk to you about using instruct
Speaker 3: lab in a way that surprised you.
Speaker 4: I mean, some people have done funky things, but sort
Speaker 4: of playing with the skills stuff, that's where I see
Speaker 4: a lot of creativity. The difference between knowledge and skills
Speaker 4: is that knowledge is pretty pretty understandable, right, oh, historical
Speaker 4: insurance claims or you know, legal contracts. Skills are a
Speaker 4: little different. So whenever somebody submits a skill, sometimes it
Speaker 4: tends to be really creative because it's not something that's
Speaker 4: super intuitive. Somebody submitted a skill. I don't know how
Speaker 4: well it worked, but it was like making ASKI art,
Speaker 4: like draw me a I don't know, draw me a
Speaker 4: dog I would do like an ASKI art dog. I mean,
Speaker 4: it's stuff that you can do programmatically. One that was
Speaker 4: actually very very helpful was, you know, take this table
Speaker 4: of data and convert it to this format, like, ooh,
Speaker 4: that's nice. That actually saves me time.
Speaker 3: How far away are we from the day when I
Speaker 3: Malcolm Globwell technology ignore Amus can go home and easily
Speaker 3: interact with instruct lab Maybe a few months, a few months,
Speaker 3: you're gonna say a few years.
Speaker 4: No, I think it'd be a few months.
Speaker 3: Wow, I hope.
Speaker 4: Hey it's power open source innovation.
Speaker 3: Yeah, oh that's really interesting. Yeah, I'm always taken by surprise.
Speaker 3: I'm still thinking in twentieth century terms about how long
Speaker 3: things take, and you're in the twenty second century as
Speaker 3: well as.
Speaker 1: I could tell.
Speaker 4: The instruct lab core invention was invented in a hotel
Speaker 4: room at an AI conference in December with an amazing
Speaker 4: group of IBM research guys December of twenty twenty three.
Speaker 3: Wait, back up, you have to tell the story.
Speaker 4: This group of guys we've been working with, they were
Speaker 4: at this conference together, and it's a really funny story
Speaker 4: because you know, it's hard to get access to GPUs
Speaker 4: and like even you know, you're at IBM and it's
Speaker 4: hard to get access because everybody wants access. They did
Speaker 4: it over Christmas break because nobody was using the cluster
Speaker 4: at the time, and they ran all of these experiments
Speaker 4: and I'm like, whoa, this is really cool.
Speaker 3: And wait. Their idea was we can do a stripped
Speaker 3: down AI model, and was the idea and even back
Speaker 3: then combine it with grantede, what was the original idea?
Speaker 4: The original idea, it's sort of multi there's like multiple
Speaker 4: aspects to it. So like one of the aspects it
Speaker 4: actually came on later, but it starts at the beginning
Speaker 4: of the workflow. Is you're using a taxonomy to organize
Speaker 4: how you're fine tuning the model. So the old approach
Speaker 4: they call it the blender approach, to just take a
Speaker 4: bunch of data of roughly the type of data that
Speaker 4: you'd like, and you kind of throw it in and
Speaker 4: then see what comes out. Don't like it, Okay, throw
Speaker 4: in more, try again, see what comes out. They had
Speaker 4: used this taxonomy technique, so you actually build like a
Speaker 4: taxonomy of like categories and subfolders of like this is
Speaker 4: the knowledge and skills that we want to train into
Speaker 4: the model. And that way you're sort of systematic about
Speaker 4: what you're adding, and you can also identify gaps pretty easily.
Speaker 4: Oh I don't have a category for that, let me
Speaker 4: add that. So that's like one of the parts of
Speaker 4: the invention here.
Speaker 3: Point number one is let's be intentional and deliberate in
Speaker 3: how we build and train this thing.
Speaker 4: Yeah, and then the next component would be okay, so
Speaker 4: it is actually quite expensive. Part of the expense of
Speaker 4: like tuning models and just training models in general is
Speaker 4: coming up with the data. And what they wanted to
Speaker 4: do is have a technique where you could have just
Speaker 4: a little bit of data and expand it with something
Speaker 4: they're calling synthetic data generation. And this is where it's
Speaker 4: sort of like you have this student and teacher workflow,
Speaker 4: so you have your taxonomy. The taxonomy has sort of
Speaker 4: the knowledge like a business's knowledge documents, their insurance claims,
Speaker 4: and it has these quizzes that you write and that's
Speaker 4: to teach the model. So I'm writing a quiz based
Speaker 4: just like you do in school. You read the chapter
Speaker 4: on the American Revolution, and then you answer a ten
Speaker 4: question quiz where you're giving the model quiz. You need
Speaker 4: at least five questions and answers, and the answers need
Speaker 4: to be taken from the context of the document, and
Speaker 4: then you run it through a process called synthetic data generation,
Speaker 4: and it looks at the documents or look at the
Speaker 4: history chapter. It'll look at the questions and answers, and
Speaker 4: then it'll look to that original document and come up
Speaker 4: with more questions and answers based on the format of
Speaker 4: the questions and answers you made. So you can take
Speaker 4: five questions of answers amplify them into one hundred questions
Speaker 4: and answers, two hundred questions and answers, and it's a
Speaker 4: second model that is making the questions and answers. So
Speaker 4: it's synthetic data generation using an AI model to make
Speaker 4: the questions. We use an open source model to do that.
Speaker 4: So that's the second part. And then the third part
Speaker 4: is we have a multi phase tuning technique to actually
Speaker 4: take the synthetic data and then basically bake it into
Speaker 4: the model. So sort of that's the approach. A general
Speaker 4: philosophy of the approach is using granite because we know
Speaker 4: where the data came from. Another approach is the fact
Speaker 4: that we're using small models that are cheap to run
Speaker 4: inference on. They're small enough that you can tune them
Speaker 4: on laptop hardware. You don't need all the fancy expensive
Speaker 4: GPU menia. You're good. So sort of like a whole system.
Speaker 4: It's like not any one component. But it's sort of
Speaker 4: the approach they took with somewhat novel, and they were
Speaker 4: very excited when they saw the experimental results. There was
Speaker 4: a meeting between red hat and IBM. It was actually
Speaker 4: an IBM research meeting that red hatters were invited to,
Speaker 4: and I think the red Hatter and Voves sort of
Speaker 4: saw the potential, WHOA, we can make models open source finally,
Speaker 4: rather than them just being these endless dead forks, we
Speaker 4: could make it so people could contribute back and collaborate
Speaker 4: around it. So that's when red Hat became interested in
Speaker 4: it and we sort of worked together, and the research
Speaker 4: engineers from IBM Research who came up with the technique,
Speaker 4: and then my team, the software engineers who know how
Speaker 4: to take research code and productize it into actually runnable,
Speaker 4: supportable software, kind of got together. We've been hanging out
Speaker 4: in the Boston office at red Hat and building it out.
Speaker 4: April eighteenth was when we went open source and we
Speaker 4: made all of our repositories with all of the code public,
Speaker 4: and right now we're working towards a product release, so
Speaker 4: a supported product.
Speaker 3: How long did it take you to be convinced of
Speaker 3: the value of this idea? I mean, so people get
Speaker 3: together in this hotel room they're running these experiments over Christmas.
Speaker 3: Are you aware of the experiments as they're running them?
Speaker 2: They?
Speaker 4: Oh, I didn't find out to February.
Speaker 3: They come to you February and they say, MO, can
Speaker 3: you recreate that conversation?
Speaker 4: Well, our CEO, Matt Hicks, and then Jeremy Eater, who's
Speaker 4: one of our distinguished engineers, and Steve Watt, who's a VP,
Speaker 4: were present I think at that meeting. So they kind
Speaker 4: of brought it back to us and said, listen, we've
Speaker 4: invited these IBM research folks to come visit in Boston,
Speaker 4: you know, work with them, like, see, does this have
Speaker 4: any merit? Could we build something from it? And so
Speaker 4: they gave us some presentations. We're very excited. When they
Speaker 4: came to us. It only had support for Mac laptops.
Speaker 4: Of course, you know Red Hat were Linux people, So
Speaker 4: we're like, all right, we've got to fix that. So
Speaker 4: a bunch of the junior engineers around the office kind
Speaker 4: of came and they're like, okay, we're going to build
Speaker 4: Linux support. And they had it within like a couple
Speaker 4: of days. It was crazy because this was just meant
Speaker 4: to be Hey, guys, you know what, these are invited
Speaker 4: guests visiting our office. See what happens. And we ended
Speaker 4: up doing like weeks of hack fe and late night
Speaker 4: pizzas in the conference room and like playing around with
Speaker 4: it and learning and it was it was very fun.
Speaker 4: It's very cool.
Speaker 3: Anyone else do anything like this.
Speaker 4: Is not my understanding that anybody else is doing it,
Speaker 4: yet maybe others will try a lot of the focus
Speaker 4: has been on that pre training phase. But for us,
Speaker 4: again that fine tuning. It's more accessible because you don't
Speaker 4: need all the exotic hardware. It doesn't take months. You
Speaker 4: can do it on a laptop. You can do a
Speaker 4: smoke test version of it in less than an hour.
Speaker 3: What is the word smoke test.
Speaker 4: Smoke test means you're not doing a full fine tuning
Speaker 4: on the model. It's a different tuning process. It's like
Speaker 4: kind of lower quality, so to run on lower grade hardware,
Speaker 4: so you can kind of see them didn't move the
Speaker 4: model or not, but it's not going to give you,
Speaker 4: like the full picture. You need higher end hardware to
Speaker 4: actually do the full thing. So that's what the product
Speaker 4: will enable you to do once it's launched, is you're
Speaker 4: going to need the GPUs, but when you have them,
Speaker 4: will help you make the best usage of them.
Speaker 3: Yeah, yeah, and no, there's all the detail. I want
Speaker 3: to go back to. Sure to run the tests on
Speaker 3: this idea way back when they needed time on the GPUs,
Speaker 3: So this will be the in house IBM and they
Speaker 3: were quiet at Christmas, So how much time would you
Speaker 3: need on the GPUs to kind of get proof of concept?
Speaker 4: Well what happens is and it's sort of like a
Speaker 4: lot of trial and error, right, And there's a lot
Speaker 4: about this stuff that like you come up with the hypothesis,
Speaker 4: you test it out, did it work or not? Okay,
Speaker 4: it's just like you know in the lab, but you know,
Speaker 4: buns and burners and beakers and whatever. So it really depends.
Speaker 4: But it can be hours, it can be days. It
Speaker 4: really depends on what they're trying to do. And then
Speaker 4: sometimes they can cut the time down, you know, with
Speaker 4: the number of GPUs you have. So like I have
Speaker 4: a cluster of agpus, Okay, it might take a day,
Speaker 4: but then if I can get thirty two, I can
Speaker 4: pipeline it and make it go faster and get it
Speaker 4: down to a few hours. So it really depends, you know.
Speaker 4: But it's like everybody's home for the holidays. It's a
Speaker 4: lovely playground to kind of get that stuff going fast.
Speaker 3: Let's jump forward one year. Tell me the status of
Speaker 3: this project, tell me who's using it, tell me how
Speaker 3: big is it. Give me your optimistic but plausible prediction
Speaker 3: about what instruct lab looks like a year from now.
Speaker 4: A year from now, I would like to see kind
Speaker 4: of a vibrant community around not just building knowledge and
Speaker 4: skills into a model, but coming up with better techniques
Speaker 4: and innovation around how we do it. So I'd like
Speaker 4: to see the contributor experience as we grow more and
Speaker 4: more contributors to be refined. So like a year from now,
Speaker 4: Malcolm Gladwell could come impart some of his wisdom into
Speaker 4: the model and it wouldn't be difficult, it wouldn't be
Speaker 4: a big lift. I would love to see the user
Speaker 4: interface tooling for doing that to be more sophisticated. I
Speaker 4: would love to see more people taking this and even
Speaker 4: using it. Maybe you're not sharing it with the community,
Speaker 4: but you're using it for some private usage. Like I'll
Speaker 4: give you an example. I'm in contact with a fellow
Speaker 4: who is doing AI research and he's working with doctors.
Speaker 4: They're GPS in an area of Canada where there's not
Speaker 4: enough GPS for the number of patients, So you know,
Speaker 4: anything you can do to save doctors time to get
Speaker 4: to the next patient. It's like one of the things
Speaker 4: that he has been doing experiments with is can we
Speaker 4: use an open source, licensed model that the doctor can
Speaker 4: run on their laptop so they don't have to worry
Speaker 4: about all of the different privacy rules, Like it's privates
Speaker 4: on the laptop right there, take his live transcription of
Speaker 4: his conversation with the patient, and then convert it automatically
Speaker 4: to a soap format that can be entered in the database.
Speaker 4: Typically this will take a doctor fifteen to twenty minutes
Speaker 4: of paperwork. Why not save them the paperwork at least
Speaker 4: have the model take a stab.
Speaker 3: Does the model then hold on to that information and
Speaker 3: he interacts with the model again when.
Speaker 4: Well, that's the thing not within struct lab. Maybe that
Speaker 4: could be a future development. It doesn't once you're doing it, diference,
Speaker 4: it's not ingesting that what you're saying to it back in.
Speaker 4: It's only the fine tuning phase. So the idea would
Speaker 4: be the doctor could maybe load in past patient data
Speaker 4: as knowledge and then when he's trying to diagnose maybe
Speaker 4: you know what I'm saying. Like, But the main idea
Speaker 4: is somebody might have some private usage. I would love
Speaker 4: to see more usage of this tool to enable people
Speaker 4: who otherwise never would have had access to this type
Speaker 4: of technology who never like you know, a small country
Speaker 4: GP doctors, it doesn't have GPUs. They're not going to
Speaker 4: hire some company to custom build them a model. But
Speaker 4: maybe on the weekend, if he's a techie guy he
Speaker 4: can say with us.
Speaker 3: Well, I mean, the more you talk, the more I'm
Speaker 3: realizing that the simplicity of this model is the killer
Speaker 3: app here. Once you know you can run it on
Speaker 3: a laptop, you have democratized use in a way that's
Speaker 3: inconceivable with some of these other much more complex. But
Speaker 3: that's interesting because one would have thought intuitively that at
Speaker 3: the beginning that the winner is going to be the
Speaker 3: one with the biggest, most complex version, And you're saying, actually, no,
Speaker 3: there's a whole series of uses where being lean and focused,
Speaker 3: focused is actually you know, it enables a whole class
Speaker 3: of uses. Maybe another way of saying this is who
Speaker 3: wouldn't be a potential instruct lab customer.
Speaker 4: We don't know yet. It's so new, like we haven't
Speaker 4: really had enough people experimenting and playing with it and
Speaker 4: finding out all the things yet. But that's that's the
Speaker 4: thing that's so exciting about it. It's like, I can't
Speaker 4: wait to see what people do.
Speaker 3: Is this the most exciting thing you've worked on in
Speaker 3: your career?
Speaker 4: I think so. I think so.
Speaker 3: Yeah, Well, we are reaching the end of our time,
Speaker 3: but before we finished, we can do a little speed round. Sure,
Speaker 3: all right, complete the following sentence. In five years, AI
Speaker 3: will be.
Speaker 4: Boring, it will be integrated, It'll just work, and there
Speaker 4: will be no now with AI thing. It'll just be normal.
Speaker 3: What's the number one thing that people misunderstand about AI?
Speaker 4: It's just matrix algebra. It's just numbers. It's not sentient.
Speaker 4: It's not coming to take us over. It's just numbers.
Speaker 3: You're on this side of You're on the team humanity. Yeah,
Speaker 3: you're one good. What advice would you give yourself ten
Speaker 3: years ago to better prepare for today?
Speaker 4: Learn Python for real. It's a programming language that's extensively
Speaker 4: used in the community. I've always dabbled in it, but
Speaker 4: I wish I had taken it more seriously.
Speaker 3: Yeah, did you say, who had a daughter?
Speaker 4: I have three daughters?
Speaker 3: You have three daughters. I have two. You're if you
Speaker 3: got three year you're you're on your own. What are
Speaker 3: you making them study Python?
Speaker 4: I am actually trying to do that. We're using a
Speaker 4: microbit micro controller tool to do like a custom video
Speaker 4: game controller. They prefer Scratch because it's a visual programming language,
Speaker 4: but it has a Python interface too, and I'm like
Speaker 4: pushing them towards Python.
Speaker 3: Good chat, bock and image generators are the biggest things
Speaker 3: in consumer AI right now. What do you think is
Speaker 3: the next big business application?
Speaker 4: Private models, small models fine tuned on your company's data
Speaker 4: for you to use exclusively.
Speaker 3: Are you using AI in your own personal life these days?
Speaker 4: Honestly, I think a lot of us are using it
Speaker 4: and we don't even realize it. Yeah, I mean, I'm
Speaker 4: a ficiano of foreign languages. There's translation programs that are
Speaker 4: built using machine learning underneath. One of the things I've
Speaker 4: been dabbling with lately is using tech summarizations because I
Speaker 4: tend to be very loquacious in my note taking and
Speaker 4: that is not so useful for other people who would
Speaker 4: just like a paragraph. So that's something I've been experimenting
Speaker 4: with myself just to help my everyday work.
Speaker 3: Yeah. We hear many definitions of open related to technology.
Speaker 3: What's your definition of open and how does it help
Speaker 3: you innovate?
Speaker 4: My definition of open is basically sharing and being vulnerable,
Speaker 4: like not just sharing gonna have a cookie way, but
Speaker 4: in a you know what, I don't actually know how
Speaker 4: this works? Could you help me? And being open to
Speaker 4: being wrong, being open to somebody helping you, and making
Speaker 4: that collaboration work. So it's not just about like the
Speaker 4: artifact or opening, it's your approach, like how you do
Speaker 4: things being open.
Speaker 3: Yeah yeah, well I think that wraps us up. How
Speaker 3: can listeners follow your work and learn more about granted
Speaker 3: and instruct lab.
Speaker 4: Sure, you can visit our project web page at instruct
Speaker 4: lab dot ai, or you can visit our GitHub at
Speaker 4: GitHub dot com slash instruct lab. We have lots of
Speaker 4: instructions on how to get involved in an instruct lab wonderful.
Speaker 3: Thank you so much, Thank you, Malcolm. A big thank
Speaker 3: you to Mow for the engaging discussion on the groundbreaking
Speaker 3: possibilities of instruct lab. We've explored how this platform has
Speaker 3: the potential to revolutionize industries from insurance to entertainment law
Speaker 3: by using an open source community, the approach that makes
Speaker 3: it easier for more people from all backgrounds to fine
Speaker 3: tune models for specific purposes, ultimately making AI more accessible
Speaker 3: and impactful than ever. Looking ahead, the future of AI
Speaker 3: isn't just about technological efficiency. It's about enhancing our everyday
Speaker 3: experiences in ways that were never possible before, like streamlining
Speaker 3: work for doctors to improve the patient experience, or assisting
Speaker 3: insurance agents to improve the claims experience. Instruct Lab is
Speaker 3: paving the way for more open, accessible AI future, one
Speaker 3: that's built on collaboration and humanity. Smart Talks with IBM
Speaker 3: is produced by Matt Romano, Joey Fishground and Jacob Goldstein.
Speaker 3: We're edited by Lydia jen Kott. Our engineers are Sarah
Speaker 3: Bruger and Ben Tolliday. Theme song by Gramoscope Special thanks
Speaker 3: to the Eight Bars and IBM teams, as well as
Speaker 3: the Pushkin marketing team. Smart Talks with IBM is a
Speaker 3: production of Pushkin Industries and Ruby Studio at iHeartMedia. To
Speaker 3: find more Pushkin podcasts, listen on the iHeartRadio app, Apple Podcasts,
Speaker 3: or wherever you listen to podcasts. I'm Malcolm Gladwell. This
Speaker 3: is a paid advertisement from IBM. The conversations on this
Speaker 3: podcast don't necessarily represent IBM's positions, strategies or opinions.