A Cultural AI Shift – Digital CxO Podcast EP89

Hello and welcome to the digital CXO podcast. I'm Amanda Ani and I'm excited to have Mo Duffy here today. She is the software engineer manager at Red Hat.

How are you doing? Pretty Good. How are you doing, Amanda?

Doing well. So happy to have you on the show today. Very happy to be here.

Well, red Hat's a pretty big name, but can you go ahead and share with our audience a little bit about Red Hat and the services that are provided? Sure. So Red Hat is essentially like the open source leader.

We have everything from OS platform, um, which is Red Linux up to Kubernetes for your applications on Red Hat OpenShift. Um, AI is a new field for us that we're starting out in. We had some announcements at the Red Hat Summit earlier this summer about some of our, uh, work in AI on top of preexisting OpenShift ai.

Um, so yeah, it's basically making open source technology accessible for enterprise businesses. Wonderful. Well, that's a good segue into our topic of the day, which is the implementation of AI and uh, achieving AI goals in companies.

But there's a lack of AI skills to meet those goals. So can you share from your experience, what are you seeing from business leaders is a real barrier or roadblock when it comes to this issue? Sure.

So I wanna say, and I've, I've been at Red Hat for 20 years now. When I started at Red Hat, um, when I started with Red Hat, I'll say Red Hat Linux, I was a high school student. That was the early days of Linux.

And right now AI feels like the early days of Linux where there were a lot of distros. Nobody was sure which projects were gonna stick. It was hard to make choices 'cause you didn't know what would have the longevity.

And I think AI is a field right now is in a similar spot. We have some things that are kind of becoming standards and some platforms that seem promising, but it's really hard to make bets on it. Um, the added dimension with AI also is hardware access.

Um, back in the old days of Linux, everybody had hardware to run it on, so that wasn't really the consideration. The consideration was more which choice I made. So, um, with AI it's, you kind of have a double whammy of you have to have the hardware access and you have to figure out what is the platform I should bet on.

Um, and so in the same way that Red Hat approaching software development in an open source manner, we basically look at upstream open source projects, try to evaluate which projects are the healthy ones, which ones seem to be taking a leadership role in the field, and which are the ones we should make bets on. We sort of take that, uh, decision making out of basically off the hands of businesses who would rather do whatever their core business need is, whether it's like insurance or banking or medical things. Like they don't need to be open source a nalytics experts.

We make those decisions for them. So in, in the same way we're kind of looking at AI and trying to see, well what are the strongest projects in the open source community can we contribute in an open source manner to AI technology and sort of help push some standards? 'cause like generally if you have a thousand different choices and different people are not really adhering to standards when they build new tools and platforms, it really makes it a mess for the the consumer trying to get some value out of those tools because you have all these different things to manage up and down the stack.

So what are some of the, the, the key concerns that business leaders have when it comes to getting everyone on board with these AI goals and some, you know, some people being concerned that it's gonna take their jobs, et cetera? Yeah, so I mean the, the first thing is figuring out what is the actual approach you're gonna take technology wise? What is the stack you're gonna take?

How are you going to provide access to GPUs to do the processing to your employees when you don't have them? Or you have to manage access to them. Um, the other thing is to think about with ai and there's a lot of sort of doism around it.

Yeah. Like, oh, it's coming to take my job. But you can kind of see this over history with different new technologies too, sort of human nature is to assume the worst of a technology.

But a lot of the things like LLMs are something, large language models are something that can kind of be a little like, whoa, it's a little creepy. It just wrote a whole paragraph, is it gonna take my job? But you know, in the end it's, it's matrix algebra, um, it's parroting, you know, it's like garbage in, garbage out.

It's parroting what was pre-trained and fine tuned into it. So I guess when you're, when you're working with LLMs, it's just a technology tool and the stuff that they do could also be done programmatically in a sense. So you could even say in the early days, software is coming to take my job.

Right. Um, and I think the better way of framing the usage of AI technology is to show how it can reduce burden or repetitive manual tasks. Like I'll give one example of what a team at Red Hat that I'm working with is looking to do with it.

Um, we use Slack like a lot of technology companies for communication between different development teams. And you know, sometimes that inter-team, uh, communication when you're working on one project and different teams are using it and they're asking questions, you tend to end up fielding the same questions over and over and over and over again. So this team is looking to use, uh, an LLM that is trained using the in instruct lab technique that Red Hat and IBM actually announced back in, um, in May at the Red Hat Summit that have that model tuned to what that project is about.

Right? So like any specific knowledge that they end up, you know, it would, ideally in the pre LM worlds you would have an FAQ and you'd have the questions and answers and point people at that. But that's a little like, sometimes you're like shoving the person aside like go read the manual.

So in this way you're sort of trading that knowledge into a model, connecting it to a chat bot in the Slack channel. Then when people come in the channel and they ask the same question for the 100th time, the chat bot can kind of, if it can recognize, oh I have the answer to that, it can intercede. And that sell saves the developers in the channel time that, you know, interruption context switching, they can continue doing their work and the chat bot can sort of filter out.

And that's work nobody wants to do. Like nobody wants to sit in a chat room all day and it's hard to make an argument to have somebody sit in a chat room all day for this specific case. So really it just saves people time.

And there's so many other examples, like you're in a meeting and you're taking notes, six pages of notes, you can't send that out to your leadership. They really just want a paragraph. At most, you can just use an LLM very easily to create a quick text summary and that's just work you don't have to do.

So Yeah, there's so many use cases for AI that are gonna make things more efficient, simplify things. But for the more advanced use cases with AI and the AI implementation, there does seem to be some skills gaps still there. And you've spoken to needing a, an entire cultural shift, um, where everybody can be an AI expert.

What do you mean by that and what does that uh, take? Well, the thing about everybody becoming an AI expert is just like anything else in technology, the AI is a tool. And the interesting bit about technology is where it interfaces with humans, where it understands their workflows for what they're trying to do to reduce repetitive tasks, tasks that aren't interesting.

So we can kind of maximize the brain power of the folks in your organization. So I don't think you necessarily need to be a deep AI expert up and down the AI stack to be able to make use of ai to be able to help power your, your company with ai. What the folks, no matter what department, what team they're on in a company can contribute with AI is their knowledge of the company's specific problems, processes, workflows, which they know a model does not know that, right?

I mean for some things it might, but for most it doesn't. And what Red Hat's approach to working with AI right now with AI models is with both in the in instruct lab project and an upcoming product we announced back in May called re ai, red Hat Enterprise Linux AI is we envision that number one, this knowledge of all of the employees across your organization is specific to your organization. It may be like a form of secret sauce almost like this is how your business gets things done.

This is the unique way that your business approaches things. For example, like you might want that tech summary in the brand voice of your business. That's not something you could buy off the shelf.

Or you may want an LLM to process, I don't know, incoming customer tickets using the historical data set of previous support tickets. And those are things you don't want to be made public. Those, those are sort of intellectual property of your company, it's part of the value you provide to your customers.

So our approach is you take small large language models and you fine tune them using your company's custom proprietary data inside the walls of your company. The models are small enough that you don't need lots of exotic hardware to be AP able to actually run them. Um, and because they're small, they're easy to run inference on, not very expensive because a lot of AI solutions are quite expensive, even on the inference side.

And you have sort of your own custom LLMs built from the knowledge contributed from all the employees across your organization, their unique insight into your business processes, workflows, ways of doing things. And that because it, you have it sort of encoded into the LLM, you can use it to again, reduce those repetitive tasks, provide a lot of value so that you can make the most of the workforce you already have. Yes.

And definitely would improve communication across all departments. Yeah, definitely. So AI is advancing very rapidly.

There are so many use cases. What do you see in the future of ai say six months to a year from now? Six months might be a little early, but a year from now I would like to start seeing AI become boring.

Um, I would like it to just be this, the standard way of doing things. Like, like I mentioned when I was in high school, I got started with Linux. You know, we didn't have an always on internet connection.

I was using a modem on one single phone line coming into my family's house. Now, at least depending on where you live in the world and always on internet connection, broadband speeds is something that a lot of us can take advantage of and just sort of take for granted, right? So that in that 20 year span, well a little bit more than 20 years, I'm not that young, but in that time span, you know, that kind of standard has changed.

And I'd like to see AI become just sort of a standard practice or a standard way of doing things. Not really anything special, not really anything exciting. Um, it has been coming out in different apps.

Like right now we're using Zoom and I see there's like an AI companion tool in Zoom. Very cool. A lot of applications and tools are starting to sort of roll this stuff out maybe a year from now that'll be somewhat boring.

However, the thing that I think that with projects like in Struc lab where we have an open source focus of the technology and we're trying to democratize the production of AI models and that that includes within a business, democratizing it. So it doesn't matter. You're in the HR department, you're in finance, you're in support, you can help contribute to a company's models and get your unique position within the company represented in that model.

I think I would really love to see that rolling out as well in the next year. That would be highly impactful. Well, if there was one key takeaway you could leave our audience today with, what would that be?

Uh, a key takeaway would be the way that I think is inevitable with any technology. And right now AI is the, one of the moment is it starts out with a small few being able to create it and as time goes on, as that technology becomes democratized, it impacts a lot of people who don't necessarily have the power to impact it or affect it in the same way. We need to make sure that we expand the scope of control over that technology.

So with ai, with open source tooling, with open source license models, with with small size models that anybody can run on modest laptop hardware, we're, we're in the same sense democratizing that technology in the same way the open source did for traditional software. Alright, well thank you so much for coming on our show and sharing your insights with us today. Pleasure.

It was a great chatting with you. Yes. And thank you to our audience and stay tuned because we have more.