The AI Chasm: Bridging the gap from pilot to production with Hewlett Packard Enterprise

I'm Mark Ser, uh, principal Solutions architect for PCAI with HPE. Um, we talk about bridging that gap from pilot to production. This is our, our interesting conversation.

Part of the problem is every week there's a new foundation model, right? Every week there's a new service that comes out that's really compelling and it's hard to under. And I'm, I live the space like every day, right?

It's very hard to keep up with it, right? Just the news releases alone, time it takes to sit down and read through that is fairly enormous, right? So it's like people are trying to, like's a train, right?

And they're trying to like throw a baseball at a train as it's driving by and they're trying to hit this car, but they accidentally, you know, missed the whole thing entirely. You just can't ever catch it at the right point, it seems, right? And people get stuck in this place of like, they want to do something, but it's like every time they kind of like start to get it, then, oh, this new foundation model came out and it's, or this new capability, it's really interesting.

Maybe we should, you know, squirrel, let's look that way. So I think that's a part of it. The data piece is, I think, an overlooked component of all of this.

Every con, every company that I have conversations with is afraid that their data is going to be made vulnerable because of ai. Because somebody puts it into the context of some call that they're making with, you know, whether it's Catch GPT or Claude, right? Or they're gonna expose their code because it would take 'em, you know, two hours to write the code themselves.

Or they could throw it in the Claude and have it in 33 seconds, right? People are afraid that they're going to have data that gets exposed. And so a lot of times that's a big concern that keeps people from acting on it.

And then just the natural time, as I talked about when it, you know, you go about that. When we build this ourselves, I have to make, you know, choices. And it takes time to make all those choices.

'cause there's a lot of teams from different business units, right? The data governance people, my development team, my infrastructure team. I have to get all these people marching in the same direction, right?

All at once. So that's a huge issue that's keeping it. And this is Underwear, nos, yes.

My, my, my fans. Um, this is how I feel like, and I've used this before and I put it in here 'cause I think it's funny. Um, this is how I feel like AI is, it's like great purchase AI and then profit.

It's like, okay, but what's the actual work that we're gonna do? Underwear, knowns, underwear. Say no more.

Yeah, I thought, I thought it was, it got rave reviews at the last time I presented. So the conversation has to turn to something, right? And the hardware side alone is not enough because it completely forsakes the software side.

And having conversations around models and different models that you can use is, again, only a small part of it. 'cause it's not couched in the broader context of the business and what they're trying to achieve. And so, I referenced this a little bit earlier, right?

You really have to, this is the conversation that you need to start, well, I have to start with my customers, right? Where do we want to do this? How do we wanna start to, to stake a claim in ai, right?

We can do it ourselves, we can do it in the public cloud or we can do it with something like PCI, right? An engineered stack about appliance. And so these benefits, I won't drain the slide because I'm, I really want to get to the demo 'cause I think it's worth saying.

Um, and I built it. So this is, you know, you'll have the slides for reference. I encourage you to take a look and pepper me with questions later on.

Um, but yeah, what it gives you is now, and this is super important, all of those kind of foundational elements are things that we've put a lot of work into. And these are concepts that we started with six and a half years ago before LLMs and Chat GPT and all this was a thing. We were building things on doing more traditional AI and ML work, right?

So like fraud, anomaly detection type use cases, computer vision type use cases, natural language processing, right? Those types of use cases. And we built a platform to support those and it trans transitions really well to the LLMs, right?

So the backbones that we built to support this as well as what we're calling the data lakehouse gateway, and I have a couple of slides on that that I think are, are valid and need to be talked about because that's the component that gives you the data management, data security data governance piece that a lot of times is missing where it's now how do I have one global namespace where I can put all of my data assets, whether they're S3 buckets or NFS shares or whatever they are tables in sql, right? How do I have them all visible through one plane and manageable through one plane and one common access point for those data assets, right? So now I don't have to care about where a particular piece of data is, as long as I know the access point and the file path.

I'm good. You get into the gateway after the demo or next, I think I have two slides on it next. Yeah.

Great. What's the, uh, what's the circuit here? The three arrows.

I didn't make the circuit The shades of blue. That's the best answer I've heard all day. Can you talk a little bit about the composable and extensible for training purposes There?

Yep. And I'm gonna get Yeah, and I'll, I'll, I'll, I'll hit that pretty hard too in the demo. Um, so the idea behind, essentially the, the extensible piece of it is that we built a, a very opinionated stack, right?

Of tools that we think are, are the best of breed in the market in each of these different places, right? And it's a broad platform. IT scan, it spans from like data connectivity to, you know, like how do we run distributed federated queries, right?

Against our data to how do we do data analytics with things like spark or airflow for like workflow automation, scheduling and stuff. How do we do the data science life cycle of it? So like things schools like cube flow, ML flow, right?

Across all of those things, we made an opinion, we took our stance on what we think are good tools. We understand that we probably aren't gonna hit everybody's. And people will always have tools that they want to use, like third party ISVs or other open source tools.

Fantastic. If you have those, bring them, you can bring those to the platform. There's a button that I'll show you in the platform.

It says import framework, assuming some of you guys are probably familiar with the helm chart. So a helm chart's, big ugly collection of yaml. It tells you how to deploy something on Kubernetes.

Fantastic. There's three or four different places in a helm chart, all mostly under the easy a namespace that you need to go and edit to tell it how to interact with the platform. And you can go and deploy that on the platform, right?

And you can do whatever you want to the helm chart around resources and all that stuff. So that's how you can bring external tools onto the platform. The only caveat is that if there's an external tool that you bring onto the platform, you are now responsible for the lifecycle of that tool.

All of the tools that are on the platform, HPE provides support for all the ones that come outta the box. We support those, we validate that they work on the platform and we support them. We're your first call.

So that, that gets us to the, the, the label on top of that big fat comparison chart that you showed a moment ago, which was engineered system. So one reason people like to see boxes like you showed initially is that that looks like a product that you go buy install. It's not necessarily an appliance that you simply plug in, but nonetheless.

So tell us how customers buy this and use it. 'cause it sounds relatively complete and I think that's an important, uh, benefit to this as opposed to a set of ref RX with three months worth of configuration and like that sort of thing. That's its biggest advantage is that it's not a reference architecture, it is a co engineered and co-developed system that we built with Nvidia.

Their engineers sat down with our engineers and we hashed it out. So when people want to ask me slowly me, right? They say, well why did you guys put this RAM in it and why did you pick that hard drive and why did you pick like it was a co engineered solution with Nvidia, we were trying to keep costs down and have performance be as good as it possibly could, right?

Keeping in mind that cost. And so, But the answer to can I put my favorite RAM in is no, no. Great.

I like That. That's it. I've had to tell a customer very recently, like last week, I had to tell them no.

Um, because that's not the purpose of it. Could we do it? Sure.

Bring just metal silicon, right? You can put whatever you want in there, but that would take it out of, out of support essentially. Um, so God 90 minutes goes a lot faster than you think It does.

Um, so yeah, this is that, that daily lakehouse gateway that I kind of promised you, right? Yeah. So this is the idea that you can use and have secure access and use your favorite sets of tools that you want.

Whether it be things like, I don't know, so Spark, which everyone's familiar with, or Presto, right? Which is a federated SQL query engine, right? That we have on the platform that I can show you, right?

Or use all of your data science to access it, it accesses all of this data that's in these different places, right? So table data, all of your file shares, right? Cloud, uh, so like S3 buckets or S3 compliant things, right?

In all of those shares, how do I have them under a single namespace, right? And so it would kind of look a little bit like this is that you have your in, and I'll show you this when we get there. You have data sources up here that are available to the platform and there's a management plane behind it, right?

That you can access. And that management plane shows you all of the assets, allows you to add assets, allows you to apply policy towards assets about who can see po these assets or not. What can they do with them, right?

And then it serves it up to you through the platform so that you can directly take and use it. And so, you know, if I wanted to take one of these tables and use it in a Jupyter Notebook, I can do that very easily. I don't have to do anything else after I provision it.

What kind of notebook? Jupiter Notebook. A Jupyter Notebook.

I know you're familiar with Jupyter. Yes. Yeah.

Mm-hmm. Yeah. Good.

'cause I'm gonna show you one just a couple minutes here. Great. Something that I built.

You have them as a service. Oh no, this has gotta be something that I didn't, I didn't catch because It wasn't here. Keep on s**t.

So the other piece I think's really important to mention here is that we have what's called our Unleash AI program, which is all of our ISV partners, we validated them to run on our platform. And so you'll see familiar names in here, right? Things like Crew and able, right?

There's all of these different tools that people could, I, I get 'em all the time. People like, what about this tool? Can I run that on your platform?

We continue to grow and validate this ecosystem of people that can run on our platform. Especially as we're getting into this agentic AI kind of world as it is now. There's tools that have done some really amazing work inside of your like ABLE is a good example.

And so we're bringing on those ISVs into our ecosystem and validating that you can run these on the platform and developing expertise around them ourselves. And then lastly, blueprints. Very important.

If you guys are familiar with Nvidia blueprints, with our partnership with Nvidia, right? Part of this is that we, we validate that blueprints that they have we'll run on our platform, right? And so the validation process is I think a little bit too onerous because at the end of the day, this is defined with a helm chart, right?

So Nvidia has for example, this uh, enterprise rag pipeline. They have one that's called multimodal enterprise rag pipeline. And so multimodal meaning that like it can do images or text graphs, right?

The thing is, when Nvidia built this, this helm chart to deploy that application, they put every single embedding model. 'cause there's different embedding models for each different modality. They put each one on its own dedicated GPU, right?

When you deploy it in a Kubernetes environment, do with docker, you only need like one GP, you do it in Kubernetes, they want you to have 10, right? And so you don't have to do that, right? You can engineer your way out.

So I'm negotiating with them to get my own cut of that. com, you can find blueprints and things that you think are interesting and compelling. And I talk to my customers about this all the time and say, great, bring that to the platform and bring that to the platform.

And you can continue to customize and build and develop applications using that blueprint. Those blueprints are kind of like those end-to-end customizable like references for what particular use cases that they're finding a lot of traction with in the market. Now we're, I'm gonna go through this very fast and then I wanna get to my demo.

I'm gonna hope to be there in six minutes. So again, we're getting it. You guys are gonna have to edit for me.

Mark Ser, PCI solutions architect with HPE. The cloud makes it easy, right? Sure.

Kind of. You don't care about money. Money, right?

But that's the truth because at the end of the day, every single one of these points is something that's true and legitimate, right? People overprovision you resources and they don't pay attention to it. They'll put long running jobs up with no check pointing, right?

And so if that job fails 10 hour job, it fails in the ninth hour, you lost all that work and you gotta go rerun it again, right? You've got things like the data piece is crazy because no one it seems in the cloud is a good corporate citizen. If anyone's ever noticed that people will always do the worst possible thing that they can do.

Hmm. So I've seen people serving static webpages from Dynamo db, like very expensive storage for doing nothing, right? And people like, well why did you do that?

I don't know. It look cool, right? That doesn't make a lot of sense.

And so there's not a lot of great corporate citizenship and that extends into people spin up instances and guess what? They never kill them, right? So those are big problems because when you start realizing that like I have a oil and gas company that I met with, you would know them.

Um, and they said our operational costs are 10 times what we thought they were gonna be. 2 million in the cloud. Instead it's costing us about $12 million a year.

They also said the same breath that we don't care 'cause it worked. I thought that was interesting. But not every company is making that type of money where they can just not care about 10 x cost over what they thought it was gonna be.

Um, and so these are very important conversations to talk about and to have and uh, this is the only one of these, I promise, right? But the IDC report about HP's position, right? In this private cloud market, um, is very strong, right?

For our private cloud, these manufactured AI systems ultimately, and I have a lot of slides that, um, they're draft slides, so I couldn't bring them up and show you. I tried to distill into something that I can talk to you about. Um, the other ones I'm sure I'll be able to talk about soon as somebody says yes.

Um, there's cost savings to be had most of the cost savings. It gets better with the larger systems over what you would have. But you can expect to see somewhere between like a 30 and 60% reduction averaging around 45 to per 45 ish percent in cost savings over doing this in the cloud.

3 70 billion model full bore for X amount of time, right? These are some of the assumptions that we can make. But when you go apples to apples for what do it cost to run these models against this hardware in the cloud versus doing it with us, it always works out in our favor.

It works out most in our favor when it's something that you manage inside of your own data center, if you want to do it with like co-location, right? Because you can, we have colocation partners, you can do it there. It adds about a 10% overhead, but there's still lots of cost savings to be had there.

And some of the places of savings that like you might not see, especially if you're doing this against like let's talk, you know, you're gonna build it and DIY and do it yourself. You have to have the teams and the resources to be able to do all of this stuff, right? So from the infrastructure layer all the way to people who are gonna install, configure all the software, you gotta have teams to support each piece of that stack.

We can get you away from all of that and get you away from the development time that it takes to do all of that. These are, yeah, these are my charts. One of the charts and I have 32 minutes to probably show you guys 32 minutes worth of a very compelling demo that I built and I'm proud of it.

So in closing, for this component, this piece of it, uh, fast time to value complete control over security data sovereignty. We haven't touched on that a whole lot, but I am less than a hundred percent sold that all of these companies that are very heavily incented to continue to grow and learn with their models, I'm less than inclined to take their word that my data is not gonna end up somewhere downstream in a, you know, following training cycle. Mm-hmm.

I know they come out with those agreements that say they're not gonna use my data. There's, uh, I think it was today actually, I saw, I read there's a, uh, there's an inquiry being launched into some of these models and the trading behind them. So it very much better for me to just have it all within my four walls and I know it's gonna stay there.

It's on this box that I can physically look, look at, see, or it's in a colo, um, predictable cost. This can be bought, interestingly, it can be bought either CapEx or opex, right? So you can do this as a subscription where you buy and then you have flexibility for capacity with GreenLake, right?

So with our GreenLake construct, if you're familiar with it, is essentially you can have re like a, like reserve capacity, but then you can have extra space that you can bump into that you don't pay for until you use it. So those GPU CPUs that aren't turned on until you need them, or how, where do you get that capacity from? Yeah.

Like we would put extra capacity on the floor for you.