Purnima Padmanabhan on Real-Time Data Strategies for Enterprise AI Applications

Hello and welcome to the latest edition of the Techstrong AI Leadership Insight series. I'm your host, Mike Bezu. Today we're with Pima Patman Aman, who is the general manager for the tan Zu division of Broadcom.

And we're gonna have a little chat about how to get all that enterprise data into our AI applications. 'cause well, it's probably harder than everybody would like. Pima, welcome to the show.

Well, thank you Mike. Good to chat with you again. And I, of course, we have known each other for a long time and uh, yes, very excited about the topic that you've raised.

It's much harder than what people think trying to get data to AI so that it can actually give you the right inferences. And what exactly is involved in that? I mean, we see a lot of data movement and there's data lakes, data engineers, and by the time you get them connected to application developers and the data science team pretty much takes a village to get anything going these days.

Nevermind updating and maintaining it, but what's your assessment of what's going on here and can this get simpler? 'cause I feel like when I talk to enterprises, they're kind of lucky if they can get two or three applications together a year, It can definitely get simpler. So that is the good part.

And, uh, in fact, that is the story. We have a great solution for that space, which we launched called Tons of Data Intelligence. But before I get into all of that, right, the problem statement is everybody's very excited about AI and you start working on projects.

Some of the easier projects to get started are coding projects that probably have require less context. So you can start greenfield coding projects pretty easily with ai, but the minutes, you start thinking about use cases that touch your core enterprise apps, right? I have a banking system or I have an insurance policy management system.

Now you, if you want to truly do something more than just some basic summarization and you wanna truly add intelligence to your app, you have to give the right sets of data to that, uh, to the AI models, right? And to the AI subsystem. And that is where you unlock the power.

Otherwise you are just, you know, maybe doing the same thing slightly better with an NLP interface. But that's not what it is. If you want to bring the true transformative power of ai, you have to unlock the power of the enterprise data.

And what happens, let us just take something as simple as, I wanna serve better policies for my customers based on their past history, based on what they have uploaded, based on their needs, let's say insurance policy. Now, if, uh, if you look at it, the data, the images that they are, uh, uploading about their cars, about their incidents, et cetera, might be sitting in an image database somewhere. And it may not be even a database, it might be a file system, multiple file systems.

The actual policy information might be in a different enterprise subsystem. The, um, user's information might be in a third subsystem. Now, if you want AI to make inferences, you have to give AI context of all these three things, otherwise you're not gonna get any meaningful, uh, ideas or meaningful intelligence.

And so the first problem that customers start facing when they start trying to think about use cases of injecting real AI into their applications is how do I give AI access to all my data? My data is distributed, it's ungoverned, it is multimodal, it has different frequencies, it has different form factors, and I need to figure out an easy way to do that. So that tends to be the first problem.

And that's where they start hiring a lot of data flow engineers, data analysis engineers, data cleanup people. And that is also insufficient exactly as you said, because even if I do some of it, how does, do I then connect it easily to the app developer who ultimately has to use the data, take the data in the right context, provide it to the LLM, get the right answer and insert it back into the application, right? So that is where I see bulk of the problems happening.

Mm-hmm. Is that meaning that, and I don't know if you've heard this term yet, but we hear folks talking about the phrase now, context engineering, which is kind of a higher elevation above data engineering, where getting the right data in the right place at the right time is the art, because that's where the context is. And that's how I add value to my AI application.

Otherwise it's just kind of like I'm slamming a bunch of data at it and hoping for the best. I absolutely hear that. And actually I hear something more.

So one is of course, we talked about how can you put the data at the fingertips of the developers so that they can provide the right context to the model for influencing. But I also tied to context engineering and context, providing this data context. I hear of this term called context rot.

And the idea is it is not enough that you just throw all the data at the AI subsystem. You have to give the right context. If you give too much context or if you are giving context of windows that are very, very large, you can actually get to bad influencing.

So how do you provide, so, so the challenge starts becoming, if I am a developer building an an, an app application, and I'm trying to endow my application with AI intelligence, then as a developer, I need to understand what is the context I need to fetch? What is the relevant context? How do I make it tight enough so that I guide the AI to make the right decisions?

You know, this idea that AI throw, throw everything at AI and it'll give you the right answer, doesn't happen. The ai AI I believe has a 95% problem, which is yes, 95% of the times it may be right, but 5% of the times it's wrong. And that is where all kinds of guardrails become important.

The first part of guardrail is in the data side, make sure you give the right context, limited context so that it is making the right decisions. The second one is make sure that whatever it is producing has got the right governance and guardrails. Especially this becomes very interesting when you talk about code generation and so on, right?

You are trying to make sure that you're restricting the kinds of code it can generate. The kinds of patterns it generates is repeatable, right? And so often what we are seeing is customers combine data flow logic to create purely high quality data that can go into context, but also once the context get used, how do you guide the AI subsystem to the right answers?

Both of those are needed. Mm-hmm. Um, when you think that through for a minute, um, I'm sure you've heard this, the MIT put out a report talking about how 90% of these AI projects are failing.

And I think that most of the reasons are for what you just described, is that we're expecting some sort of, um, outcome that will be predictable, but the LLM, you know, might do the right time it runs, yeah, might do the right thing nine outta 10 times, but that one 10th of a time is probably gonna involve your best customer to being that Murphy's Law. So what do we gotta do to kind of simplify this enough so that we can figure out what the right context is? Because a lot of the folks building the apps don't always have the context themselves.

So where do I get the context from? So, well, I love this question, and by the way, this is a, a, a big platform question. So you need to think about not just data and how do you curate your data and how do you define a data intelligence story there, but it also ties very ties together to what is your application platform for building ai, um, infused apps or for building even for even running AI generated apps.

So let us start with the first part. So if indeed you have a problem where the 95% raw AI problem, as I said you have, that you may want to have, be able to have multiple runs of a given prompt or a given solution to see what kind of results it gets produced, right? And, um, so right, right there, you need a platform, right?

And that is where some of the enhanced use cases for Tan Zu platform around AI have been coming in. Imagine I can have, I'm, I'm asking, I I define my application. First of all, the entire ecosystem of the application can easily run on a platform.

So I only define the business logic, how it connects to the models, how it connects to the vector stores, how it connects to the data sets is all automatically managed by the platform. But rather than thinking of single run of applications that we would otherwise think of in a normal environment, you may want sandboxed environments that are running multiple simultaneous threads of a given prompt or a slightly very slight variations of the prompts. And that is what helps with the 95% problem, which is you can now start seeing how these results come out up, start applying reinforcement learning in it and guide, start guiding the AI to a better guardrail, right?

So that is the first part of the puzzle, which is you need an application pass so that you can truly build and iterate on AI based apps. The second part is you have to, uh, before you even start talking about context and all, you wanna start being able to interact with your data in a more simple way as a developer, right? As a data user.

So you want an interface that takes your multimodal data, data that might be sitting in data lakes, that might be sitting in file systems that might be sitting in, uh, structured databases in federated stores like S3 buckets, and be able to look at it in a more unified way and talk to it in a more unified way. And that's what we are doing with tons of data intelligence, right? Bringing your multiple multimodal data with variety volume and velocity, different types of data together.

But with this patented technology called PXF, we allow you to do a single query across that and start talking to that data, being able to have a metadata catalog on top of the data. So you can say, what is the business value? Where is the user?

What is the RAC on that data, et cetera. Once you do that, then now you can start saying, okay, I have a given application. I'm talking to my data to understand what kinds of data I have, which then allows me to create a data flow which is baked into our platform today to then feed the context in, right?

So that is one part of the puzzle of the data. The other thing that I say is, once you've identified what data you need for your AI application, imagine as you collect the data itself, right? A lot of this is real time data.

So as you collect the data and the data is streaming in into your data int intelligence solution, you call an embeddings model and vectorize it. So you are storing vectorized data. So when it comes to actually applying an influencing, uh, stage, all you're doing is just a similarity search, right?

You're not trying to again go and convert data and then try to do a match of a vector and so on. So there are many techniques that customers can do if they have a good platform. So what you need is to first get some sanity across your multiple multimodal data, be able to have a query layer, be able to have an interaction layer using MCP tools or something like that, and then be able to take that data and take the relevant context streaming context of that data and cache it for an application to be able to use it.

Are we kind of melding together two worlds? And I'm asking the question because you mentioned PAs and I know you guys were talking about earlier how um, you kind of move back to the cloud foundry platform, and yet we're also talking about adding data lakes and real time streaming. So is this the definition of a new platform that's kind of combining some of the best of the old world with some of the new core requirements?

And we're kind of melding this and going Forward, I I think, um, at least our thesis is that AI has created a new jump ball and for the first time it is forcing what might have been perceived as disparate worlds of app and data to come together. So I definitely agree with you there, but what it has also, as you look at it a little bit more carefully and say, okay, what is these piece pieces of building blocks of ai? What, as we have been building using our own products to build AI software and AI infuse software, um, it has become clear that an AI application is no different from a traditional or a modern application.

It's microservices based agents you can think of as microservices. It needs to connect to an ecosystem of things like models and tools and vectors. If you think about what is MCP, it's just a nice formulation of an API that I can easily connect to, to do an action, right?

So What was the core principles? Don't, don't throw the the and say, okay, I'm going to now create a new AI current enter and allow for it to connecting to a model whether I'm connecting to a vector databases. These are things that PAs solutions do very well.

And that is why I said PAs is important. Similarly, while we talk context and embedding and vector stores, at a basic level, you need to be able to connect to your multimodal data. You need to be able to ask questions, you need to be able to put the, pull, the relevant set of data, cache it and make it available to the agent or the app.

And that, if you think about it, is a set of technologies that we know very well, but how do you connect these dots is where the problem happens. And that is really what we have set out to do with our tansu platform and tansu data intelligence solutions. Do you think the way it organizations are structured will need to be reorganized in this new age?

Because historically we had all these different tools and platforms and each one required a specialist, and maybe there's another way to think about all this stuff going forward because the cost of the platform requires a specialist and then that increase the total cost of it to a point where, well, it's not maybe feasible. Could be. And that is where I guess, um, that that is something that is continuously the industry keeps moving towards that, right?

So if you think about it, just on a different business of on or in, in Broadcom site, we do a private cloud. And in the past private cloud was 10 different domains. There'd, there'd be somebody responsible for compute, then somebody else for virtualization, somebody else for storage, virtualization, somebody else for networking.

And now what has happened is more and more organizations are organizing themselves into a private cloud group or into a cloud group. And they have all the disciplines within the, within the, or within the same team, but they are trying to build a, um, API surface of IAS or CAAs out to the cus consumers. And so that convergence has happened and of course as, as VMware we have seen that convergence happen right?

Within organizations. Similarly, um, platform teams have come together in most organizations and platform teams have done that bridging between security compliance and app requirements and the platform and the infrastructure requirements saying, Hey, I have got a PAs solution that a single team brings together. But it has different disciplines.

And what we are seeing is a transformation of the data discipline definitely happening. Uh, we feel, even when I talk to some customers, there are some customers who have thought of as a DA really data warehouse was in the past thought of as this static solution that sat in a corner that produced a report once in a month or once a week. And that's all right.

But now you're looking back at that same solution and saying, okay, now I I make it into a data lake. Why should it be a static solution? It's continuously getting streaming data in.

Imagine if I can take, if I can keep, first of all vectorize the data on fly and keep it to use for, for inferencing, if I can get the right streaming out of data so that relevant data can be cached and made available to applications, the same technologies and same subsystems start becoming more powerful and more relevant. I do think you're a hundred percent right. It does require that transformation happen in these data teams.

Hmm. But AI has been the jump ball to open that, right? We, we wouldn't have otherwise had a transformation in the industry, but AI is forcing CIOs and forcing our enterprise organizations to go back and say, okay, I need to connect to the data, solve the problem for me.

Mm-hmm Isn't this kind of bringing us back to the future in another way where I kind of think it was always about the data in the first place, but somehow or other, and we managed to get distracted over the years, but you know, are we coming full circle? We are coming full circle in so many different ways, right? Uh, I uh, think as I am, especially when I think about my tan zu business, right?

We talked a lot about lower level infrastructure and containers at one point, but now I'm really coming back to my core business which came from the pivotal heritage, the Cloud Foundry heritage, which is the PA business, right? Because as AI abstracts more and more things out on the coding side, you wanna a platform to abstract everything on the infrastructure side. And so definitely I'm seeing a back to the future.

In fact, that's exact word I use with my team here. It's a back to the future on PAs, which is developers are building code, whether they're building it themselves or AI generated, they just simply want to push it to production. And that is the true a sense of what we did with Pivotal and Cloud Foundry long back.

And I see a resurgence of that similarly, lot of times the data was very important in all these decision making, but it was hard to access. It was hard to access near real time. And now everybody's saying, why am I not getting that data near real time?

We have proven this in other industries we have and, and of course, uh, with our um, incredibly powerful foundational models that are out there, I can ask any questions about the general world. Why can't I do the same thing with my enterprise world? And that has definitely come back a full circle.

Mm-hmm. So what is your best advice to the IT leaders out there? Because I think that they look at all of this and it's a little daunting and it's not just the fact that we want AI and that'll be great, but the, the political capital required to kind of drive this is significant and you have to kind of bring these teams together in ways that is gonna be challenging.

'cause they all have their own cultures and their own silos. So how do I kind of get this ball rolling? First of all, don't make it very complex.

And, and, and here's how I would break it down. Think of AI as just yet another AI and ai um, enabled app or AI infused app as just yet another application in your portfolio. So then you start saying, okay, you already have an existing platform.

How can you make that, how do you look for platforms that are available that allow you to build AI apps just like you build any other modern microservices based apps? So don't make it too complex. The second one is really start with your use cases because there is so much, uh, hype at the same time there is so much further about ai, right?

Because like you said, things are not getting to production. People are not seeing ROI. And I would say in my experience, the places where I've seen customers get ROI are the following, right?

Think about the use start use case first. Don't start technology first. Don't start saying, oh, I have to get in media GPUs, I have to put the model up or I have to get the platform.

Don't start with any of that. Start with what use cases you want. And there are a few use cases we are seeing are high impact.

Definitely on the development side, the coding assistant use case is very powerful. So in that particular case, you'll say, okay, great. I'm trying to either refactor code, I'm trying to um, uh, modernize code, or I'm trying to write some greenfield apps.

How can I get velocity? Those are great use cases except you now need to make sure now when the code gets generated, can I get it out to production using a PAs? Think about that.

The second type of use cases that we are seeing is where you have an existing enterprise application often written in Java Spring, and you want to be able to add simple things like, Hey, I want my customer now to be able to have a natural language interface to be able to talk to the subsystem. Okay, that's a good use case. That's a good ROI, maybe it'll reduce the amount of support calls you get.

How would you infuse the existing application? And that's where I would say definitely consider looking at things like the spring framework and what we have done with spring ai because we are bringing AI to the enterprise Java apps, right? That have already been there.

Um, you want to infuse AI into that, it's much easier. Standardized APIs, standardized paradigms, you start doing that. The third use case would be, Hey, now I, I've already got an application and I want to offer more data to it, right?

And identify what is the subset of data you need where the, what is the location of the data that is there? And look for simple solutions that, uh, give more value of what you have already out there, right? Do you already have a warehousing solution?

Like maybe, uh, what we have from, uh, many of my customers have tan zu, uh, green plum, right? Green plum at its heart is warehousing. But by a adding the data lake capability, they're able to bring the unstructured data in.

And by adding the PXF querying capability, which is all just extensions to what they have, they can now offer that data to the applications. So it's as simple as anything else. Start with the problem.

Start with the use cases. Start pick simple use cases that can drive ROI and then start pushing on the systems that you have today for your modern apps. They have to support AI-based apps.

All right folks, you heard it here. Hey, you know, sometimes you don't need to do the great rocket science thing. You just need to do what everybody else is doing to get to get your feet wet and then figure out what the awesome thing is you're gonna do next.

Proma, thanks for being on the show. Thank you, Mike. All right.

ai Leadership Insights series. com, and we invite you to check out all those episodes as well. And until then, we'll see you next time.