Open Context Is the New AI Moat

AI Leadership Insight Series. I'm your host, Mike Bizos. Today we're with Suresh Srinivas, who's the CEO of Collate, and we're having a chat about the need for a more open approach to the context that we show AI agents, because, well, without context, they're just going to get in trouble.

Suresh, welcome to the show. Hi, Mike. Nice to be here.

So explain what's going on here and how do we get to some sort of open context, because I think everybody and his brother who has some sort of data management platform is finding a way to expose it to AI agents, but that just seems to create a whole lot of different metadatas and things I got to master. So is there a more open approach to all this that maybe will be simpler? Yeah, even before coming to open context, let's look at what is going on.

Context has become a big buzzword now, right? AI is changing everything in a big way, right? Coding, data, how we do so many of the things.

And especially with data, AI, even with the latest models, the accuracy is pretty poor. Our own benchmark AI accuracy is around 60% for the natural language questions you ask and the answers you get. And the reason for that is the context gap.

And it's now pretty well established that AI needs to be grounded in context to improve its accuracy and improve its efficiency. And that's how context has become the buzzword. And what we believe is better models that are coming are not improving the accuracy of AI, right?

It's the context, the missing gap. And given that context is how you teach AI about your data, about your business, and about what your business has learned, it's a moat that you're creating as a business. As we see, models are going to be swappable.

There are newer models that are coming, there are less expensive models, open weight models. Models are not where you create value, right? The moat is created by the context.

And given context is very important moat, this context needs to be open. It cannot be vendor locked in, in proprietary format. A company should be able to own its own context.

And then with that context, it can enable, with different AI models, all the AI for data things that are, transformation that is going on. So there's no shortage of these data lakes that are out there, and people theoretically have aggregated their data, or they might not have, but they all seem to be proprietary platforms. So how do I approach this in a way that is a little more open and I'm not trying to figure out how to do metadata and labels for everything that's out there, because that's a lot of work.

Yeah. So, as we go into this, let's define what context means, because there's a lot of confusion, and this has become a big buzzword. Now, how we define context as is metadata, which tells you what data exists.

Meaning, what are your business concepts and entities that you have within your business? And then finally, memory. What has organization learned about how you do things with your data?

The experience that an organization has acquired. So for us, context means metadata, meaning, and memory. Now, you were asking about what does open mean here, right?

Open standards and formats have been becoming more and more important. You talked about data lake and data warehouses and all of that. There, open formats started coming in long back, and it's being adopted from proprietary formats.

Now, in our case, the layer that we have is metadata that sits on top of these data infrastructure and platforms and tools. There we have JSON schemas that are standardized and defined that describes your metadata. And when it comes to meaning, there's a lot of work that has happened, even before how critical meaning has become now in the world of AI.

And it started with semantic web, and there's a lot of work that went on in modeling the world and the meaning through ontologies. And so there are standard technologies that are available, like OWL, RDF. And many domains have already been modeled, such as FIBO for finance and HL7 for healthcare, and there are many vocabularies that are already available.

So what we do is we model the meaning through ontologies, through RDF and OWL, using the standard vocabularies that are already available. And in terms of meaning, again, meaning is something that there's work that is beginning to happen. We again do it through RDF and OWL, and connect all of these different concepts in a knowledge graph and make it accessible to AI.

That's what we do at Collate and open the data. So, does each organization going to need its own ontology for using AI, or will there be industry specific ontologies, or how do these things come about? Yeah.

There are a couple of ways this will work. So some of the organizations have already defined their ontologies. They should be able to reuse the ontologies that they have defined themselves.

Now, the second part is there are domain-specific ontologies that are already available. You should be able to build on top of that. You should be able to import, for example, FIBO if you're in finance.

And then you should be able to customize it, specialize it for your organization. That's how it is going to be. And we have agents that can help you in building ontologies by looking at the data that you have and the unstructured documents and stuff like that, where you talk about your own businesses and policies and things like that.

But this requires human beings to be involved. In the end, different people within the organization have to agree upon this meaning. That consensus needs to be established.

What we believe is in the current world, AI will automate a lot of these things, but there will still be human in the loop. Mm-hmm. As we go forward with all this stuff, what level of accuracy can we get to if we get the context right?

Yeah. So, our own benchmark, the latest models stagnate at 60% accuracy. That means 40% of the answers are wrong.

No business can depend on those kind of answers. So when we ground AI with the context, context of metadata, meaning, and memory, we have come up to 96%. That's the initial work.

Now, again, 96%, where 4% of the answers are wrong, is still not good enough. This requires additional work. Additional work of grounding and specifically the concepts are AI has generic concept.

Grounding it into your business concept and continuously learning and improving is going to be important to even further get above 96%. There also seems to be a correlation between context and cost, and from what I can tell, the AI agents in the absence of context will go out and try to get that context themselves, and then they'll consume more tokens doing that, and they may be locked up in some infinite loop as a result. So is the ROI on the context pretty critical in terms of lowering the total cost of AI?

Absolutely. So, in our benchmark, and this is coming out even with, I think OpenAI uses open metadata. Even in their own, they actually build a context layer.

And it's pretty evident that if you don't provide the right context, not only AI agents will try to build their own context right At the expense of spending a lot of tokens on it. And we have seen 75% reduction in the token cost when you provide them curated context, when they're trying to answer the question. The second thing that also happens is, especially in data, if you don't provide this context, AI agents will do a lot of query on your data warehouses or your metadata systems, and that increases the amount of compute that is used for building this context.

So we have seen 87% less compute when you provide the right context. So in our benchmark, there were 202 queries that AI agent was making when we provided the right context, the number of queries dropped down to 27. So you'll get better accuracy as we already talked about with the context, 75% reduction in the token spend, and then 87% less compute.

So that's the efficiency gain, the accuracy gain with context. Right. I think most organizations leapt into AI with two feet, but they're now discovering all these issues, and so most of them I talk to are kind of revamping their AI strategy.

So how long will it take for us to get aligned around the context and everything that we need to really drive AI into business processes that, for the most part, they need to be done the same way every time, and AI is probabilistic and rarely does anything the same way twice? Yeah, I think the first impression you get out of AI is like magic, right? You're amazed by what it can do.

But when you start productionizing it, that's when you realize that it's not a magic wand. You need to actually have AI-ready foundation before you can productionize some of these agents within your enterprise. I think many organizations are coming to a realization, and now it's pretty well established and understood that context is important.

You need to have AI-ready foundation. How to go about it is where Collate is much farther ahead. We're more comprehensive context and how context is exposed through knowledge graph, where AI can actually infer the meaning, learn about other concepts, starting from one concept.

So that's the realization that is coming in. So I think it's a journey. So you go through the magical experience of AI, and then you try to productionize it, learn from it, and then find the missing gaps, and then try to have AI-ready foundation.

That's the journey many organizations are in. One of the other benefits I think of this approach to context is that a lot of organizations, there is tribal memory. Yeah.

And it exists in the humans, AKA wetware. And every time those humans leave the company, they take that tribal knowledge with them. So are we in the process of maybe capturing a lot of that tribal knowledge in a way that makes it reusable and things that we can repurpose across a lot of different applications?

Absolutely. So when we started OpenMetadata, we were actually building context for people. And so that's why we are called OpenMetadata.

That's why we started our journey with metadata, which was to help people understand their data, what it means, what is its quality, is it ready for use. And then once AI came in, we found a big gap. When people got to know about their data, there were domain experts who understood the business.

It's all tribal knowledge. It's knowledge in head. Now you need to externalize it for AI because AI doesn't stop to ask questions.

You need to now teach AI what people had in their head, the domain expertise, the learnings that they've had, how to do the things the right way, and things like that. All of these things need to be captured so that AI agents can benefit from this human knowledge. It's knowledge organization around your enterprise.

And that's where meaning and memory in addition to metadata became very important. And you're absolutely right. Two things need to happen.

One, what people know, AI must know. What people have learned, AI must have access to that. And then the other thing is what AI now knows, has learned, people should actually be able to know and have consensus with AI.

There should be common understanding between AI agents and human beings because in the end, human beings in the loop has to make sure that the work that is done is right. So it's going to be very important to capture this tribal knowledge in a standard way, using open standards in a portable manner that both human beings and AI can consume. If I have a context graph, to what degree do I still need to go in and clean up my data?

And I'm asking this question because just about every enterprise I know, well, they wouldn't get a good housekeeping seal of approval for the way they've been managing data for the last two decades. So does a graph figure out what data matters and therefore maybe I can get away with maybe not doing as much grunt work? Yeah.

So there's an 80/20 rule. Maybe in case of data it is 95/5 rule. 5% of data is what you use for your critical enterprise needs.

95% of the work, especially in a world where self-service data creation, where a lot of people are creating a lot of data is not going to be production ready. It's just done for individual projects and things like that. So what we do is we have an AI agent that can actually figure out what is your important data.

That way you focus your energies on making the data high quality, well documented, through ontologies, you map the meaning and things like that. You don't need to do it for all your data. You focus on your most important data that has the biggest blast radius for your enterprise application.

And so we do have an agent that we call it as a tiering agent. Tier one is your important data. Lower tiers are not so important.

You focus on making your tier one data trusted and governed. Now, again, when we are talking about metadata, meaning, and memory, still the data that is used by your AI has to be high quality and trusted. If it is not high quality, even if AI gets everything right, the data is wrong and the answer is going to be wrong.

And so we also have AI agents actually that improve your data quality, suggest data quality tests, looks at observability signals and failures that are happening in your data pipelines and things like that, and then can tell you if the data is high quality, is it ready for AI to be used? Then the meaning, the memory, all of that kicks in when AI starts using the data. Are people underestimating how much data we may need to manage and provide context to because we are deploying all these AI agents and they too generate data alongside humans.

So how much data do you think we're ultimately talking about? When we are talking about AI agents, we are looking at a bunch of stuff. AI agents do certain things.

You need audit trails. And then through human interaction, what we call as memory is when people say this is not the right answer, you've used the wrong data, things like that. Corrections are continuously made, which we call as memory that is captured at an organization level, becomes reusable across different agents.

But still, compared to the amount of data people are collecting, this will be a fraction of this is sort of like metadata in the context. But over a period of time, it is going to be accumulating. So you need some concept of aging and curation because otherwise if you have too many memories and too many corrections and things like that, over a period of time some of them become irrelevant.

You need to be able to age it and get rid of it. You also would need human curation. So what I feel is this data will not compare to the amount of data you have in your data warehouse, but can grow significantly and can cause a lot of confusion.

There will be inconsistencies, disagreements between what you have learned over a period of time. There needs to be curation, automated curation, human involved curation, both will be needed. So ultimately at the end of the day, are we at some point where we're looking at some sort of renaissance in terms of data management and context and all these things that we're talking about here?

And if we get to this renaissance, will we start to deploy more AI applications in production next year or the year after? Yeah. I think if you look at it, even without AI, we have been going through this data quality and give context for people.

But it becomes more important in the world of AI. The scale at which AI will be using data compared to human beings as you unleash agents on them is going to be 100 folds. Which means getting your house in order becomes more important than ever.

And so this becomes a priority for an organization. And there's going to be a lot of effort. There will be humans involved, but there will be AI agents themselves that will be automating a lot of these things.

We have what we call as context agents that can continuously curate your context. It's humanly impossible to do this at the scale where AI is consuming the data. So I believe a lot of work will be going on.

And then just like any technology, there are early adopters. We have some of our customers like Scout24. So they're doing fantastic work with AI agents with the context provided by Collate.

And there are people who are going to be slowly adopting and then coming on board. So there are early adopters, there are going to be people who will adopt it next year and year after that. But I think what I feel is people who start with few use cases and then slowly expand, learn from those few use cases and expand are going to be the winners here.

Oh, cool. Folks, I think you heard it here. If you want to succeed with AI, maybe you should start with the context and work backwards from there.

Hey, Suresh, thanks for being on the show. Mike, it was a pleasure talking to you. All right.

And thank you all for watching the latest episode of the Techstrong AI Leadership Insight series. You can find this episode and others on our website. We invite you to check all those out.

Until then, we'll see you next time.