The Rise of Context Engines – Why Data Management is the New AI Frontier

Hello and welcome to the latest edition of the Techstrong AI Leadership Insights series. I'm your host, Mike Fazar. Today we're with Simba Kada, who is context engine lead for Redis, and we're having a little chat about, well, just what are context engines and how they're gonna rise in the age of ai.

Simba, welcome to show. Thanks for having me, Mike. I think what's becoming apparent to everybody is that, well, at the end of the day to make AI work, we gotta get the right data at the right place at the right time.

So the question is, is well, how do we go about doing that? Because moving data around in real time is a bit of a challenge and, um, frankly, um, we don't know what data is always gonna be needed. Exactly.

When, so from your perspective, how does a context engine kind of arise to address these issues? Yeah, I mean, taking a step back, um, the thing that we're seeing happen, which is really fascinating, is these agents, they're able to run for really long periods of time and solve tasks and solve problems. For example, there was this metric, um, where an agent can run for 45 minutes straight and solve a software task.

80% of the time. That number, that 45 minutes, that's doubling every seven months. So like in a year or ish, we're talking three hours of an agent running independently solving problems for us, and that completely changes workflows like a three hour run does not look like a chat bot anymore, right?

Um, it's something completely unique. And the thing that changes with that is the bottleneck is no longer, uh, the, the intelligence. It's quite rare, even for my software tasks, that it's just not smart enough to solve the problem.

It's almost always a context problem, getting the right data to the agent. How we've done this historically in the first iteration of agents where they would run for maybe 30 seconds uninterrupted, was via things like rag, which still have their place and are extremely powerful. But the idea of RAG is that you are kind of feeding the agent context.

You're pretty much finding the context for the agent and providing it to the agent in these, in a three hour run, I always joke like if you were to put me or anyone in a room for three hours and tell 'em to think about something, you know, we would start hallucinating just like an agent. And so the idea is you need to give it something to, to learn and reason about. If you ever use like, uh, open eyes, uh, you know, the, the pro style where you ask a question, it goes for like three hours and it research is everything.

The thing that it's doing is it's reading, it's learning, it's reasoning, it's making more decisions, making more tool calls, and we need to provide something similar to agents. If reasoning's not the bottleneck anymore, it's context. We need to provide a way for the agents to be able to find context that they need, retrieve it quickly, no matter where it is or what form it's in.

And also we need that context to get better as the agent uses it, it needs to be stateful and needs to be able to kind of improve. And the companies that build this thing a context engine are companies that will succeed in this ENT world and be able to take full advantage of all these reasoning increases. To your point, I also think it's part of the other extreme is sometimes in the absence of any guidance, the AI agent will consume more data than it needs and wind up doing all kinds of things or trying to do things that maybe are just simply unproductive.

I think that's true. Um, I think that we're all kind of figuring out how to, we're all learning this together in, in a way, it's, it's, it is a, a very exciting time to be a builder, um, in this space. I haven't been like as excited, um, in in, I mean since like the mobile wave, um, to just feel like anything's possible and no one knows what can be done.

But yeah, you're right. I do think that there's also another aspect which is, is cost mitigation amongst other things. Like one thing that's really come into play is stuff like semantic caching where you can actually help an agent that's making very sim or if you're making similar calls to agents, you can kind of pre-cash responses that have the same intent.

But yeah, there's systems to be built around this where when you need the full reasoning, you need to enable the agent to take full advantage of that reasoning. And when you don't, you need to have ways and kind of outs to allow it to, to do things in a way that is, um, cost effective or productive. So what exactly is a context engine in that regard then, and where does it fit in the ecosystem of infrastructure that we've already got out there, which is rather exhaustive to begin with?

Yeah, I think the main consumers of data, um, in pre agent were mostly people via dashboards, maybe ML models. If you're doing like recommendations and fraud, those things remain very relevant, but agents are inherently different. The thing that's different about agents is rather than in like a BI tool or an ML model, you kind of specifying here's what you need, here's the dashboard.

Or like for an ML model, literally, here are the features that are gonna go inside with the agent. There's kind of a discovery phase. So one of the most important parts of building a context engine is building what we would call I Redis a contact surface for it to be able to see all the domain specific entities for relationships and be able to navigate that to get the context that they need.

So one aspect is almost providing this different surface layer. 'cause agents inherently, again, they consume data differently. The way agents consume data is also quite different.

The types of data structures that you would need, they might need a geo index, a time series data structure. It doesn't look like sql. The way they consume is actually closer to a person in, in, in an interesting way, and people don't think in sql.

Um, so the data structure is building kind of this almost context view with a proper navigation surface on top. That is, I think the first stage that is of building a context engine. It's almost like an ETL that builds like a context view and a surface on top of semantic layer, if you will, on top of that.

And the part that makes it an engine is as data changes, as agents learning and interacting with things, it's able via memory to enhance context, create new context. And as the systems of record change, the context surface and the, the, that context view is also updating. So it's an architecture style that if you do it correctly, you have a constantly up-to-date context surface that's getting better with usage and is navigatable natively by agents in a way that is fast.

So some people are saying to themselves, probably right about now, is this something that I buy and install? Or is this something I kind of build? Or is it more like a discipline and an expertise That's an extension of what we used to call a data engineer.

I would say it's a funny mix of all three, um, because like a data engineer too, data engineering is a, um, a skill. It's, it's a discipline that you have to learn how to do. And there were many different ways of thinking that came of that skill.

Like the rise of data scientists changed how we interacted with and worked with data. There's obviously a set of tools that made doing that easier, like being a data engineer if Spark and Snowflake didn't exist, like doesn't really make sense. Um, so it it's a mix of both.

It's a discipline, but that discipline is powered by, by tooling, by having proper databases and abstractions and applications on top of that database. Something like red is where with a Redis, you can kind of build, because of our, our versatility, we have all these different data structures that you can build upon. Um, we have speed by default.

We have applications like our agent memory server, um, contact surfaces, other things that we're, we're building to, to enable you to take advantage of all of reds, uh, engine via the agent application layer. But there is just giving you that alone isn't enough. I can't, because you understand your business, you're, you're the person who's gonna be able to take this powerful, um, engine that we give you these tooling and plug it in based on what you know of your business to build the proper, uh, context engine for your, uh, application and company.

Do you think that the AI agent's behavior will be predictable enough that I can kind of maybe start figuring out what the cash and what they're gonna need and what that context is? Or are these things basically just gonna be, you know, querying everything in anything at random moments and we're never gonna be able to predict what they're doing? So I'll put in two fronts.

One part is arev, um, deterministic, and the answer is no, they're not inherently LLMs are non-deterministic, but they're so good at reasoning. I give a demo where I actually use an LLM, I do it live every time, every single time I do it. It's slightly different.

The wording's a little different. It uses a different word here or there, but it always solves the task exactly the same. I run it live every time and I'm confident it'll work every time.

Um, so because the reasoning is getting so good, you can feel confidence, but it will solve the task at hand. We'll do it exactly the same one-to-one. Can you write like a unit test, like an engineer speak to like, uh, to test us?

No, it's a little different. Um, but the example I like to give is, is this, if I'm build, if I'm have a customer support agent, like a person and I call them and I'm getting, let's say a, a food delivery and I'm complaining it's late, they have this portal in front of them and that portal will have like a search bar. It's gonna have all these things I can like click into to get all the information I need as an agent to be able to, a human agent to be able to help the, the caller.

But if I do it and someone else does it and someone else does it, all three of us are probably gonna click around in slightly different order. We're gonna respond slightly differently. It's not gonna be deterministic, but we're all solving a task at hand because if you give the proper data in front and proper training, we can solve.

The problem You mentioned Brian earlier, is, so is the whole notion of what we first call retrieval, augmented generation rag, is that becoming quickly obsolete because we're gonna move to something that functions more in near real time. So, um, so rag fundamentally, like if you just look at the, the words like retrieve augmented generation, like I'm retrieving some data to augment the generation of a response. What I've described of a context engine actually arguably is the same thing.

I'm giving it something to retrieve data from and it's gonna augment its response based on data that it retrieves. So I think rag as a concept, it's the same, it's the same problem, really the problem that rag solves. And I think one of the fundamental problems of building agents is I have a context window.

It's yay big. I need to fill it with the most relevant, useful pieces of information as possible. As the reasoning has increased for certain use cases, it makes sense to take this different style where we let the agent actually decide what data it needs and give it again, that context engine to retrieve it.

Um, for some use cases that are maybe more fast or very, uh, maybe a little more, uh, very short doing rag the traditional way. Traditionally when people do rag, they take data, typically unstructured data, they chunk it up, they put it in a vector database, they do a search on it, and then they feed that to the model. We're pretty much hard coding the context into the model.

It's dynamic, but they always feed it into the model. The model doesn't decide what context it retrieves that will still exist. But I think to take full advantage of reasoning, you, you're not gonna be, if you have a three hour run, it's not gonna make sense for you to go retrieve all the context up front.

You won't even know the model's gonna be able to know better than yourself. The other thing is RAG is very focused on unstructured data. There's this metric that like 90 something, I dunno what the number is.

90 something percent of the world's data is unstructured. And that's probably true, but it's not a fair metric because it's talking about size of data. One byte in Redis and a structured database doesn't have to be any sort of structured, like a SQL database is gonna be worth way more than a by in some random JPEG or a byte in some random PDF that I have.

So my most valuable data is structured data, almost always, but via this unique, the the specific way that we typically do rag, we can't unlock that data. So what we're moving towards is two things. One is allowing the agent to decide and reason and decide what context it needs and actually be able to get context, learn, make a decision, get more context based on what it's thinking, and take full advantage of that reasoning.

Window two is unlocking all of the data, unstructured data, structured data, semi-structured, everything in between memory to fully, uh, enable this agent to solve hard problems. Will we kinda have to develop a greater appreciation for the nuances of data management? And I asked this question because as historically, well we've had databases, but I would argue that most organizations have not been especially good at managing their data.

Very few of them would probably get a good housekeeping seal of approval for the way the data's been managed. But, um, you know, is AI gonna finally force us to revisit that conversation in that whole discipline? I think every few years we kind of revisit that, and every few years we find out, hey, there's even more value if we can do this better.

And I think agents is like a massive increase in that. Um, I think the reality is is like these system of records will remain there and they, they will, will costly be fighting to keep them in order and, and try to organize them and do them well. Part of also building a context engine and where this is also very powerful as what a lot of teams end up doing is they create context views.

They join across all these system of records, they build this view that is inherently scalable. It's inherently fast, it's secure in the sense that I know exactly what's in it. I'm not giving it direct access to my core system of record.

I have a view that I created with the context that I know that an agent may will want to use to be able to solve my set of problems. And so the short answer is yes, data has been this ever evolving problem. As data sizes increase and the ways we can take advantage of that data increase, um, it becomes more critical than ever.

And yeah, I think building a context engine in a way is, is a form of, of, of data management. Um, but it's a form of building context views in Redis or or wherever else to, um, power these agents. So what are you seeing people doing today that just makes you shake your head a little bit and go, folks, we need to be a little bit smarter than that.

Um, I've seen, uh, so one thing I see a lot, it has a use case, but I think it's being overused, is things like text to SQL where you're letting the agent actually create SQL and run queries directly against a database as opposed to building a contact surface and, and reticent and building kind of that view of data. The problem there is, uh, I mean we were already just talking about the non-determinism of L LMS and safety, like, do I trust giving an agent direct SQL execution access into my database? No.

Do I trust it to be able to navigate a context view that I've created? Yeah, for sure. So I think that's one that is, uh, ripe for, for error and ripe for, uh, disastrous outcomes.

Ultimately. Do you think that the organizations that wind up winning in the age of AI aren't necessarily the ones that build or have access to better LLMs or they're not necessarily, uh, the AI side of the equation as much as it is just how they manage the data and what data gets exposed and, and just quite literally the fundamentals of data management? There's a, so the paper that sort of, let's say invented LLM is a paper called Attention is All that you Need or All, all you need, and I would argue today if you're an enterprise context is all you need.

So the main problem to be solved for most companies is the ability to deliver the right context at the right time quickly to agents. And those are the problems that at Redis we are fundamentally focused on solving and, and are solving for a lot of different companies. Um, so yes, I I think con building these context engines is what's gonna enable the, the, uh, enterprises and companies that are able to actually benefit from this agent weight as opposed to be completely disruptive by them.

All right folks, you heard it here. The one thing that humans and AI agents have in common when it comes to making the decision context matters. Hey Simba, thanks for being on the shelf.

Hey, thank you very much, Mike. All right, and thank you all for watching the latest episode of the Techstrong AI Leadership Insight series. You can find this episode, others on our website.

We invite you to check those all out. Until then, we'll see you next time.