Search Infrastructure Wasn’t Built for AI Agents

AI Leadership Insight Series. I'm your host, Mike Vizard. Today we're with Ben Flast, who's senior director of product management for MongoDB, and we're having a little chat about the impact AI agents are having on search because, well, they're just pulling more data than ever.

Ben, welcome to the show. Thanks, Mike. Good to be here.

All right. So walk us through what's happening here, because I think people are deploying AI agents and not entirely understanding the implications on infrastructure and in particular search, but as we deploy more of these AI agents, what should we be expecting in terms of what's going to happen with search? Sure.

So AI agents are really a fascinating evolution of applications, right? So agents at their core are a large language model that is then able to interact and take action in the world. And what's really interesting for search is that AI agents rely on high quality search to be effective in the work that they're doing, right?

So an agent has a large language model with a context window, and the more context you put into that window, the more chance you have to steer it the wrong way, the more expensive it gets, the more it takes time to process the data. And so by having search attached to your agent, you're enabling the agent to more easily get the right information so that it can take effective actions and do them much quicker. And if I don't do that, they will spend more time trying to figure out what that context is, and the total cost of running the AI agent will start to increase, correct?

Exactly. All right. So as we kind of play this out, what should we be thinking about the way we deploy search and the tools that go with that?

Because most of what we currently have was designed for humans rather than AI agents. Yeah, absolutely. So one of the things that's crucial with search is that for agents that are using search, you need the agents to have search connected to the data that they themselves are producing.

And what I mean by this is, as an agent is going through and taking action, and changing the state around them, you need that new updated kind of definition of the world to then be reflected in the very data that it is going to query. And so one of the things that you want is the search to be very closely connected to the data that the agent is then going to take action on. And so I think you're going to see more and more places where search needs to be very high performance, because you need to feed the agent data quickly, but it also needs to replicate from the actions that the agent is taking quickly back into the search index so that it can keep then processing those results and keep going.

So does that mean we're going to fundamentally change the underlying databases and the search infrastructure that it sits on top of to accommodate these AI agents? And are people prepared for what might be involved in doing that? Yeah, absolutely.

And I think that's exactly what we're seeing with a lot of our customers. So, with MongoDB's approach, the search index is replicated right off of the database. And so if an agent is going through, taking action, updating a flag inside of the database, copying in some information that it's going to work through, we're seeing that they're then using those same pieces of data that they just created or modified and searching over it to then feed that data back into the agent so that it can take the next step.

And so what we see is that performance bit of not only how fast is the search query, but how fast is that search index being updated, being really crucial, and then obviously the quality of search, so the quality of your results, is also all the more important because that's what ends up guiding the agent on the action it's going to take. So you really need that high quality search result, powered by oftentimes really high quality embeddings or reranker models, and then also you need it to be really low latency over the data that is both being created and then also the queries that are being returned. Mm-hmm.

Do you think as this all plays out that we're all about to have a greater appreciation for some of the nuances of data management? I think maybe we've taken databases for granted over the years, and maybe is that going to change as we roll out these AI agents because, well, everything seems to be a lot more latency sensitive, shall we say. Definitely.

So, it's not just that it's latency sensitive, though. That's obviously a crucial part of it, but the other bit is just the scale, right? And it's not just the data scale, but the amount of consumers of data has just radically changed, right?

You have a ton of autonomous agents who are all taking action, and so if I used to myself go in and book a ticket, if I now have 10 agents looking for the best ticket for me, then I've just multiplied the amount of traffic that my old request used to take by 10. And so the volume of requests is going to change for sure. And then also the interface with which the agents need to engage with the data is going to change as well, because they have a different perspective or a different way that they want to engage with the data and query it than a human does.

Oh, and then the last thing that I would mention is that the access patterns and role-based access controls are definitely going to need to change as well because agents have a totally different kind of pattern of interacting with data, but also amount of access that you want them to have. So lots of things are going to change as agents become much more prevalent on the web. Are there things to be done as it relates to the database to provide more context to those AI agents?

Because one of the issues you hear people struggling with is, AI and generative AI doesn't do the same thing the same way twice. It's probabilistic, and most of our workflows are deterministic, so how can we kind of strike a balance between these things with a little help from search? Sure.

So what we see from this perspective is that things are changing so fast. So obviously agents are a whole new paradigm. We weren't talking about them a ton last year, though they had started to show up obviously, but now that's all we're talking about.

And, in six months there'll be something new, right? And so what we see is that the customers who are being most successful building new applications, building out for new use cases, are relying on technology that allows them to easily change things. And so one of the things that we're very focused on is having a flexible document model for how you store your data and then how you can query it.

And so as your data changes, you don't need to make huge changes to the underlying data model to support the application. In fact, the database is just going to very elegantly take in a new data model, or a new shape of data. And that really undergirds a lot of what we do, which is building the database to support the expectation that data will change.

Right? Really being built for that type of change. And I think customers that are finding the most success are using tools that let them do that and also thinking about their build process, with that in the back of their mind.

Right? We need to be ready for things to change. How do I apply guardrails to all of this?

Because there's a lot of conversations about AI agents going rogue, and whether they went rogue or not is up into debate, but the point is, is that they are going to aggressively try to complete whatever task assigned, and I need to figure out some way to make sure that whatever they're searching for isn't sensitive data that I don't necessarily want them seeing. Yeah. So, there are two things I would highlight there, and both of them are actually kind of related to other things that we've chatted a bit about today.

So the first is that using search to power agents is a great way to make them consistent, right? You choose exactly what data goes into them and when it goes in, and that is the best way to make sure that they behave in a consistent and effective manner. And so oftentimes when we're thinking about guardrails, we're thinking about what will we put into the context window to make sure it behaves in a certain way, as well as how are we going to constrain their tool use and make sure that they can only do the certain set of things that we're comfortable with?

So that's one bit of it. And then the other side is the governance bit. So like, what do we give them access to?

What do their roles and permissions inside the database or inside the application actually allow them to do? And so setting up clear rules of the road for the agent that are not things that it can choose to ignore, but instead are kind of built into the application, built into your data layer, is really crucial to make sure that you can have well-controlled and orchestrated and consistent agent behavior. How do you envision the IT teams will be structured as this all comes together?

Is there going to be any fundamental changes? Because today I feel like developers are building the agents, but there are still databases and DBAs that need to be managed. But is that converging, and if so, how?

Yeah, I think that's a big question for what's coming in the future. From what I see, as it pertains to database management versus application engineering versus agent engineering, I think the past few years, we had only continued the move from the separation of concerns to you build it, you own it, kind of DevOps approach to building and managing solutions. I think that just continues now, and the same developers who are building are, in many cases, responsible for managing their databases and using more managed services, so it's easy to do so.

I think the other thing that is changing is some of the roles that participate in building of applications and agents as things like business stakeholders and designers and other roles are kind of collapsing and merging more, and you're seeing more shared work across the different domains. So I see that changing, but I think the amount of change happening within the engineering role around infrastructure and management and deployment, that seems to be somewhat consistent. Of course, at this point, every organization I know of has multiple databases, different formats.

Are we going to bring all that back together in the age of AI, or are we going to continue to have all these different databases supporting different data types, or will there be, I don't know, maybe one database to rule all types? What can happen from here? Yeah, I think we are seeing databases offer more capabilities in adjacency to the core online transactional processing workload.

That's certainly happening. I think that at the same time, you do still have different use cases with different personas and different groups using databases. That means that you're not going to see just a complete collapse into a single system.

But for application databases, I think many capabilities are really folding into the core transactional database, and it's able to serve more workloads than ever before for agents and applications. So ultimately, what's your best advice to the IT teams that are trying to deploy all this stuff? Because I think in a lot of instances, they're a little overwhelmed.

It's a lot of new stuff, new tactics and techniques to master, as it were, but where to get started? Yeah. So my best advice would be really two things.

First, assume that the components you will use inside of your application, inside of your agent will change, and so choose core pieces of your infrastructure that make that easy. Whether it's a flexible document model or a range of different query capabilities, building to have that capability for change is crucial. And then the other side of it is generally with your generative or AI-powered features, use some of the highest powered or highest quality capabilities to start with, and then tune back from there to optimize for price and performance.

The reason I say that is oftentimes you don't know what exactly is going to be possible for your application since we're bringing in these very stochastic components in the form of large language models or embedding models or re-ranker models, and so you really want to use the highest powered or highest capability ones. And then once you find fit and find value inside of the product you're offering, then it's a good time to start paring things back and figuring out how do you do this in a sustainable manner that makes sense for the business. All right, folks, you heard it here.

The laws of physics as they apply to data have not been suspended in the age of AI, so plan accordingly. Benjamin, thanks for being on the show. Thanks, Mike.

ai Leadership Insight series. You can find this episode and others on our website. We invite you to check all those out.

Until then, we'll see you next time.