AI Data Infrastructure Swings Back Toward the Edge
AI data infrastructure is facing demands unlike anything before. For 30 years, systems were designed for humans. AI agents put a very different kind of pressure on networks, systems and databases.
Couchbase SVP of AI Tim Finley examines that shift with Alan Shimel at KubeCon + CloudNativeCon in Salt Lake City.
Righting the One-Trick Ponies
Finley spent years as a customer before moving to Oracle and AWS. He joined Couchbase because it already had the components he would want in an AI system.
Over the years, specialized databases emerged for graph, time series, cache and vector workloads. Each did one job well. But AI runs on context, and scattering data across many systems scatters that context too.
Couchbase has pursued convergence from the start. Fewer systems means fewer integrations and a smaller security surface to manage.
The Pendulum Swings Back
For 15 years, modernization meant moving to the cloud. Now token costs, latency and hardware scarcity have some enterprises bringing workloads home.
Finley sees the pendulum settling in the middle. Analysts expect much of the data from agents to be generated at the edge. Couchbase customers already run projects on oil fields, airplanes, cruise ships and coffee shops.
Since no one knows exactly how AI will evolve, flexibility matters. Your AI data infrastructure should follow you wherever customers are.
Semantic Caching and What Comes Next
At its Catalyst conference, Couchbase made Agent Memory and an MCP server available as managed services. It also added an Amazon Bedrock integration with semantic caching.
Couchbase says that cache can cut LLM calls by 40% to 60%. Teams can keep using more capable models longer before costs force a trade-off.
Finley also previewed the next Couchbase Server release. He says it brings much faster cluster rebalancing and new vector index options. A conversational query assistant will let users ask questions in natural language.
After years of being seen as utilitarian, he says, databases are cool again.
Transcript
Hey, everyone. We're back here live with our KubeCon coverage from Salt Lake City. You're watching Techstrong TV.
My next guest is Tim Finley. Tim is with Couchbase. But Tim has got a great story in the industry.
We're going to hear about it. But before we jump into it, I want to tell you, the buzz on the floor this year is really palpable, right? Seeing good things happen, good stuff.
It's a very lively, jazzed up, no pun intended to the Utah Jazz. But a very jazzed up kind of audience and group here this year. Tim, this is your first interview with Techstrong.
Yep. Awesome. Thanks.
Not Techstrong. You were in person, but I mean with me here- Yep ... on this.
Yep. Give people a sense, though, of your path, Tim. How'd you wind up here?
Why'd I end up here? That's a great question. So, as you mentioned, I've been in the industry for a while.
I was a customer for 13 or 14 years, then I decided to go over on the vendor side. I went to Oracle, went to Amazon, had good runs there. I saw something really special at Couchbase.
When I look at how the industries are going, when I look at all the trends right now in agents and infrastructure and AI in general, there were things that I came to realization that Couchbase had that I wanted to put my thumbprint on, and it turned into a really great opportunity. I've been there for about seven months now. Okay.
And this is the most fulfilling role that I've had probably in my 25-plus-year career. You have a very interesting title there, though. Share that with the audience, if you don't mind.
Yeah. So there's a couple of things. When you look at where AI is running right now, and I came from Amazon, I had exposure to a ton of different customer situations and use cases.
You start to realize that where the world is going, system scale is actually changing a little bit. Yeah. When you look at the demands of agents and whether that is the pressure that it puts on network and systems, it's like nothing that we've seen before.
We've been designing systems for humans for 30 years, and now that's changing. Sure. And then you take a step back and you look, if I were to build a system that would support that today, what components would it have?
And if you go and you check down the list, what I realize is Couchbase has all of those things. Yeah. Including the ability to run where your AI needs to run it, which isn't always going to be in a cloud data center.
No, absolutely. You know what's interesting to me, Tim? Look, I've been around the block, too, and I've watched the database industry move from SQL relational to NoSQL.
Then I would say over the last, let's say five years, seven years, we've seen, I don't want to call them exotic, but different kinds of databases. Yes. Graph databases, time series databases, all different databases tailored almost like they're tailored to a single need, a single want.
But behind them all was this move towards real time. Yes. Right?
Away from systems of record to systems of engagement, to being able to take data lakes, to take ever larger chunks of data- Uh-huh ... and make that actionable, make that discoverable, whether it's through an AI agent or some other kind of automation or what have you. And I think when you look at the story of Couchbase- Uh-huh ...
it's that story. Yes, it quickly became one of the leaders in NoSQL when NoSQL was the cat's meow. Yeah.
Right? But it went from there, where NoSQL became, okay, NoSQL. We know what that is.
Yeah. We got all these other exotics to play with. But through it all, and now into the age of AI, we're recognizing, you know what?
NoSQL still does the job for that system of engagement on, close by where I need it. Uh-huh. On the edge, on-prem, on device, wherever, in the hypervisor.
It scales. And with AI, scale is important, right? Yep.
So, it's an important thing, and I think that's really the strength of Couchbase. You actually talked about an important part of our history. Go ahead.
Well, I lived that. Well, you live long enough. Yeah.
Well, go ahead, man. Jump in on it. You started it up, now I'm going to give it to you.
Go ahead. Because you're right. If you go back 15 years ago, we said, "Okay, what we want to do is we want to build a NoSQL database that can defy the laws and the physics of a relational database," and we did it.
And we wanted to run on pretty inexpensive hardware and still scale and be available. And then we had a realization after doing that, and we went, "Well, the world's most well-known language is SQL. " And that's when we built SQL++.
And then you see all these things that have been popping up over time, time series here, cache here, graph here, vector here. At that point in time, they had a valid purpose, because they were doing one job really, really well. They're all one-trick ponies in many ways.
Exactly. Right. But when you look at what has happened over time, especially when this world of AI evolves around context, when you start to have all of these one-trick ponies in your data spread everywhere, you're spreading out all of your context.
Yeah. It becomes very, very difficult to manage. It's chaos.
Yes, exactly. So we've been committed to this convergence, which is righting the wrongs of the one-trick ponies over time from the get-go. And, that is one of the biggest advantages that we see in this world of AI today.
Sure. Because not only are you managing fewer integrations, one of the things that I talk with customers about that is even more important, their operational surface area for the security that they have to manage. Yeah.
Surface area. Because your normal AI stack has four or five things, right? Yeah.
That means you need to manage the security of those four or five things differently. In this world of AI, that's difficult. So the more you can squeeze these together in a single operational surface, you start to have the benefits of that convergence effect.
Agreed. Here's another thing. We talk about history and how things, sometimes the more they change, the more they stay the same.
The other thing I've seen in my years is sometimes the pendulum swings this way and it swings back, right? 26, cloud comes on. For the last 15 years, it's all been- Yeah ...
are you moving to the cloud? Are you modernizing? That's like code word.
To move to the cloud was all about moving to the cloud. Yeah. And now with AI, between tokenization and latency and scarcity of hardware and infrastructure- Uh-huh ...
we're starting to see people say, "I don't know if I want to move this to the cloud. " Yeah. " Yeah.
" Mm-hmm. " You're exactly right. The pendulum is swinging back a little bit.
It is. It is, and I think it's one of those things where it swung to the right, cloud or bust. Right.
And now it's probably come back down to the middle, and I think that probably feels a little bit more correct. Yeah. Not for all businesses- No ...
but if you consider our business, enterprise, which is what we're focused on, AI has triggered a realization where running in one specific spot has actually become a little disadvantageous, actually. It limits your options if you've got to pay- It limits your options to the models. Yeah.
And the majority of the data, especially with agents, and this isn't Tim, this is analysts smarter than me. Right. Over the next few years, two thirds of it or more is actually going to be generated at the edge.
Yeah. Which means if you do not have infrastructure that can deal with that stuff at the edge, you're going to be at a competitive disadvantage. Absolutely.
Even if the analysts are wrong about that and it's not two thirds, maybe it's 40% there. Maybe a good chunk of it runs locally on your phone, your laptop- Yeah ... your iPad, whatever.
The real issue is we don't know. We don't. And what we have learned over the last couple years since this AI burst onto the scene is that stuff is happening so quickly.
It's hard to... You don't want to get too far out in front of your skis because you're not sure how it's going to happen. I know.
But you don't want to be the last guy to the party either. Yeah. And so you need systems like a Couchbase- Mm-hmm ...
a database like Couchbase, solutions that are agile, small A agile, right? " Yeah. If it goes that way, a little to the left, a little to the right, I still got it covered.
Yeah. I can't afford to go on one thing that puts me too far one way or the other. Exactly.
I need options. I need versatility. Realistically, your system should be able to follow you as you go to where your customers are- Right ...
based on whatever your business model is. And right now, in the world of AI, it's like you said, we don't know what we don't know. But I know what I see.
Right. And I can tell you some of the most interesting projects that we're involved in now. Oil fields.
Mm-hmm. So heavy manufacturing. Airplanes.
Cruise vessels. Yep. Right?
Coffee shops. None of these things are sitting within a neat combine of a specific- Oh, okay ... cloud or even a specific data center.
The workload, the valuable part, is actually originating on the outside. Right. And then the work is to bring it in.
And that is what we make it easy for customers to do. Love it. If you don't mind, I want to pivot.
You guys just did your annual user conference down in Fort Lauderdale. I think it was around my birthday, around October 20th or something. Yeah.
You made several announcements on AI as well. Mm-hmm. Share with our audience, if you don't mind, a little bit.
Yeah. A few things. So if you remember back in June, we announced AI Data Plane.
AI Data Plane had all the components, MCP, Agent, Memory, the things that you need to do in order to make AI infrastructure effective, more effective. So in a one quarter follow-up, we actually announced Agent, Memory, and MCP are all available as managed cloud services now as well. Good.
Now, go back to our point before. You can't shift the pendulum too far in any direction. No, no.
That's the lesson here. You see us launching AI features for self-managing customers to run wherever they want to, and for a lot of customers that say, "Cloud service, please. " Right.
I'm just not ready for that yet. One of the most interesting things, though, that we announced was a very cool integration with Amazon Bedrock. And basically, the problem that this solves, and you've talked to customers, I'm sure, like I do, customers right now are realizing, especially if they're more mature, model diversity is actually a very powerful lever for them to control their, let's call it, the cost versus the value that they're getting out of AI.
They maybe choose a less expensive model in order to balance it out when they get to scale. So it has to be easy to change those models. But then the second thing is, especially for customers that aren't as mature, they'll start on a very powerful frontier model.
They'll love the results in pilot, maybe even early production, but then when it goes viral and it takes off, they don't want the bill. Yep. And so what our integration does is it makes it easy to change the models if you want to, but you can put the models behind our semantic cache.
And our semantic cache reduces the trips to the LLM by about 40% to 60%. And so you can hang on and not have to trade off model intelligence a little bit longer. That's a very powerful- Love it ...
value proposition for our customers. It absolutely is. Now, that was October 20th.
Fast-forward to today. These things, they're available out there if people go to Couchbase now- Yep ... and check them out.
Let's talk about KubeCon. Beyond that. Yeah.
You know what they say, Tim. I was going to say. If you're not moving forward, you're dying, right?
Yeah, yeah, yeah. So I'll meet you in the middle on one thing that we're going forward, but it's in preview. Okay.
And that is our new server release. 5. Cool.
It is our next generation server. People can go to the website. They can request a preview.
The level of scale, Couchbase has always been known very well for scale. Scale. The scale that we're doing now in the new server version would just blow your mind.
You're talking cluster rebalancing and cluster expansion that used to take somewhere in the ballpark of 24 to 30 hours online for a large multi-terabyte cluster, we're doing it in under 15 minutes. Wow. Which is significant.
Crazy. It's crazy. Very significant.
New AI features. So the more advanced you get, and you start to realize that in the world of AI, context isn't just the quality of your data, but context is actually a little bit into how you architect your vector in regards to the types of search and the type of questions AI is going to ask. Yep.
So we will be releasing new sophisticated vector index support, sparse vectors, being able to combine results from sparse vector indexes and dense vector indexes, and a general lexical search with a re-ranker in the back that then provides the customer the best possible answer. That really cranks up the accuracy. And then the last thing, and I think this is one of the things I'm most excited about, is we believe that natural language eventually should rule everything.
And you shouldn't even- So do I, for the record. Yeah, exactly. Love it.
You shouldn't have to be a Couchbase expert to get value out of Couchbase. And so we will be releasing something called CQI, which is a conversational query assistant. I love it.
So talk to it in natural language. It'll go, it'll locate the data in your cluster for you, build the query, come back, give you the result. And if it's not quite what you expect, you just continue to have a conversation with the system until you're there.
So you can have a conversation like you're sitting at the couch. Just like you would talk to your BA guy or your DBA. Yep.
And we will support that, not just if you're in cloud. That will be supported and self-managed for customers that want to be- Yep ... running on-premise.
What does that mean for DBAs? All the way to the edge. Next at the edge.
So one of the cool things on the edge, I don't know if you saw it, we announced it at Catalyst. Yep. We give people the ability with agent skills to build a full-fledged mobile application with cluster replication, all of the things that you would expect, that typically takes a group of four to five people months.
Months. And you can build it in under an hour as a single human. It'll provision all the infrastructure.
Yeah. It's amazing. Let me ask you a serious question.
In all your years in databases, did you ever think we'd come to this? No, but I love it. So- It's a great time, isn't it?
There was a period of time, I remember, probably leading up until about five years ago, where when you were a lifelong database person, you started to get the feeling that databases weren't cool anymore. They became almost sort of utilitarian. Yeah.
Lower in priority, one of those things that you just had to have. Yeah. But people weren't necessarily seeing the value in it.
Wasn't fancy. But then AI comes on the scene. Yeah.
And then generative AI comes on the scene. Then agentic AI- Yeah ... and physical AI, and everybody goes, "Oh, my.
The most important part of this system is going to be my database- The data ... " And so we have our little badge now back in the cool kid club. And it's great.
I've seen the entire spectrum going all the way back for like four or five- We talked before about the pendulum. Yeah, exactly. Nice.
Hey, man, thank you for coming on here. Thank you. We appreciate that.
I enjoyed it. Keep up the great work. Come back and keep us posted.
Will do. Appreciate it. All righty.
We're here, we're live at KubeCon with Couchbase. We'll be back with more in a second. You're watching Techstrong TV.