Beyond RAG — Inside Pinecone Nexus and the Knowledge Engine for Agents

Hey everyone. Welcome back here to another Techstrong TV interview. I'm happy to have this fellow back on.

We've had to go through some hoops to reschedule a few times because I've been running around, but I'm glad he stuck with me. It's my friend Jeff Zhou from Pinecone. Jeff, how are you?

It's good to see you again. Doing great. How about yourself, Allen?

I'm glad to be home finally, at least for a little while. I think for the rest of next week I'm home, and then back on the road again. But that being said, how have you been?

What's going on? I'm good. It's been a, as you can imagine, a pretty crazy time at Pinecone, and I think we've been cooking up some really, really exciting, I would say, transformations of the company, honestly, as really rethinking where Pinecone's place is in this agentic storm that we're all entering together.

So it's been an exciting time. Absolutely. I've been on the road.

I've probably been in about five different user conferences in the last couple weeks, and I think we're all going through this storm, as you called it, right? Yep. Reinventing companies, reinventing businesses, but it's also so damn exciting.

I was talking to a CEO yesterday at a conference I was at in Dallas, and he said, "Just what an exciting time to be an entrepreneur, to be in business, to be in one of these forward areas where what we used to think was impossible is now possible," and how do you plan for the impossible that's possible now? So crazy times. Jeff, before we jump into it, though, I wanted to give people a sense of your role at Pinecone and your background so they know who it is they're listening to.

So if you don't mind, give them just briefly. Yeah, for sure. I've been at Pinecone for roughly four years.

I run product here at Pinecone. I'm the VP of product. And before this-- It's a funny story, actually.

Before this, I spent most of my career at Microsoft working in Bing as a machine learning platform PM there. So essentially running a bunch of their distributed GPU training optimizations, accelerating A6 on inference and GPUs. But also, most notably, I also bootstrapped their vector search platform for web scale search from scratch, which was perhaps the obvious reason why I came to Pinecone, because it was such an interesting, fascinating problem.

But the funny story I always like to tell everyone is that I joined four months before ChatGPT came out, which is actually a really funny thing because the moment that happened, my entire world changed- Upside down. Yep ... turned upside down and completely shifted everything that I've even known about vector search, building all my time at Microsoft.

So it's been a fascinating journey. And even now today, I feel like that same exact process is happening again, where I have to completely rethink how I operate, how the product operates, how we all operate as a company. It's been such a fascinating time to be building.

Absolutely. I told someone in another conference I was at this week, that whole ChatGPT generative AI thing is so 2025. It's like that's old hat now.

What are you doing for me lately with your agents? It's- Yeah ... really is a crazy time.

We're talking about Pinecone. Of course, I'm really familiar with Pinecone. You live Pinecone, but not everyone out here is maybe as familiar with Pinecone, or they think they're familiar with Pinecone.

Jeff, how would you describe Pinecone to them? Yeah. So Pinecone's mission, even from when we first were founded many years, I think almost six years ago, was all about making AI knowledgeable.

Right? It was how do we actually give AI the context, the knowledge that it needs to operate? And I think that's a really fascinating thing, is that what that meant, of course, maybe four years ago when I first joined, was a very specific thing.

Typically, what most people know as retrieval-augmented generation, RAG- RAG ... was a very traditional pattern. But obviously, with the world that we live in now and how to give context and knowledge to agents and LLMs now is a fundamentally different world.

And I think that itself has also evolved over the time. But at its core, Pinecone's mission has always been to really empower AI to have the right set of the knowledge, especially across your private information, at your enterprise, within your own domain itself, to actually make AI useful for what you need it to do. I agree.

That's a great description. Thank you for that. All right, let's jump into the news.

You guys made an announcement, something around Pinecone Nexus, is that right? Yeah. That's correct.

Great. Take it from there, Jeff. I think as I mentioned before, when I joined I would say four years ago, there was a very specific meaning to how people called Pinecone to get knowledge to their LLMs.

And I think a lot of that was a fundamental shift in terms of, hey, a lot of times that pattern was oftentimes, hey, a person comes in and do a chatbot. " That then, of course, triggers a call to Pinecone from the LLM to get the top K context chunks and then return it back to them. So there was a very simple, I would say, approach where every single query, there was a human behind it.

Where that query, that human was actually triggering some sort of interaction, calling Pinecone, and then returning. But what we started to see over the last year or so was a really fundamental shift in how the access patterns on how Pinecone was being used in the first place. We all started seeing, essentially, the patterns become a little bit more, I think at a very simple way, agentic.

Where you see a lot more bursty patterns around queries being solved at the same time, a lot more continuous high QPS and bursts as well. So that already was giving us a signal on our side was like, wait a second, the fundamental consumer on how the infrastructure is being used is shifting from a human-driven interaction into more of a more autonomous, agentic kind of a driven interaction in the first place. And so that was really the impetus for us to really fundamentally rethink, okay, this shift is happening.

And so we really wanted to rethink from a core concept of what would it take for us to fundamentally provide, be the really effective knowledge layer for agents in the first place, right? And so that's how Pinecone Nexus came about, where ultimately, we kind of made two key realizations, right? I think the first one being is that a lot of people, when they want to get context, it kind of lives everywhere, right?

You think about just any sort of enterprise, right? You have your data warehouses, right? You have your Snowflakes, you have your big queries, you have all sorts of structured data warehouses, right?

You have your documents living in all sorts of repositories. You have ticketing systems with all sorts of other information about your support system, and then you also have Slack conversation, Gong recordings, all these transcripts as well. All your contacts are scattered all over the place, right?

But what ended up happening is the typical approach, especially with agents today, is that what they would do is that, "Okay, why don't I just sit here? I'm just going to simply connect my LLM or my agent to all these things with a CLI or an MCP, right? Cool.

" So what it does is that I have access now to my BigQuery, to all my structured data warehouses. And what ends up happening is that it does what actually agents are really good at, right? Which is doing exploration, right?

These agents are really, really good at saying, "Okay, given a specific task, let me go explore the data. " And it does ultimately 20, 30 tool calls ultimately to figure out, okay, now I finally have the information that I needed. Then I can actually then go and actually execute whatever you want me to do, right?

And so that was the pattern that we were seeing over and over again, and that's how knowledge effectively was being given to agents in the first place, was that agents were rediscovering the knowledge every single time on every single call, right? And so that was a pattern that leads to, as we mentioned before, lots of token usage, really, really slow execution, your unreliable kind of task execution as well. And so that becomes like while it's very, very simple to do in many ways, it's really an anti-pattern when it comes to building agentic systems, right?

Because it's really, really just simply inefficient, right? And so that's kind of where Pinecone Nexus came about and said, okay, what if we changed the entire concept of knowledge? Instead of doing this exploration, we actually want to do the work up front, right?

What we want to say is, "Hey, why don't we actually take the knowledge that they want to use, and instead of just leaving it in the formats that it's been existing in for forever, we'll actually use an engine to then transform that knowledge into the representations that the agent needs to be fast, efficient, and accurate," right? And so that's the core idea around Pinecone Nexus, is that it's a knowledge engine that not only just takes your sources of data and enables it, but also is able to really transform that information into task-optimized artifacts that then the agent can use to really, as you mentioned earlier, Alan, around, oh, we have faster time to success, lower token usage, and ultimately all the benefits that we talk about in our Nexus blog as well. I love it.

Jeff, I've seen a few cycles in my day right, of tech. This is probably going to be, I think, the biggest, bigger than dot-com, bigger than it all. But the way these cycles work is first is the experimentation, called the Cambrian explosion of life, right?

We were trying a million different things, and some things have eight eyes, some have six eyes, and then eventually we settle on two eyes as a good life form, right? But inevitably what happens is after that initial explosion, efficiencies start becoming important. Cloud is a great example, right?

" And we've got to get a handle on this. We need deterministic, as they say, right? We need real numbers.

I think we're just now starting to see that with the AI and especially the agentics. We could call it the just because I can doesn't mean I should era, right? Yeah.

And that's something I think Nexus is picking up on, right? In terms of, all right, we could do it. There's several ways we could do it.

Let's do it the most efficient way, and let's try to save you some token burn. Yep. That's exactly spot on.

And I think that's the thing that we've been hearing again and again from our customers is that, look, what we've been doing for a long time right now is exploring and testing these agentic workflows, right? But now, when you want to start moving these agentic workflows and agents into production, you have a whole host of, you're no longer just worried about, can it do this? You're now worried about, can it do it efficiently?

How much is it going to cost me? Can it do it accurately? Can I trust this in terms of its governance and actually make sure it's not hallucinating?

All these things all of a sudden become production-grade kind of concerns. And now that we've proven the core technology itself is capable, it is all about how do we then actually scale it and build it into production. And that's exactly where Nexus is designed to come in and ultimately help you bring these production agentic experience and workflows in a way that is manageable and economic and ultimately accurate for what you need to accomplish.

Love it. Just want to make sure we hit no QL or no? Yeah, we did not hit no QL yet, but we can also- Let's talk a little bit about that.

Yeah. Again, part of this efficiency and maturation thing. Yeah.

So the other part, I think that's a great lead in as well, because one of the other things that we realized as we were designing Nexus, as we were designing thatWe were actually really struck by what agents need to be able to interface to actually request the knowledge that they want in an efficient way, actually required a fundamentally different interface or query language that did not exist. So today, the way that most of these agents interact with systems are through the APIs. Through the MCP or CLI or through whatever, a generic API, right?

But when you think about what does an agent actually want out of a knowledge system, it actually wants to be able to do things, for example, like tell me, "Hey, instead of giving me the chunks or whatever the raw data, I actually just want the answer. " Think about the example I always give is the same thing with Google's webpage. We don't look at the 10 blue links anymore.

We just want the answer upfront that actually answers the exact need, versus us going through all the data and then figuring out the answer ourselves. But you also want to do things like, don't give me back a paragraph response. I actually just want the exact structured output or structured thing that I need.

I don't need a paragraph of prose. Just give me answer and company or something like that as well, just side by side next to it. Don't give me anything else.

So that's a structured output concept. But there's also other things that are very specific to agents that have not been expressed in other query language. For example, hey, I want to be able to talk about grounding.

I need to understand where this information came from, where are the citations coming from, so I can make sure I'm not hallucinating, which is a very unique agentic kind of capability or restraint that needs to be represented. And so that's ultimately where we came into this new concept of knowledge query language, NoQL, which is how does an agent declaratively define all of these concepts, in a way that's not vague or confusing or using natural language that can be interpreted in multiple ways. It's in the same world of how SQL and GraphQL came about.

Yeah. And being able to actually declaratively demonstrate what you want, and then ultimately then having an engine that will actually be able to execute that query language itself. So our finding was that we needed both the interface to be actually changed, as well as the core engine to accept that interface.

And those two things together ultimately are the things that allowed us to drive the efficiencies that we see with Nexus. I love it. I think the lesson here, though, and you touched on it a little bit earlier, is moving beyond just RAG.

Yes. As we get more deeper into this agentic AI future, we need products like Pinecone, solutions like Pinecone- Yeah ... to move beyond just RAG.

Again, that's 2025, right? I want to go somewhere, Jeff, and we didn't really prep on this, but I'm interested in your take. Had another interesting conversation with a CEO this week- Mm ...

around, they call it the autonomous enterprise, what it's going to take to make a really autonomous agentic enterprise. And one of the pillars of it was too many organizations spend too much of their IT budget on keeping the lights on stuff, sort of SaaS-based product, monthlies, and it takes 75, 85% of your IT budget is on keep your lights on SaaS. Sure.

Keep doing what we're already doing is 85% of my budget. And a lot of it is because a lot of these SaaS-based licenses are, they're not based truly on consumption. They're based on how many seats you have or how many CPUs you have, or whatever the number is.

And he said, in this new age, organizations need to cut that down. Spend 40, 45% on your existing software, free up money to spend on R&D and new things and on growing the business rather than just keeping the lights on. Pinecone is offered as a SaaS, right?

And so are you guys seeing that at all? Are you reflecting that in your pricing? How is this whole new way of doing things going to change?

People talk about SaaSpocalypse. It's not an apocalypse. I'm talking about a license model change.

Yeah. So Pinecone itself has always been, while we are offered as a service, I would say, obviously, we're more infrastructure, so people- Mm-hmm ... build on top of us to obviously have their offerings.

So I would say it's not quite SaaS in the traditional sense of- Right ... productivity capability. But what I will say, though, what we have even been doing ourselves is fundamentally rethinking what is necessary for us to run the business.

I'll give you some key examples. For all of our business analytics, if you think about our dashboarding and all the classic business analytics, we've actually started to build that in-house. It's actually powered, we have what we call DataBot, which is essentially a way for us to ask natural language questions over our BigQuery data.

But that is using, by the way, Nexus as a semantic layer on top of our- Really? Great ... BigQuery data.

But that allows us, essentially, we very, very few dashboards now, to be honest. Almost everything now is just you ask it a question, and our knowledge, because all the knowledge is within Nexus as a semantic layer. It has all the things like, oh, when you say ARR, I know how to calculate ARR because this is what we've done in the business.

And it has all the context like, oh, right, and you also have a dot time lag. So you always want to, with a SQL query, always make N minus 2, because there's a little bit of time lag before everything catches up. So we have essentially, we've been even ourselves transforming our own business operations- Yeah ...

using something like Pinecone Nexus to fundamentally change. And we are very quickly no longer going to be heavy on dashboarding. We're going to remove ourselves from a lot of those things.

So a lot of the efficiency itself is by fundamentally rethinking our own business processes. We live it. We obviously dogfood it.

We actually are the one to transform ourselves because these are the types of things that you can do with Pinecone Nexus as a first place. And this is whereUltimately, I do see the transformation happening in the industry, where you're now able to really fine-tune your actual application and your needs by using agentic workflows that are tailored to exactly what your company wants, versus what is an arbitrary decision, a bundle of features that someone has decided. So in many ways, that's kind of where I see is, for us, it's not so much a SaaS apocalypse for Pinecone specifically, but rather I see us enabling so many more companies to build the fundamental agentic infrastructure themselves to actually redefine how they operate and run their business.

You think anyone's going to cry over a couple less dashboards? I know I could do without some more dashboards, that's for sure. Right.

Hey, Jeff, last question, because we're over time already. Nexus is available now. People can go to Pinecone and check it out?

It is currently in early access. We'll send a link as well if you guys are interested. Obviously, we'll be moving very quickly.

We're still accepting additional entries if you guys want to apply for access during early access phase right now. And of course, we'll be moving into public preview shortly in the upcoming months. So, I think other than that, we're really excited to kind of show everyone and really excited to talk more.

And of course, reach out to us if you guys are interested and have some sort of use case you guys want to explore with Nexus. More than excited to talk to all of you. Love it.

Jeff, man, you're up to your ears over there in this stuff, I know. So I appreciate you taking time to come on here. Keep it up.

Keep us posted on what's happening new with Nexus and everything else at Pinecone. We'll talk to you soon. Absolutely.

Thanks, Alan. Really appreciate it. Thank you.

Jeff Zhou, Pinecone, here on Techstrong TV. We're going to take a break. We'll be back in a little bit.