Graph Memory for LLMs and the Enterprise AI Stack

Hey, everyone. " My next guest is a friend of mine. We were talking off camera, on camera, but not on this camera.

I think we missed each other for a whole year. The last time I might have seen him was KubeCon North America last year, and if I'm not mistaken, Steven, you were just in for the Monday of that one. You presented on a Monday, and then you had to take off to yet another event.

Yeah. So I would say that if anything, I've traveled more in the past year than I did probably the whole time during, well, of course, COVID-19, but like the 2000 to 2024. There's so many AI conferences popping up everywhere across the world that I think people and developers are trying to figure it out, and they want to hear from the experts.

Right. They want to hear from folks who have done it before, who have the experience, and who have more insight into how it applies for enterprise and business workloads. I agree with you.

I think also last year, though, it was a little bit if Muhammad can't come to the mountain, the mountain comes to Muhammad. So rather than getting 100,000 AI developers to come to some gargantuan thing going on in Vegas or Moscone or something like that, there was a lot of decent AI events. And some of them were 10,000 people, 8,000 people, 12,000 all over the world.

And so it gave, at the end of the day, well, for someone like you, yeah, put a lot of miles on your body, right? Yeah. But for the AI developers out there, they didn't have to go get budget to take a big trip halfway around the world to be in on hearing from the latest and greatest thinkers on it.

So now I think that's going to change, Steven. I think we probably will start seeing some winners, if you will, among the AI conferences, though they'll always be the Unconferences, the Besides, the DevOpsDays kinds of conferences that I think will develop for AI as well. Yeah.

And I think the other kind of trends is pretty much every conference is an AI conference now, right? Well, so is every story we do here . And it's kind of this norming where AI is getting integrated into everything we do.

It's getting integrated into development workflows, it's getting integrated into product launches, it's getting integrated into multimedia, like all the shows which you produce. So I think that we're finally finding out and figuring out where it makes sense, where you get business value out of it, and where it's not as applicable, where it has rough edges, it has difficulty, produces hallucinations, frequently gives answers which are incorrect, has a lot of trouble with doing very good technical writing. So there's a whole bunch of things which it's not good at.

Yeah. But, well, okay, so I think this is something interesting as well. So the Anthropic and OpenAI and all the big labs, they're pushing towards AGI, right?

5 trillion parameters, five trillion parameters, the bigger the better. 8 to Opus 5, it keeps getting smaller and smaller. And we're going to achieve AGI, and we're going to figure out that humans, well, computer humans, like computers which pass the AGI test, are no better at solving complex business problems than we are.

And so that's where you need to look at knowledge systems, enterprise knowledge layers, like applying things like graph technology to actually give more structured contextual reasoning to solve harder problems. Because no individual human can solve complex business problems. We solve it by working together, by having shared systems, by building databases and architectural systems.

And those same constructs apply to how we're going to solve real business problems with AI. Steven, I apologize. We jumped into things.

I didn't properly introduce you, so let me make up for that. So beyond being my friend, Stephen Chin is also the VP of Developer Relations at Neo4j. Steven was the Java man for a long time, talking about Java around the world.

He then was over at JFrog and talking SecOps and all things around that for a long time. You've been at Neo4j now, what, about maybe four years? Has it been four years, I guess?

It seems like a lifetime, but only a few years now. Yeah. And talking about graph memory and graph database and so forth.

0 is, what I find more interesting is the gap closing, let's say, between Anthropic versus some of the best open-weight models, right? That gap has really shrunk. You look at Kimi and some of the latest, greatest things coming out of China, mostly with these open weight.

And they're right behind Anthropic. According to if you believe the ratings and everything, they're right there with OpenAI and Grok and the best of the American ones. Yeah.

There's a lot of buzz about the Kimi K3, which came out of Moonshot, I think, in China, and some of the kind of frontier models which are very catching up with the Anthropic. But I would say also to take things with a grain of salt, because there's a lot of benchmarking going on where the models are- Well, they're optimized for certain benchmark tests, right? Exactly.

And again, this goes to what you were talking about before. Look, here's the bottom line. You've got a non-deterministic type of technology here.

So if you're going to ask it to do deterministic type of calculations or problem-solving, that's not what it's made for, right? Tigers don't change their stripes. Yeah.

So there's that to it. And then there is, this is like do IQ tests determine how smart you really are? Well, to a certain degree, they do.

Do these benchmark tests determine how good a particular model is? To a certain degree, they do, but you could game them, too, by optimizing. Yeah.

Let me give an example of where the benchmarks don't meet actual what people are doing in enterprises. So as an approachable example that I did for my last AEngineer World's Fair talk, I took a digital twin of my home lab. So basically, I have some Proxmox servers, some Docker containers, some Minecraft servers for my kids.

Probably way more infrastructure than I should have for a home. But nothing compared to a large enterprise and all the servers and infrastructure and stuff which they host. And I did a proper digital twin with standard vector database.

So I used pgvector, and I used GraphRAG, which basically is an approach for storing in a graph database using graph algorithms like k-nearest neighbor or the Louvain algorithm to group communities to kind of pull more relevant information forward. Now, when you give it a small dataset, let's say something which fits in a 200K context window The models do a wonderful job. They don't need structured information.

They can give you the answer straight away because it fits in their envelope of what they can process. When the data set expands to something beyond a million token window, so I have enough data in my home lab where asking it security questions, asking it about versions of software, different infrastructure holes, it no longer can reliably solve those sort of problems at scale because it can't fit it in memory. Whereas if you take a graph database and you do a query and you say, "Okay, pull back all the information related to this domain.

I want security holes, I want these services," and you can, that to the LLM, it can give you a very well thought out and structured response, which is correct reliably. You can see that reliably will always give back the right answer when it's given the right data. But when you give it a mass of unstructured data, it can only reliably do that with basically a small data set size.

So this is where the benchmarks fall apart. Most of the benchmarks, they're not long-running agentic tasks with lots of turns, they don't have large data sets, they don't model real-world problems, and therefore, you can create a model which can solve those problems. And the frontier models do a very good job of solving problems where the data set of the sample size is small enough, they completely fall apart from most enterprise workloads.

And this is where having that semantic knowledge layer, that kind of memory context for LLMs makes all the difference for solving really, really hard problems. It's the only way to get LLMs to align with critical enterprise applications. I agree.

Steven, we didn't do right by Neo4j here. We're running low on time, so I need you to rescue it. Not everyone out here knows Neo4j and what they do and everything else and how to reach them and when to reach them, and that could suck.

Can you give me a little Neo4j background? Yeah. So I think Neo4j is kind of unique that we were doing graph databases way before graph was cool.

And graphs are kind of the underpinning of a lot of the technical innovation. The Google PageRank algorithm, which, I mean, people are all familiar with Google, that's a graph algorithm that's based upon graph technology. Pretty much all of the fraud detection, the Panama Papers, they discovered a lot of the fraud by using graph databases and graph research.

And it's a very interesting technology which is used by a lot of researchers or a lot of PhDs to solve very complex problems. And it pairs really well with AI technology. So I think that it's a great time for folks who aren't familiar with graph databases to take a Graph Academy course, we have free online training, to maybe watch some talks or kind of bone up on the latest advances in using agent memory together with graphs, and it helps you solve complex problems which LLMs are not going to solve.

Even the generations coming out next month, next year, a couple years from now, it's theoretically impossible for them to give good results without the right data. Absolutely. Let me ask you a question for the people out in the back row in the, way in the back here.

They're not AI experts per se. The idea of going to HuggingFace and downloading a model and setting it up to run locally, is beyond them. They love the convenience of just firing up ChatGPT or Claude or Grok or whatever you want to use.

Can they still benefit from using Neo4j graph technology with these frontier commercial models without having to run their-- Not everyone's Steven and runs a lab at home like you do. Yeah. No, absolutely.

And, actually we did this with a bunch of kids for AI Engineer World Fair. So we did AI workshops, got all the kids set up with little Codex instances, running OpenAI, building games. And the great thing about basically AI tooling, so the most important thing I would encourage people to do if they're still typing in ChatGPT in the browser is try Codex or Claude Code or an IDE, which will let you interact with models locally.

And it can automatically do things like it can install graph database technology like Neo4j. You could start cataloging, for example, your shows, Allen, or different YouTube videos. It can do transcripts of them, kind of record all the key moments, and put those in a knowledge base.

And once you start actually curating and building a knowledge graph which stores all that data and provides it in an easily accessible format, then the real power of the LLMs comes out because you can ask them hard questions, you can have them do deep research tasks, and they're not just guessing. Now they actually have the knowledge to give you factual answers back. I love it.

Steven, we're about out of time. Actually, we're over time, but whatever. Hey, I hope to see you in Salt Lake City in November in person.

Your hair has gotten longer since the last time I saw you, I think. But as I said, I hope to see you in Salt Lake in November. But until then, man, keep doing what you're doing.

It's always a pleasure to have you on here and talk to you and catch up with where your head's at on things, so I appreciate it. Yeah. No, thanks.

Always glad to be on the show, Allen. All righty. Stephen Chin, VP of developer relations at Neo4j.

I'm sorry this was maybe a little less structured if you watch a lot of these at home, but it's such are the conversations that Steven and I have. So that's the way that goes. We'll be back.

We've got more Texture on TV coming your way. Stay tuned.