AI Coding Hits the Context Window Wall

Hey guys, thanks for the intro. We're here with Neeraj, who's the field CTO for Blitzy, and we're having a chat about, well, maybe we got the cart in front of the horse when it comes to AI and application development, because, well, we got to ingest the code base entirely before we start just writing code. Neeraj, welcome to the show.

Thank you for having me here, Mike. Excited to be on the show today. So walk me through what's going on here, but it seems to me a lot of folks are just grabbing tools and starting to write code, and they're not really thinking through the workflow.

But from your perspective, what needs to change? I think what needs to change is a couple of things. One is expectations.

People often kind of expect AI to be this miracle worker, which it's not. As smart as AI can be, it can also be incredibly dumb, in the sense that it needs to be told precisely and exactly what you want. And if you don't do a good job of that, AI can do a lot of things, but it can't read your mind.

And so it's going to come up with its own ideas about resolving any ambiguities in your commands, and then it's going to start doing things in interesting ways that may not be what you expected. And that's where you run into issues. Because as these kind of little diversions build on top of others, pretty soon you are so far off your desired course, that the outcomes are pretty much useless.

And you see a lot of that with codes lobbed that people generate with AI. And oftentimes it is, you need to be prescriptive in what you want. So am I achieving that goal somehow or other?

When I talk to people, there's this whole movement around context, and people are trying to wrap all kinds of stuff around the AI coding tool to provide that context. Or should I just load the code base directly into my AI model so that the AI model understands the context better from the get-go? Right.

So excellent question. Context is absolutely important. I mean, everybody has accepted, and this is true, AI can actually write really good code.

There is no doubt about that. People have experienced it firsthand through their ID tools, copilots, cloud codes, codexes and whatnot. The question now is, as you kind of scale up the use of AI to basically write larger and larger parts of your code, and as you work with code bases that get increasingly bigger, does AI understand the system against which it is writing that code?

And the reason for that is something you write today in one place, it may have downstream effects, several files, several modules, maybe thousands or hundreds of thousands of lines away. And that is basically the information that you often find missing when you're going with AI. Now, yes, your idea is pretty solid.

Let's just feed the entire code to the AI, and it'll have all the information. Works great in theory, not so much in practice, because just like people who have a limited capacity for storing information in their brain at any given time, AI also has a similar limitation. We call that the token context window, because typical-- Even if you take one of the best and biggest models from the frontier lab today, those have a kind of advertised token context window of about a million tokens.

But a lot of that is actually used for other purposes, like bookkeeping, like keeping track of your instructions, like keeping track of history, of what has happened in the past that has bearing on the future, and so on and so forth. And so the effective context that you're left with is much smaller. It's about 200 to 300,000 tokens, and that's good for about 20 to 30,000 lines of code.

And if your code base is bigger than that, now you're going to get into trouble because the AI is simply not able to keep track of that extra code. And so you're going to start compressing things, which means they're going to be lossy, which means you're losing information. And so the outcomes are going to start degrading the more and more code that you start stuffing into the AI at this point.

So what is the right answer here then? How should I approach this to get better outcomes? Because I think a lot of people are struggling to figure out right now how much of the code that's being generated using AI tools is actually making it into a production environment.

Right. So the answer is twofold. One is that you essentially want an army of AI agents working together to solve these large problems.

But you also need to make sure that every agent in this swarm of agents that you're working with has access to the right context information at the right time and in the right place in the kind of overall code context that you have. And when you do that, and when all of these agents know that they are working together and they can kind of coordinate with each other, in terms of what they're doing, and everybody has just a piece of context that they need to know and nothing else to kind of clutter their context window, then you have a lot better outcome. How do I orchestrate that?

Because what you're describing is a small army of AI agents, but I personally might not have the cognitive load to keep track of what all those AI agents are doing. That's what Blitzy does, right? And that's why we are here today.

So Blitzy effectively has solved that problem. But Blitzy has gone one step further, and not just kind of created this agent swarm that has a knowledge layer to back it up in terms of capturing the relevant context for the entire code base. And no matter how large that code base is, it could be hundreds of millions of lines, that's not a problem.

Blitzy has also integrated extreme validation as part of their code generation. So these agents, when they're writing code, not only are they generating code, they are also at the same time making sure that the generated code is being tested to function correctly and to align with whatever problem that the human user has asked them to solve. So if I'm making that code change for a particular purpose, for a particular reason, one of my responsibilities as the human in the lead is to also provide what constitutes success, what constitutes a correct outcome for me, and what my success criteria are.

And when I define those, then the agents basically take that as something to validate against. And so they create tests not just for the kind of functional correctness and the language correctness for the code, but also for that behavior correctness to make sure that it's doing exactly what I asked it to do, and it is satisfying the criteria that I had defined for it to be considered successful. And if these agents continue to do this, they will generate code, they will evaluate this, they'll validate that.

" How do I keep the agents from kind of squabbling with each other because they don't always see the same thing the same way, and sometimes they might have even competing missions? Excellent question again. So again, that kind of all-pervasive knowledge layer that I talked about, which we create by reverse engineering the existing code up front, is your friend here.

Because when we When the human in the lead provides that task description to what is effectively a prompt at that point, one of the things that we'll do first is the agents will create an implementation plan. We call that the agent action plan here. So before writing a single line of code, we are essentially going to create a specification for what that implementation looks like, and we'll share that with the human in the lead to review.

And so they can identify if there are any gaps from their vision or if the AI has made any decisions that are kind of unexpected or not aligned with how they wanted things done. So you can revise that plan, you can iterate over that until you are satisfied. Then you can ask the agents to go ahead and start writing the code.

And even at that point, the first thing the agents will do is they will kind of figure out in what order different parts of that plan can be executed. So there are certain things that can be done in parallel. Those will be distributed across different agents to work in parallel.

There are certain things that, by definition, need to be sequential. Certain things have to happen before you can start on the next phase of that implementation. So there are planning agents which will then take that spec and essentially scope out the serial and parallel versions of the different aspects of the task, define what order things get done in, and then hand off those tasks to the appropriate agents to execute.

And so these agents that way are well-coordinated. They don't have to fight against each other. I think there's an expectation that people think that their existing DevOps platforms and whatever else they're using will evolve to provide this kind of capability.

So why would organizations need to kind of move on to something different or kind of shift gears altogether? Because as you well know, a lot of them are reluctant to rip and replace things. That's true.

But here's the problem. The context window, which I talked about a few minutes ago, that's the real limitation. So unless you kind of change your entire architecture for how these platforms behave, that is not going to be addressed.

We have agentic frameworks that are being used by a lot of the solutions out there. So it's not like there aren't multiple agents working together, but oftentimes they have the same kind of context limitation because either they don't have as deep a knowledge layer backing the work that they're doing. Maybe they don't have the coordinated access to exactly the right kind and right amount of context at the right time, or they have kind of a shared context that they are all using together in order to keep that kind of collective knowledge in their shared mindset.

Any moment you have that kind of shared context, not only are you hitting that context window problem, you're hitting it a bit sooner because now multiple agents are all kind of availing of that same limited context window. So there are issues there. The other aspect of this is most AI companies are interested in getting you to use more and more tokens because that's how they make their money.

It's consumption-based. And so the more tokens you use, the better off they are, and that's their motivation for keeping things the way they are. So as you talk to a lot of these organizations that are building software using AI, are there things that they're doing that just make you shake your head a little bit and go, "Folks, I think we need to be a little bit smarter than that"?

I think the bigger issue that I see with a lot of people who are trying to adopt AI and not seeing the gains or the solutions is that it's not simply a matter of plugging in AI tools and expecting results. The entire process, what we call the software development life cycle, SDLC, has to be kind of adapted to be more AI-centric, to be more agentic. And that is a bigger change management problem than simply a technical problem.

Because what AI does is it effectively accelerates the process that it's applied to. And what that means is if you have certain deficiencies in your existing processes that were, until now, hidden or covered by the kind of inherent slowness of a human-centric kind of workflow, and now AI has kind of sped up parts of it. Now you kind of suddenly come to a screeching halt when you reach a stage which was actually problematic and now becomes a bottleneck.

Oftentimes, you'll see that happen with code reviews. AI basically generates code at a much faster clip and at a much higher volume as a result because you can generate more code in the same amount of time. Well, how do you review that code that has been generated?

Suddenly you have this mountain of generated code that has to be reviewed, and that's a huge bottleneck. How are you going to test all of this code that is being generated? Blitzie does all of that validation testing as part of the code generation itself.

But if that's not how you're doing things and you're keeping it in the old kind of sequential fashion where code gets written first and then it gets tested by your QA team, now your QA team is facing a mountain of these PRs or new implementations that they need to test. So these are all problems that now become evident, which were not so before, and that's a process change issue. It's not really a technology issue.

All right. Well, folks, you heard it here. Hey, the issue at the end of the day is that coding is only a small piece of the entire equation, and we basically have now created more code than ever.

But essentially, in a lot of cases, that means we're running into the same bottlenecks 10 times faster, and that may not help you- Exactly ... at the end of the day. That's right.

All right. Hey, Neeraj, thanks for being on the show. My pleasure.

Thank you for having me. All right. And back to you guys in the studio.