Battle of the Bots – Which AI Assistant Delivers with Calvin Hendryx-Parker

I'm excited to be here with you all at AI Field day seven. So in April, I did a version of this talk where I actually went and took five different ag agentic AI tools and demoed them live to a crowd. Uh, that takes about an hour, hour and hour and a half to go through this talk.

Will a follow on to that, that actually tells us about everything that's happened since then, which I can tell you is a lot, there's a lot going on in the AG agentic AI world. A lot going on, obviously in the model world and in the, in the coding, uh, world. So we're gonna take a, a trip down, basically a state of the AG agentic AI Union.

Uh, feel free to stop with any kind of questions along the way. And, uh, this shouldn't take again, the two hours to do, or I, I'll, I'll actually show you a couple cool things as we did this. Uh, but why is this even important?

I, I think this, this stat's actually from 2024. So by this point, we're almost two years old on the state of the AI report that came out from GitHub. The 97% probably now is about 110%, uh, which is not even possible, but I'm sure there's not a developer on the planet who isn't using, who aren't using these kind of tools.

Uh, it is just so prevalent right now that that's why we have to be talking about it. Uh, the, the AI assistance are basically transforming how we operate as developers. So I, I really get excited about the developer experience as part of what makes me who I am and what I love doing.

And so this journey is about choosing the right tools, helping everyone understand what's going on, understanding there's not really magic under the covers. This is a tool that's gonna augment human capabilities and make us all hopefully, uh, better real world developers. I added one to the list.

So originally I had done, uh, er, Claude Code, cursor, goose, and Juni, and I added, uh, OpenAI Codex because it actually appeared on the scene after I gave this talk back in April. Uh, now there's obviously many contenders who are not on a list. I didn't include Windsurf, don't have Devon.

There's a bunch of other players out there in the market. I picked this grouping, uh, mostly because most of them, at least a good number of them on this list are open source. So there's, there's open, the clients themselves are open source.

They may be talking to proprietary models. Some of 'em may be, may allow you to bring your own models. So I will, I'll kind of go over that specifically as we, we talk about this.

So again, what makes this an AI agent? Uh, I'll briefly touch on this, but it's really that it perceives environments and tools via APIs. Uh, we're gonna talk a lot about MCP, and I'll go over quickly about that too.

That it can make decisions and work autonomously. Uh, some of them can be semi-autonomous or, or wait for user interactions, but the idea being that they could just go off and, and keep going at it. Uh, they've got tools that not only can talk to APIs, but those tools can talk to your own local file system.

They can access the Unix utilities that you have in your file system, for example, and make decisions and, and explore the file system and understand the world around it, and try and bring that all in. And then make decisions on how to proceed with the requests that you've been giving it as part of the prompt. Yes.

Can you, um, uh, further define or explain what's two words on this slide mean? The first one being perceives and the second one being learns. So the perceives is gonna be its ability to discover what's on the file system, what you're attempting to do, what, um, MCP servers may be available to it.

And that learns, I guess, is more of in the context of the session. I can take the output from other commands and you incorporate that into the context window. So learning isn't so much like training a model as it is.

I'm enriching the context window with additional data about the task at hand. What's failed, what's succeeded? Did the test pass the agents themselves?

Read that in as part of the, the experience depending on which agent you're using. Uh, some of them do better jobs other than others. Do you control that explicitly?

You can, uh, depending on the agent. Uh, now some of them will, again, will give you the ability to say explicitly run this command and include the output of the command as part of the context window moving forward. Some of them will do it without your, uh, kind of unsupervised.

They'll, they'll, they'll continue very hard to make this go. Like, for example, cursor will try very, very hard, uh, to make things succeed. Uh, whereas some of the tools like aid or chat will ask you, do you want to include the output of that last command run in your context window, or do you wanna ignore it?

Uh, uh, think of one of the big differentiators between a lot of these tools is gonna be how they manage the context windows. And I'll go over that as I talk about each tool and kind of hopefully highlight at each tool where it's got kind of like where one of 'em shines and where one of 'em may, may not operate as, as efficiently. So that, that's part of the goal here too.

And while We're talking about the vocabulary, can you just define MCP? Alright, there you go. So the next slide was literally, what is MCP?

Nice. So for, for a lot of these age agentic tools to work, we needed a way to access tooling or APIs or sources of data. And MCP is basically fronted by our been put forward by Anthropic.

So the creators of Claude and Quad Code, they're the ones who also started the MCP standard, which is a model context protocol. I mean, in this diagram, you can see here, uh, some MCP services run locally. Most of them are running locally on your local laptop and maybe accessing data that's local.

They are MCP servers that have capabilities that can go beyond your laptop or beyond your data center and out into the internet world and access data in other places. Uh, some examples of this, for example, AWS has made all of its documentation available as MCP servers. So if you're dealing with cloud formation templates and trying to create specific kinds of infrastructure, you could enable that CloudFront and MCP documentation server.

And now your, uh, your AI or agents can actually ask questions to the documentation that would be relevant to the task at hand and incorporate that additional information back into the context window as you're creating, uh, for example, cloud formation, uh, you can use playwright, uh, which is a, a, a driver to, like a browser to browse the web in an, in a actual graphical user interface. You can control the browser, you can grab the content and the context from that session. You can summarize web pages, for example.

You can tell it to, uh, put something in a search field and click search. So for example, if you wanted to go have it scrape LinkedIn, uh, as part of your agent AI workflow, you can say, navigate to LinkedIn, search for Calvin Hendrix, Parker in the search field. And then it'll, you know, it'll do the searches and bring back the, the actual dom uh, into your context window if you wanted it to.

But you have to be careful. You're, again, flooding your context window with potentially lots of noise. Uh, from that, there are other cool services out there that can turn those pages into straight, which makes it easier to, to consume into that context window.

The last one on this list as an example, there's tons and tons of these, uh, MCP servers out there is a Serena language server. This is awesome because it takes your code base, looks at it, uh, from a, a code like language server perspective, and can actually give you answers about the structures of the code, the kind of languages you're using. So it should, it's like if you're thinking it like an IDE in your ENT workflow.

So you know how, if you're using a traditional visual studio, IDE and you start typing code, it starts auto completing based on the knowledge of the code base. This gives you that same capability inside your Agentic workflow. And that's all just built in as an MCP server.

It runs locally, it never talks the outside web, but basically builds a, a model model of your code base and an understanding of your code base and an understanding of the languages you're using to be able to, uh, augment the AI's capabilities when talking to code. Can I ask you a quick, quick analogy question? Yes.

Sorry, I've been waiting patiently. Um, so I start with a physical or virtual server running IP based services, right? I I have an FTP server.

Um, I might have an SSH server, right? Um, I might have a web server right? All running on well-known ports and all clients know how to get to those ports.

Yes. Uh, this to me seems like this on steroids where I can define anything as a service, and I'm not using any well-known ports. I just need a mechanism to direct to the right services To the ai.

That's how it looks. Okay. So, so to the AI MCP is, uh, appears as natural language.

Okay. The, they're like function calls, but they would be like, for example, I use the fireflies MCP server all the time to grab transcripts, uh, from my previous meetings. Sure.

And I say, grab the last transcript, uh, from a meeting and, you know, let's talk about it. And they'll, it'll do a search because it has a function. Mm-hmm.

It's like search transcript. Well, it's, it's written in human English as opposed to like a, what would be a function call on Python, you know, which looks almost English, but it's not English. And though those function calls will have descriptive text along with them.

So the AI can decide which tools to pick up and which tool function or which tool call to make based on, uh, human language and, and, and semantic meaning, uh, behind it. So that's what those tools this MCP has brought to the table, is a more human language interface to calling APIs that it can now understand and use. I like to think of it as basically you're given a chat interface to every single one of your endpoints.

Okay. Yeah. Pulls up.

Yeah. And, but you, you then, as you as an MCP developer, I would develop those capabilities. What does, what's possible, describe them in a way that the LLMs can access them.

It still comes down to prompt engineering and context engineering. 'cause you don't obviously wanna not flood people's context windows, again, with, with maybe noise. So developing an MCP that performs well is also kind of a skill in these, in this this day and age.

So I mentioned your name in practice. I already talked about some examples, like we may use the AI assistant to get AWS documentation that can be embedded in your editor. Like if you're using Cursor or if I need for example, using er, uh, those different tools can now access that data and add it enrich and enhance the context window as part of the development process.

So I'll go through each of the tools with a quick introduction, talking about some of the highlights, uh, from each of them. The first one I love, uh, that I started, actually, this is one of the first ENT tools I started off with as this kind of revolution started was was eight or chat. chat.

This is a open source unbased, agen, uh, AI tool. It does support multiple models. You can bring your own model as well.

That's probably one of its key features. Uh, so all each release will provide support for like, say G PT five. So it's got full support for GPT five and G-P-G-P-T five codex.

But you can also bring support through third parties if you're using, for example, um, uh, right now one of the, well, I'll go, I'll get that on the next next slide. You can actually bring your, bring a local model like llama models if you wanted to not actually have the data leave your own like laptop or workstation. It is semi agentic.

And the reason I put that in there, it will ask you probably more questions than some of the other ones would if you put them into kind of dedicated dangerous mode. Uh, this, this one kind of airs a little bit on, uh, let's talk about it before we go implement it. Uh, but it does have two nice features, or at least two nice modes, uh, where you can use architect mode or the code mode that architect mode's kind of a planning mode.

Talk about how I want to implement something, propose, propose some possible ways of doing it to me. And then I'll drop down into code mode and actually have the coding ai, uh, do that, which brings up, you can actually use multiple models in the same session. I can have one model, for example, I maybe, I think deep seek R one is a better, uh, architect model.

And I want to use Claude Sonnet four five as my coding model, because I think, or Haiku four five as a coding model, because it may be quicker and more lightweight, uh, for different, you know, levels of token usage. You can tune that experience based on what your ex, your experience are with these various models. The reasoning models may be better at architecting 'cause they're thinking through it a little deeper than the, uh, standard gener, uh, generative ai like chat style models.

Or you may be bringing a, a coding model specifically to it, like a dev stroll or GBT five codex, for example. Uh, it also has a voice integration. So technically you can be using three to four different models simultaneously in a single eight or session.

So it uses GT's Whisper or open as whisper model, um, behind the scenes. But you can talk to your terminal, and this is all in the terminal. This is totally a terminal based, um, VER agent, and there's no gooey to it.

There is a web UI for it. So you can actually start up er web and it'll launch a webpage. I don't typically use it that way.

I'm not a gooey editor type person. So it's maybe a personal preference. So I don't have a lot of experience with the Web ui.

Most of these tools have started implementing some kind of a web UI that looks like a chat GPT interface to these tools. I prefer the terminal based, uh, interaction with them. Uh, and I'll get a little into that more as I talk about, uh, cursor, for example, since I gave this talk in April, uh, it's added in support for all the latest models.

Uh, that's all up there is also added in the ability to tune the reasoning, uh, capability. So if you're using a reasoning model or model that supports reasoning, you can give it, uh, effort or token quantities to kind of constrain it a little bit. So you can tune that experience of how many tokens it's gonna try and burn before it gets into a a, a good state with whatever you're asking it to do.

Uh, you've got, uh, the user experience has been, uh, fixed, has been enhanced a little bit there with the context and editor. So it can drop you into like a VI style or EMAX style, uh, multi-line editor mode, uh, in inside your shell. One of them, I, I am, I kind of forgot to mention this on the previous slide.

One of my favorite features of the ER chat agent is its height integration with Git. Uh, of all the ones I'm reviewing here, the ER chat has probably the best integration with, with Git natively, in my opinion. It's undue function.

For example, when you do slash undo, literally rolls back a commit. Uh, it uses, it tracks all the history as commit, and then it's really very gi aware. It has a slash gi um, command built into it.

You can turn it on for like auto, uh, auto committing, uh, but you can also have it do get commands for you and read that, get history or get output into the context window depending on what you're trying to do. So if you're looking for one that really interoperates with a kind of a get mindset like atomic commits and thinking through what is gonna like put into the commit messages and conventional commit messages, all those kinds of things that, that works really, really well here. The last bit that's kind of at the bottom, you can use it in the terminal and simultaneously be using your editor edit files.

So if you've got an editor that can edit a file and, and update with live changes, if as something else changes it, uh, er inside the terminal can also be watching that same file. And if you give the file, if in the file you put specially formed comments like, AI, implement this for me, er will see it and then take action on the side. So it's almost like having a coding expert on the side.

If you wanna stay in your editor and actually edit, uh, through your normal text editor, it can be editing or the aider chat on the side could be actually working on problems as you comment them inside the code. So Yes, Um, this is a little bit different than the tools like Cursor that I'm super familiar with. And this is this the semi agent nature that you're talking about because, and then I think it, there's different target audiences for this.

There's me who I could care less about the code code because I'll Never make something happen. I'll, I'll never be in the editor itself and I'll always be in the chat versus someone else who wants the, uh, you know, let's say you're a, uh, you're primarily a coder and you need network ports opened and that has to happen. Is this where eight or steps in and says, okay, you, you have the code and you need to do something in, in your Google Cloud, uh, compartment.

Let me go out and do that for you. All the tools have that capability mm-hmm. To say, do you know, understand maybe I'm using Kubernetes, I need to open ports or forward something back or run a Cube CTL Exec Command to be able to, to do some part of the operation.

The, I think what you hit on is definitely the user experience piece of it, right? Uh, I, I almost believe the opposite. I don't like cursor experience because I, I am, I, I don't care to see the chat window and a thin bar on the side.

I like having a full terminal window open that I'm having a conversation in and can read, you know, full, uh, like conversations with the AI that I, I'm reviewing. But I, I actually like the ER chat. You, er, goose and Claude Code and Codex all kind of follow the same terminal ui.

You'll, you'll see the new term TUI, you know, so that's, you know, the COIs are basically now turning into the T UIs, which are these very fancy, you know, graphical interfaces in a terminal. 0, which which Hasn't CLI yet I, I found that the, the comparison to the two, you, you hit the answer right on the head. I don't think you realized it.

You don't care about the code. Mm-hmm. You don't care what's being written.

You don't care about the file system, you don't care about the tree, you don't care about the cps. You want to focus on what it needs to deliver. That is what a terminal is for.

If you care about the code or you care about, you May want it in your id. Yeah. ID is what you want.

So the, uh, a great example for the, like, the thing that I've struggled with the most, it's the same thing that we all struggle with, is the not having access to an MCP server. You're dealing with these clouds and the SDKs are moving way faster than the documentation that I can find. Yeah.

And, uh, this solves both problems from when I'm working from the terminal, if I using the terminal to talk to the cp Yeah. That has the documentation then in theory, And so this, so er may not be the one for you, uh, because ER actually does not have MCP support of all the ones that are in here. There is a wrapper around ER and some community support for it, but officially, ER does not have MCP support er can read webpages.

You can hand er a start getting started page for your favorite dependency you just added into your project and it'll now know how to do those things, but it doesn't have an Mc P capability built into it. So that's why I'll, I'll, as I go through, I'll try and point out some more things that are kind of developer friendly. I just need to get something done friendly.

Uh, and, and I think that hopefully should help answer the, that questions. Uh, I'll keep moving here 'cause there's a lot to go through. Claude Code is developed by anthropic.

It is a closed source, uh, terminal UI based, uh, agent. The latest version of it is gonna be using the latest versions of the models sonnet four five KU four five and Opus four one. Uh, again, it's proprietary models and proprietary tooling.

So people reverse engineered it and, and forked it into some what looked like open sourcey things. But I wouldn't touch that stuff with a 10 foot pole. I, I think you should trust what an mean, not implicitly trust what Anthropics putting out there, but they're probably the most authoritative source to be delivering, uh, this version of tool.

It is fully ag agentic. It has the planning modes, tools, modes, subagents. Uh, that's probably one of the newest features that most of these have gained over the last few months has been subagents.

You can now, if you carefully craft your prompts, spin up multiple, uh, versions of the agent in one session to go after multiple tasks in parallel without having to leave one terminal window. It used to be, if you wanted to do this, you would have to spin up five, you know, terminal windows or five cursor instances to be able to have multiple agents running, kind of like my DevOps agent, my front end agent, my developer agent. You can now do a lot of that without actually leaving the same terminal window.

I did notice that, um, subagents came very, very quickly with all of them. Yeah. But they all varied in adding the task manager to handle it.

Yeah. And so some of 'em, you very specifically had to add your own task manager. Yeah.

Um, Well that's why Ruben Cohen had the early start on the cloud flow stuff is that he basically went out there and created the orchestration engine and then Claude went ahead and Put it in. So obviously cloud code's gonna have MCP support. It started off with MCP support since it's anthropic, uh, it does ship as a terminal app.

There are some native vs code extensions, uh, for CLO code as well. If you want to use that inside and have kind of a more cursor like experience with, uh, cloud code since April. Again, it was the new models, uh, the new VS code extensions.

There's some new commands. Uh, there's a rewind command, which is kinda the equivalent of the undo from, uh, ER chat, which basically will take you backward in history. Uh, you can get more usage like token usage chat because you can consume a lot of tokens with these, these tools.

Uh, there's new skills, uh, which skills are kind of like encapsulated little prompts of, of information. Goose has a similar feature called recipes, which I think is a little more, uh, mature in this way. Uh, then you've got the Sandbox Options and hooks.

That's probably actually one of the more interesting ones in here is with Claude Code, you can actually hook into the event process that's going on in there, like before Prompt, I think I, I think in the notes I put down a little more information on the link to the hooks, but there's an event life cycle going on as I type in a prompt, as I hit return as a user for the prompt, as the AI returns, I can hook into many of the events that are inside the, the session protocol that's going on and actually have it call deterministic code, for example, as part of this process, or spawn sub prompts or subagents to go do specific kinds of tasks. Uh, one of the things that AI agents are very bad at is if you tell them, don't use comments in my code because I don't care about comments in my code. I want just the clean code.

I don't want all the extra noise. They will not obey that. Like up and down.

You can tell 'em all day long you want to, but you can use the hooks to run an a post prompt basically to remove comments or run a post, you know, script that removes comments. And that's probably a more reliable way of actually achieving that goal of having no comments in that code, for example. So from a practical sense, I, I just started using Claw in general this as week, the talks talk about the, the token limits relative, like what is the community's ments on the token limits of a claw code versus a what I'm more familiar with Cursor, which seems to be unlimited in comparison to Claw.

So it's not unlimited, it's not. But, So each, each of them are based on the context windows of the LLM models you're talking to. I could give a whole nother talk on context window management.

But these tools right now in the current state, actually probably since April, have more context window management built into them where they will auto summarize or auto compress, like there's a slash compressed back, you know, command built into these now. Or if you get to 80% of your context window, you can configure it to auto summarize or auto compress. And so it'll try and pick out key points in the last, you know, context window.

You've got stuff those over into memory, again, refeed them back in as part of this, the session. And that's what Curs cursor under the covers has been doing for you all along is it's been trying to do this context window management for you and kind of like leaving it off to the side. FOD code, er, goose, uh, give you more explicit control over it.

And some of 'em have automatic control, like Cloud code and Goose. Both have very automatic controls where you can set a threshold for automatic summarization. Uh, and there's also slash commands in each of these for clearing windows.

You may wanna be in a, in a session and you may know that you're heading onto the next feature you may do slash clear, clear the context window, start over with say, read an X, Y, Z file and now we're gonna attack this problem. And now you don't have all the muddiness of the previous context, you know, hanging out in there, but you're still in the same session. You haven't had to exit the window and you can kind of keep in the flow.

Which Brings me to the question around sub agents are the, and the subagent basically have completely different contexts than the Yes. Than from each other. Subagent at least, I don't know.

I'm not sure in cloud code that, for example, skills which are basically treated like tools, they can also have their own clean contact that you've, you populate as you like, like function calling. You tell it what information it needs to know to do in its past, and it hands back to you just the results and doesn't muddy up your contact window with all the things that happened inside that subprocess. So if I wanted to just manually it process, Yeah, you could put into a recipe or a, in this case a skill.

Uh, if you had a skill, like you have a specific process for, again, maybe GI management, you know, maybe you're squashing, committing, merging, pushing, that can all happen outside of the main flow and not, again, muddy up the target of what you're trying to do. So Cursor is probably my least favorite on the list, but is, that's mostly because I'm a terminal person. Uh, it is, you know, an independent company is proprietary editor based on open source components.

So it's vs code that has been, you know, made basically proprietary. It supports what models it has outta the box. Uh, you get a full agent, uh, with project navigation.

So it will, you know, read the tree of files and try and understand the structure of the project. It does support, uh, MCP some new things in Cursor since, uh, April, the VMs, GitHub, uh, A-I-I-A-M integrations. There's a bunch more of enterprise level authentication support for like OAuth and, and authentication pieces that are in here.

Uh, the agents md most of the tools are moving to an agents md, uh, con like context file for configuring the conventions of your project. So that's something to kinda look out for. Uh, they also have the dot cursor directory where you can put your rule files in there.

I'll talk more about those in a second. Uh, it has a agent ask and plan mode so you can actually review, plan, do a plan session and review it. That seems to be a, a new feature.

Most of them are all, uh, tending to spawn. Uh, Memory is another one that they just added. They added with that, yeah.

0. But it's there, which Plus multiple agents running simultaneously with the ability to, to Manage. Yeah.

0 has, you know, like the eight update subject. Yeah. Yep.

So in the headless CI thing, you can also run in the cloud. They've got the, the in the cloud versions of it. So you can run agents in the background that are not, Or spend your limit quicker.

Oh, yeah. You can spend a lot of to very, very quickly. Uh, again, I, it's, it's a great tool.

It's just not my favorite, um, because of the UI aspect of it. And I, I just, I, I want to have the Open Terminals text expansiveness to, to work in for myself. Uh, goose is probably my new favorite right now.

Um, I, this one's developed by Block the people behind Square or we will rename themselves from Square. It's open source. Uh, so you can go check out the code.

You can actually like see what's going on behind the scenes. It's very active. There must be three or four releases a week, uh, on the GOOSE project.

Currently. It does support multiple models. So very similar to, um, aid or chat.

Uh, one of the things I forgot to mention about Aider Chat and when I got into the new improvements was, uh, open Router has become supported by many of these tools. So if you have an open router account and you've got access to, you know, tens or hundreds of like different models, you can bring all those to play inside of Goose. It is fully agentic, uh, with the integrated tooling.

So it's got some NCP stuff built into it. And it supports also open route outta the box. The new things are recipes and then subagent.

The recipes are like skills. These are like tools that you basically can kind of prompt and build up. Uh, it does have the ability to integrate with containers, so you can actually get some isolation on your local running machine.

So if you're kind of worried about what these MCP and Agen tools may do, it can actually spin up those, those commands inside of containers themselves to hopefully limit the blast radius for what can happen inside of there. UNI is by JetBrains, uh, it's built into the ides. So you have to have kind of a paid subscription.

This is probably the most restrictive of the ones I've, I've got up here. You kind of already gotta buy in on the IDE and then you gotta buy the AI extensions to the IDE to enable, um, the full Juni operations. They've switched up.

When they first launched it was Claude, uh, under the covers. Now it's GPT five and GPT five Codex by default. Uh, again, fully ENT project navigation, much very similar to Cursor itself.

Uh, it is now public and EAP as of the last talk. It wasn't, it was a private closed beta. Uh, it does have now MCP support.

It did not have MCP support in April. Uh, it does support Mac, uh, Linux, but not Windows yet, unfortunately. And then they've been working on a lot of performance pieces there.

The last one I added, which was Claude Coda, or sorry, Claude Open AI Codex. So many of these tools, uh, this one's developed by Open ai. It is an open source CLI, uh, wrapped around their specific models.

So you've got the choice of any, any OpenAI model you like. Um, it's, it's available in there. It does run locally.

It can read and modify. Uh, it does support, uh, MCP, uh, and it runs on all the major platforms. 'cause it's just gonna use the Linux subsystem for Linux.

Uh, if you're gonna run it on Windows, the big deal here in OpenAI, it, it's probably not as mature as some of the other ones are 'cause it's relatively new to the market. But it does have the cloud S-D-A-S-D-K, uh, options here. You can talk to it from code programmatically and use the, the Codex, uh, agent, uh, to do your bidding from your own application.

So you Claude and, uh, open AI's Codex and both be built into other kinds of applications and not just be a developer tool only. So I got a quick tool comparison. The thing I really wanted to get to was this, the, uh, agen coding models leaderboards.

So both er, and I'll talk about another one here have, uh, leaderboards that they run on the regular with some, um, eval benchmarks that they run. And you can see down here, it didn't show it, but the next one down is actually, um, oh three Pro. And that one takes about $170 to run that benchmark.

But they're running these on the regular to basically evaluate. And this is a polyglot benchmark across multiple languages. How accurate are these AI models getting to what is considered like a standard, uh, output for that, that version of the model?

Uh, this links ranks a whole bunch of different models in here. You can actually see it where it combines different architect models, but different coder models. So I'd recommend checking this out and kind of washing this as the LLMs gain different levels of maturity.

Another interesting thing, this is news from yesterday. Uh, jet Brains just released this developer productivity AI arena. Uh, and they're working with other vendors in the market.

You can see here we've got, uh, Claude Codex Uni and Gemini, uh, listed up here. But they're encouraging other people who are building ag agentic AI tools to submit their benchmarks into this same set of tooling. All the tooling, all the methodology, all this is open source as well.

So if you go and check that out, there's a GitHub, uh, that you can go to if you wanted to participate and kind of dive deeper into what they're using for their methodology for this. But you can see like they're having these ais run for multiple days straight, uh, against certain benchmarks that are going on in here and giving you their, um, uh, various scorings and costings. But if you're kind of wanting to track what each of the, the models under the core are, are doing, this one may be a better, uh, benchmark to look at if you want.

Because each of these are affected by the context that's given to them. The Quad Code adds into the system. Prompt Goose adds into the system prompt.

Each of these has their own flavors of, uh, system prompts. They're adding onto whatever you are telling it to do. This is gonna be a better reflection of how these tools compete when you're using a specific Agen AI with a specific model.

So you may see this list grow. Have Codex mentioned multiple times about different versions of the model with the Codex version and a non Codex version, for example. Uh, I'm not gonna get into the demo 'cause we, I did mention it takes about an hour from here to do all six of these tools to demo it.

Uh, but there are, in each of the demo some notes about the best usages for these tools. So if you wanna check out the slides later, you can see like how they specify the conventions, for example, ERs, users in conventions, markdown file, uh, Claude Code for example, has really, um, granular control over what tools on your own system it can use. So you can actually allow and disallow, uh, very easily what which things are it's allowed to do.

And let's see if you're of the other ones that are in here Cursor actually, well, the other thing, I know I kind of pooed Cursor, cursor just a little bit. One of my favorite things about Cursor actually is this directory that is version controlled full of rules. And you can have multiple convention files in that directory.

And with a glob you can say these for Terraform files, you are a DevOps master, blah, blah, blah, blah, blah. Perform actions as if you understand infrastructure. Use this MCP service for getting documentation.

If it's Python files, you are a senior Python developer. If it's, you know, next JS files, you're a front end developer. So Cursor gives you the ability to actually have file specific inventions that can get applied where most of the other tools rely on you specifying that in one single file.

So you can actually get a little more concise, uh, in your context window as you're using these various tools. So that's one of the superpowers. I think Cursor has new features to generate cursor rules.

So if you don't have rules currently in your project, like many of these tools you can type slash a knit or slash startup or whatever cursor just added. Now the ability to have it generate rules, it will analyze your current code base and write a appropriate set of tool, uh, rules for you. Same thing with cloud code.

You, you just type slash a net. It's gonna look at your code base and be like, oh, you've got a Python project with a Django back in it. It's running in Kubernetes and, and it starts writing these things for you.

So you can help give it some more guide guardrails where the whole thing died. Go end Your agent Croaked over notable mention, by the way, if you like Rust, is to use Zed. Zed is a entirely rust based ID for agen coating that is Z-Z-Z-E-D.

Yeah. And they, they do a really good job. It's super fast.

It looks like cursor, it acts like a cursor, but it is completely organic and it's all in rest. Yeah. So that's actually one of the things with, uh, goose, goose and Zed have implemented, uh, a CP.

So you've seen MCP as part of like the, the model context protocol. A CCP is gonna give you this agent control protocol where you can actually have multiple agentic tools talk to one another inside the same session. So if I was using Goose for example.

Yeah. So if I'm using Goose and I've got Pie Char running at the same time, it will know that I've got pie char running and can be like when I come over my goose window. Did you wanna like use that pie charm tab of whatever file you have open as the context for this chat session so they can actually talk amongst each other using this a CP protocol, which is again, very, very new, uh, to the scene.

Is that ibm, The A CP? Yeah. Uh, I don't know who developed a, I know Juice has been talking about it.

'cause they, they, they implement it pretty quickly and Zed has it as well. I I, I actually as I'm developing two related programs, but I want to keep the Yeah. Development track separated.

I'm like why can't the two agents just talk to each other? Yeah, Yeah. Well There was the A DA, which was Google stuff, but that's more on the, uh, as agents be deployed, you know, things are going out there.

So yeah. A CP is IBM, right? Yeah.

Or IBM led. Yeah. The, the other couple I got, I've got in here, which were the uni notes and the Codex notes.

I mean the, again, early days, early days for Codex agents mark down file for all the conventions that are in there. Uh, Junie the big deal there is they've added MCP. And if you're already a pie charm user, it's, it's already pretty amazing, uh, being in pie charm and using this tool.

And it just seems to flow very naturally in, in the whole pie charm experience. But it's kind of the, the, the kind of wrap up of this. Like if you ask me which one is my favorite, uh, I can't tell you.

Uh, because I use most of all of them, uh, on a day-to-day basis. I'm constantly jumping back and forth between each of these. Uh, the new one to me was Codex.

I actually had never used Codex until this morning when I wrote a Macintosh app to show Blinky lights. And I, I won't show that right now because it's Well in Codex, to be fair, codex started out as an evaluation protocol in open ai, my Royalty checks. So this morning when, when Carl mentioned, uh, we should have a nap that shows blinky lights, I was like, this would be a perfect excuse for me to pull up in Codex.

And so I literally made a blank directory, I type CD into the directory and type codex, and then I cd into that directory and I typed a knit, which is blank, so there's nothing going on in there. So I wrote some standard stock rules that would guide the development process. And then I wrote it a simple prompt, which was where, might even prompt Somewhere in there.

Yeah. Uh, that was, that was, can you ship me This thing? I can, yeah.

Uh, I got, I finally got like fed up with it. I was like, I don't need anymore. I don't care.

Uh, that's all the agents markdown file still. I, here we go. So it tells me I created the, here we go.

This, this is the only prompt I gave it. And, uh, and you, I watched what I showed. This will be an electron app that compiles into a macros application, build the application that recreates a system administrator's nostalgia for blinky lights at the data center.

The app will be called Blinky Lights. And it output to me this app called Blinky Lights, which I'll move over here. Wait, those are Blinky lights.

A meditative And it's a full Mac OS app called Blinky Lights with an icon. And actually, I had a note, The column heads rack one, rack two. Yeah.

Escape their, uh, So, so you'll see a lot of these tools will make their own internal plans. A recommendation I have for you is they're using these tools, have it output those internal plans to markdown files in the folder so that you can keep track of what it's actually doing. Uh, behind the scenes.

It's doing the same thing in its own memory. But here it built up a to-do list of like how I was gonna build a, a Blinky Lights app. It made a main js, it bundled it all up.

And then at the very end, it actually asked me, well, do you wanna build it? And I said, go for it. And so I'd never left this session.

I let the AI drive the whole thing. 'cause it said, here's your next steps. I said, go ahead and do the next steps.

Why do I need to type stuff in? Uh, you're, you're a robot. You should do it for me.

So I approved Codex to run it, uh, from here on out, it built it, it recognized some errors actually in the build, uh, at some point in here and, and changed up its plan. It fixed the code to fix the build. And then, uh, I told it to generate an icon so it actually installed pillow, uh, but it did it wrong.

I didn't want it to use the global, uh, Python for installing pillow. So I literally interrupted the conversation. I said, use UV to handle any Python package operations.

I don't want you muddying up my system Python. And it's like, cool. I upgraded, I updated the plan and then it, uh, made a directory for the icon set.

I approved it to run this icon utility and it built the icon for me. And then I think at some point, yeah, generat the icon using UV run. And that was it.

Uh, that, that was the whole prompting session to get to a Mac app running, uh, called Blinky Lights. Uh, I'll be posting this in the app store. It'll be nine, 9 cents.

Make some money on it. Gotta charge something. Yeah, exactly.

So, So, so look, I've only met you today. It's clear you're not an average developer. You're kind of a developer's developer and like kind of a big deal.

Kind of a big deal, kind of a kind of a big deal. Um, sorry, very poor Ron Burgundy. And the fact that I had to explain that, you know, shows how bad it was.

Um, what do you think about certain tools becoming dominant? Like, you, you clearly like to try different tools and you don't seem bothered by your workflow being interrupted by jumping around to different tools. Correct.

Um, but from a productivity perspective, we know, um, changing context has overhead. Yeah. Um, who do you think is gonna win?

Like if you were to pick two, which ones do you think will be the most fully featured in the next 12 months? Goose or Claude Code? I think, uh, I, I really enjoy Goose.

I didn't show you, but Goose also has a gooey version of the app. They develop Goose with a mindset. It's not just for the developers.

Goose is MCP for every human being in in the organization. The rest of us. Yeah.

Yeah. For, for all of us. And so Goose's gooey app is meant to edit spreadsheets and open up word and browse the web and, and, you know, work on office type documents in addition to being a coding agent.

And it has the, the, one of the like really, really cool features in Goose, I didn't talk too much about is the multi-agent feature in Goose has the ability to specify a lead model, which can be an expensive model and a worker model, which can actually be a very, very cheap model. Like you can have sauna and Haiku paired together, and if the worker model gets off track, you can, you can specify how many rounds it can try before the lead model kicks back in and double checks its work. So it, it can basically have one model watching another model simultaneously doing the work.

But behind the scenes, as a non-developer, I just see success because it's, it's, it's actually putting a lot more thought into it. Not just reasoning, but also it's actually monitoring itself kind of a, a model, judge model as a judge, um, you know, type situation there. Sure.

So I, I think because they're, they're, they're heading toward the, the gooey market. They're heading toward the enterprise. They're heading toward the office worker and not just the developer.

Goose on the command line is fantastic. I I love it. Uh, I think I, again, it's one of my favorites to interact with.

Although this code experience was not bad. Um, and I had a very small prompting session. I think it was successful because I knew things as a developer's developer, how to make it do what I wanted.

I recognized quickly it was doing the wrong thing, you know, had to interrupt it, tell it no use uv. For example, if I was a junior developer, I may not have done that. And now I would've had a polluted python and that in my, my system.

And now, now future stuff is gonna break. And, and there was all these kind of cascading things that would've happened if I didn't know what I know. And so I, I, I, I'm concerned about junior developers in these tools.

I, I don't know how they're gonna learn some of the craft other than if they were to maybe pair program more often with senior developers, kind of a journeyman apprentice style operation. I think that's where the future, a lot of these tools are gonna be. Bring back the guilds.

Yeah. Well, I'm blessed, I gonna say here was develop developer, because if every developer was I, you, I'd be in trouble. Right?

Uh, I think one of the things I I, um, my guess feedback on Claude, I don't like it. Like I love it from token limits. No, it's not tokens like Claude Code Or Claude Claude Code and the, the, uh, and the LLM that back, it seems more geared towards geeks than they are towards the citizen developers.

Interesting. So my, my, my CEO and partner and I talk about this quite often, and when we're negotiating and talking to other folks, we may use Claude more because we feel like it's empathetic to the discussions and the negotiations negotiations we're having as opposed to like what GPT five may be doing. I think from that perspective it is, but Not from a developer, From a developer's perspective.

Okay. Interesting. Yeah.

It's, it may be a little more unforgiving if you are, if you are more senior at boy, it sings, it's fine. I think when it comes down to who's gonna win, it's gonna be the ones that, it's gonna be anybody that can manage to encompass the user experience from novel to or novice to expert in a way that's meaningful. 6 billion or something like that.

Who gives a s**t right. When it comes down to it, it's one of those situations, like valuation doesn't necessarily net out the applicability or usefulness of it. Anthropic went through their five different changes in terms of their costing model.

'cause people were exploiting and abusing the service once they figured out you could run swarms. Well, they had to readjust. It doesn't mean that they're, yeah, they're worse.

But, you know, to the point earlier, like it's, it's definitely always been designed around a certain type of user. It's never been, and uh, what we would call an accessible interface to the proletariat, right. Versus the bourgeois coders.

Right. But anybody who can democratize that to a level, like you're saying, I I'm looking at the goose gooey, you know, is up on my screen right here. It's great.

It's, it's great. It's approachable, it's usable and it's not the, these type of things, It's not their business model to sell you goose. Like they're in Theran, you know, financial transactions, business and space.

They've just found that this tool is generically useful to the world. They probably built it as a, you know, ship a day type project over our, you know, 24 hour kind of thing. And they're like, oh, this is cool.

We should just release it. And now they went crazy with it because it keeps continually delivering value to them themselves. So there's no, and it's not a threat to their own business model to do this.

Yeah. They're not worried about the curses the world. It's crazy Thing to watch over the past three to five years is to see how fast the AI market has been slamming shims in and expanding and growing.

Yeah. And not necessarily in a negative way, but all of the steps of maturity through the life cycle is, is going so much faster than anywhere else I've seen. And even like what we're pushing through with frameworks, I mean, they're everywhere now and it took less than a year for frameworks to market.

And I think the, the tools are gonna be open and allowing you to bring your own models. The tools are gonna allow you to have local models. That data, your data never leaves to the desktop.

We're heading into, I mean, this is a M four, you know, max, um, Mac, it can run an incredible number of models on there at a relatively high speed. Like when I spin up, um, minstrel small or the Srel model on here and say, develop me a with a single one-liner, develop a jengo app, it gives me a running app at the end of it. And no data left my machine.

Any tool that's gonna work with that, I think it's gotta edge in my, in my book. Like because of the needs of the enterprise and, and where we're going with all this.