Beyond Tokens per Second: Measuring Real-World Agentic AI Work with Signal 65’s Pinnacle Benchmark | Utilizing AI Episode 45

A good benchmark can provide incredible insight, allowing independent comparison of various models and platforms. This episode of "Utilizing AI" features Ryan Shrout, president and GM at Signal 65, discussing Pinnacle, a new agentic AI benchmark, with co-hosts Brad Shimmin and Stephen Foskett. Welcome to "Utilizing AI," the podcast focused on practical applications of artificial intelligence and part of the Futurum Podcast Network.

Every Wednesday, we explore news and use cases of the ways in which AI is transforming enterprise IT and the industries it serves. I'm your host, Stephen Foskett, president of the Tech Field Day business unit here at the Futurum Group. Before we dive into this discussion, let's meet who's on the panel today.

Hi, I'm Ryan Shrout. I'm the president and GM at Signal 65, one of the main developers of the Signal 65 Pinnacle benchmark, and excited to be here. Thanks for having me.

And hi, everybody. I'm Brad Shimmin. I also work at the Futurum Group.

I'm VP and practice lead for data intelligence, analytics, and infrastructure, and I'm an analyst, if you're wondering what I do. You know, I've been wondering what you do, Brad. I thought you came on the podcast- I do every day.

Every day. No, honestly, it's really wonderful to have you all here. And just to be clear, Signal 65 is a sister company for the rest of us here at the Futurum Group, but though it is an independent company.

Ryan, thank you so much for hopping on here. I have been in the IT industry for a long time, long enough to say that benchmarks are like belly buttons. Everyone's got one.

But the truth is, most benchmarks are not all that useful, to be honest. And I think that the challenge that people face, and I'm not criticizing benchmarkers here, I think a lot of them want to develop something that is generally useful and interesting and helps people to understand hardware. But ultimately, it's really difficult to do that and to make something that is generally applicable, that it can be run broadly, and that actually reveals some true insight into the capability of the product being tested.

Now, you and your team have bravely taken on the effort to develop a new AI benchmark. So tell us a little bit more about Pinnacle and what your goals are and where you've gotten with it. Yeah.

Let me start out by just saying your kind of interlude here about the value of benchmarks and what they can bring or what they can miss on things. I've been in performance analysis, competitive analysis my entire career, whether it was as an independent technical reviewer or as an analyst, my time at Intel running competitive analysis labs. And what I'll say about benchmarks is they are a necessary, I won't say necessary evil, but they're a necessary item in how we evaluate things.

You need to have standardized ways with which to measure and compare different platforms, configurations. It could be hardware, it could be software, it could be input and output, just different paradigms. And they all have different pros and cons.

And like I said, I've been doing this for 25 years now, and we've yet to find a perfect one. And the genesis of what we built with Signal 65 Pinnacle was, we started this world of AI measurement, and the toolset that exists for that is pretty limited. Right?

If you look back in through history and time of benchmarks like 3DMark or MobileMark or CrossMark, a lot of marks actually as I named them all out loud. They had sustained life, and the workloads that they were measuring tended to be pretty static. It might be a couple of years before you would have some new benchmark update to Cinebench, right, as an example, one of the popular CPU tests.

But in the world of AI, you can't get away with that anymore. And we quickly have seen going from measuring tops and flops to measuring tokens, to then trying to add other vectors of tokens per dollar, tokens per watt metrics. And then it's now down to how do you measure work?

And we've seen other things out into the world about, is tokens per second the right metric? Is a model grading comparative system, which model has more intelligence, the other one is really interesting. And what we saw was kind of like a gap.

None of those were connected in the right way to measure what is the effective business outcome, what's the effective work completed, for any given AI scenario. And so what Pinnacle attempts to do is provide a marriage between a model grading system and the capability of that infrastructure to get you to a new, unique kind of bespoke metric of correct work measurement, which I think is very new. Yeah, Ryan, as a practitioner/analyst, I suffer greatly from this sort of the evilness of benchmarks.

And they are evil, I think, if you just blindly follow them, or if you're chasing them, even worse, because there is no shortage of said benchmarks, as we saw in the last week with suddenly everyone saying, "AGI is here because we had a model score 99 whatever on CGI," whatever that is, ArcGIS. That is ridiculous, because it doesn't really map to actually pragmatic in-the-field performance. And so I, like you, have been really frustrated, because I'm charged with actually running something that's consistent over time, that is performance, that doesn't eat a bunch of money, and does what I need it to do without being stupid.

And so when you look at all the benchmarks out there, sure, I can kind of triangulate on some qualities like an ability to do tool calling, for instance. But I really can't get a good grip on what the darn thing costs in terms of what I want it to be good at. So I think Pinnacle shows me a lot of what that would look like.

So there's a couple of interesting points, right? One is existing benchmarks that are out there that measure model capabilities or tokens per second throughput and all that, they all have a place, and it all depends on what your focus is as a practitioner. Maybe you are measuring only silicon to silicon.

You only care about that kind of maximum throughput. But what we were targeting was more people who have to make financial decisions on what AI to go implement. " But what we did was the Pinnacle benchmark goes a step further and measures correctness of models in agentic workflows.

So we have real agentic workflows that are happening inside of these sandboxes. The agents don't know that they're being benchmarked. We have some really interesting and unique methodology behind it.

I encourage everybody to go to the website, and kind of read the methodology paper because the idea is basically every workload that we kick off is kind of unique. The answer key is not predefined, so you can't benchmark. You can't have benchmarks or model companies gaming the system this way.

So the scenario is created, the answer key is generated, the agents go through their workflows, and then they are checked against that answer key at the end. So there's not an LLM as a judge that introduces some squishiness into the output. It's not human as a judge, so we can run it very reliably and very quickly.

It is very deterministic, which is something that's sometimes difficult to get into from an AI benchmark perspective. And so with that, we kind of run this tool, and this gives us our model grading capabilities for it. We measure accuracy.

We measure the likelihood of fabrication, of hallucination in these models. We can set up really interesting scenarios like we ask it a question that we know the answer does not exist in the data set that we've provided it. And if it provides some answer, we know that it hallucinated, and we can go back and track and see what that is.

So there's a bunch of really great stuff. We worked with our partners at KamiWaza to kind of develop this harness and this infrastructure to go do it. And then we do measure the performance of infrastructure.

So in our initial results, we looked at servers with H200 GPUs, B300 GPUs, MI300 GPUs. We have MI355X coming on board hopefully this week. And we can look at what is their capability of throughput of these different agentic workflows.

And then what I think is the most differentiated, compelling part is our grading system at the end of this is how many sustained agents can each of these platforms support at any given time. And that is a combination of correct work being completed, the throughput capability of the hardware, all into one kind of unified outcome. So it's been very interesting to see the reaction to it so far.

Well, and I want to really focus in on what you just said, because that is the key for me. The question is not how much can I pour through this thing or how fast is it or how many tokens or whatever. Ultimately, the question that I have as an AI user is how much correct work am I getting done?

And that's what you just said, and I think that that is really the key to understanding the question you're trying to answer. And I think that that's a good question because it really matches with what enterprise users of AI are trying to answer, which is, I don't just want a thing. I don't want a black box.

I don't want a thing that can just, I don't know, pour out tokens. I want work, and I want it to be correct, and I need to know that this work is correct, and I guess as a side effect, that it's not being manipulated or gamed or biased or whatever. But really, how do you answer that?

I mean, the word correct is doing a lot of work here. Yeah. So, like I said, we generate these answers, and we go through hundreds of simulations per model in several different ways of looking at the data sets too, right?

So we might do, I think it's 280 individual bespoke runs of each agentic workflow, right? 1, we didn't have any models that would complete 100% without fabrication as an example, right? And now we're just kind of getting to the edge of that.

So we have the ability to judge when they're correct and when they're not, when they're making stuff up and when they're not. And we've also tried to simulate what I would say are two scenarios of kind of data readiness for these enterprises. One of them we call governed and one of them we call as found.

And governed is this absolute perfect state that doesn't really exist as far as I can tell, where everybody's data is very properly organized, all the file names are very correctly named, everything's in the right order, there aren't dates misaligned, there aren't missing fields. And what we found is a lot of these models do exceptionally well there. Even pretty mid-size models now can do those very effectively, 95%-plus completion rate, which is very good.

Then we get into what we call as found, which is a little bit more like how I organize my life, which is, I don't know, you just throw stuff in a folder, you got your downloads folder, sometimes you drag stuff over to someplace else. Things are going to be superseded. Maybe you've got two file names in the same, one is in the archive folder and one is in the main folder, and how good is the model at figuring all that out?

And those percentages tend to be significantly lower in completion. But the way to get to your point about how do we get to the correctness of it, it's statistical simulation of these real workflows. Like I said, if we have roughly 300 of these that go through and 30 of them don't complete the workflow correctly, right, then you've got a 10% failure rate.

And one of the interesting things that has come through with this is agentic measurement or agentic workflows are mini-step workflows, and any given step can break the chain and thus undo all the good work and tokens that you've spent on all the steps up to that point, which is a really interesting concept. And what we've done is we've built that into it. It's like, okay, your token consumption actually accounts for all of the work that was done that was incorrect, that has to be redone and recalculated as well, which is an interesting part of it.

So I think correctness is inherent in kind of the harness that we've built out of it, and then the modeling of how that translates into a cost per complete work item is kind of like the mathematical statistical piece that comes out of it. Let's talk a bit more about that agentic tool set here, because everyone is building out their own harnesses and their own tool chains right now, and a lot of that is sort of dependent upon how effective and efficient these models are at very distinct tasks. " And it just shows you how bespoke and how actually participatory models are in that tool chain.

And so it's such a difficult thing for practitioners to really measure how fast, good, correct an agentic workflow is, especially as we start to get into these long-horizon tasks that depend upon those kinds of internal discussions, like Astra complaining about Quen. How does Pinnacle look at that sort of, I want to say Wild West, but it is kind of the Wild West right now in terms of how people construct these harnesses that they're using to drive these workflows? I would say, so today we don't do multi-model heterogeneous testing of that, although I can definitely see a pathway that we could go to that where you have kind of larger models that are going to do some of that orchestration and put off some of the work into lower, smaller models, maybe more efficient, more cost efficient.

But it's an interesting characteristic, right, to have one model recognize at least what it believes to be inefficiency in another model and make a recommendation around it. Down that same path, one of the things that we have found is the model's token efficiency vary quite a bit, meaning how many input or thinking rounds, and then how many output tokens do you need in order to get to whatever that correct answer is. 1s and that class of model, these brand-new massive ones, are actually way more token efficient than the smaller models.

The smaller models, older models have to do a lot more thinking, a lot more rounds to kind of get to that completion. And you might see an order of magnitude, I think, on one of the data sets that we're hoping to put out here in the next week that looks at Gemini. It might be two orders of magnitude more tokens consumed to get to the same level of accuracy that we're seeing out of a GPT-6 Astra, right?

And that's super important looking at the cost of all these things, right? Because everybody knows that Anthropic and OpenAI's API costs are super high. And that in theory, open models running on on-prem hardware can be way more cost effective.

But if you're using, GLM happens to use 10X the tokens of something else, maybe some of that starts to degrade. And we have a view of that, we call it like a solutions view on the website as well, that maps a frontier Pareto curve of intelligence versus cost, right, to get to each action. And it's maybe a little bit surprising about some of the results that you'll see on there.

Yeah, I love that, Ryan, because in my travels, I have been lamenting but also celebrating the fact that Google rolled out four models within two months in the same Flash family. And so it was not misnamed. And what I came to, where I started liking what they were doing was when I realized that instead of using different models for different tasks, that depending upon their price and performance and capability, that I could simply toggle some of the effort and the number of thinking tokens that they were allowed to do the task, and that resulted in very different models and very different pricing personas, if you will.

And I noticed that you guys in Pinnacle are doing the same thing. 1, I think. What do you see when you look at those thinking max, thinking medium, thinking low?

Because like you said, efficiency sounds like a great deal. I'd be willing to pay for a bigger model if it's more efficient. And I think it's an interesting experiment and because I think many of us, at least I myself am, are the kind of person that's always like, "Well, I could probably get away with a smaller model and less thinking, but whatever.

Let's just make sure we get the best answer out of it," right? And you just always turn it up to the max, and you always are kind of using the stuff. And that makes sense if you're an individual and you're on an individual subscription or something like that, you're paying 100 bucks a month.

But if you're selling enterprise with 10,000 consumers, that's a significantly larger problem. One of the interesting things that came out in our new GPT-6 testing from just a couple of days ago was that we did this. We tested max and default, which I think is medium for OpenAI reasoning.

And the intelligence, the grading of the output was almost the same. There's almost no difference in the answers we came to or the fabrication resistance that the model had. But what we did see was in the max mode, you use a lot more tokens, right?

So the token efficiency, your kind of correct answers per million tokens, which is the metric we utilize for that, went down pretty far in the max mode, even though you didn't get much intelligence from it. But there are some other examples, like GLM-52, I'm just looking at this page on my laptop here, where when you go from instruct to thinking, your score goes from 1,800 points on our metric to 3,200 points, right? Okay, now it's a huge difference.

You are paying more for those tokens, but you're getting significantly lower fabrications, significantly better agentic outcomes out of it. And so that's why we include both instruct, no thinking, and thinking modes for all the models that support it in our grading system and in our kind of TCO and ROI model, because we think that's an important piece to consider. Yeah, that is really interesting stuff because I think that it shows just how hard it is to evaluate these models today.

I mean, number one, as you're pointing out, there are different settings even within the same model. And of course, many of these labs and service providers have multiple models on offer. " And then there's, of course, the under-the-covers thing where some of the models are burning more tokens and burning less tokens, and the users have very little clarity on that.

I guess, could you give us a little help? I don't know, maybe give us a little tease of some of the cool facts that you've found so far, and then sort of how should users be thinking about this in the future so that they can make better choices going forward? Yeah, I would say some of the interesting things that we found early on were a little bit surprising, like Claude Haiku is a horrible model for agentic tasks.

You should never be using it. There are lots that are way better, more intelligent, and also cheaper and more efficient. How?

Answers are questions. It's like, what's a tool? Yeah.

I mean, some of this comes down to that, right? And you would see Gemma 4 28B is outperforming that particular model, right? And so there's some interesting things that come out with that.

3, and GPT-6 Astra all kind of release in the same, I don't know, inside the same four or five-day window. They're all a significant step up above every other model we've tested. And it seems to us that there has been some kind of threshold unlocked, some kind of next-gen model step has been taken in this way.

And it's part of what leads us down our roadmap discussion around building more agentic workflows and tasks. They're going to be more complicated, longer step chains, longer workflows to go evaluate as well. 8 Flash Next is one of the absolute best open-weight models of any size, right?

Let alone kind of that medium class, 120, 150 billion parameter model. So some of these things are really standing out, and it gives a whole lot of credence to what the NVIDIA and Hugging Face and kind of this influence, or I would say not influence, but kind of emphasis on open weight being strong once again. Yeah, I think it's always been strong in terms of being a foundation upon which the broader ecosystem can build upon and share knowledge.

And we've seen that accelerate, I think, the frontier and the closed work as well. You just don't see direct lines driven, drawn between them. But what I want to ask you, Matt, is to what do you attribute this bump?

Because I'm looking at your, was it model score versus token efficiency? And those three models you mentioned, Muse Spark, Astra, and Claude Fable 51, are significantly higher in terms of their pinnacle model score, and they are much more efficient. Like crazy.

Is that because we're training on more data? Is Meta actually in destroying books, doing the right thing here for AI? Or is this really down to some of the refinements that we are seeing in terms of what we're learning about how we train these models, how we fine-tune them and align them, and how we architect them to actually carry out these tasks?

So training data or architecture, what do you think? I think it's more on architecture, and I think it's a little bit more on just kind of opening up the model's ability to self-learn, right? Look, I'm not a model scientist in the way that I think I can answer that question.

There's a couple things. 3, is flagship level now? No.

They spend enough money on people and on resources, I would say, that this is the outcome that you should expect. And if it wasn't any better than this, that would be significantly problematic for them. I also look at, and this is just my own kind of thinking out loud is there was Mythos came and went, and there was all this kind of worry about was Mythos too strong, too dangerous to release out into the world and all this stuff.

And it turns out it existed for a while, and people used it, and everyone was like, "No, you know what? " And I kind of get the sense that these vendors were kind of holding back a little bit to see how this security-focused emphasis was going to be before they kind of took the next step up on these models, right? I don't know if that's the case, but it kind of seems to me like that was the inflection point for a lot of the changes that we see.

Interesting. Well, I hate to do it, but we have to kind of summarize here. So again, let me, Ryan, this is new.

You all just started running this, but you've already run it across an incredible number of different models. You've already generated an incredible amount of data. Are you seeing any patterns, any trends, anything that people should basically sort of take away from this discussion and plant in their mind and say, "You know, this is what I've got to think about when I'm evaluating models going forward"?

I think the trends are that things continue to improve maybe faster and in larger steps than we thought were really possible or were going to continue to happen even a couple of months ago. I think that's the case for the frontier AI labs. I think it's also the case for the open-weight labs.

1, and they will start to address all this. And so we're looking forward to adding those tests into our pinnacle benchmarking system as well. So I mean, that's what I'd be on the lookout for.

We also have a lot of new hardware coming online that we're utilizing. We've had great partnerships already with AMD and NVIDIA providing us access to their latest GPU platforms. We're getting ready to onboard our GP300 systems this week, so we can really dive into the multi-node configuration side of this because that's Huge enterprises, that's what they're going to be interested in.

But more complex tasks, more models, more hardware, and continuing to push and evolve this cost per second. And it was a great partnership with us and the team at Kamiwaza. They have great engineers there that helped us build a lot of this harness out and understand what are these agentic workflows and how can they be implemented.

And so it's been a great partnership there too. So we've been very fortunate. And I'd like to just add, we didn't get a chance to really talk about it on today's call.

But Ryan mentioned this at the outset, and that is that this doesn't suffer from what we see with a lot of benchmarks, where you have the training data showing up in the evaluation set. And so you're just basically over-fitting, and everything looks great, but it's not great. And what we didn't talk about too much, but I want everyone to know, is that Ryan and his team built out data sets that are specific to their workloads.

These aren't just borrowing basic data sets and using them that are out there in the wild. These are specific to a number of tasks and personas. " So I love it.

Yeah. It's funny, Brad, I was looking at the IT, what is it, the IT professional persona in the same way, and I'm going down the list and I'm like, "Yeah. " Yep.

And I think that's great. And also, it's kind of cool to see how these personas, how the ranking of models changes for different personas. It's not one size fits any.

You have to really get in there. And I think that that's both the positive and the negative thing of this world that we're living in. As you mentioned, we had three incredible models released in the last seven days.

Right. And basically, by the time this recording goes live, there's probably going to be two or three more models released that we didn't even talk about, that you haven't looked at yet. Yeah.

But you're going to try to keep on top of these things, right, Ryan? Yep. That's the goal.

Within 24 hours of any new model's release, we have API access and we have that capability. It gets pretty expensive to test GPT-6 Astra at the level that we need to test it. But it's worth it.

It's been an interesting experiment. com. All the data is there.

Go explore, go look at it, and I'm open to any suggestions or input or questions that anybody has, anytime. That's awesome. And I've got to say, as soon as I saw that go live, I was like, "Whoa!

" And I've been playing around with it. I recommend the listeners to do that. Where else can they connect with you and learn more and perhaps work with Signal 65?

Yeah. com is where all of our content and reports live. You can find me, I'm on X, @ryanshrout, and on LinkedIn as well, and I post random musings in all those locations oftentimes.

Great. com publishing all the time. Everything we do as analysts is actually available to the public.

We very much like to share what we learn. So I invite everyone to check that out. Yeah, and that's really the differentiator for the Futurum Group overall, is the fact that we're trying not to hide this stuff behind paywalls and under a bushel, as they say.

And trying to do a lot of things publicly. We've got these podcasts that we're producing here on the Futurum Podcast Network. You mentioned the website of the Futurum Research branch, as well as the Signal 65 website.

And of course, the Techstrong family of websites, where you can find all sorts of Futurum media content as well, including coverage of this. Also, you'll see me most Mondays on Techstrong Gang. A lot of fun to do that as well.

I know that you folks like to hop on there from time to time as well. And of course, like Brad and Ryan mentioned, you'll find me on LinkedIn. Thanks for listening to "Utilizing AI" today.

If you enjoyed this discussion, please do subscribe. You'll find us on YouTube or in your favorite podcast application. And please give us a rating and a nice review.

We'd love to hear from you. This podcast was brought to you by the analysts and experts from the Futurum Group, where insights meet AI. ai, the "Utilizing AI" YouTube channel, or the Techstrong TV app on your favorite device.

Thanks for listening, and we'll catch you next week.