Tricentis on Why AI-Generated Code Needs Agentic Testing

Hey everyone, welcome back here to Textrunk TV. My next guest is Dave Caldwell. Dave is the VP of AI, artificial intelligence, and machine learning over at Tricentis.

It's good to have someone from Tricentis on here. I don't think we've interviewed anyone in Tricentis in quite some time. So Dave, welcome to Textrunk TV.

It's great to have you here. Thanks, Alan. Happy to be here.

Some exciting topics we've got ahead of us. Yes, we do. Dave, before we jump into them, though, I wanted to give people a sense of who they're watching.

Right? I mentioned you're VP for artificial intelligence and machine learning. And quite frankly, I know a fair bit of Tricentis history.

They've always, well, as long as I know them, they've had machine learning capabilities built in the product. But of course, over the last three, four years, that has morphed into AI. But give people a sense of how you got here, Dave.

What's your journey been like? Well, a lot of my journey's been with Tricentis, so I'm glad to hear that you have seen our AI products in Tricentis. I built most of those.

So I started the AI department in Tricentis back in 2018. And then I've been running it, building it, and expanding it ever since. But before Tricentis, I actually worked in the QA field.

I was a director of QA for a consultancy. I ran QA rollouts across companies like Vodafone Australia, doing the 4G rollout, and tech firms, telcos, finance groups. So I lived in the trenches of quality assurance for a long time, longer than I would care to note.

I'm about five kids old at the moment, so that puts me- Mm-hmm ... somewhere between like, I don't know, 38 and 96. So, depending on how well-behaved they've been on any given day.

Five kids. God bless you, man. You know what?

I don't know. I had two. We have two.

We still have them, though they don't live at home anymore. I can't even imagine. You run out of hands, and then at some point, you try to chain them together, and I don't get any real success out of that yet.

The eldest is 13. No. Wow.

But putting aside the kids for a second, I joined Tricentis because I saw the QA world was underserved with the tools that it had. There was plenty of cool tools for developers, and I'd been contributing to products like Selenium for a while, trying to improve those. And then I saw Tricentis as a, this is a company that cares about the experience of the tester.

It wants to make the world of QA better. I think I can work there. I went over here.

" A story for another time on how I got into AI. It involves Halloween decorations. But we- Oh, you can't leave that dangling.

But that's a reason for you to come back. We'll discuss it next time. All right.

Well, we just passed Halloween here locally. I don't know how long it takes to produce these, but there was plenty of Halloween decorations in my AI past. And I actually went to the founder of the company at the time, Wolfgang, and said, "Hey, we should be using AI to make these products better.

I think it can really help with things like stability and object identification. " So I had all these ideas for how we could be using what's now called machine learning, like neural network models, computer vision models, statistical models. And so we trained those.

We actually got some patents in world record-setting OCR models and object detectors. And then language models appeared, and they sucked. " Got them aside.

Some time went by, and then in about '22, we took another look at them. We're like, "They don't suck anymore. " So we've been riding on those language model trains, building assistants, and then those assistants turned into copilots, and the copilots turned into agents.

And the goal has been the same the entire time. How do we make it so that testers can test more when the pipeline of work coming towards them starts flooding them? How do we help them keep up?

And so we're really just trying to build more and more of these more capable AI capabilities that are going to make it so testers have to spend less time with their hands on the keyboard and more time doing the things that actually matter to testers. Finding bugs, finding the gaps in assumptions, and identifying what the customer is not going to like about the product so that they can be that customer advocate, protecting customers from, well, it used to be bugs in development. Now the tester's job is to protect the customer from AI slop getting out there.

Yeah. So the role hasn't changed. But the way you do it has changed, the tools you use have changed, and we in Tricentis have changed to adapt to that.

I think the word there is change, right? Everything is changed. Dave, I want to jump into where rubber meets the road here around code and testing code, specifically testing AI-generated code.

Before we do, though, let me just get a little housekeeping out of the way. com, the website. Yeah.

People want to get more information. And by the way, look, Tricentis has been a leader in QA testing for, I don't know, probably what, 10 years, 12 years, something like that? Yeah, I think we're on 11 now.

11 years of being around. Yeah. I was pretty close, 10 to 12.

So there's a lot under that tent. com is probably the best place for you to go see the whole breadth of... Because believe it or not, not everything is AI today, but to see the whole breadth of the Tricentis offerings.

But Dave, let's turn now to AI, right? Mm-hmm. Look, a lot of people are saying maybe a majority of our code today is AI generated.

Mm-hmm. Many of my friends who used to be called developers and are now called software engineers, haven't written a stitch of code in a long time now. They prompt their code, they vibe coding, whatever you want to call it.

However, depending who you talk to, AI-generated code suffers from a lack of quality. Now, is it worse than human-generated code? I think that depends on what human generated the code, right?

And I've seen it go both ways. But clearly, quality of code, especially in this post-Mythos, we haven't mentioned that, but- Mm-hmm ... with Mythos, you're not going to be able to get away with putting vulnerable code out there anymore because someone's going to find it really quick and exploit it even quicker.

Mm-hmm. " Talk to me how Tricentis is helping there. Yeah.

So I think the best way to understand this immense amount of change that's happening in software engineering is we're speed running 40 years in about two years. So if you think about 40 years' worth of software development, it used to be there was an engineer who would sit down, and they would write a Linux app or a Windows app or something like that, but they would write the whole thing, and there would be basically one, two, three, four engineers. And there wasn't pipelines, there wasn't a lot of structure around it.

It was like that engineer generated the code, and then it went to a tester who would kind of... Even testers were somewhat rare in the '80s and '90s. It was really after Y2K that it became a major role.

I mean, it was still there, but it wasn't as much of a fun role. So you can almost imagine that that period in our history is vibe coding. It's where it's like an engineer pushing the code out, and you get about as good quality output as the engineer parallel is about as good quality as the model.

What we're doing now is we're kind of going through the 2000, 2010 time, where we're moving away from just vibe coding into agentic engineering, software factories, this type of thing, where you're starting to put the structure around the code and say, "Okay, it doesn't really matter who wrote or generated the code. That's irrelevant. " How do we know we've got the right tests that guard the bugs from going into production?

Do we have code cleanliness? Do we have checks and balances? Do we have security scans?

All of these things are reappearing for the agentic coding era. And I would be completely unsurprised if all of the things about CI/CD deployment, everything else, also reappear for the agentic era. I've seen Git rewritten for the agentic era by Anthropic and OpenAI.

All of the tools are likely to change, but the lessons that we learned from the last 40 years are just repeating. And so what is the testing world for agentic coding? Without testing, you have no idea if that agentic code actually worked, if it was good, if it was bad, if it has bugs, if it does not.

And you can't trust the tests that the agent wrote for itself. Now, I won't dig too deep into the reasons why because that's a 20-minute conversation, we only have 15 or so. But suffice it to say, the models were trained on human outputs and are very similar to humans in how they work, and in the same way that when a developer writes code, they'll...

We're not saying developers write bad code. They write the best code that they feel they can, but the assumptions that they've made as they wrote that code are not going to be challenged by the tests that they write. It's all in their brain.

The assumption is already baked into the way they're working. Same thing happens for models. It makes an assumption about the way you want something done.

That assumption could be erroneous, but it will build tests that validate that assumption. It will build code that works according to that assumption, and that assumption will end up in production, and you hope that assumption was right because the model didn't document it. Creating separate testing pipelines, separate testing agents, gives you that independence that allows another agent with different assumptions to be able to come along and say, "Hang on, I'm not looking at the code.

My job's not to build it. I'm looking at the requirements of the user story, the context of the organization. And I'm trying to figure out not does the code do what the code says it should do.

I'm trying to figure out does this work? " So you'll find that those are like guardrails that fits on the outside of the agentic coding life cycle. Now you get safety out of it.

But the second thing you get is you get velocity. Because we, again, are rediscovering the history. If you have good tests, you can code much faster because you know when it's right and wrong much earlier.

So having a good testing agent for your coding agent allows your coding agent to bounce off those tests and get to outcomes before it even deploys something. " "Oh, I understand. Yep.

" Now deploy, and you have a much lower failure rate on your deploys. You have much less time spent with the agent having to go back and forth with logs and figuring out what's working and what's not, and you save a ton of tokens. So you get better accuracy, you get less time spent, and you get less money spent on tokens if you inject agentic testing into your agentic coding.

Absolutely. Dave, there's two things that I hear from people. We talk to a lot of companies, a lot of users, practitioners.

Two things about AI code. Number one, trust. Trust factor, right?

If I can't trust that this is quality, I just... Because if I have to sit here and go through every single thing by hand, by human, I might as well just write the code. You get no productivity.

Right. So trust is number one. And then number two is, it's like a theory of constraints argument.

Once I trust the code to turn that spigot on, where's the next bottleneck? Because now I'm moving at AI scale, and I can't just write code at AI scale. I've got to have my whole pipeline working at AI scale.

Otherwise, it's the old story about the internet. I may have a humongous pipe here, but if the pipe coming to your house is only that little, that's the maximum speed we're talking about. It's the same thing with software pipelines, right?

And so is what Tricentis doing now, removing the next bottleneck after, assuming we trust code, after AI scale code generation, now Tricentis is bringing AI scale testing. Correct. So you've got to imagine the production pipeline very similar to a factory where you've got- Right ...

things need to move through this. And I was a student of the Toyota lean manufacturing principles. It's like- Uh-huh ...

okay, if you've got areas where you have to have handovers that have got manual injection, then what you're going to have is buildup of inventory behind that manual area. So making the production track faster up until that point just results in more inventory building up at this. So what are those constraints?

What are the restrictions in flow that you need to deal with? And there is ample evidence out there from Anthropic, from OpenAI, from Faros AI, from GitHub and GitLab, you name it. Everyone is coming to the same conclusions.

That you run into two main constraints immediately after you start using agentic coding. You run into the trust constraint first. But it's not called trust in your pipeline, it's called review.

So everyone will say, "All right, we need human review of code. We need to have four eyes on everything," et cetera. And then you'll start to try to find ways to trust it more.

Now, some ways you'll use tools. You use code review tools, you use agent reviews, you try to throw that at it. But you'll still see defects creeping through into the product because the code review isn't really reviewing the fundamental assumptions of whether the code was written the right way for the right requirement.

It's just reviewing, is this code good code, right? So you get some trust level there, but you'll see that trust crumble when defects make it through to the latter product, or when your users pick it up and go, "What are you... " So then you'll say, "Well, okay, we need to solve these two problems of trust and validation," which is the next constraint that people come to simultaneously.

We need to have validation injected back into the pipeline so that when we get it to that review point, it's not, is the code good? We go, we know the code works, we know it meets the requirement, and we've got easy-to-read, understandable evidence that it meets the requirement. So tick, this is ready.

I've got demo videos, I've got tests passing, I've got traceability. This is the right product. Now I need to review, was it written the right way?

And that's where code review then solves that problem for you. If you don't do those two, then you'll gain 10% to 15% productivity out of 100% agentic code generation. If you solve those, you can see orders of magnitude.

You can see multiples in productivity. This is a rule that internally, there's probably a better term for it. We call it the last mile problem.

You can almost imagine that 80% of any problem is solved by a language model. But that remaining 20% not being solved means that a person needs to check it out. So you're now 80% more productive.

But if you can say, "Hang on, I can make that 20%, 10%, 5%, 2% go away," the moment you can make that go away, that person can now do 20 times, 50 times, 100 times more work. So solving the last mile is where we're all at, and testing is probably the number one blocker to most companies solving the last mile of shipping to production. Dave, we're over time.

I'd love to continue this conversation, right? Because I can tell you a story about in the dot-com days, I helped a company, we were what we called an ASP, application service provider. Mm-hmm.

And we had a similar 80/20 last mile situation. But we could talk about that, Halloween, and then what's the next bottleneck after testing in this whole software delivery life cycle. But we'll do it on another day.

I want to thank you for coming here on Techstrong TV. Say hello to all our friends at Tricentis. I wish you luck, and come back soon.

Thank you, Alan. Happy to be back anytime, and thank you for having me. All right.

Dave Caldwell, VP, Artificial Intelligence and Machine Learning over at Tricentis. We're going to take a break here on Techstrong TV. We'll be back in a moment.