Transforming Test Automation with SmartBear’s Frank Roe and Dan Faulkner

This is Textron tv. Hey everyone, welcome to another Textron tv Mond here. You know what, this gentleman reminded me that we haven't spoken, it's been a couple of years.

I didn't realize that much time had passed, but let me introduce you to Frank Roe. Frank is the CEO of SmartBear, and, uh, as he said, it's been a few years. Frank, welcome back to Tech Drug tv.

It's great to see you. Thanks, Alan. It's great to see you as well.

I, I, like you couldn't believe it's been, uh, been just about three years since we last spoke. Yeah. Uh, really excited to get you caught up on everything that we've done and, um, you know, just, just, uh, have another great session with you.

Absolutely. And then we've got a, a, well, he, he, it turns out he's not a new guy. He's also been there almost three years.

But joining us for the first time here on Dex Junk tv, we have Dan Faulkner and, uh, Dan, I, I take you your chief product officer or something of VP product. So, well, I'll let you introduce yourself, Dan. Why don't I do that?

Thanks, Al. It's very nice to meet you. And, uh, yes, uh, I'm Dan Faulkner.

I'm the Chief Product Officer at SmartBear. I've been here now for just about three years. Excellent.

So you came in right after I last interviewed Frank and Frank, you were kind of newly installed as CEO at that point, weren't you? Yeah, It'd probably been, uh, about a year, year and a half, uh, since I've become Yeah. Fairly and, yeah.

Um, you know, and, and Alan, if, if you can go back in that way, back machine, three, three plus years ago, everyone is really kind of coming outta Covid a little bit. Um, and what we saw was an absolute acceleration in software development, right? There was just a backlog of, and a lot of companies really had, um, kind of got themselves caught.

They, they hadn't invested enough in the digital automation and, uh, and they found out, uh, through the hard way. And so we saw just a massive uptick, uh, in our own business. And that was really driven by the amount of software, uh, development that was, you know, was being put forward, the amount of code that was being delivered.

Um, and maybe as a quick refresh on SmartBear and who we are and what we do, um, we have 32,000 customers, uh, we operate, uh, in, and then we have of, of those 32,000 customers, we have 20 to 30 million developers. Uh, we operate in every major, uh, country of the world. And ultimately what we do, our, our tagline is revolutionize software quality.

And so we can unpack that, but fundamentally, uh, we help developers, QA, engineers, and companies across the globe make sure that the software that they're developing works. It performs on time. And that's from, that's across the entire SDLC.

So whether that's through the API lifecycle, uh, all the way, uh, across testing and into production. And, uh, and so, uh, we saw when we last spoke a few years ago, we saw an absolute explosion in our business because of covid. And I'm happy to tell you, and I think you probably know this, but, uh, gen AI is now another acceleration, uh, in terms of the amount of code that's being developed.

And Dan will get into, uh, this as well. We're, we're leading, uh, in this area, both in terms of internal use here at SmartBear, but also bringing, really proud to say, uh, we launched Halo ai, which is our gen ai, uh, solution. And we've done it a little differently than most.

Instead of it being a brand new product that nobody has used or, you know, someone needs to buy it and then deploy it, we actually put Halo AI into our existing products, so into the hands of those 30 to 2000 customers. So is that like a co-pilot kind of thing then? Or You could think of it as that, you could think of it as a co-pilot for test automation, uh, and the acceleration of that.

And, and, uh, and, and Dan, please, please, uh, weigh in, uh, 'cause because you're, you're leading this effort here at Smart Bear. Sure. The, the, um, I, I think certainly, uh, you can think of, uh, halo IRI as as having relationships, kind of the copilot theory, the or or approach.

I'd say the main difference is the copilots being heavily used as kind of a chat, uh, interface for a developer. Um, and we have very consciously prioritized AI implementations where the product will interact with the LLM, and that allows us to make sure that we are providing all the right context examples, controlling the sequences of information that go from our products to the LLM, um, so that we can take full advantage of the creativity of LLMs, but uh, also get the right level of determinism that we need to be valuable to our end users. If we just open up a chat window, I think you, you run the risk of suffering some of those, uh, well-known challenges that can come up with, uh, generous ai.

Sure. Um, you know, just the whole issue of LLMs themselves, uh, is a minefield, right? Are you using, you know, are you using some gigantic l lms, kinda like what OpenAI is trained on, or are we going to use, you know, custom s SLMs as some are calling them, right.

Small language modules, right. That are much more geared towards specific use case, you know, and personas and so forth. Um, you know, it's hard to believe, but it's been what, well, I guess November will be two years since kind of OpenAI chat.

GPT kinda opened the floodgates, if you will, and like nothing's ever been the same, Right? When I look at, when I look at things like Halo ai, for instance, right? They kind of fall in two camps for me.

One is, I, I think a really good solution in this space enhances the human user. It enables him 10 Xs them or whatever. It doesn't have to be I'm right.

And I think that's what every vendor aspires to that. No, I'm not looking to replace a developer. I'm not looking to replace a QA person.

I'm not looking to replace people. I want to enhance their ability to deliver value to the organization or the, I want to deliver value to them. And though I think everyone strives for that, there's a part of the audience out there, you know, c-level people, Frank, I'm sorry to say that.

Say, Hey, can I, this, can I cut heads with this? Yeah. Right.

Can I, can I be more profitable with this? So when we look at Halo here, where does that fall? Dan?

I'm gonna put it on you. You are the product guy. Yeah, no, I'm glad you did.

Um, we, uh, are 100% convicted around the idea of this in enabling people. Um, and there's a couple of reasons for that. Um, but from a pure technology perspective, we are a long, long way away from anything being able to be completely autonomous.

Um, it would be a, um, uh, a, a crazy risk to wave through, um, AI generated code, then AI tested code, AI security validated code, and just deploy it so that that just isn't on the horizon. Um, our view generally is that when it comes to code generation, you are seeing an acceleration. Um, and, and it's really helping around areas like, um, improving test coverage, particularly unit test coverage, um, documentation.

Um, however, when it comes to generating new code, we're actually seeing the level of code degrade slightly. Um, and that's because the code that's generated has errors and it's gotta be reviewed and people are bad editors versus creators. Um, so there's, there's relatively, uh, high number of errors are being introduced into the world's repos at a record velocity.

Um, that's actually going to increase the burden on testing. And our opinion is that the testing burden has already pretty much broken, um, the current system already. Uh, so the world is gonna need a set of solutions that actually 10 x the people, 10 x the testers free up those developers to make sure they review the code properly and push the testing burden onto people who A, want to do the testing.

B are really good at testing. That's testers. Uh, and it kind of runs against some of the code-based testing solutions that we've seen, uh, emerge over the, over the last few years.

So the paradox for me though, is that we wanna go faster, we want to do more. We, as you said, the, the adult testing kind of processes that we've put in place is broken by the sheer volume of, of code coming through. And yet we still wanna, you know, let's go 10 delivery, 10, you know, deployments a day, like everybody needs 10 deployments a day.

But, um, but again, the paradox is, is how do I do that and still keep people in the equation? Uh, exactly. And, and you can generate the code, let's say, multiples faster or, or, you know, you have a real acceleration of generating the code, but if you think about the total amount of work that's going to go into that code over its lifetime, uh, the jury's out on whether that's actually coming down or not, um, and you know, the amount of refactoring you'll need to do, are you actually generating brand new, new pristine code, or are you generating tech debt at ripple rates?

Um, and I think the initial data's indicating that it's, it's a mixed bag. Um, so it, there is no e even medium term perspective that, that we can see where people are not heavily, heavily involved in that loop. Uh, so, so you're gonna have to enable people to deal with the increased volumes, the increased speed, and that's where things like Halo, I, halo AI come in because it's entirely designed to help people be more effective at dealing with that greater volume.

Yeah. And in fact, to that point, right, um, we, we did think through, uh, our approach and our strategy, you know, uh, our purpose is to revolutionize software quality that's SmartBear's purpose. And so when thinking that through, we really made a very conscious decision to say, in order to help all of our customers and all of our users, let's embed Halo AI into our existing products versus making it a forced standalone new purchase.

Um, because Alan, as you said, you know, we we're hearing from our largest, to our smallest customers, and everybody in between help us figure this out because we're seeing it. We, we are seeing an, an explosion in code. There's all the standards that need to be applied, security policies.

How do we get out ahead of this? And it was pretty clear from our customers, give us something that we're already using, right? Yeah.

Yeah. Versus having to go out and buy something brand new. And, and so proud to say that that's what the, that's the whole genesis of Halo AI is it's embedded into our existing products to get it into the hands of our users right away.

So is, and I don't wanna get in your shorts here, but is there an upcharge for using Halo as part of the SmartBear In certain scenarios there? Yep. Yeah.

Of, of, uh, in certain scenarios there are, and others, there, there are not. Um, we've, we've also made that a very, you know, conscious decision to try and lower that bar, but there are certain areas that Halo AI provides, uh, that value across the SDLC for our customers that they even told us during beta. Uh, not only sh you know, would we pay for this?

You should charge for it very incrementally, um, based on usage. And, and that's exactly what we're doing. So this a question Go ahead, Dan.

I'm Sorry. I was gonna say maybe it might be useful to give a couple of examples of, yeah, I was gonna add, that was my next question. I've asked this question before of companies that are introducing ai, and I, I find it fascinating as to where that line is.

Yeah. So, uh, no Way it changes. Go ahead.

I'll give you a few examples, um, of sort of Halo AI in action. Um, the first is in our application testing space where, uh, we are using Halo AI to take tests that are written in plain English, that have lived in kind of test management tools that are very often carried out by a manual QA person. Um, and Halo AI is able to understand that English and very effectively automate the test.

Um, and if you think about the potential extension of that, the giant number of tests that are written in plain natural language around the world today that are being carried out manually, we believe we can actually automate a huge number of those, um, and, and alleviate the need for each of those to be done person by person. Imagine a world where actually the test automation language of choice is English rather than one of the automation languages that are out there. Um, quite a different use case is we have a product called Pact Flow, which is used for API contract testing popular product.

Um, but, but as people adopt contract testing, one of the biggest hurdles for usage is to write their first contract test. It is a little bit more technical than, than some of the other testing tasks that QA engineers may need to do. And so we are, are using Halo AI to generate drafts of contract tests for them, just get 'em over that usage hump so then they can take it on, um, uh, uh, much more quickly and adopt the product much more quickly.

So there's just a couple of examples that Sure. You know, we've already integrated into our products Mm-Hmm. You know, I, and I've been in, in technology a long time, right?

30 plus years. You guys have been around two a few times, right? Yeah.

Have you ever seen a wave like this that moves so fast as with seeing gen AI evolve? Not for it, not in this time compressed, right? I think we, that's what I Mean.

I mean, right in 99, 2000, we saw it, uh, clearly post covid, we saw it, but nothing this time compressed, right? I think, I think what's, what's going on in the last two years would've taken five, right? So Agreed.

And that even in a very compressed manner. So yeah, I think, uh, I've, we've never seen anything like this. We, we use it internally and, and I, we, you know, we talk to our customers.

I, I talk to 'em, Dan does as well. You know, we've got large, you know, uh, very heavily regulated industry companies, and they're struggling with trying to figure out, Hey, wait, I've got 10,000 developers. If only 10% of my developers are using Gen ai, that's still a thousand developers.

How do I get my, how do I harness what's going on here to small companies that are saying we only use Gen ai? Um, and again, the, the intent is to try and increase, um, productivity. The question is, is it increasing productivity just yet?

Um, or is it changing productivity, right? Meaning it's, it's more lines of code are being written, but at a, at a poor quality. Uh, which means, uh, there, there's a lot.

The, the pressure has now moved, right? And it's moved right into the square of our customers and our users. And, and that's why we're really proud of introducing Halo AI in into the marketplace.

I, I think you're hit the nail on the head. If, if more code's great, but if the code's not quality, not so great. Yeah.

Um, guys, we only have 15 minutes and we're at our end here. I'd love to keep going in on this. Maybe we could pick it up, Frank.

It can't be three years till you come back on here. Okay. I'll Make that, I'll make that commitment right here.

That, uh, we'll, we'll, All right. I'm gonna hold you to that. And, and, and Dan, you don't have to wait for him.

So if we, we, we can continue the conversation, love to do it. What we did mention for people maybe wanting to get more information about Halo ai, where, where can they go on the SmartBear site? Do you know The homepage?

It's right there on the homepage. com or That's right. com.

And look for Halo ai. Yep. It will, it will be there.

Waiting for you. I love it. Hey guys, congratulations and keep up the great work.

It's always a pleasure to have you on Halo AI from SmartBear helping, uh, accelerate your test automation. We're gonna take a break on Textron tv. We'll be back in just a moment.