How AI Is Redefining DevOps and Quality Engineering | CTRL + ALT + DEPLOY Ep 06

Hey, everyone, it's Alan Shemel, and welcome to another episode of Control Alt Deploy. Control. Alt Deploy is a, uh, a podcast series that we've been doing with our good friends over atex now for more than six months, I think.

And we discuss different areas around DevOps, that SecOps software development and deployment, CICD testing, et cetera, but really with an eye towards, you know, what, what's happening in the space. And of course, the biggest thing happening in the space is probably ai, uh, as it's taking a, a, an effect on everything. So in today's episode, we're gonna look at AI and the role of, and, and the role it's playing in testing, automated testing, continuous testing, whatever you want to call it.

I'm really happy to be joined for this episode, and it's just me and him. This one, I know many of the others, we've done big panels, but this a one-on-one with Jan Yav. Sayers.

Yanev is the, uh, a DM Chief architect at OpenText. Yav, welcome to Control Alt Deploy. How are you?

I'm doing great, and thanks for having me, Alan. My pleasure. Yav, before we jump into what we want to talk about around testing and ai, give people, it's, I, I gave them your title, but what do you actually do over at OpenText and tell us a little bit about your journey.

Uh, sure. So, so thanks again, uh, Alan, for having me. So I've been in, uh, OpenText for the, uh, last, uh, 20 years in different engineering position, uh, as a developer, as an engineer, uh, running internally our DevOps, uh, uh, platform implementation.

And, uh, I'm now running our application delivery management technology and innovation. And a lot of my focus is on, uh, on ai, how to utilize ai, uh, both in our products, both in our own DevOps implementation, in testing, uh, and, uh, very excited about it and opportunities around it. Absolutely.

Thanks for joining me today, Jan. If, um, so if we were talking before we started recording, we, we actually had a good talk. We should have recorded that.

com in 20 13, 12 years ago, almost 13 years ago now, testing was always one of the poster childs for DevOps success, right? With DevOps, we, we did much more automated testing, much more continuous testing. And, you know, and that old adage of freeing up the human to do higher level work, instead of doing the same mundane, running those mundane tests over and over, it freed up the human to almost be more of a test architect, if you will, right?

The, he, the human, he or she, right? The human would, would write the script for the test coverage, right? What do we want to test?

And then we can automate the actual running of the test. And then, you know, based upon test results, again, humans would get involved, remediation, uh, whatever needed. You know what, depending what came out of the test, and it, it made a huge difference.

I don't know how long you're doing or been involved in, in, in the space, but you remember before there was a DevOps and it wasn't so automated, right? Mm-hmm. The life of a, a tester was, was kind of really, you know, doing the same thing over and over and over a lot.

Again, it made it, it made it better, and our testing got better, right? With automated testing, we, we had better coverage. We, we did more.

You know, I guess the real question is, did it make our software better? Mm-hmm. That's, that's a great question.

That's the question. Um, but now AI's kind of changed the game, so I've set the table. Why don't you explain to the audience how AI is kind of changing that game?

I, I, I think that, uh, what we're seeing, especially in the software, uh, delivery industry is, uh, a significant, uh, shift, uh, once, uh, uh, AI and especially large language model were, uh, recently introduced. And it started mainly by adopting a code assistant, whether that's, uh, co-pilot, uh, uh, or, or others. And, uh, I think where we are today is that, uh, many organization, uh, including ourselves, realize that it's, it's great to have a code assist, but then you, uh, then you have kind of an influx of new code, and, but eventually your outcome isn't just generating code, it's, it, it's having new capabilities or feature or delighting your users and, and customers.

And, uh, the rest of your DevOps pipeline or, or value delivery, uh, may introduce bottlenecks. So just generating more and more code, uh, isn't, uh, uh, isn't eventually improving your, uh, your outcomes. Uh, so, um, uh, uh, I think kind of the, the maturity here, or the evolution here is, uh, to see how you can now leverage AI across that, uh, value stream across that, uh, DevOps pipeline.

So it's not just generating more code, it's actually making sure that what you're building is at the right quality, both by meeting the business goals, the business expectation, and also validating that it's secured, that it's performance, that it meets the enterprise's, uh, requirements, uh, uh, and, uh, uh, and controls. Uh, so it is a, it is a matter of maturity. It's a matter of evolution.

But, uh, I think that now the way we are looking at, at AI is how can we inject and leverage AI at every step in the software delivery lifecycle from the planning to delivery beyond just coding and coding a system? And clearly, uh, we are a large enterprise. Uh, there are no controls, regulations, risks that we need to manage.

And being able to, uh, make the most of code assistance, that actually means you need to add these guardrails through AI across the lifecycle. You cannot just look on the coding part. You have to look on testing, you have to look on security.

You have to look on how you validate eventually, uh, uh, I'd say the outputs of your code assistant, of your copilots and the like. Agreed. Agreed.

Um, let me, let me get, let me talk on behalf of the humans out here, though. What do you think the role of the human is in this new way or this AI enabled or empowered way of doing testing? Is there a place for humans?

I think that's a fair question. It Is a fair question. And, and, and I think, as you know, uh, uh, as we were looking back into, uh, uh, previous, uh, technology, uh, enhancements and disruption, whether that's around, uh, uh, the shift to or from manual to automated testing, uh, the production of, uh, mobile and mobile application, et cetera, eventually also here, the role of the tester will, will evolve.

It has to evolve. I don't think that, uh, uh, humans are, uh, obsolete. Uh, I don't think that the human, uh, uh, testing is obsolete.

There is still a role here for the human, and, and we'll discuss that in a, uh, in a second, but it definitely has to change. Uh, the way we're looking at it is, you know, eventually testing evolves to become more q quality engineering. Uh, that means that, uh, the current tester would need to know how to best utilize, uh, the new tools, the new capabilities of AI across that live cycle.

And that means, for example, uh, when you're looking on, you know, your requirements or the design, uh, how to best utilize AI to better plan, to better design, to identify risks earlier in the life cycle. Uh, so if you have now, uh, a new feature being designed or, or planned validating that there are no missing aspects in the requirements, may have validated that there are no ambiguity in the definition, uh, doing earlier in the lifecycle, threat modeling, utilizing AI to identify security risks upfront, et cetera, uh, we think that this is a key area where, uh, the quality engineer, uh, can level up and leverage AI to look across that software delivery lifecycle beyond just testing. Testing is a key, is a key part of it is a key control.

But beyond that, so you can, early in the life cycle, do the shift left to identify risks, uh, identifying a bigot in the design, in the planning, et cetera, that then becomes a better input eventually also for code assistant and the like. Uh, and then the QA engineer is the one that is responsible to validate eventually that output, that output from that a DevOps cycle when you get a business goal, a business requirement, validating that it was actually met and defining which tools, how you utilize AI to put these controls, to put these validations in the steps in the lifecycle. So we do see that evolution from the shift from manual to automated testing and up to autonomous testing.

Yeah. Eventually the tester or the qa, the QA engineer, uh, to come and define the methodologies, the practice of the tooling that he's going to inject into that lifecycle to control the agents, to control the agent that are going to build potentially more and more of the software. Sure.

So, hey, I, I, I've had this discussion with people before. So if we're going to use agents to build the software, then we're going to use agents to test the software, then we're going to use agents to remediate or re, you know, re recode or, you know, re remediate the, from the test results, validate the test results, remediate the test results, where is the human in the loop? Mm-hmm.

So it's, it's a, it's a great question. And, and, and I think here again, it's not, uh, uh, black and white. Uh, I think it really depends on the domain.

It it depends on the, the, the maturity of the organization and, uh, you know, the, uh, the criticality of the, the application and the software. Then, yeah, there could be cases where it's a relative simple application, uh, that follows a pattern. And yeah, you could, through a vibe coding, uh, uh, get a, a great outcome or a good enough outcome.

But in many cases, especially when you are looking on large enterprises, uh, and large scale applications, uh, it's, it's more complex than that. Uh, so, uh, in that case, yeah, I still see engineers, humans, but leveraging AI and assistant to augment them, whether that's in the planning phase, whether that's in the development phase, in the testing phase, in the deployment phase, et cetera, uh, to, uh, uh, accelerate the delivery to actually improve quality by better validating or earlier identifying the risks and concerns and mitigating that upfront, uh, and, uh, uh, validating the outcomes or say the outputs of the agents. So I don't see in that these scenarios, especially if you're looking on financials, healthcare, et cetera, uh, are more, let's say, a critical application mission, critical application kind of, uh, just, uh, uh, uh, fading out.

Uh, so I do see a more leverage of ai, but then empowering the human, rather replacing the humans in that, uh, in that scenario. And, uh, and, and I think there is a, uh, an another factor here. I mean, there is, uh, up to a certain scale where you can know, uh, uh, offload to an ai, uh, agent.

Uh, there are many cases where, uh, you know, an ai a an AI agent can't yet, uh, uh, tell you, uh, what is the real human experience of an application? How does it make you feel? Uh, that's something that you will still need in many of these scenarios, uh, to have the human and the human in the loop, not just to develop, not just to, uh, utilize the agents, but really to provide the human aspect, the human experience.

Eventually, we are the ones that are experiencing the applications, experiencing the software, and, uh, that's something that the agent cannot yet replace. And I don't see that happening in the, uh, in the near future. I agree.

I, I don't, I agree. I don't disagree at all with that. Wanted to, you know, a lot of what we're seeing in the end, in, in the media now, right?

This report that says 95% of, or 90% of ai, uh, programs, projects are not contributing to the bottom line. Mm-hmm. Uh, another, another study said, uh, 80% were deemed not successful.

We've seen other studies where, and this is kind of counterintuitive, 70% of the IT workers are saying, or 75, more than 75%, like critical mess, 77% are using ai, but two thirds don't trust it, but they use it anyway. And when we think specifically about testing yav to use a tool that you don't trust, is, is that the case? First of all?

Is that that valid? Do you know? And I think it is, I think a lot of people are using ai, but they don't trust ai.

Right. And what does that mean for the quality of the code that's being passed on via testing right now? Yeah.

That, that's, that's a valid concern. And, and actually, uh, we do see that as in, in, in, in many organizations becoming a, potentially an inhibitor for utilizing, uh, a ai. Uh, and that's why my point that was that, you know, in one hand you see more and more usage in one hand of code assist, but a lot of concern that is being introduced, a lot of being introduced on what is the output of these code assistant?

What is the quality of the code they generate, uh, the security of that code, et cetera. Yeah. So you, you have to, you, you have to control that.

And again, the, the, the, the more mission critical the application here is, uh, uh, uh, the more, uh, a large enterprise is, the higher the risk is. So you have to counter that, or you have to balance that. And part of that balancing act is also validating it, having the proper testing and tooling in place, uh, putting the proper still code reviews, human in the loop code scanning, test automation, autonomous testing, as we discussed, to mitigate that risk.

And, and still it's there. I mean, you still have that, that risk. And I would say it's more than risk.

You, you have here new challenges that you didn't have before. So traditionally, uh, when you are looking on the traditional applications, they were pretty deterministic, right? You had an expected outcome or expected behavior of the application.

Now, with the introduction of, uh, uh, of AI and large language model that are being injected or becoming part of the software, they're becoming less deterministic. And that requires different practices on how you test, how you validate, how you measure the quality and the experience of these applications. And it requires taking more statistical, uh, approaches, uh, creating baselines using techniques like, uh, LLM as a judge and so on.

So also, the way you test and validate your applications has to evolve in order to meet the new technology of introduction of AI within applications. So it's, uh, to kind of, uh, uh, uh, a closure to your question, yeah, there is more, I'd say, risk that is being introduced. I think that some of that is acceptable.

I mean, we're, all of us are using GPT, all of us know that there could be hallucinations, and we accept that, and we're, I wouldn't say, uh, uh, uh, uh, necessarily, uh, happy with it, but we are, we accept it. We, we know how to adjust to it and how to handle it. Uh, with that in mind, um, uh, you do need to put, to put in place the right controls, the right validations, the right, I'd say, uh, uh, balancing, uh, uh, uh, uh, tooling, processes, et cetera, in order to still be able to provide enterprise grade applications to your customers, end users, and so on.

And that requires also introducing, uh, new methodologies and practices, especially when you were coming to test and validate AI powered applications. Fair, fair. So, Yanni, let me, let me move from the, you know, from the hypothetical to the real Mm-hmm.

And I'm asking you now on behalf of OpenText. Yep. Where does the rubber meet the road?

Have you rolled out products around this already? Are people using the mm-hmm. What's the experiment been like?

Yeah, Definitely. Uh, we have, uh, uh, we have introduced, uh, a, a, a variety of, uh, uh, I'd say, uh, uh, smart assistant, we call them aviators. Uh, so we have our, uh, our own, uh, uh, DevOps, uh, avior testing aviators that are, we're using that internally and also with our customers, using that to generate tests, to identify risks and mitigate that by generating tests, whether this could be manual tests or automated tests.

We know how to, for example, and we're using that, not now a lot, uh, uh, improve our planning. So we're using our aviators to identify earlier risks, to do threat modeling, uh, to identify, uh, ambiguity in the requirements and improve that. Uh, we are using our own aviators, uh, for validating our security, uh, for, as I mentioned, generating tests, uh, for planning, uh, breaking features into user stories, into tasks and so on.

Uh, so, uh, uh, a variety of, uh, uh, uh, of aviator of smart assistance that are being used in every step in the software delivery, uh, life cycle. Uh, we're measuring that continuously, both the, the success, both the usage, meaning to what extent, uh, uh, or how many tests were generated in a, for an AVIOR versus the human generated test. And, uh, our at least, uh, uh, experience is that, uh, the trending is very, very positive.

Meaning we do see more and more shift towards, for example, tests being generated more by AI versus the human. So that's one measure that we're looking at. Uh, same goes with, you know, uh, uh, uh, the agile planning with a user story definition, with requirement definition, et cetera.

Uh, we're measuring that shift from human driven activities to agent driven activities and to what extent that is actually being, uh, being adopted. And I'm, I'm, I'm happy or confident to say that no, we're getting these feedbacks also from our customers. So that's kind of the feedback loop we're operating in.

Uh, each new agent we introduced, uh, we validate with our partners, uh, with our customers, we were getting that early feedback, and then we evolve accordingly. And it looks very, very promising. Very promising.

Excellent. Excellent. Where can people get more information on that yev?

So you can, you can look on, uh, uh, uh, on OpenText website, look for DevOps, uh, the DevOps aviator. Uh, you'll find more information, demos, uh, references, et cetera. Absolutely.

And then, you know, we only have a minute or two left, but I, I want to, as if we're not looking forward enough, I re, I wanna spend the next minute or two looking maybe a little more forward, right? So now we are, like you said, we're getting our heads around the risk of using ai. We're getting comfortable with AI mapping out our test, uh, coverage and, and platform, uh, you know, uh, protocol and stuff.

Where do you see this going? I think that, uh, uh, I, I, I envision eventually a shift or an evolution to, uh, uh, a, I would say a, an autonomous delivery team. So if today we have, uh, you know, uh, uh, a performance engineer, a functional tester, a security expert, uh, a product owner, an engineer, software engineer, collaborating together to, uh, to deliver a business goal, uh, I believe that, uh, with, uh, the evolution of ai, we'll see a collaborative team of humans and agents, uh, to achieve a joint goal.

And, uh, I, I, I think that this is, uh, uh, you know, this is evolving as we speak, uh, and we see more and more, uh, smarter agents augmenting humans. And in some cases, yeah, you can offload the activity to agents to be delivered, whether that's fixing a defect, implementing a new feature, or a complete, uh, uh, a complete application that's not that far, uh, farfetched. And, uh, beyond that, I also see or envision, you know, more adaptive applications.

So applications that are changing their behavior on the spot based on human feedback as they are being consumed, as they are being used. And just think of it that what does it mean to have this kind of, uh, uh, ever living software and application that is continuously changing, uh, based on as they're being used? Sure.

By us. That's a totally different ballpark to what we're common to use today. That's almost like evolution right before your eyes, right?

Because yeah, as it changes with the humans, you know, it, it's, uh, I was talking to someone the other day about this. It's such an exciting time to be involved in technology with this right now, because the, the window for innovation, the, the possibilities are probably greater than they've ever been. I mean, I, you know, I was around when we, the internet went commercial, right?

And, and wow, what a, that just opened up so many things connecting the world. And every business would have a place on the web and, and all of these things, right? This is maybe even bigger than that.

So I agree, interesting times. Jan, if thanks, thank you for being our guest here on Control Alt Deploy, and thank you for all the work you're doing at OpenText. Good luck to you.

And thank you for listening in on our conversation today. We hope this is giving you a little bit of a peek into how AI and, and agentic AI generator of AI ml are all playing into today. And tomorrow's changing landscape for, for testing software.

This is now Shimel on behalf of Techstrong and OpenText. Thanks for listening.