Navigating to AI-Assisted Continuous Testing in DevSecOps With Marc Hornbeek at AIE 2024

Welcome to my talk, navigating to AI assisted continuous testing in DevSecOps. My name's Mark Hornbeck. I'm known as DevOps the Gray, because I've been practicing what is now called DevOps and DevSecOps and continuous testing for many years, believe it or not, since the late 1970s, long before those were even words.

Um, I'm kind of dating myself now, but you know, even back in the 1980s, I was leading technologies for testing at Bell Northern Research, including automated generated test methodologies from executable specifications and automatically analyzing large scale test results using combined outputs from network systems and products. Uh, since then, I've played leading roles in developing international standards for protocol testing, and I've been a leader over developing many test products for, uh, networks and commercial enterprise systems. Uh, I'm a a 48 year lifetime member of IEE.

Way back in 2016, I was awarded the outstanding IEE engineer by, uh, the engine, by the IEE, of course, uh, for my career achievements in the area of automated testing, DevOps, and automation of software protocols and network systems. I founded a company that, uh, uh, develops a product to automate test environments using robotics. More recently, I've served as a strategic transformation consultant for network manufacturers, operators, and enterprises.

Today, I offer freelance consulting services through my company, engineering DevOps Consulting for organizations that need senior guidance for transformations of their continuous testing, DevSecOps, and reliability, uh, practices. This talk reflects my recent experience of incorporating tools using machine learning and generative AI tools, such as chat, BT copilot, asage perplexity, and specialist tools into the DevSecOps continuous testing transformation projects. In my talk today, I will explain how to transform an organization that is currently using continuous testing as part of their DevSecOps solution towards AI assisted continuous testing.

I'll cover what is AI continuous testing. Uh, I'll compare AI assisted testing to existing continuous testing. I'll highlight some of the key challenges that I have found during different types of transformation projects that use AI enabled tools.

I'll explain some tools and transformation practices that I find useful to ensure success with assisted, uh, continuous test transformations for DevSecOps. Finally, I'll conclude with my own view of benefits and, uh, some radical prediction for the future of DevSecOps. I'll wait for that.

Um, so first, uh, I think it's important you know, that you, you, we have a common understanding of what is continuous testing. There, there is no standard definition in my book, uh, engineering DevOps. I say that continuous testing is a strategy designed to reduce lead times and failure rates and continuous delivery pipelines and continuous operations through the use of automated and iterative testing processes, all aimed at decreasing time from code, commit to production, deployment, and reduce failure rates in production.

A key, you know, myth I feel like, and, uh, a point of, uh, clarification is that continuous testing is a lot more than simply automating the execution of tests. In my book, I predicted that AI would someday enable what I call in the book autonomous continuous improvement in which DevSecOps systems can improve themselves. I believe we're on the verge of that now with generative AI tools and ML tools and related tools.

This rather complicated looking diagram, which we don't have time to go through in detail, but it's a blueprint for my representation of continuous testing processes and high level workflows in relation to product value streams with embedded DevSecOps pipelines, certainly there are a lot of parts to get right, uh, with continuous testing. Software changes are assessed for quality and security throughout the pipeline, and tests and security checks are shifted as far left as possible to avoid finding problems late in the pipeline where it is less efficient to fix them. I define nine pillars of practice for continuous testing that are all important to achieve a well-engineered continuous testing solution for DevSecOps.

Uh, this includes special continuous testing tenets, practices for leadership, special practices for culture test strategies, management, data automation, tooling environments, and results analysis. Actually, all of these re uh, practices are important to get right and all of them can be in fact assisted by AI enabled tools. I completely agree with the report that's shown on the left here.

Uh, the 2023 state software quality report that says generative AI and machine learning are game changers for the area of continuous testing. Every aspect of the testing process can be, you know, transformed, starting with test requirements through test generation, synthetic data generation, defect and vulnerabilities, detection and correction, test optimizations, test maintenance, test report, generation analysis, scheduling and management, as well as improving the testability of the applications themselves. In general, my experience with AI assisted testing shows that the cycle from test strategy to getting test results can be reduced from days to hours if they're well engineered, you know, solutions.

The current state of the art or AI assisted continuous testing requires that a continuous testing domain expert guide AI agents and tools, because AI tools do not yet have the capacity, nor can they be trusted, uh, to run and forget, you know, from the beginning of a test, uh, to the end of the results analysis. Um, so you need detailed supervision at each step to really monitor and supervise the testing workflows. Among other reasons listed here, it's vital that for an expert to set standards and con and the context for each of the testing practices and workflows, the expert work side by side specialized AI agents, which perform specialized roles for content evaluation and refactoring under the direction of the expert.

Some of the recommended practices is to have dedicated AI agents with personalities for each function role such as developer, qa, architects, security operations, and various others. Uh, you can have the bottom line is, at least for now, the effectiveness of AI assistance depends on the wisdom and skills of the domain expert and the combined capabilities of those AI agents. Uh, this is my blueprint for AI assisted continuous testing.

You can see that AI enabled tools are configured in a tool chain for orchestration, automation and results analysis of the tests. And together with AI agents and, uh, you know, a domain expert directs the test workflows for all the different testing stages across the continuous testing spectrum to ensure consistency. And this is really a critical point to ensure consistency and repeatability of testing and test results.

Every workflow is needs to be supported by templates for artifacts such as test strategies, test plans, test cases, test scripts, environments, results, analytics. Uh, the templates are needed, um, to make sure you have, uh, consistency and usability throughout the process. And they're created.

The, the templates themselves can be created, validated, and managed by AI assisted workflows to ensure that the AI tools can use them during the workflows. All the templates and artifacts are need to be version managed to ensure traceability and to support diagnostics as well as rollbacks. Besides the usefulness of the gen AI chat tools, uh, AI and machine learning has given rise to many new tools for that are useful for testing and new variations of existing test tools that leverage the new opportunities that AI and machine learning provides.

The chart shows some of the notable examples. Uh, as you can see, there are a rapidly growing number of AI enabled tools that specialize in different testing tasks, such as generation of tests, management of testing, uh, data generation, user testing, and many other functions. Every continuous testing transformation must evaluate the set of tools and tool chains that are relevant for each specific DevSecOps transformation and tech stack that is relevant for the an application.

There's no, uh, ones tool fits all at this point, and not sure there will be any time soon. Uh, given the lack of hard standards and the rapid pace of changes to tools, it's important for organizations to build solutions that are flexible to changes in those tools as well as the technologies at the workflow level. Continuous testing consists of a series of steps that are both macro and micro flows.

Ev this is true for AI as well as you know, prior to ai. Uh, every step in the macro flow consists of five micro flow steps, including research, artifact creation, artifacts, review, artifact modification, and approval. In general, the micro flow is repeated for each micro step, sorry, each macro step, including requirements analysis, test strategy, test plans, test cases, test scripts, test environments, campaigns, executions, reports, reviews, remedies and, and test regressions.

Without ai, each micro flow may take hours or even days to complete, and of course, the macro flows may take, you know, uh, days, weeks with AI assistance, the micro flows can be completed in seconds or minutes. I found that a well engineered AI assisted workflow can dramatically improve overall lead times by as much as 50 to 90%, and labor costs can be reduced by as much as 70 to 95%. So it's, uh, you know, with the right, right engineered solution, you can get major, major gains.

When we talk about transformation of continuous testing to ai, it's important to know that there are different types of transformations. Each transformation requires specific considerations. In an AI assisted continuous testing transformation strategy, technology transformations are intended to transform existing continuous testing tools and processes to use ai.

Procedural transformations are intended to transform existing tests, creation, execution, and data generation to AI generated, uh, artifacts. Organization transformation, focus on re-skilling and up-skilling the workflows, teams and cultures to understand and leverage AI tools effectively, and especially to identify and build up the, you know, the, um, domain experts that are gonna direct the AI tools. Uh, and finally, quality assurance transformations focus on quality improvements using AI to enhance test coverage, accuracy, continuous improvements, and overall improved compliance and security.

So in a specific transformation project, it may be unrealistic to transform all of these things at once. Therefore, you know, the, the organization needs to clarify its priorities in its transformation roadmap and strategy anyway, um, you know, a lot of things can go wrong in these transformations. There's a lot of parts here to get right.

Recent state of the DevOps surveys confirm what I see in practice with, you know, transformation clients. Um, DevOps transformations start off with high expectations always, but unfortunately they often get bogged down or stuck in the middle of level of performance. Uh, and as a strategic, a strategic transformation consultant, I see too many organizations kind of overly anxious to jump into the implementation phase and skip the, so-called non-value added strategic alignment and planning phase, which is a huge mistake I know consistently, uh, when you skip that, there are plenty of pitfalls that can derail the transformation implementation.

Uh, if you do that, I mean, the very power of AI and automation without the right strategy and controls can quickly lead to inconsistent and unrepeatable test results. Tests that are hard to understand, tests that are not relevant, tests that require overly complex test environment variations, lack of aversion management tests that depend on unavailable observability features in the application. And a lot of other factors I'm showing here.

The point is, is that you definitely, you know, the recommendation is that you do have a strategic transformation, uh, in mind. Um, before you get going on your journey for transforming to ai, in fact, you need to for, you know, um, a basis for continuous testing in the first place to even augment it with ai. So this is my transformation blueprint, and, uh, just briefly we'll talk about it.

The, the chart illustrates, um, that, um, you know, there's a logical path to success, and if you follow this path, it absorb, it avoids, uh, typical transformation pitfalls. I mentioned the blueprint is based on the same blueprint I have in my book seven Step transformation blueprint. Um, but it's, you know, it works the same for ai.

One of the main differences is that it puts particular emphasis on getting, uh, standards and tools in a more, you know, executable format that the AI can process. It starts with vision and goal alignment. Assessment then includes things like discovery surveys, gap assessments, value stream assessments.

Then there's a solution arc, you know, alignment, which includes getting a strategic plan and a roadmap. And that provides guidance for the, you know, the implementation, including, you know, building a an an example solution, uh, operationalizing that solution and then expanding it to other applications. So that's the logical recommended flow.

All of these transformation phases are accelerated using generative AI assistance guided by a transformation consultant. Uh, this is an example of an AI assisted continuous testing strategic roadmap. Uh, a roadmap is a logical progression, and in this case, we're starting with a theme focused on preparing AI templates, AI tools, selection, uh, defining the priority processes and, and then training people for AI assisted test generation and metrics to monitor changes.

The second theme, building on the first theme standardizes AI tools into the continuous delivery part of the pipeline and applies more automation to the results side of testing. And the third theme is, um, uses AI to expand test coverage for delivery and release automation. The final, uh, stage or theme automates testing for deployment to production and ongoing optimization of test activities.

Every organization should develop their own roadmap tailored to their own vision, goals, preferences, tech stacks, and so on. So it's my experience that each of these themes takes about four to six months to implement when properly resourced. In a large enterprise, a critical decision for every transformation is to pick and organize an i an organization team implementation pattern.

This chart, uh, illustrates four different patterns that I found do work. Uh, selecting the right pattern can lead to a smoother transformational, higher team adoption rates, more substantial improvements in testing quality security feedback, and generally speaking, less confusion as you go through the implementation phases. For example, the hybrid dedicated slash outsource team pattern offers a pragmatic solution for organizations aiming to enhance their organization capabilities while maintaining strategic control over key processes.

It combines the depth of dedicated teams with the breadth of outsourcing to provide flexibility, cost efficiency, and access to specialized skills. However, the success of this pattern largely depends on the effective integration strategies, strong management practices, and careful selection of outsourced partners. The right pattern should align with the organization's specific needs, strategic goals, and the inherent risks and benefits associated with combining external team dynamics.

Certain implementation patterns shown here have pretty much consistently shown to be less effective or even detrimental. Uh, siloed department teams suffer from poor communication and efficiency, and delays over reliance on outsourced teams suffer from a lack of control, knowledge, drain, and security risks. Top down hierarchical teams suffer from resistance to change, lack of motivation, and slow adoption.

Fixed teams, uh, without cross-training at least suffer from resource vulnerabilities, reduced flexibility, skill stagnation. And finally, project isolation suffers from reinventions, you know, not invented here type of thing. Missed synergies, inconsistent standards in general, identifying and avoiding unfavorable implementation patterns is also critical to, uh, have an effective transformation to effectively motivate and manage the transformation.

Start by educating stakeholders about the benefits and practical implementa implementa, sorry, implications of AI assisted continuous testing. Then align the initiative with the organization's broader goals to gain buy-in and alignment at all levels, especially top management. Uh, use pilot projects to demonstrate the concrete benefits of AI and testing.

Select projects that are manageable in scope and likely to show quick wins, such as reduce testing times, improve coverage, quicker identification of bugs, then invest in training and development for your team to build AI and testing competence, uh, simultaneously upgrade your IT infrastructure to support the new tools and technologies that are essential for AI assisted testing. Encourage collaboration between developers, testers, AI specialists to create unified approaches. Implement robust feedback mechanisms to gather insights.

You know, by employing all these strategies, organizations can, you know, not only motivate the team, but smoothly transition, uh, leading to significant gains in productivity to effectively measure the value of AI assisted continuous testing. A combination of Dora metrics, uh, and other key performance indicators, uh, should be used. These metrics help assess the impact of AI on the testing processes and DevSecOps.

Uh, it can help track improvements in efficiency, quality, and overall performance of the development cycle. So the four door metrics, if you're not aware of them, uh, they're pretty well known as lead time for changes, change, failure rate, time to restore service, um, that's only three oh oh yeah, deployment frequency as well. They're critical for understanding that performance of software development and delivery processes.

Uh, there's lots of other metrics that can be valuable, especially for continuous testing. So test coverage, defect detection, efficiency, automated test pass rates, flakiness scores on the tests, how good are they in the reliable test execution time, the cost for a defect, uh, feedback loop duration, ROI of testing. So by tracking these metrics, uh, you know, you can quantitatively assess the impact of integrating AI into testing practices and guide decisions throughout the transformation.

Calculating the return on investment for a transformation do AI assisted testing involves considering both the tangible and intangible benefits, uh, versus the costs incurred, the calculation, you know, helps in determining the financial viability and strategic value of adopting AI technologies and testing initial and ongoing cost factors to consider our development. Integration costs, software costs, hardware costs, training costs, maintenance costs, uh, the quant, the tangible benefits tend to be, you know, efficiency related, reduce labor costs, improve quality co, you know, uh, improvements, I should say, and, and enhanced test coverage, uh, and benefits related to the, to that. Um, in addition, you know, there are intangible benefits such as improved morale, productivity in general market reputation that should be at least mentioned in any kind of business case.

So a comprehensive understanding of the ROI, uh, allows for more informed, you know, business case decision making. Okay, so by now, you know, I've shown you that a transformation to AI assisted continuous testing requires investment in a strategic approach. The good news is that continuous testing has proven to deliver major benefits that make the investment worldwide.

Uh, the world quality report indicates major benefits to on-time deliveries, improve productivity, reduce cost to quality, improve customer experience, improve product. You know, I said that already. Quality release frequency, security and compliance.

In fact, up to 70% efficiency gains have been reported in reduced cycle times even without AI assistance, uh, just to, you know, transform to continuous testing. Uh, so in my estimate with ai, uh, continuous testing, you can expect an additional, you know, 50 to 70% on top of that. So where is AI continuous testing leading in the future?

Continuous testing dramatically shortens continuous delivery to production cycles by shifting testing and security activities left, so they can be dispatched efficiently. I envisioned perhaps radically that as cycle times continue to shrink and smart automated testing activities are ever more efficient in combination with effective use of smart feature flags, we'll get to a point where DevSecOps can be replaced by Dev and ops. There will be no need to shift left.

Uh, I'd love to hear whether or not you agree with this prediction, and if so, when will it become the norm? So this concludes my talk on navigating to AI assisted continuous testing in DevSecOps. I hope you enjoyed it.

I'm currently writing another book on continuous testing, quality security, and feedback for DevSecOps that's scheduled for publication this year. The book includes use case for AI assisted tools, not only for testing, but also related topics of quality, security and feedback. com.

Thank you.