Kubernetes Deployment Validation in Two Minutes

Kubernetes deployment validation can eat up hours of an SRE’s week. Alan Shimel welcomes Sai Joshitha Kathari, Senior Site Reliability Engineer at Visa. Furthermore, she recently wrote about cutting validation time from 45 minutes to two minutes for Cloud Native Now.

A green deployment is not a healthy application

Sai has spent more than six years as an SRE working on cloud infrastructure and distributed systems. In addition, she focuses on making deployments safer and removing repetitive manual work. A deployment can report success while pods still crash or fail to come up. Consequently, engineers often log into every cluster and check every dashboard by hand.

Building Kubernetes deployment validation into CI/CD

Sai moved those manual checks directly into the Jenkins pipeline. Meanwhile, the pipeline captures the state of each cluster before the rollout begins. After the deployment, it checks pod readiness, failure rates and workload stability. As a result, the pipeline flags crash loops and failed pods right in the Jenkins logs.

Teams running 20 or 30 clusters no longer have to inspect each one. Therefore, engineers can go straight to the cluster that failed with the logs already in hand. Furthermore, the time saved can go toward troubleshooting and new automation. Sai stresses that a deployment is not finished until validation also passes. As a result, every SRE team can benefit from automating the checks it already runs by hand.

Sharing knowledge both ways

Alan praises Sai for sharing her experience with the wider tech community. In addition, Sai hopes other SREs and DevOps teams will adopt the approach and suggest improvements. Consequently, she sees reliability work as a two way exchange of ideas. Alan calls it one of the easiest interviews he has ever done.

Explore more DevOps coverage and the latest Techstrong TV interviews on Kubernetes deployment validation and cloud native operations.

For more information please visit cloudnativenow.com