Accelerating Delivery with CI/CD – Navigating the Journey
Many published articles discuss the benefits of using Continuous Integration/Continuous Delivery (CI/CD) in software development. One benefit would be improved code quality with bugs found earlier in the development life cycle. Another is that bug fixes can be delivered to customers more quickly. And yet another is that new features are delivered when ready rather than having to wait for the next major release cycle. Although CI/CD offers many advantages, making the transition from traditional development can be tricky. The transition is a journey. It is replete with rocky roads, peaks, and valleys. Also, it does not come with a map.
Some of you have already made this journey or some are just starting. As mentioned in a previous post, SAS recently transitioned to a CI/CD approach. We moved from a single daily build of software to multiple builds per day, consisting of several stages modeled around automation and quality. In this post, I will discuss some of the challenges we faced with accelerating the delivery of our software. Then I will share how collaboration and creating a new tool helped us traverse the CI/CD experience more effectively and efficiently.
A major benefit of CI/CD? Quick turnaround on the validation of current software with automated builds and automated test execution in a pipeline.
A big challenge? Quickness also implies failing fast and failing often. Although that could sound negative, it’s actually a CI/CD principle that evokes the embracing of failures so you can learn from them and continually improve. Since the builds are occurring multiple times per day, there are now multiple times during the day when something can negatively impact the build. These could include areas such as hardware issues, connectivity issues, new containers being added, and new code being introduced. Multiple moving parts equals multiple points for possible failures.
At SAS, Jira tickets are automatically generated for tracking problems discovered by our CI/CD pipeline. These tickets are often technically challenging to diagnose and remediate. As the influx of Jira tickets continued to rise, SAS realized the necessity of building a cross-divisional and cross-expertise triage team. The team consists of developers, testers, IT staff, and others who collaborate on ways to minimize the mean time to remediation (MTTR).
We use several products to assist us in the CI/CD journey. These include Jira (for tracking issues), Grafana (dashboards about cluster health, CPU usage, memory usage, and so on), and Kibana (a user interface for searching deployment logs), just to name a few. But given all of that, where do you start when you are investigating a ticket? Which of the products should you use? What should you look for? How do you identify the most common issues and relieve developers and testers from time-consuming diagnosis of potentially non-software related tickets? Everyone on our team approached triaging these Jira tickets from different perspectives. It was that combined mix of ideas that started leading to a recognizable pattern in the triage process. And from this process, the internal tool getTicketInfo evolved.
Several years ago, I watched the Disney® movie “Robots” with my kids. A quote from that movie stuck with me:
“See a need. Fill a need” – Bigweld (Robots)
If you have seen the movie, you are well acquainted with Bigweld’s motto. He believed that anyone could contribute to solving a problem. With SAS’ journey into a CI/CD world, that quote took on a life of its own.
As a principal developer in R&D on the Analytics Frameworks team, I represented our department on the newly formed triage team. The team consisted of developers from divisions throughout SAS, testers from those same divisions, managers, and people from the DevOps team.
