5 minutes read

POSTED Dec, 2022 dot IN CI/CD

8 CI/CD Metrics You Should Monitor

Tolga Takir

Written by Tolga Takir


Software architect

linkedin-share
 X

In a modern software development process, several steps are typically carried out to create a software system. These steps often build on each other, with each step providing the foundation for the next.

One important step in this process is integrating code from different developers into a single codebase. This typically involves merging the code changes made by different developers into a common repository, such as a version control system. This step is important because it allows developers to collaborate and work together on the same codebase, which can help ensure that the code is consistent and high-quality.

Another significant step in the software development process is deploying the codebase to some infrastructure. This typically involves transferring the code from the development environment to a production environment, where users can access and use it. This step is important because it makes the software available to users and ensures it runs in a stable and secure environment.

These two steps are crucial for creating a high-quality software system, typically carried out multiple times throughout development.

With a CI/CD pipeline, we can automate most work related to integration and deployment, enabling high development velocity. These pipelines allow us to react to changes in the market quickly and ship new features without a manual approval process. They also help get bug and security fixes into production fast, so users aren’t annoyed or at risk of a breach in their private data.

But with all automation comes the requirement to see—from the outside—what’s happening. This is especially true if you deploy your applications to an external cloud provider’s infrastructure. That’s why it’s crucial to monitor your CI/CD pipeline, just like you would monitor your other software systems used in production.

The problem with monitoring is that it’s hard to know what to measure. You could gather many metrics for your pipeline and makeup infinitely more on your own. But nobody has time for trial and error here; that’s why we’ll look at some of the most well-known metrics CI/CD pipelines in this article.

1. Test Failure Rate

The pipeline that runs our test is itself software that can have errors. The same goes for the software that is tested by your tests. You must ensure that everything runs smoothly and find the causes of these errors quickly.

Again, Foresight has capabilities that can help mitigate the amount of work that usually goes into debugging tests. Foresight maintains a separate list for tests that errored because the pipeline failed or the tested software was buggy.

2. Test Durations

Testing can quickly become the most significant step in your pipeline. More tests can lead to better quality but also slow your development velocity. It’s important to observe and optimize your tests to ensure they aren’t slowing down development and that they provide you with the insights you need.

Monitoring systems like Foresight are tailored for automated testing and can help keep things in check. With Foresight, your tests are sorted by their execution time, so you know what’s happening at a glance. If you instrumented your production code with Thundra APM, you could even dissect slow tests to determine which code or service is causing the slowdown.

3. Test Coverage

Test coverage is a measure of the amount of testing that has been performed on a software system. It is typically expressed as a percentage of the number of lines of code, branches, or other software units executed during testing.

For example, test coverage of 80% means that 80% of the lines of code in the software have been executed during testing. Test coverage is an important metric for assessing the thoroughness of a software testing process, and it can help to identify areas of the code that may not have been adequately tested.

By ensuring high test coverage, developers can increase the likelihood that their software is free of defects and high quality.

4. Change Volume

How many changes are included in each cycle? Depending on your cycle time, you might need to optimize your change volume. The higher the overhead of each cycle, the more changes per cycle you’d want to make.

The change volume metric is a measure of the amount of change that has occurred in a software system over a given period of time. This metric is typically calculated by counting the number of code changes (such as lines of code added or removed) made to the system during the period of interest.

The change volume metric can be useful for understanding the level of activity in a software project and can help to identify periods of time when the codebase is undergoing significant change.

This information can be useful for planning and prioritizing future development work and identifying potential risks or issues that may arise due to the changes. Additionally, the change volume metric can be useful for identifying trends or patterns in the rate of change over time, which can provide insight into the overall health and stability of the software system.

5. Deployment Frequency

How many deployments are done per period?

Again, some companies deploy hundreds of times per day; others only deploy once or twice a year. As your application accumulates features, it also gathers code. This code slows down your pipeline, making you less agile over time.

A slowdown in development velocity is expected, especially in the first year of a new application. But if you see this metric break down suddenly, there might be something seriously wrong in your codebase, and you should investigate further.

6. Cycle Time

The cycle time is the time your whole pipeline takes from start to functional deployment. Some companies take days to deploy a new version after all the coding is done, while others deploy hundreds of times a day.

The shorter your cycle time, the faster you can deliver. If you only need a few minutes for a deployment, you can fix a problem in production quicker, freeing you to try new things without the fear of losing hours of work. Taking advantage of these experiments, so often postponed out of fear of failure, are what will get you ahead of your competition.

7. Team Retention Rate

What do the people who use the system think about it? Are they avoiding using it for their changes, preferring to “drop into VMs with SSH” instead? If your teams circumvent your pipeline, you should ask why. After all, it costs money to run a CI/CD pipeline, and it should save your team valuable time; if this isn’t the case, you don’t need the pipeline in the first place.

If the team retention rate goes down, ask your developers what’s wrong; sometimes, a few changes in the pipeline settings or a bigger VM can be enough to make people use it again.

8. Rollback Speed

How quickly can you restore an older version in the event of a significant error? The faster you can do it, the less money you’ll lose when it happens. Even if every other metric is good, a slow rollback can still break a company. Things can and will go wrong, and if you take too long to recover, your customers might flee to a competitor.

At the same time, if your team members think a breaking error could cost them their job, they might pour more time into quality assurance than is needed, slowing down the whole development process. This can ultimately lead customers to choose a competitor anyway.

Conclusion

After automating all of your integration and delivery tasks, make sure they are—and keep—working correctly. This is just as true for your CI/CD pipeline as it is for your production software.

You can add countless metrics to your repertoire over time, but the ones we’ve covered here are a good place to start. With experience, you’ll gain an intuition for what’s missing and make more informed decisions about what metrics to remove or add.

If you’re concerned about your test suites, gain visibility in no time with Foresight.

blog-banner-1 (1)