Measuring the quality of software projects Using DORA to determine the DevOps ROI
17.02.2022A guest post by Sujeevan Vijayakumaran *
When collecting DevOps metrics, it is important that a meaningful benchmark is used. So now the question arises: what is being measured? How is it measured? And: What conclusions are drawn from the data that result from this?
Related companies
DORA has developed four metrics to measure and evaluate the performance of software development teams.
Many companies have been using DevOps for several years. One of the essential features of DevOps is the collection of metrics. This makes it possible to assess on a fact-based basis how good the software that is currently being developed is, where there is a need for optimization and where perhaps even more work would have to be incorporated in order to increase productivity.
The larger the company, the more common it is that there are concrete key figures for various projects; especially with regard to the financial aspect, e.g. the IT expenses for hardware, software, personnel and infrastructure are well known.
Things are a little different at the moment when you look at the productivity of software development teams. It is often only developed in advance, without it being clear what the final result looks like in detail and whether there is potential for optimization.
In this context, one can often even speak of a development in blind flight. Changes in the culture and processes of the teams have a direct impact on productivity and the results (of the teams). But here the essential question arises: how is it measured and why exactly so?
Because even with DevOps and agile processes, it is possible to identify efficiency and relate it to business value. It is important that the entire organization is involved, which works together on the same goal and on the basis of the same vision. Simply put, the better the possibilities for collecting clear measurement data, the better you can focus on ROI.
DORA, which stands for “DevOps Research and Assessment”, offers help here. The team behind DORA conducts research within the framework of DevOps. They have developed a total of four metrics to measure and evaluate the performance of software development teams. This makes it clear what the maturity level of DevOps is in the organization.
The DORA metrics
In total, DORA consists of four metrics that are relatively easy to implement and provide a good basis for the Metrics initiative.
1. Deployment Frequency
Specifically, it is about measuring how often an organization can successfully deploy to the production environment. The further away the organization is from DevOps principles, the longer the time span between two deployments. In traditional organizations, it is often considered normal for the software to be rolled out on production systems only once a quarter, for example.
In DevOps teams, on the other hand, the frequent rolling out of the software is the rule in order to make the changes available to users faster and to be able to react much faster to errors. The higher the degree of automation, the more often a deployment is performed. Furthermore, the higher the deployment frequency, the more likely the organization and its developers are to rely on their own software and its quality.
2. Lead Time
The second metric is the question of lead time. How long does it take for a commit to be created and rolled out to the production environment. This metric is also an important indicator for a well-functioning and productive software team.
3. Change Failure Rate
The first two metrics are more focused on rolling out the project in production environments and less on frequency and speed. The other two metrics focus on errors that occur in production environments.
The Change failure rate shows the frequency of errors in deployments to the production environment. What percentage of the changes lead to problems in terms of reliability for users, after which it is necessary to intervene again? The fewer errors of this kind, the more productive the team.
4. Time to Restore Service
The fourth and last metric measures the time from the point at which an incident or a serious error occurs until it is corrected and the application is available to users as usual. Again, the faster problems can be eliminated, the better the quality of the entire project.
In addition to these four metrics, there is always the question of whether there are possible security vulnerabilities in the project. Unsurprisingly, the lower the potential security gaps, the better.
Complexity is not negligible
The DORA metrics are a good and reliable means to make the quality of software projects measurable with regard to DevOps. However, collecting these data points is not easy. Often, a number of tools are included in the DevSecOps toolchain, which make the whole thing much more difficult.
The main problem with this: the data comes from different tools and is stored in different places. So a lot of time is wasted trying to capture these metrics. This is a loss of efficiency, because these systems also have to be maintained so that the instrumentation works without problems. This often results in a rather complex collection of the data for the metrics. Or this data is not collected at all, if not even working with incorrect data.
A way out is possible if a fully integrated DevOps platform is used, in which these presented DORA metrics can be automatically recorded. The organizations can then better understand and assess the speed, overall quality and stability of the projects by means of fewer steps.
This results in a consistent transparency that is easy to understand for all teams, including CISO. This also includes aggregation at group and instance level, including a superordinate trend. This makes it clear how the situation is at the moment and whether it may not be necessary to take countermeasures – and that at the various levels. Also, improvements can then be implemented almost directly, since everything is in one tool.
Conclusion
Sujeevan Vijayakumaran
The DORA metrics make the quality and productivity of DevOps in the organization more usable and support an efficient ROI analysis. This approach takes a clear look at any problems and challenges that still need to be solved in order to ultimately deliver better products for end users.
* Sujeevan Vijayakumaran is Solutions Architect at GitLab and author of the book “Versioning with Git”. Every day, he supports customers with the transition to GitLab, so that they not only use source code management, but also use as many functions of the DevOps platform as possible to accelerate their software development cycle.
