How do leaders accurately measure their team’s productivity in order to remain successful, agile and competitive? Human resources technology company Bullhorn uses a variety of metrics, some of which include the number of story points completed per sprint, points completed per person, and percentage of planned versus unplanned tickets.
Vice President of Software Engineering Ben Levine said that measuring these data points — and a handful of others — allows the team to gain a holistic view of the health of the team as well as each project.
However, while measuring productivity is important for improvement, it’s the insights and actions derived from those measurements that actually lead to positive results.
“That information, coupled with a wider review, enables our teams to share ideas about how exactly to improve productivity.” Levine said.
What metrics do you use to measure the productivity of your engineering team, and why?
We leverage a variety of metrics that are intended not just for leadership consumption and review but for the whole team to use and digest. We believe the team is best positioned to see changes in data and act on it quickly, so everything we review at the leadership level is open and visible. Metrics that we leverage to access productivity include velocity (measure of story points completed per sprint), points completed per person, points completed per day (this includes developers and testers), hotfix deliveries per release (hotfix is any ticket delivered outside of a planned release candidate), minor releases per cycle and size of releases.
Additionally, our metrics include the percentage of planned versus unplanned tickets (planned tickets are those committed to by the team at sprint planning), percentage of tickets groomed at the start of the sprint, unit test coverage per repo and automation test coverage.
We leverage this set of data points because it allows the teams and their leaders to gain a holistic view of the health of the team and each project. It also gives us the ability to deliver a consistent product to our customers while reducing the benefits of trying to game the system by seeking to drive one measure in an unnatural manner. And lastly, it allows each team to see early indications of trouble and pivot quickly.
The metrics help the teams constantly improve.”
What’s an example of insights you’ve derived from these metrics, and what actions came out of them?
In leveraging the review of hotfixes and deployments per cycle, we found that the level of hotfix delivery and associated churn was being amplified by the smaller but rapid nature of hotfix vehicles being delivered. Since each vehicle created a level of overhead and need in deployment, by narrowing down the number of vehicles to set days of the week and packaging a couple of items in each vehicle, we were able to reduce the churn in the teams and provide for more consistency in sprint delivery.
In addition, in looking at both velocity and points per person per day, we can see how changes in team capacity and alignment have an impact on the team’s delivery capacity. This helps us to understand the impact of making resource shifts across the teams and roadmap, and allows us to be much more deliberate and strategic in making team changes to minimize the disruption that the changes can cause. It also helps us to ensure we are still supporting changing customer and business needs.
What impact has tracking these metrics and acting upon the data had on your team’s productivity?
Tracking these metrics and, in particular, making them a part of the conversation that the teams have in retrospective and other forums, has had a positive impact on the teams’ productivity.
The metrics help the teams constantly improve. It allows for direct and timely feedback on the impact of changes that the team makes. It also allows them to understand what’s working and what isn’t. That information, coupled with a wider review, enables our teams to share ideas about how exactly to improve productivity.
Through the course of the pandemic, we saw our teams continue to improve in their delivery numbers across the board despite the chaotic nature of the shifts in work environment and world state occurring around them.
