The service I’ve been working on for the past few months is a bit different to most government services: it doesn’t involve any transactions.
This means we’ve needed to be innovative with our ways of measuring how successful it is.
Most government services involve some kind of transaction, such as a payment or registration. This helps when it comes to measuring success - you can use metrics such as the number of transactions or how many people achieve their goal.
Measuring success is really important, as it helps us to know:
- how much value a service is providing
- whether a service is meeting the needs of its users
- when something might need improving.
Identifying key performance indicators (KPIs) and having a way of measuring against them is also part of meeting the Digital by Default Service Standard.
But how can we measure success when a service has no transactions?
The challenge of non-transactional services
My team has developed a guide that gives information to parents resolving disputes over child arrangements, so they can make informed decisions and reach the best outcome for their child.
Our users need to understand what their options are - including going to court, using a mediator or negotiating through a lawyer.
This raises a couple of questions:
- How do you measure the success of a service which doesn’t have any transactions?
- How do you know what to benchmark it against when there isn’t an existing service to compare it to, and when users aren’t compelled to use it?
What we did
For our private beta we needed to:
- be confident that the service was meeting the user needs we’d identified
- understand the value the service was providing.
Before accessing the service, users were screened to make sure their situation was appropriate for this kind of guidance. We asked some demographic and contextual questions, as well as finding out what people were planning to do next to resolve their dispute.
We then used Google Analytics to understand how people were interacting with the service:
- which pages they were viewing
- for how long
- which external links they were visiting.
We also added custom metrics to measure things like how far down the page people were scrolling.
Finally, we had a feedback survey to get some more qualitative data. We also repeated the question asking what people were planning to do next.
What we learned
The key to our approach was the use of unique user IDs, which allowed us to match people’s responses in the screening to their behaviour on the service and to their feedback.
This gave us a detailed picture of our users and allowed us to answer questions like:
- were users more or less satisfied with the guide if they spent longer reading it?
- did users feel more or less informed if they viewed more pages or visited more external links?
- did users change their mind about what they would do next after reading the guide?
This analysis helped us learn a lot about the service and its users, and gave us the confidence to move into public beta. It also helped us identify several areas for improvement to better meet our users’ needs.
We know that there’s more to making sustainable child arrangements than understanding different options for resolving disputes, and ultimately it would be difficult to attribute a particular outcome to our guide. However, we can still understand the value of our service to parents who are going through a separation.
Since moving into public beta, we’ve removed the screening but we’ll continue to report against our KPIs using a mixture of Google Analytics and a shorter feedback survey.
It isn’t always straightforward to measure value without transactions, but it’s still important to do so. There are ways of measuring the value of content, such as traffic volumes, net promoter scores and SEO rankings, but we also want to measure the value of the guide at a service level.
We hope to use our service as an exemplar to help policy teams develop products or services.
If you’ve tried something different to measure the success of a non-transactional service then it would be great to hear from you in the comments below.