The service I’ve been working on for the past few months is a bit different to most government services: it doesn’t involve any transactions.
This means we’ve needed to be innovative with our ways of measuring how successful it is.
Measuring success
Most government services involve some kind of transaction, such as a payment or registration. This helps when it comes to measuring success - you can use metrics such as the number of transactions or how many people achieve their goal.
Measuring success is really important, as it helps us to know:
- how much value a service is providing
- whether a service is meeting the needs of its users
- when something might need improving.
Identifying key performance indicators (KPIs) and having a way of measuring against them is also part of meeting the Digital by Default Service Standard.
But how can we measure success when a service has no transactions?
The challenge of non-transactional services
My team has developed a guide that gives information to parents resolving disputes over child arrangements, so they can make informed decisions and reach the best outcome for their child.
Our users need to understand what their options are - including going to court, using a mediator or negotiating through a lawyer.
This raises a couple of questions:
- How do you measure the success of a service which doesn’t have any transactions?
- How do you know what to benchmark it against when there isn’t an existing service to compare it to, and when users aren’t compelled to use it?
What we did
For our private beta we needed to:
- be confident that the service was meeting the user needs we’d identified
- understand the value the service was providing.
Before accessing the service, users were screened to make sure their situation was appropriate for this kind of guidance. We asked some demographic and contextual questions, as well as finding out what people were planning to do next to resolve their dispute.
We then used Google Analytics to understand how people were interacting with the service:
- which pages they were viewing
- for how long
- which external links they were visiting.
We also added custom metrics to measure things like how far down the page people were scrolling.
Finally, we had a feedback survey to get some more qualitative data. We also repeated the question asking what people were planning to do next.
What we learned
The key to our approach was the use of unique user IDs, which allowed us to match people’s responses in the screening to their behaviour on the service and to their feedback.
This gave us a detailed picture of our users and allowed us to answer questions like:
- were users more or less satisfied with the guide if they spent longer reading it?
- did users feel more or less informed if they viewed more pages or visited more external links?
- did users change their mind about what they would do next after reading the guide?
This analysis helped us learn a lot about the service and its users, and gave us the confidence to move into public beta. It also helped us identify several areas for improvement to better meet our users’ needs.
We know that there’s more to making sustainable child arrangements than understanding different options for resolving disputes, and ultimately it would be difficult to attribute a particular outcome to our guide. However, we can still understand the value of our service to parents who are going through a separation.
What next?
Since moving into public beta, we’ve removed the screening but we’ll continue to report against our KPIs using a mixture of Google Analytics and a shorter feedback survey.
It isn’t always straightforward to measure value without transactions, but it’s still important to do so. There are ways of measuring the value of content, such as traffic volumes, net promoter scores and SEO rankings, but we also want to measure the value of the guide at a service level.
We hope to use our service as an exemplar to help policy teams develop products or services.
If you’ve tried something different to measure the success of a non-transactional service then it would be great to hear from you in the comments below.
10 comments
Comment by Shane Parsons posted on
This is really interesting, but there's one thing that puzzles me.
This sounds an awful lot like "content" and I thought government policy was that all government content should be on http://www.gov.uk ? Why is this being set up outside of that? Has the policy changed?
Comment by emilyhallstrutt posted on
Thanks for your comment, Shane. We spoke to GDS about the best approach and they have quite strict guidelines on what can go on a GOV.UK content page - it has to be only informative and cannot be in any way advising people. Our service was deemed to be providing advice and therefore needed to be setup as a service on our justice.gov.uk domain. I hope that helps!
Comment by Tom Wynne-Morgan posted on
Great work guys!
Comment by Johanna posted on
Hi,
What survey tool did you use? And how did you attribute unique IDs to the survey responses?
Comment by Alejandra Diaz posted on
Thanks Johanna for your question. The team used Surveymonkey. The ID attribution was done manually, so a lot of handy work to match users and responses.
Comment by Lorra Noels posted on
Are there plans to share in a future article more details on what exactly were areas of improvement and what has been done to improve them?
Comment by Alejandra Diaz posted on
Thanks Lorra for your question.
Yes there are - As Richard commented, we are trying to find out how this product adds value at service level. On that, this information tool has helped us create a content strategy approach for messages around mediation and how we best support the sector with consistent information that support our strategy.
Additionally, we are working on other transactional services that use this product as a basis. More to come in future weeks, so watch this space. For another related article on the work on Family Justice please see: https://mojdigital.blog.gov.uk/2017/10/20/try-before-you-apply-prototyping-policy/
Comment by Alex posted on
This sounds really good. How many users did you get? Or in other words - how many responses do you think you need before you can have confidence in your findings?
Thanks!
Comment by Matrrix posted on
Great stuff!
Comment by Will Turner posted on
It's been a while since this blog was published. I'm wondering if those involved have had the opportunity to build on this approach, and if there is anything else to learn from this?
I'm only asking because this is SO useful. Thank you for sharing.