Criterion 3. Measure the success of your digital service

Criterion 3 – Measure the success of your digital service

Monitor success to understand if your digital service is effective and working well.

Your responsibilities

To successfully meet this criterion, agencies need to:

understand what success looks like for the digital service
identify the most appropriate measure to monitor the success of the digital service
regularly measure and monitor the effectiveness of the digital service and act to improve outcomes.

When to apply

Apply Criterion 3 in the Live environment and consider it during Discovery. Collate metrics and monitor your digital service with a holistic approach. Report your results to help government understand the impact and benefits of its ICT investments.

The Digital Performance Standard sets the expectation that, at a minimum, agencies should be monitoring how well they support users to finish the tasks they start in digital services. Where your agency has a mature monitoring framework in place, a more comprehensive monitoring approach that captures data along the user’s digital journey should be implemented as best practice.

Monitor Criterion 3 to evaluate the effectiveness of the digital services and identify areas for continuous improvement.

Questions for consideration

What does success look like for the digital service?
What rate of users complete their end-to-end transaction online?
What data can be collected along the user journey?
How can agencies support users to finish the online tasks they start?

How to apply criterion 3

Understanding cohorts: Overlaying demographic data, such as location or socio-economic data, may help agencies to understand the outcomes of different cohorts and their interactions with digital services. If a cohort has a low success rate in completing transactions online, it may signal the need for digital service improvements.
Compare performance to non-digital channels: While a user’s whole activity may involve multiple channels, including non-digital channels, in meeting Criterion 3, it is important that agencies measure the digital component(s) of the activity separately from the non-digital channels. For example, if an end-to-end service requires multiple digital transactions and an in-person assessment prior to the service being delivered, each digital component should be measured separately.
Understand the touch points of a user’s digital journey: Capture data along the user’s digital journey. Map out the user flow and capture data from the start to the end of their journey, marked by the successful completion of their transaction. This data will help you recognise potential hurdles and drop-off points, to optimise how users complete their transactions online.
Enhancing the overall experience: Many government services have both digital and non-digital channels, designed to work together. Consider assessing the user’s journey across different channels to help understand user behaviour and identify ways to enhance the digital experience, aiming to make the whole service smoother.

Guidance to measure the success of your digital service

Begin by understanding what success looks like for your transactional digital service:
- Identify the user's objectives and goals through research, affirm their expectations, and make sure they match the agency's completion expectations. See also,Define clear objectives.
- Identify the key tasks and/or transactions that users need to complete to achieve their goals. Break down the user journey into individual, manageable components, focusing on each critical interaction point. For example, if the service involves updating personal information, key tasks might include logging in, navigating to the profile section, editing details and saving changes.
Determine the measures of success for transactions:
- Identify KPIs that indicate how well the service supports users to finish actions or tasks in the digital service. KPIs will depend on the nature of your service, and your user’s goals. For services involving applications or submissions, KPIs could include the number of applications started versus completed, the average time to submit an application, and the number of users who encounter errors and abandon the process. Refer also to Choose relevant metrics.
- Where feasible, DTA recommend measuring the number and percentage of digital tasks that are started and completed successfully versus abandoned.
- Additional metrics to monitor service success include:
  - the time spent by users on each touchpoint, task and the overall transaction
    the sources of traffic that lead users to the digital service
    errors, bugs, or technical issues encountered by users
    user behaviour patterns, such as clicks, scrolls, mouse movements, and keystrokes
    user demographics, such as age, gender, location, and language spoken
    Capturing a Customer Effort Score (CES) with a free-text feedback dialogue at points along the user journey. See also Criterion 4.
  - Once you have determined KPIs, choose tools to collect, store and analyse data. These tools should integrate with your systems and provide real-time insights and comprehensive reports. See also Choose monitoring tools and methods.
Once the digital service is operational and monitoring tools are implemented:
- Regularly analyse data and report insights within your agency to build an organisational understanding of how helpful your digital service is and identify areas for improvement.
- Act on the findings and implement improvements or changes to the service design, delivery, or maintenance, as needed.
- Test the improvements and measure for impact by monitoring for changes in data.
- Ensure that measures of success are embedded into an ongoing process of continuous improvement. See also Develop processes for continuous improvement).

Digital Performance Standard