Criterion 1. Implement a monitoring framework

Criterion 1 – Implement a monitoring framework

Develop a digital performance monitoring framework to fit your digital service.

Your responsibilities

To successfully meet this criterion, agencies need to:

define clear objectives and goals, based on users’ needs
identify and choose relevant metrics that align with organisational goals, meet Digital Performance Standard criterion and capture the user experience
articulate how you will implement the monitoring framework
develop processes for continuous digital service improvement and share insights.

When to apply

Apply Criterion 1 during Discovery of ICT investment planning and as part of your continuous improvement processes in the Live environment.

We recognise agencies have different digital and ICT capabilities. The Digital Performance Standard sets out application approaches based on the maturity of your monitoring framework. Every agency should work towards applying best practice monitoring and reporting methods. At a minimum, your agency should implement a monitoring framework that focuses on providing digital service outcomes based on user experience and feedback.

As best practice, agencies should implement a monitoring framework that embeds user-centric performance monitoring across all your informational and transactional services.

Questions for consideration

What are the online needs and expectations of the users?
How does the framework fit into the strategic landscape?
What resources are required for successful implementation and maintenance of the monitoring framework?
How can baseline metrics be established to understand the current state of digital performance and facilitate future comparisons?

How to apply criterion 1

Establish a performance monitoring framework: Use a performance monitoring framework to understand the digital platform’s real-world impact and how users interact with digital services. The framework should be established from an end-user perspective, not from the perspective of an agency’s infrastructure. Use clear objectives and goals framed in the context of what users need and expect from the digital service.
Key performance indicators: Apply measures to achieve the outcome as set out in the Digital Performance Standard and to support your organisational goals. They should be specific and measurable and further your agency’s understanding of how users interact with your agency on digital platforms. Metrics need to be meaningful to understand and improve user experience. Meaningful metrics are crucial to the overall success of the framework.
Apply a best-practice approach: Implement a performance monitoring approach that is comprehensive and focuses on the end-user experience. Where best practice cannot be achieved or does not line up with your agency’s other metrics, strive to introduce best practice concepts over time.
Continuous improvement of the user experience: Integrate processes for continuous improvement with a focus on user-centric benefits. Data and feedback should be regularly analysed to find improvement opportunities to enhance overall user experience.
Use a baseline to measure performance: Establish a baseline for your digital service performance with data gathered from your digital service. A baseline can identify areas to improve a digital service in line with user expectations.
Share insights and learnings: Share your insights and learnings with the DTA and other agencies. A collaborative approach to digital experience will support whole-of-government standardisation of digital services, build digital and ICT capabilities and deliver a consistent customer experience. The DTA will support agencies by incorporating insights and best practices in its guidance documents and toolkit.

Guidance to implement a monitoring framework

To develop your monitoring framework, start by setting clear objectives and goals for your digital service based on user needs:
- Conduct user research to identify the needs, expectations and goals of the users of the digital service.
- Use service design methods to further interpret user research and document the user perspective.
- Define the desired outcomes and impacts of your service for your users based on the user perspective.
Choose metrics that align with organisational goals and meet Performance Standard criteria:
- Consider the user experience, service benefits and individual agency objectives.
- Create key performance indicators (KPIs) that are meaningful for your service and your agency.
- Make sure the KPIs can be monitored over time to enable continuous improvements.
- Check your KPIs align with criteria 2, 3 and 4 of the Performance Standard.
- Choose metrics to measure against your KPIs — for example, user satisfaction, completion rates and service availability.
Choose appropriate monitoring tools and methods to collect, store, and analyse performance data:
- Research different tools and methods that can capture, store, and analyse data related to your KPIs, referring to guidance for criteria 2, 3 and 4.
- See what tools and methods other agencies use to understand best practices.
- Collate your research and create a shortlist of tools and methods.
- Make sure tools and methods meet the legal and ethical requirements for data collection and analysis.
- Document your choices and rationale to make an informed decision, for example by conducting a cost-benefit analysis, considering factors such as:
  - alignment with your chosen performance indicators
  - ease of use
  - adaptability, compatibility and ease of integration
  - ability to produce comprehensive reports
  - scalability
  - cost
  - security and reliability.
Articulate the monitoring framework in a document that covers data collection reporting:
- Include information such as data sources, methods, frequency, formats, and the roles and responsibilities of staff and teams.
- Establishing clear rules and maintaining thorough documentation to detail quality assurance mechanisms will provide clarity for staff involved in data processing.
- Data validation (the process of ensuring data is checked for errors and anomalies), data verification (ensuring data is accurate and consistent against source or reference points) and data cleansing (identifying and correcting inaccurate data) procedures will ensure the collection and use of high-quality data.
- Plan how data and insights will be presented and communicated, for example a dashboard, spreadsheet or a report.
- Explain the process for reviewing and acting on the performance data, include the key decision makers, responsible persons and key stakeholders.
- Review and update the document regularly based on the effectiveness of the monitoring framework, new information and changing needs or priorities.
Benchmarks are reference points to compare your performance with other services, agencies and sectors. They help you set realistic and achievable targets or benchmarks for each KPI based on user expectations and best practices.
Setting benchmarks and targets:
- Review the benchmarks and targets in similar services within your agency, government, or the private sector.
- Use the benchmarks to inform the targets for each KPI. For example, you could examine the average task time or drop-out rate for completing a similar task in different services — these targets should reflect user expectations, best practices and the strategic goals of your agency.
- Check your targets and benchmarks are specific, measurable, attainable, relevant and time bound (SMART). Targets and benchmarks should reflect the available resources of your service.
- Document the choices and rationale for setting these targets or benchmarks, and communicate them to relevant stakeholders, senior management or other agencies.
Develop processes for sharing insights and continuously improving the digital service:
- Make sure the people responsible for performance monitoring work closely with those delivering the digital service.
- Frequently assess your service performance data and metrics and insights in comparison to your benchmarks or targets, pinpointing areas where you excel and where improvements are needed.
- Use data-driven insights to inform decisions and actions for enhancing the quality, efficiency and effectiveness of your service.
- Implement agile and iterative methods for testing, experimenting and deploying changes to your service.
- Use prototypes, minimum viable products, or beta versions to validate assumptions and hypotheses and measure the changes in your performance data.
- Document findings, learnings and best practices from your service improvement processes. Share these with your team, senior management, other agencies or the public.
- Foster a culture of learning and innovation in your organisation and encourage collaboration and knowledge sharing across teams and agencies.
- Seek feedback and input from your peers, mentors, experts, or external partners.
- Participate in communities of practice, forums, or events related to your service domain.
Combining user research efforts across the DX Policy and its standards can help to reduce duplication and the cost of research.

Digital Performance Standard