Pilot AI assurance framework guidance

If the data used to operate, train or validate your AI system, or any outputs from your AI system, are Indigenous data in line with the Maiam nayri Wingara definition above, you should refer to the guidelines in the GID.

This includes applying the principles of respect for cultural heritage, informed consent, privacy (including collective or group privacy) and trust, to all stages of the ‘Data Lifecycle’. These concepts, including the FAIR (Findable, Accessible, Interoperable, and Reusable) and CARE (Collective Benefit, Authority to Control, Responsibility, Ethics) principles, are described in the GID.

Relevant practices to consider in this context include:

• Checking if datasets used to train the AI included diverse and representative samples of cultural expression, artifacts, languages and practices. This supports the AI system being able to recognise and appropriately respond to a greater range of cultural contexts in a less biased manner.
• Describing any mechanisms in place for engaging with Indigenous individuals, communities or group representatives and collecting and incorporating their feedback on the AI system’s performance, especially regarding cultural aspects.
• Describing processes to review documentation and protocols that ensure the project has incorporated the GID principles. Look for evidence of meaningful engagement with and input from suitably qualified and experienced Indigenous individuals, communities and groups. Assess if the system includes features or options that allow Indigenous stakeholders to control how their data is used and represented and describe how benefits of the project to First Nations Peoples, to which the data relate, have been considered.

5.3   Suitability of procured AI model 

Also consider the use of Indigenous data in the context of the United Nations Declaration on the Rights of Indigenous Peoples and apply the concept of ‘free, prior and informed consent’ in relation to the use of Indigenous data in AI systems.

If you are procuring an AI model (or system) from a third‑party provider, your procurement process should consider whether the provider has appropriate data management (including data quality and data provenance), governance, data sourcing, privacy, security, intellectual property, and cybersecurity practices in relation to the model. This will help you to identify whether the AI model is fit for the context and purpose of your AI use case. 

The relevance of the data used in training the AI model may influence the output and may not be relevant to the use case (and Australian context). Consider whether the model is likely to make accurate or reliable predictions concerning matters relating to Australian subject matter if it has been trained on, for example, US‑centric data.

In addition, there are a number of other considerations you should take into account when selecting a procured AI model. The following considerations may be relevant to your use case.

• Does the AI model meet the functional requirements needed for your use case?
• How was the model evaluated? What test data and benchmarks were used? 
• How is versioning for the AI model handled?
• What support does the provider provide for users/procurers?
• What provisions apply regarding potential liability issues? If the product fails, is accountability clear between your agency and the provider?
• What security precautions have been taken? What residual risks remain and how are these being mitigated?
• Are there any guarantees that data handling and management (for the entire lifecycle of the data) for the procured model meet internal agency and legislative requirements? What guarantees are there regarding the robustness of the model?
• What measures have been taken to prevent or reduce hallucinations, unwanted bias and model drift? 
• Is the explainability and interpretability of the model sufficient for your use case?
• What computing and storage capacities are necessary for operating the model on‑premises?
• What capability is needed to maintain the AI model? Can this be done in‑house, or will this need to be sourced externally?
• If you are considering using a platform as a service (PaaS) to run and support your AI system or AI model, have you considered risks associated with outsourcing?

5.4   Testing

Consider also how your agency will support transparency across the AI supply chain, for example, by notifying the developer of issues encountered in using the model or system. 

Testing is a key element for assuring the responsible and safe use of AI models – for both models developed in-house and externally procured – and in turn, of AI systems. Rigorous testing helps validate that the system performs as intended across diverse scenarios. Thorough and effective testing helps identify problems before deployment. 

Testing AI systems against test datasets can reveal biases or possible unintended consequences or issues before real-world deployment. Testing on data that is limited or skewed can fail to reveal shortcomings.

Consider establishing clear and measurable acceptance criteria for the AI system that, if met, would be expected to control harms that are relevant in the context of your AI use case. Acceptance criteria should be specific, objective and verifiable. They are meant to specify the conditions under which a potential harm is adequately controlled. 

Consider developing a test plan for the acceptance criteria to outline the proposed testing methods, tools and metrics. Documenting results through a test report will assist with demonstrating accountability and transparency. A test report could include the following:

• a summary of the testing objectives, methods and metrics used
• results for each test case
• an analysis of the root causes of any identified issues or failures
• recommendations for remediation or improvement, and whether the improvements should be done before deployment or as a future release.

In your explanation, outline any areas of concern in results from testing. If you have not started testing, outline elements to be considered in testing plans.

Model accuracy

As an example. model accuracy is a key metric for evaluating the performance of an AI system. Accuracy should be considered in the specific context of the AI use case, as the consequences of errors or inaccuracies can vary significantly depending on the domain and application.

Some of the factors that can influence AI model output accuracy and reliability include:

• choice of AI model or model architecture
• quality, accuracy and representativeness of training data
• presence of bias in the training data or AI model 
• robustness to noise, outliers and edge cases
• ability of the AI model to generalise to new data
• potential for errors or ‘hallucinations’ in outputs
• environmental factors (such as lighting conditions for computer vision systems)
• adversarial attacks (such as malicious actors manipulating input data to affect outputs)
• stability and consistency of performance over time.

Ways to assess and validate the accuracy of your model for your AI use case include:

• quantitative metrics
• qualitative analysis (for example, manual review of output, error analysis, user feedback)
• domain-specific benchmarks or performance standards
• comparison to human performance or alternative models.

5.5   Pilot

It is important to set accuracy targets that are appropriate for the risk and context of the use case. For high stakes decisions, you should aim for a very high level of accuracy and have clear processes for handling uncertain or borderline cases.

Conducting a pilot study is a valuable way to assess the real-world performance and impact of your AI use before full deployment. A well-designed pilot can surface issues related to reliability, safety, fairness and usability that may not be apparent in a controlled development environment.

If you are planning a pilot, your explanation should provide a brief overview of the pilot's:

• scope and duration
• objectives and key results (OKRs)
• key performance indicators (KPIs) 
• participant selection and consent process
• risk mitigation strategies.

5.6   Monitoring

If you have already completed a pilot, reflect on the key findings and lessons learned. How did the pilot outcomes compare to your expectations? What issues or surprises emerged? How did you adapt your AI use case based on the pilot results?

If you are not planning to conduct a pilot, explain why not. Consider whether the scale, risk or novelty of your use case warrants a pilot phase. Discuss alternative approaches you are taking to validate the performance of your AI use case and gather user feedback prior to full deployment.

Monitoring is key to maintaining the reliability and safety of AI systems over time. It enables active rather than passive oversight and governance. 

Your monitoring plan should be tailored to the specific risks and requirements of your use case. In your explanation, describe your approach to monitoring any measurable acceptance criteria (as discussed above at 5.4) as well as other relevant metrics such as performance metrics or anomaly detection. In your plan, you should include your proposed monitoring intervals for your use case. Consider including procedures for reporting and learning from incidents. You may wish to refer to the OECD paper on Defining AI incidents and related terms.

Periodically evaluate your monitoring and evaluation mechanisms to ensure they remain effective and aligned with evolving conditions throughout the lifecycle of your AI use case. Examples of events that could influence your monitoring plan are system upgrades, error reports, changes in input data, performance deviation or feedback from stakeholders.

Monitoring can help identify issues that can impact the safety and reliability of your AI system, such as concept or data drift. 

• Concept drift refers to a change in the relationship between input data and the feature being predicted
• Data drift refers to a change in input data patterns compared to the data used to train the model

Vendors offer monitoring tools that may be worth considering for your use case. For more information, see pp. 26-27 of the NAIC’s Implementing Australia’s AI Ethics Principles report.

5.7   Preparedness to intervene or disengage

Relevant stakeholders, including those who operate, use or interact with the AI system, those who monitor AI system performance, and affected stakeholders identified at section 2.4, should have the ability to raise concerns about insights or decisions informed by the AI system.

Agencies should develop clear escalation processes for raising concerns, such as designated points of contact, guidelines and criteria for when human intervention is necessary and timelines for response and resolution. Agencies should also consider documenting and reviewing any interventions that occur to ensure consistency and fairness.

In addition, agencies should be prepared to quickly and safely disengage an AI system when an unresolvable issue is identified. This could include a data breach, unauthorised access or system compromise. Consider such scenarios in business continuity, data breach and security response plans.

Agencies should consider the techniques below to avoid overreliance on AI system outputs. 

System design stage

Build in transparency about system limitations

Incorporate prompts to remind users to critically analyse outputs, such as explanations of outputs, hallucination reminders, and accuracy scores.

Build in 2-way feedback pathways

Prompt users to assess the quality of the AI system’s outputs and provide feedback.

Similarly, provide feedback to users on their interactions with the systems (e.g. feedback on ineffective prompts, alerts when the user has accepted a risky decision).

Prompt human decisions

Consider designing your AI system to provide options for the user to choose from, rather than a single solution, to encourage user engagement with AI outputs.

Evaluation stage

Ensure regular evaluation

Involve users in regular evaluations of your AI system. Encourage users to assess the effectiveness of the AI system and identify areas for improvement.