Pilot AI assurance framework guidance

Attachment

Risk consequence rating advice

Negatively affecting public accessibility or inclusivity of government services

Insignificant

• Insignificant compromises to accessibility or inclusivity of services.
• Minor technical issues causing brief inconvenience but no actual barriers to access or inclusion.
•  Issues rapidly resolved with minimal impact on user experience.

Minor

• Limited, reversable compromises to accessibility or inclusivity of services.
• Some people experience difficulties accessing services due to technical issues or design oversights.
• Barriers are short-term and addressed once identified, with additional support provided to people affected.

Moderate

• Many compromises are made to the accessibility or inclusivity of services.
• Considerable access challenges for a modest number of users.
• Resolving access issues requires substantial effort and resources.
• Certain groups may be disproportionately impacted.
• Affected users experience frustration and delays in receiving services.

Major

• Extensive compromises are made to the accessibility or inclusivity of services, may include some essential services.
• Ongoing delays that require external technical assistance to resolve.
• Widespread inconvenience, frustration, public distress and potential legal implications.
• Vulnerable user groups disproportionately impacted.

Severe

• Widespread irreversible ongoing compromises are made to the accessibility or inclusivity of services, including some essential services.
• Majority of users, especially vulnerable groups affected.
• Essential services inaccessible for extended periods, causing significant public distress, legal implications, and a loss of trust in government efficiency.
• Comprehensive and immediate actions are urgently needed to rectify the situation.

Unfair discrimination against individuals, communities or groups

Insignificant

• Negligible instances of discrimination, with virtually no discernible effect on individuals, communities, or groups.
• Issues are proactively identified and rapidly addressed before causing harm.

Minor

• Limited instances of unfair discrimination occur, affecting a small number of individuals.
• Relatively isolated cases, and corrective measures minimise their impact.

Moderate

• Moderate levels of discrimination leading to noticeable harm to certain individuals, communities, or groups.
•  These incidents raise bias and fairness concerns and require targeted interventions.

Major

• Significant discrimination results in major, tangible harm to individuals and multiple communities or groups.
• Rebuilding trust requires substantial reforms and remediation efforts.

Severe

• Pervasive and systemic discrimination causes severe harm across a broad spectrum of the population, particularly marginalised and vulnerable groups.
• Public outrage, potential legal action, and a profound loss of trust in government.
• Immediate, sweeping reforms and accountability measures are required.

Perpetuating stereotyping or demeaning representations of individuals, communities or groups

Insignificant

• Inadvertently reinforce mild stereotypes, but these instances are quickly identified and rectified with no lasting harm or public concern.
• Minor
• Isolated cases of stereotyping, affecting limited members of community with some noticing and raising concerns.
• Prompt action mitigates the issue, preventing broader impact.

Moderate

• Moderate stereotyping by AI systems leads to noticeable public discomfort and criticism.
• Disproportionally affecting certain communities or groups.
• Requires targeted corrective measures to address and prevent recurrence.

Major

• Significant and widespread reinforcement of harmful stereotypes and demeaning representations.
• Causes public outcry and damages the relationship between communities and government entities.
• Urgent, comprehensive strategies are needed to rectify these representations and restore trust.

Severe

• Pervasive and damaging stereotyping severely harms multiple communities, leading to widespread distress.
• Potential legal consequences, and a profound breach of trust in government use of technology.
• Requires immediate, sweeping actions to address the harm, including system overhauls and public apologies.

Harm to individuals, communities, groups, businesses or the environment

Insignificant

• Inconsequential glitches with no real harm to the public, business operations or ecosystems.
• Easily managed through routine measures.

Minor

• Isolated incidents mildly affecting the public.
• Slight inconveniences or disruptions to businesses, leading to manageable financial costs.
• Limited manageable environmental disturbances affecting local ecosystems or resource consumption.

Moderate

• Noticeable negative effects on the public.
• Businesses face operational challenges or financial losses, affecting their competitiveness.
• Obvious environmental degradation, including pollution or habitat disruption, prompting public concern.

Major

• Significant public harm causing distress and potentially lasting damage.
• Significant harm to a wide range of businesses, resulting in substantial financial losses, layoffs, and long-term reputational damage.
• Compromises ecosystem wellbeing causing substantial pollution, loss of biodiversity, and resource depletion.

Severe

• Widespread, profound harm and severe distress affecting broad segments of the public.
• Profound damage across the business sector, leading to bankruptcies, major job losses, and a lasting negative impact on the economy.
• Comprehensive environmental destruction, leading to critical loss of biodiversity, irreversible ecosystem damage, and severe resource scarcity.

Compromising privacy due to the sensitivity, amount or source of the data being used by an AI system

Insignificant

• Insignificant data handling errors occur without compromising sensitive information.
• Incidents are quickly rectified, maintaining public trust in data security.

Minor

• Isolated exposure of limited sensitive data affects a small group of individuals.
• Swift actions taken to secure the data and prevent further incidents.

Moderate

• Breach of moderate amounts of sensitive data, leading to privacy concerns among the affected populace.
• Some individuals experience inconvenience and distress.

Major

• Serious misuse of sensitive private data affects a large segment of the population, leading to widespread privacy violations and a loss of public trust.
• Comprehensive measures are urgently required to secure data and address the privacy breaches.

Severe

• Significant potential to expose sensitive information of a vast number of individuals, causing severe harm, identity-theft risks; use of sensitive personal information in a way that is likely to draw public criticism with limited ability for individuals to choose how their information is used.
• Significant potential to harm trust in government-information handling with potential for lasting consequences.

Raising security concerns due to the sensitivity or classification of the data being used by an AI system

Insignificant

• Inconsequential security lapses occur without actual misuse of sensitive data.
• Quickly identified and corrected with no real harm done.
• These types of incidents may serve as prompts for reviewing security protocols.

Minor

• A limited security breach involves unauthorised access to protected data affecting a small number of records with minimal impact.
• Immediate actions secure the breach, and affected individuals are notified and supported.
• Incident is catalyst for review of security protocols.

Moderate

• Security incident leads to the compromise of a moderate volume of sensitive data, raising concerns over data protection and privacy.
• The breach necessitates a thorough investigation, enhanced security measures.

Major

• A significant security breach results in extensive unauthorised access to sensitive or protected data, causing considerable concern and distress among the public.
• Urgent security upgrades and support measures for impacted individuals are implemented. to restore security and trust.

Severe

• A massive security breach exposes a vast amount of sensitive and protected data, leading to severe implications for national security, public safety, and individual privacy.
• This incident triggers an emergency response, including legal actions, a major overhaul of security systems, and long-term support for those affected.

Raising security concerns due to implementation, sourcing or characteristics of the AI system

Insignificant

• Inconsequential security concerns arise due to characteristics of the AI system, such as software bugs, which are promptly identified and fixed with no adverse effects on overall security.
• These issues may serve as lessons, leading to slight improvements in the system's security framework.

Minor

• Certain characteristics of the AI system lead to vulnerabilities that are exploited in a limited manner, causing minor security breaches.
• Immediate remediation measures are taken, and the system is updated to prevent similar issues.

Moderate

• A moderate security risk is realised when intrinsic features of the AI system allow for unintended access or data leaks.
• Incident affects a noticeable but contained component of the AI system.
• Prompts a comprehensive security review of the AI system and the implementation of more robust safeguards.

Major

• Significant security flaws in the AI system's design result in major breaches, compromising a large amount of data and severely affecting system integrity.
• Incident leads to an urgent overhaul of security measures and protocols, alongside efforts to mitigate the damage.

Severe

• Critical security vulnerabilities inherent to the AI system lead to widespread breaches, exposing vast quantities of sensitive data and jeopardising national security or public safety.
• The incident results in severe consequences, necessitating emergency responses, extensive system redesigns, and long-term efforts to recover from the breach and prevent recurrence.

Influencing decision-making affects individuals, communities, groups, businesses or the environment

Insignificant

• Decisions lead to negligible errors, swiftly identified and corrected with no harm to the public, business operations or the environment.
• Incidents may serve as learning opportunity for system improvement.

Minor

• Decisions result in minor inconveniences or errors affecting the public, business operations or finances or slight environmental impacts.
• All impacts reversible with prompt action.

Moderate

• Decisions cause moderate harm to the public, business operations or finances or noticeable environmental degradation.
• Targeted interventions are required to mitigate these effects.

Major

• Significant harm to the public, substantial business financial losses or operational disruptions, or significant environmental damage.
• Loss of confidence in government, operations, service delivery and partnerships.
• Significant harm to a wide range of businesses, resulting in substantial financial losses, layoffs, and long-term reputational damage.
• Compromises ecosystem wellbeing causing substantial pollution, loss of biodiversity, and resource depletion.

Severe

• AI's influence on critical decision-making processes leads to severe and widespread harm to public, business operations or finances or the environment.
• Potentially endangering lives or significantly impacting public safety, rights and trust.
• Causes massive job losses, undermining business economic stability and viability.
• Catastrophic loss of ecosystems, endangered species, and long-term ecological imbalance or severe resources depletion.

Posing a reputational risk or undermining public confidence in the government

Insignificant

• Isolated reputational issues arise, quickly addressed and explained.
• Causes negligible damage to public trust in government capabilities.

Minor

• Small-scale AI mishaps lead to brief public concern, slightly denting the government's reputation.
• Prompt clarification and corrective measures minimize long-term impact on public confidence
• Seen by the government as poor management.

Moderate

• Misapplications result in moderate public dissatisfaction and questioning of government oversight.
• Requires remedial actions to mend trust and address concerns.
• Seen by government and opposition as failed management.

Major

• Widespread public scepticism and criticism, majorly affecting the government's image.
• Requires substantial efforts to rebuild public confidence through transparency, accountability, and improvement of AI governance.
• High profile negative stories, seen by government and opposition as significant failed management.

Severe

• Severe misuse or failure of AI systems leads to profound public distrust and criticism.
• Significantly undermining confidence in government effectiveness and integrity.
• Requires comprehensive, long-term strategies for rehabilitation of public trust, including systemic changes and ongoing engagement.
• Seen by government and opposition as catastrophic failure of management.
• Minister expresses loss of confidence or trust in agency.

Risk likelihood table