Search

Appendix

Methodological limitations

Evaluation fatigue may have reduced the participation in engagement activities.

Several agencies conducted their own internal evaluations over the course of the trial and did not participate in Digital Transformation Agency’s overall evaluation.

Mitigations: where possible, the evaluation has drawn on agency-specific evaluation to complement findings.

The non-randomised sample of trial participants may not reflect the views of the entire APS.

Participants self-nominated to be involved in the trial, contributing to a degree of selection bias. The representation of APS job families and classifications in the trial differs from the proportions in the overall APS.

Mitigations: the over and underrepresentation of certain groups has been noted. Statistical significance and standard error were calculated, where applicable, to ensure robustness of results.

There was an inconsistent roll out of Copilot across agencies.

Agencies began the trial at different stages, meaning there was not an equal opportunity to build capability or identify use cases. Agencies also used different versions of Copilot due to frequent product releases.

Mitigations: there is a distinction between what may be a functionality limitation of Copilot and when a feature has been disabled by an agency.

Measuring the impact of Copilot relied on trial participants’ self-assessment of productivity benefits.

Trial participants were asked to estimate the scale of Copilot’s benefits, which may naturally under or overestimate its impact.

Mitigations: where possible, the evaluation has compared productivity findings against other evaluations and external research to verify its validity.

Statistical significance of outcomes

The trial of Copilot for Microsoft 365 involved the distribution of nearly 5,765 Copilot licenses across 56 participating agencies. As part of engagement activities — consultations and surveys — the evaluation gathered the experience and sentiment from over 2,000 trial participants representing more than 45 agencies. Insights were further strengthened by the findings from internal evaluations completed by certain agencies. The sample size was sufficient to ensure 95% confidence intervals of reported proportions (at the overall level) were within a margin of error of 5%.

There were 3 questions asked in the post-use survey that were originally included in either the pre-use or pulse survey. These questions were repeated to compare responses of trial participants before and after the survey and measure the change in sentiment. A t-test was used to determine whether changes were statistically significant at a 5% level of significance.

Guidance for staff training on AI

Version 1.1

Use the following information to support your agency’s implementation of the policy for responsible use of AI in government.

The policy recognises that AI is used in many areas of the APS and everyday life. As adoption grows, staff at all levels will interact with AI and its outputs, directly or indirectly.

The policy strongly recommends that agencies implement AI fundamentals training for all staff, regardless of their role. Agencies should also consider if it is appropriate for staff to complete annual refresher training. 

Providing foundational training will help staff become confident using AI in a responsible way. 

What’s covered in the training

The Digital Transformation Agency (DTA) has developed AI fundamentals training for agency use. The training will be updated periodically to address changes in the AI landscape, such as evolutions in the technology and its use in government.

The training provides knowledge on AI that includes:

• an introduction to AI
• an explanation of generative AI 
• foundations of safe and responsible use.

As staff will likely interact with generative AI now and in the future, it is an area of focus for the training.

The training is designed for all staff regardless of their experience using AI. It takes approximately 20 to 30 minutes to complete.

It does not cover advanced topics such as model training, system development or instructions for specific technologies or platforms.

Implement the training

Agency’s Learning and Development specialists can access the training for download through the APS Learning Bank. The training is provided in a format compatible with most e-learning platforms. Alternatively, agencies can choose to let their staff access the module directly through APSLearn.

Agencies can use the training module as provided, or choose to modify it or incorporate it into an existing training program based on their specific context and requirements. 

Agencies are encouraged to consider additional training for staff in consideration of their roles and responsibilities, such as those responsible for the procurement, development, training and deployment of AI systems.

Report completion rates

Where an agency implements the training, accountable officials should monitor completion rates and provide this information if requested by the DTA. This is in line with the activities to measure the implementation of the policy under the Standard for accountable officials. 

Questions about implementation

Agencies can contact the DTA with questions about implementing the training by emailing ai@dta.gov.au.