-
-
Copilot survey questions
We have published the questions from the Microsoft 365 Copilot trial’s evaluation surveys to support organisations to create their own generative AI evaluations.
The evaluation was designed in consultation with the Australian Centre for Evaluation.
More information about the evaluation approach can be found in Appendix B of the full report.
About the surveys
Participants in the whole-of-government Copilot trial were encouraged to complete all 3 evaluation surveys, including the:
- pre-use survey, first issued on 29 February 2024
- mid-trial ‘pulse’ survey, first issued on 3 May 2024
- post-use survey, first issued on 2 July 2024.
Information collected
The surveys were designed to keep participant responses anonymous but still allow the evaluation team to link responses across the 3 surveys.
Participants were required to consent to their responses being used in the evaluation. They also had the option to allow their free-text answers to be quoted anonymously in future reports.
The surveys also collected participants’:
- APS classification
- job family
- managerial responsibilities.
Optionally, participants could provide their gender identity to support cohort analysis.
-
Featured initiatives
-
Pre-use survey questions
OffQuestion Options How long have you been using Copilot? (in weeks) Number Have you used, or do you currently use, any other generative AI tools besides Copilot? Single choice:
- Never
- Yes - in a personal capacity
- Yes - have used another generative AI tool to assist with work tasks
How frequently do you use the following Microsoft products?
- Teams
- Outlook
- Word
- PowerPoint
- Excel
- OneNote
- Loop
- Whiteboard
Single choice:
- Rarely/never/NA
- Once a month
- Once a week
- Few times a week
- Daily
How many meetings do you attend in an average week? Single choice:
- 0
- 1-5
- 6-10
- 11-15
- 16-20
- 21-25
- 26-30
- 31+
Please estimate how much of your average week you spend on tasks that you could perform more effectively with the help of automation. These could include creating a slide deck, summarising long documents, writing weekly reports, taking meeting minutes, analysing survey results or preparing proposals. Single matrix choice:
- Less than 20%
- 20 - 40%
- 40 - 60%
- 60 - 80%
- 80 - 100%
Please estimate how often you experience the following:
- I struggle to find the information or documents I need to complete my job
- I spend too much time reviewing and responding to emails
- I am required to switch tasks at short notice
- There are many meetings I attend to just ‘receive information’ (I do not need to contribute to the discussion)
- It can be difficult to keep on top of my workload
- I don’t have enough dedicated focus time
- I feel rushed and don’t feel I have put forward my best work
Single matrix choice:
- Not at all
- Rarely / A few times a year
- A few times a month
- A few times a week
- A few times a day
- Most of the day
Please estimate how often your role requires the following:
- Summarising large amounts of text-based information
- Undertaking data analysis
- Writing code in a programming language
- Preparing briefs and proposals
- Preparing speeches and talking points
- Preparing visual communication products e.g. presentations, slide decks, infographics, process diagrams
- Preparing public communication products for diverse audiences (e.g. website/social media content, fact sheets, publications)
- Undertaking user research and consultation
- Working collaboratively on documents
- Other
Single matrix choice:
- Not at all
- Rarely / A few times a year
- A few times a month
- A few times a week
- A few times a day
- Most of the day
Please estimate how many hours you spend on the following tasks in an average week:
- Searching for information required for a task
- Summarising existing information for various purposes (email updates, talking points, briefs, papers, minutes, etc.)
- Preparing meeting minutes
- Preparing first draft of a document
- Undertaking preliminary data analysis
- Preparing slides (for both presentations and information)
- Communicating through digital means other than meetings (1:1 calls, Teams chats etc)
- Other
Single matrix choice:
- 0
- 1 - 4
- 5 – 8
- 9 – 12
- 13 – 16
- 17 – 20
- 21 – 24
- 25+
Which of the following best describes your sentiment about using Copilot? Single choice:
- Very pessimistic
- Slightly pessimistic
- Neutral
- Slightly optimistic
- Very optimistic
To what extent to you agree or disagree with the following statements?
I believe Copilot will…
- Improve the speed at which I complete tasks
- Improve the quality of my work
- Make valuable suggestions to enhance my work
- Help me spend less mental effort on tedious or mundane tasks
- Allow me to attend fewer meetings
- Allow me to spend less time in emails
- Allow me to quickly find the information I am looking for
- Allow me to reduce task switching
- Free up more focus time for important work
- Be a net positive on my work
Single matrix choice:
- Strongly disagree
- Disagree
- Somewhat disagree
- Neutral
- Somewhat agree
- Agree
- Strongly agree
Are there any other impacts you believe Copilot will have on your work not mentioned above? Free text If you have not started using Copilot yet, what Copilot features are you most looking forward to using?
If you are already using Copilot, what Copilot features are you finding most useful?
Free text Are there any cases where you believe Copilot will not be suitable for your work? Free text Do you have any other concerns about using Copilot? Free text -
Mid-trial survey questions
OffQuestion Options In your use of Copilot, how much do you trust the results that are returned? Single choice:
- 0
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
Do you manage staff who have access to Copilot? True/false Do staff consistently flag they have used Copilot to prepare their outputs? Single choice:
- Never
- Sometimes
- About half the time
- Most of the time
- Always
Are you confident you could recognise the difference between outputs produced with Copilot and those produced without? Single choice:
- Definitely not
- Probably not
- Might or might not
- Probably yes
- Definitely yes
As a manager of staff using Copilot, how much do you trust outputs you receive from staff prepared with the assistance of Copilot? Single choice:
- 0
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
Thinking overall about your interactions with Copilot, how little or how much do you agree with the following statement: ‘Copilot has met my expectations.’ Single choice:
- Disagree
- Somewhat disagree
- Neutral
- Somewhat agree
- Agree
Thinking about your interactions with Copilot, how little or how much do you agree with the following statement?
Copilot features have met my expectations in…
- Excel
- Outlook
- PowerPoint
- Teams
- Word
Single matrix choice:
- Disagree
- Somewhat disagree
- Neutral
- Somewhat agree
- Agree
How little or how much do you agree with the following statement: ‘I feel confident in my skills and abilities to use Copilot.’ Single choice:
- Not at all confident
- Not very confident
- Moderate
- Fairly confident
- Very confident
Connect with the digital community
Share, build or learn digital experience and skills with training and events, and collaborate with peers across government.