Copilot survey questions

We have published the questions from the Microsoft 365 Copilot trial’s evaluation surveys to support organisations to create their own generative AI evaluations.

The evaluation was designed in consultation with the Australian Centre for Evaluation.

More information about the evaluation approach can be found in Appendix B of the full report.

About the surveys

Participants in the whole-of-government Copilot trial were encouraged to complete all 3 evaluation surveys, including the:

pre-use survey, first issued on 29 February 2024
mid-trial ‘pulse’ survey, first issued on 3 May 2024
post-use survey, first issued on 2 July 2024.

Information collected

The surveys were designed to keep participant responses anonymous but still allow the evaluation team to link responses across the 3 surveys.

Participants were required to consent to their responses being used in the evaluation. They also had the option to allow their free-text answers to be quoted anonymously in future reports.

The surveys also collected participants’:

APS classification
job family
managerial responsibilities.

Optionally, participants could provide their gender identity to support cohort analysis.

Question	Options
How long have you been using Copilot? (in weeks)	Number
Have you used, or do you currently use, any other generative AI tools besides Copilot?	Single choice: Never Yes - in a personal capacity Yes - have used another generative AI tool to assist with work tasks
How frequently do you use the following Microsoft products? Teams Outlook Word PowerPoint Excel OneNote Loop Whiteboard	Single choice: Rarely/never/NA Once a month Once a week Few times a week Daily
How many meetings do you attend in an average week?	Single choice: 0 1-5 6-10 11-15 16-20 21-25 26-30 31+
Please estimate how much of your average week you spend on tasks that you could perform more effectively with the help of automation. These could include creating a slide deck, summarising long documents, writing weekly reports, taking meeting minutes, analysing survey results or preparing proposals.	Single matrix choice: Less than 20% 20 - 40% 40 - 60% 60 - 80% 80 - 100%
Please estimate how often you experience the following: I struggle to find the information or documents I need to complete my job I spend too much time reviewing and responding to emails I am required to switch tasks at short notice There are many meetings I attend to just ‘receive information’ (I do not need to contribute to the discussion) It can be difficult to keep on top of my workload I don’t have enough dedicated focus time I feel rushed and don’t feel I have put forward my best work	Single matrix choice: Not at all Rarely / A few times a year A few times a month A few times a week A few times a day Most of the day
Please estimate how often your role requires the following: Summarising large amounts of text-based information Undertaking data analysis Writing code in a programming language Preparing briefs and proposals Preparing speeches and talking points Preparing visual communication products e.g. presentations, slide decks, infographics, process diagrams Preparing public communication products for diverse audiences (e.g. website/social media content, fact sheets, publications) Undertaking user research and consultation Working collaboratively on documents Other	Single matrix choice: Not at all Rarely / A few times a year A few times a month A few times a week A few times a day Most of the day
Please estimate how many hours you spend on the following tasks in an average week: Searching for information required for a task Summarising existing information for various purposes (email updates, talking points, briefs, papers, minutes, etc.) Preparing meeting minutes Preparing first draft of a document Undertaking preliminary data analysis Preparing slides (for both presentations and information) Communicating through digital means other than meetings (1:1 calls, Teams chats etc) Other	Single matrix choice: 0 1 - 4 5 – 8 9 – 12 13 – 16 17 – 20 21 – 24 25+
Which of the following best describes your sentiment about using Copilot?	Single choice: Very pessimistic Slightly pessimistic Neutral Slightly optimistic Very optimistic
To what extent to you agree or disagree with the following statements? I believe Copilot will… Improve the speed at which I complete tasks Improve the quality of my work Make valuable suggestions to enhance my work Help me spend less mental effort on tedious or mundane tasks Allow me to attend fewer meetings Allow me to spend less time in emails Allow me to quickly find the information I am looking for Allow me to reduce task switching Free up more focus time for important work Be a net positive on my work	Single matrix choice: Strongly disagree Disagree Somewhat disagree Neutral Somewhat agree Agree Strongly agree
Are there any other impacts you believe Copilot will have on your work not mentioned above?	Free text
If you have not started using Copilot yet, what Copilot features are you most looking forward to using? If you are already using Copilot, what Copilot features are you finding most useful?	Free text
Are there any cases where you believe Copilot will not be suitable for your work?	Free text
Do you have any other concerns about using Copilot?	Free text

Question	Options
In your use of Copilot, how much do you trust the results that are returned?	Single choice: 0 1 2 3 4 5 6 7 8 9 10
Do you manage staff who have access to Copilot?	True/false
Do staff consistently flag they have used Copilot to prepare their outputs?	Single choice: Never Sometimes About half the time Most of the time Always
Are you confident you could recognise the difference between outputs produced with Copilot and those produced without?	Single choice: Definitely not Probably not Might or might not Probably yes Definitely yes
As a manager of staff using Copilot, how much do you trust outputs you receive from staff prepared with the assistance of Copilot?	Single choice: 0 1 2 3 4 5 6 7 8 9 10
Thinking overall about your interactions with Copilot, how little or how much do you agree with the following statement: ‘Copilot has met my expectations.’	Single choice: Disagree Somewhat disagree Neutral Somewhat agree Agree
Thinking about your interactions with Copilot, how little or how much do you agree with the following statement? Copilot features have met my expectations in… Excel Outlook PowerPoint Teams Word	Single matrix choice: Disagree Somewhat disagree Neutral Somewhat agree Agree
How little or how much do you agree with the following statement: ‘I feel confident in my skills and abilities to use Copilot.’	Single choice: Not at all confident Not very confident Moderate Fairly confident Very confident

Question	Options
Which of the following best describes your sentiment about using Copilot after having used it?	Single choice: Very pessimistic Slightly pessimistic Neutral Slightly optimistic Very optimistic
To what extent do you agree with the following statement: ‘I want to continue to use Copilot after the trial’.	Single choice: Strongly disagree Disagree Neutral Agree Strongly agree
Please explain why you would or would not continue to use Copilot.	Free text
How frequently did you use Copilot during the trial?	Single choice: Not at all A few times a month A few times a week A few times a day Most of the day
During the Trial, which Microsoft products have you used with Copilot features? Select all that apply.	Single choice: Copilot chat Teams Outlook Word PowerPoint Excel OneNote Loop Whiteboard
(Conditional display) How frequently did you use the following Copilot chat features? General queries Interface with other Microsoft products Document search / retrieval	Single matrix choice: Not at all A few times a month A few times a week A few times a day Frequently throughout the day
(Conditional display) How frequently did you use the following Teams features? Meeting summaries Real-time answers Task management	Single matrix choice: Not at all A few times a month A few times a week A few times a day Frequently throughout the day
(Conditional display) How frequently did you use the following Outlook features? Email summarisation Drafting assistance Meeting follow-up	Single matrix choice: Not at all A few times a month A few times a week A few times a day Frequently throughout the day
(Conditional display) How frequently did you use the following Word features? Summarisation Rewrite suggestions Tone adjustments Formatting assistance	Single matrix choice: Not at all A few times a month A few times a week A few times a day Frequently throughout the day
(Conditional display) How frequently did you use the following PowerPoint features? Content creation Design suggestions Summarisation	Single matrix choice: Not at all A few times a month A few times a week A few times a day Frequently throughout the day
(Conditional display) How frequently did you use the following Excel features? Data analysis Formula assistance Insight generation	Single matrix choice: Not at all A few times a month A few times a week A few times a day Frequently throughout the day
(Conditional display) How frequently did you use the following OneNote features? Content generation Summarisation Task management Collaboration enhancement	Single matrix choice: Not at all A few times a month A few times a week A few times a day Frequently throughout the day
(Conditional display) How frequently did you use the following Loop features? Draft page content Content editing Summarisation Idea generation	Single matrix choice: Not at all A few times a month A few times a week A few times a day Frequently throughout the day
(Conditional display) How frequently did you use the following Whiteboard features? Idea generation Content summarisation Content organisation Interactive collaboration	Single matrix choice: Not at all A few times a month A few times a week A few times a day Frequently throughout the day
Which Copilot feature(s) had the most positive impact during the Trial? Please describe any particular use case(s) of how you used the feature(s) in your job	Free text
What other generative AI products do you use in a work capacity to support your role? Select all that apply.	Multi choice: I do not use other generative AI products in a work capacity ChatGPT Gemini Claude Meta AI Bing AI Perplexity Midjourney GitHub Copilot Other
How does Copilot compare to other generative AI products you have used?	Single choice: Other generative AI products meet my needs significantly more than Copilot Other generative AI products meet my needs slightly more than Copilot Copilot and other generative AI products meet my needs to the same extent Copilot meets my needs slightly more than other generative AI products Copilot meets my needs significantly more than other generative AI products
How little or how much do you agree with the following statement: ‘I feel confident in my skills and abilities to use Copilot.’	Single choice: Not at all confident Not very confident Moderately confident Fairly confident Very confident
How did you learn to use Copilot? Select all that apply.	Multi choice: My agency provided Copilot training Microsoft provided Copilot training I found Copilot resources on the Internet I experimented with Copilot to learn its functionalities Other
To what extent do you agree with the following statement: ‘The Copilot training I received was useful.’	Single choice: Strongly disagree Disagree Neutral Agree Strongly agree
To what extent to you agree or disagree with the following statements? Using Copilot has… Improved the speed at which I complete tasks Improved the quality of my work Made valuable suggestions to enhance my work Helped me spend less mental effort on tedious or mundane tasks Allowed me to attend fewer meetings Allowed me to spend less time in emails Allowed me to quickly find the information I am looking for Allowed me to reduce task switching Freed up more focus time for important work Been a net positive on my work	Single matrix choice: Strongly disagree Disagree Somewhat disagree Neutral Somewhat agree Agree Strongly agree
On average, how many hours per day has Copilot helped you save in the following areas? Searching for information required for a task Summarising existing information for various purposes (email updates, talking points, briefs, papers, minutes, etc.) Preparing meeting minutes Preparing first draft of a document Undertaking preliminary data analysis Preparing slides (for both presentations and information) Communicating through digital means other than meetings (1:1 calls, Teams chats etc) Attending meetings Writing or reviewing code in a programming language Other	Single matrix choice: Copilot has added time to this activity 0 0.5 – 1 1 – 2 2 – 3 3 – 4 4+ N/A (My job does not require me to complete this task)
To what extent do you agree with the following statement: ‘Copilot has enabled me to allocate my time to perform tasks that are higher value and/or more complex.’	Single choice: Strongly disagree Disagree Neutral Agree Strongly agree
(if agree or strongly agree) Please describe the tasks that are higher value and/or more complex that Copilot has enabled you to complete.	Free text
To what extent do you agree or disagree with the following statements? Copilot has improved the quality of my… Searches for information required for a task Summaries of existing information for various purposes (email updates, talking points, briefs, papers, minutes, etc.) Preparation of meeting minutes Preparations of the first draft of a document Preliminary data analysis Preparation of slides (for both presentations and information) Communication through digital means other than meetings (1:1 calls, Teams chats etc) Coding Other	Single matrix choice: Strongly disagree Disagree Neutral Agree Strongly agree N/A (My job does not require me to complete this task)
If you selected 'Other', please describe the output(s) delivered and Copilot's impact on quality.	Free text
Please describe how Copilot has affected the quality of your outputs.	Free text
(if managing staff) What is the impact of Copilot on the quality of your team's outputs?	Single choice: Negative Somewhat negative Neutral Somewhat positive Positive
(if managing staff) What is the impact of Copilot on the quality of your team's outputs?	Single choice: Negative Somewhat negative Neutral Somewhat positive Positive
(if managing staff) What is the impact of Copilot on the efficiency of your staff?	Single choice: Negative Somewhat negative Neutral Somewhat positive Positive

Question

Options

Which of the following best describes your sentiment about using Copilot after having used it?

Single choice:

Very pessimistic
Slightly pessimistic
Neutral
Slightly optimistic
Very optimistic

To what extent do you agree with the following statement: ‘I want to continue to use Copilot after the trial’.

Single choice:

Strongly disagree
Disagree
Neutral
Agree
Strongly agree

Please explain why you would or would not continue to use Copilot.

Free text

How frequently did you use Copilot during the trial?

Single choice:

Not at all
A few times a month
A few times a week
A few times a day
Most of the day

During the Trial, which Microsoft products have you used with Copilot features? Select all that apply.

Single choice:

Copilot chat
Teams
Outlook
Word
PowerPoint
Excel
OneNote
Loop
Whiteboard

(Conditional display)
How frequently did you use the following Copilot chat features?

General queries
Interface with other Microsoft products
Document search / retrieval

Single matrix choice:

Not at all
A few times a month
A few times a week
A few times a day
Frequently throughout the day

(Conditional display)
How frequently did you use the following Teams features?

Meeting summaries
Real-time answers
Task management

Single matrix choice:

Not at all
A few times a month
A few times a week
A few times a day
Frequently throughout the day

(Conditional display)
How frequently did you use the following Outlook features?

Email summarisation
Drafting assistance
Meeting follow-up

Single matrix choice:

Not at all
A few times a month
A few times a week
A few times a day
Frequently throughout the day

(Conditional display)
How frequently did you use the following Word features?

Summarisation
Rewrite suggestions
Tone adjustments
Formatting assistance

Single matrix choice:

Not at all
A few times a month
A few times a week
A few times a day
Frequently throughout the day

(Conditional display)
How frequently did you use the following PowerPoint features?

Content creation
Design suggestions
Summarisation

Single matrix choice:

Not at all
A few times a month
A few times a week
A few times a day
Frequently throughout the day

(Conditional display)
How frequently did you use the following Excel features?

Data analysis
Formula assistance
Insight generation

Single matrix choice:

Not at all
A few times a month
A few times a week
A few times a day
Frequently throughout the day

(Conditional display)
How frequently did you use the following OneNote features?

Content generation
Summarisation
Task management
Collaboration enhancement

Single matrix choice:

Not at all
A few times a month
A few times a week
A few times a day
Frequently throughout the day

(Conditional display)
How frequently did you use the following Loop features?

Draft page content
Content editing
Summarisation
Idea generation

Single matrix choice:

Not at all
A few times a month
A few times a week
A few times a day
Frequently throughout the day

(Conditional display)
How frequently did you use the following Whiteboard features?

Idea generation
Content summarisation
Content organisation
Interactive collaboration

Single matrix choice:

Not at all
A few times a month
A few times a week
A few times a day
Frequently throughout the day

Which Copilot feature(s) had the most positive impact during the Trial? Please describe any particular use case(s) of how you used the feature(s) in your job

Free text

What other generative AI products do you use in a work capacity to support your role? Select all that apply.

Multi choice:

I do not use other generative AI products in a work capacity
ChatGPT
Gemini
Claude
Meta AI
Bing AI
Perplexity
Midjourney
GitHub Copilot
Other

How does Copilot compare to other generative AI products you have used?

Single choice:

Other generative AI products meet my needs significantly more than Copilot
Other generative AI products meet my needs slightly more than Copilot
Copilot and other generative AI products meet my needs to the same extent
Copilot meets my needs slightly more than other generative AI products
Copilot meets my needs significantly more than other generative AI products

How little or how much do you agree with the following statement: ‘I feel confident in my skills and abilities to use Copilot.’

Single choice:

Not at all confident
Not very confident
Moderately confident
Fairly confident
Very confident

How did you learn to use Copilot? Select all that apply.

Multi choice:

My agency provided Copilot training
Microsoft provided Copilot training
I found Copilot resources on the Internet
I experimented with Copilot to learn its functionalities
Other

To what extent do you agree with the following statement: ‘The Copilot training I received was useful.’

Single choice:

Strongly disagree
Disagree
Neutral
Agree
Strongly agree

To what extent to you agree or disagree with the following statements?

Using Copilot has…

Improved the speed at which I complete tasks
Improved the quality of my work
Made valuable suggestions to enhance my work
Helped me spend less mental effort on tedious or mundane tasks
Allowed me to attend fewer meetings
Allowed me to spend less time in emails
Allowed me to quickly find the information I am looking for
Allowed me to reduce task switching
Freed up more focus time for important work
Been a net positive on my work

Single matrix choice:

Strongly disagree
Disagree
Somewhat disagree
Neutral
Somewhat agree
Agree
Strongly agree

On average, how many hours per day has Copilot helped you save in the following areas?

Searching for information required for a task
Summarising existing information for various purposes (email updates, talking points, briefs, papers, minutes, etc.)
Preparing meeting minutes
Preparing first draft of a document
Undertaking preliminary data analysis
Preparing slides (for both presentations and information)
Communicating through digital means other than meetings (1:1 calls, Teams chats etc)
Attending meetings
Writing or reviewing code in a programming language
Other

Single matrix choice:

Copilot has added time to this activity
0
0.5 – 1
1 – 2
2 – 3
3 – 4
4+
N/A (My job does not require me to complete this task)

To what extent do you agree with the following statement: ‘Copilot has enabled me to allocate my time to perform tasks that are higher value and/or more complex.’

Single choice:

Strongly disagree
Disagree
Neutral
Agree
Strongly agree

(if agree or strongly agree)

Please describe the tasks that are higher value and/or more complex that Copilot has enabled you to complete.

Free text

To what extent do you agree or disagree with the following statements?

Copilot has improved the quality of my…

Searches for information required for a task
Summaries of existing information for various purposes (email updates, talking points, briefs, papers, minutes, etc.)
Preparation of meeting minutes
Preparations of the first draft of a document
Preliminary data analysis
Preparation of slides (for both presentations and information)
Communication through digital means other than meetings (1:1 calls, Teams chats etc)
Coding
Other

Single matrix choice:

Strongly disagree
Disagree
Neutral
Agree
Strongly agree
N/A (My job does not require me to complete this task)

If you selected 'Other', please describe the output(s) delivered and Copilot's impact on quality.

Free text

Please describe how Copilot has affected the quality of your outputs.

Free text

(if managing staff)

What is the impact of Copilot on the quality of your team's outputs?

Single choice:

Negative
Somewhat negative
Neutral
Somewhat positive
Positive

(if managing staff)

What is the impact of Copilot on the quality of your team's outputs?

Single choice:

Negative
Somewhat negative
Neutral
Somewhat positive
Positive

(if managing staff)

What is the impact of Copilot on the efficiency of your staff?

Single choice:

Negative
Somewhat negative
Neutral
Somewhat positive
Positive

Australian Government trial of Microsoft 365 Copilot