Ten Commandments for creating a survey

: 06 Dec 2017

Thanks to computer software and the web it is now easier than ever to create a survey. The technical side is so easy that people forget that there is more to it than just adding some questions to a form.

I was fortunate to attend a lecture by Marguerrite Cox, a biostatistician from Duke Clinical Research Institute in the USA, and I thought it might be useful to share her 10 Commandments.

(I have changed the examples to be more relevant to a tech audience)

Before you even start with the questions you need to:

Decide on the survey’s goals,
Identify the target population
Determine the sample size
Select a survey mode eg internet, phone, in-person et

Ten Commandments

1. Thou Shalt Not Be Vague

Know exactly what you want to ask and be specific and add a reference frame.
"How often do you install x per year?" instead of "how often you install x?"

2. Thou Shalt Not Make Unto Thee Complicated Questions

Be clear and simple
Avoid jargon
Avoid complex sentences

3. Thou Shalt Not Ask Double-barrelled Questions

Example: "How satisfied are you with the software's support and documentation?"
Break these questions down into multiple questions:
"How satisfied are you with your software's support?”
"How satisfied are you with your software's documentation?”

4. Thou Shalt Avoid Leading Questions

Example: "Do you believe the documentation is good enough?"

5. Thou Shalt Ask as Few Questions as Possible

Trade-off between getting all the information you need and reducing the amount of non-responses

6. Honour Thy Order of Thy Questions

Questions should flow logically
Questions at the beginning of the survey may influence answers to later questions
Easy, general questions should be at the beginning
Sensitive questions should be placed at the end of the survey
If the questions are similar, random ordering can be helpful

7. Thou Shalt Anticipate All Possible Answer Choices

Example: "How would you describe your programming skills (check all that apply)?":
novice, new to this language, still learning, moderate, experienced, practically invented the language, Other (specify), Prefer not to answer.

8. If Thou Uses an Ordinal Scale, Thou Shalt Use a Balanced Scale with 5-7 Choices and a Clear Middle Category

Ordinal data: categorical data where there is a clear and consistent ordering
Examples:
Strongly agree, agree, neither agree nor disagree, disagree, strongly disagree
Very good, good, fair, poor, very poor
Never, rarely, sometimes, most of the time, always, No chance
very little chance, unsure, some chance, very good chance
Just as the order of the questions is important, the order of the answer choices is important
Primacy Effect: respondents pick the first choice (paper and internet surveys)
Recency Effect: respondents pick the choice most recently heard (in-person and phone surveys)

9. Thou Shalt Not Have Overlapping Categories

How many web sites do you manage?": 1-10, 10-20, 20-50, 50-75, Over 75
What if they manage 20? Some of your respondents may check 10-20, while some may check 20-50. Some may be confused and not respond.