The Survey Doctor says, “the right questions eliminate order bias”
There are many opportunities to introduce response bias into marketing research survey design. The Survey Doctor encourages removing order bias from research questionnaire designs.
The Survey Doctor has recommendations on how to write the right questions. This article is focusing on once you have the right questions, how to design and order them. There are two common order bias mistakes in survey design.
What exactly is order bias? It is a type of response bias in which the order of the choices within a question or the order of the questions biases the answers. This typically happens in 2 ways:
1) Order Bias
This can occur when a respondent is given the opportunity to select answers in multiple-choice question design. Here’s an example from the Survey Doctor:
What color do you consider to be your favorite?
- Blue
- Yellow
- Green
- Orange
Respondents have a propensity to choose from the beginning and ends of lists.
This is particularly problematic with a list of choices. Often respondents associate first choices with correct choices. In order to remove order bias within a survey question, we recommend randomizing the multiple-choice responses. If you are surveying a sample of the population, let’s say for this example 100 people, ensure that the order changes and all of the colors have the same opportunity of being in any given choice placement by using a randomization scheme.
2) Order Effect
This is often referred to as order bias; however, the order effect is more accurate. When response bias is introduced within the questionnaire design an order effect can occur. This happens when a question is asked at the beginning of the survey that can influence other responses. An example from the Survey Doctor is asking a person’s income level at the beginning of a survey and then asking a number of price-related questions. This may influence how your survey respondents answer the subsequent questions.
A common error in customer satisfaction questionnaire design is asking detailed questions before obtaining the larger answer. The detailed questions may influence or bias the respondent’s initial response. For this reason, we strongly encourage questionnaire designers to ask overall satisfaction prior to asking questions about various components that could influence the overall satisfaction levels.
If you still have questions we would love to hear them. Call us, we have answers.
by Lynnette Leathers CEO of Mindspot Research and The Survey Doctor
