In 2016, the Government of Canada commissioned Vox Pox Labs, a Canadian statistics company, to create a survey and report on the data on Canadian opinions with respect to government.  The purpose of the survey was to start a discussion on electoral reform, or a possible change to the elections process that the country currently has in place for federal elections.

15 million Canadian households received an invitation to complete the online survey with 383,074 unique respondents. The survey results, an extensive 134 page report, can be found at mydemocracy.ca.  Note:  p. 21-25 of the report discusses the demographic breakdown of respondents vs. the total population.

Bias in Collecting Data

When writing survey questions, it is important that the true beliefs of the survey respondents (definition:people who are answering the survey questions) being sampled is measured.  As with all bias, it may be impossible to fully eliminate it, you can only work hard to reduce it.  Recall, variability is inherent in data and will always exist in some way.

Measurement Bias

Definition:  Measurement Bias exists when there is some flaw in the measurement tool.  For surveys, it is when the survey has been poorly designed.  There is always some margin of error when collecting survey responses.  In a well-designed survey, the error is random, and does not result in an opinion being overstated or understated.  When a survey has measurement bias, the survey responses are either overstated or understated.

Recall the example from Activity 3:  "Since cars pollute the environment, do you think there should be a better public transit system in our city?"  The opinion that public transit should be improved will be overstated by the wording of the question.  By taking out the first part of the question, you will remove the measurement bias.  A better question to ask is:  "Do you think there should be a better public transit system in our city?"

Measurement Bias Questions

Three common types of questions that contain measurement bias are leading, loaded and double-barreled questions.

1. A leading question is a question that influences the responses in some way.  The "Since cars pollute the environment" part of the question above influences responders to say "Yes."  
2. A loaded question is a question that assumes something about the respondents.  An example would be "What do you love about baseball?" This question assumes that the people being surveyed have something they love about baseball.  The question "What do you love about baseball?" seems very simple, but needs to be thought through very carefully to get true results from respondents.
   • One possible solution to help minimize the measurement bias is to remove the word "love" to make it easier for people to answer.  There are cultural biases toward the word love.  
   • You can also only ask the question to people that said they enjoy baseball.  Online surveys will have ways of skipping questions for given answers.
   • You can also include options that allow respondents to say "Prefer not to answer" or "I do not enjoy anything about baseball."  
   • You would also want to consider if the options you give are an exhaustive list of everything you can think of and then you include "Other" as an option and allow people to write in other ideas.  
3. A double-barreled question is a question that asks about more than one issue, but only asks for one response.  An example of this regarding brushing your teeth would be:  "How important is the time spent brushing and number of brushes per day?" Time spent brushing your teeth is a different idea from the number of times you brush your teeth.  If you want to know about each of these ideas, it would be better to ask them separately.  

Response Bias

Definition:  Response Bias is a type of measurement bias where the survey respondents were influenced to answer in a certain way often because the survey is not anonymous or because the respondent thought that they had to answer a certain way.  

Example

Making a survey anonymous is a good way to limit response bias, as respondents will be comfortable stating their true beliefs or opinions.  Even with making the survey anonymous, some people may still tend to answer questions the way they feel the surveyor wants them to be answered.  It is best if questions are written in a way where people feel that all answers are equally accepted.

Designing a Survey

It is very important that a survey is written to minimize measurement bias so that the true beliefs of the sample respondents are measured.  

Types of Questions

In this activity you are exposed to a lot of different types of questions to collect data.  For information on more specific types of questions, read the following article from QuestionPro.

Writing Effective Questions

In addition to the knowledge of different types of measurement bias, a survey writer needs to also keep the following in mind.

1. Ethics:  When designing questions, it is important that the survey writer is in the correct frame of mind and is looking to get the true beliefs of the sample.  If the goal of the survey is to measure what people actually think, rather than get people to say what we'd like them to think, this will help avoid phrasing questions in ways that manipulate the opinion of the respondents.  This isn't only the right thing to do, it's the best thing to do in the long run.  

   Example

   Say you are a marketing consultant for Coca-Cola.  You want to know if people would like a chocolate flavoured soda.  You ask questions that suggest that people are overwhelmingly in favour of this by being vague and focusing on the appealing aspects of chocolate and soda. If the conclusion is that people will love this, it may result in a lot of time and money going into developing the product, only to find out later that people don't want it at all.

    
2. Writing Demographic Questions:  When asking personal questions, it is important to ask if you really need this information.  It is very important to do your own research (definition:Web search on best practices when writing survey questions.) as to how to ask the particular question you want to ask.  Asking the question in the wrong way may result in people not answering your survey or continuing it but not answering truthfully.  It is important to get a truthful answer from all respondents.

   Example

   When asking for gender, it is best practice to ask an open ended question such as "To which gender identity do you most identify?" 

     
3. Cultural Bias:  It is important to ask yourself whether or not some of your questions may result in cultural bias.  Cultural bias is when respondents tend to answer surveys in different ways depending on their culture or nationality.  As an example, consider the Likert scale. This is just one example of cultural bias, and each survey question needs to be evaluated for whether or not the true beliefs of the respondents are being measured.

   Example

   The Likert scale was invented by American psychologist Rensis Likert.  The typical Likert scale asks respondents to agree with a statement with a choice of the following:  

   • Strongly Disagree,
   • Disagree,
   • Neither Agree or Disagree,
   • Agree,
   • Strongly Agree.  

   Other rating scales can be used that ask people to rate on a scale from a min to max (from 1 to 10, for example).

   There may be a tendency for some cultures to avoid saying "Strongly" and for other cultures to be more likely to answer "Strongly."

    
4. Randomize the Questions:  There is chance that people may answer questions in a different way depending on what order they are being asked the questions in.  One way to limit this bias is to randomize the questions, if possible, so that they appear in a different order for each respondent.  This will make the error that would be caused by the order of the questions more random.  This also requires us to use technology to administer surveys.

Creating an Online Survey

The following example is of a survey created using Google Forms.  There are other internet tools that will allow you to create a survey, with functionality that will allow you to minimize the measurement bias. Consider the Self-Assessment from last activity. 

Example

Watch the following video for a demonstration on how to create the survey, shorten the url so that it is easier to type in, and look at responses in a spreadsheet: