Copyright

Identifying Bias in a Survey

  • 1.

    A high school is attempting to decide on a new school mascot, and wants student input. They randomly select 100 current students to survey, and task the rather opinionated gym teacher and football coach to perform the survey. Does this process show any potential evidence of bias? Why or why not?

    Answers:

    • No, there is no evidence of bias in either research process.

    • Yes, because the method of selecting students may introduce a possible bias with the results.

    • Yes, the sample size selected is not large enough to get an accurate picture of the student body opinion.

    • Yes, because the person selected to perform the survey may intentionally show bias or influence the student choices based on their relationship with them.

  • 2.

    Jupiter high school has decided to make a new school mascot and wants student input on the matter. They randomly select 100 current students to survey. The survey asks to choose a preference between a "a strong and powerful tiger" or "a bird of some kind". Is there evidence of bias in this survey? Why or why not?

    Answers:

    • Yes, because the sample selection process itself shows evidence of bias.

    • Yes, because there isn't an "other" category for write-in suggestions, meaning it biases the students towards answers that the school already selected.

    • Yes, because strong descriptor terms attached to one choice but not the other may bias the students into choosing one choice over the other.

    • No, there is no evidence of bias in the survey.

  • 3.

    A bakery wants to find out what new types of pastries they should introduce to the store, and they decide to conduct a survey. They hire a temporary employee to perform a survey with any customer that spends over $40 over the course of a month. Does this process show any evidence of potential bias? Why or why not?

    Answers:

    • Yes, because the person selected to conduct the survey has enough connection to the customers and bakery that it may influence the choices of the customers.

    • Yes, because a month is not a long enough time to have a proper variety of survey respondents. You need more time for the survey to run to gain fairer results.

    • No, there is no evidence of bias in the survey process.

    • Yes, because the people selected for the survey are only customers who spend a good sum of money at the store rather than all customers. The opinions of customers who spend a lot of money may differ from those who spend less.

  • 4.

    Eva's Bakery wants to know what new types of pastries they should introduce to the store. To find out, they decide to conduct a survey with their customers. Each customer is given a long list of potential pastries that takes a full page. There is only one version of this survey, so everyone gets the exact same one. Is there a potential issue of bias in the survey? Why or why not?

    Answers:

    • Yes. Because there are so many choices, the customers may have trouble deciding between what they want.

    • Yes. Because the selection process doesn't include people who aren't their customers, it introduces a bias towards people who already have some attachment to the store.

    • Yes. Because the list is so long and there are no different versions of the list, the pastries listed first may have an unfair advantage as they'll be the names that every customer sees first.

    • No, there is no evidence of bias in this scenario.

  • 5.

    A local election is coming up, so an independent pollster organization conducts a poll to see candidate favorability. They call 1000 households in the voting district at random, and ask questions about their opinion on candidates. They also send text messages out to another 1000 random numbers, in hopes of catching people who don't answer their phone. Is there evidence of potential bias in this survey process?

    Answers:

    • No, there is no evidence of potential bias in this survey process.

    • Yes, because the sample size is not large enough to reflect the general opinion of the population.

    • Yes, the selection process is selected in a way that may introduce bias into the responses.

    • Yes, the people administering the survey may influence survey responses as they are too connected to the situation.

  • 6.

    In a few months a local election is coming up, so an independent pollster organization conducts a poll to see candidate favorability. They call 1000 households in the voting district at random, and ask about which candidate they favor. They make sure to change the order of the the candidates listed, and avoid any adjectives to describe the candidate. Is there evidence of bias in their survey?

    Answers:

    • No, there is no evidence of bias in their survey process.

    • Yes, the sample size is not large enough to reflect the general population.

    • Yes, by changing the order of the candidates names, it may introduce confusion.

    • Yes, by leaving out adjectives, it may make respondents not understand anything about who they are selecting.

  • 7.

    Richard is a medical student who is performing research on people living with chronic back pain. He decides to conduct a survey, and to get his sample, he posts online advertising a $200 cash reward for people with chronic back pain to participate in his survey. Is there any potential issue with bias with his selection process? Why or why not?

    Answers:

    • Yes, because having that high a monetary reward will heavily bias respondents into giving answers that Richard wants, rather than the truth, to ensure they get paid.

    • Yes, because requesting only people with chronic back pain will bias the responses to be from that specific viewpoint.

    • No, there is no evidence of bias in his selection process.

    • Yes, because posting an advertisement online means only people with internet access could participate in the survey, which could potentially bias their responses.

  • 8.

    Richard is a medical student who wants to do research on people living with chronic back pain. After finding a suitable sample size of people for his survey, he askes them a lengthy questionnaire. Richard is more educated about lower back pain, so a significant number of questions focus on that type of back pain. Is there a potential bias in Richard's questions? Why or why not?

    Answers:

    • Yes, because asking just about back pain and not about other types of pain may bias the research results, and not provide a full picture of what the people responding are going through.

    • Yes, because giving a lengthy questionnaire might cause issues of candidates maintaining attention for long enough to answer all the questions, so the later questions may not get as much attention.

    • No, there is no evidence of bias in his questionnaire.

    • Yes, because there are many different varieties of chronic back pain, and having a significant number of questions focus solely on lower back pain will bias the results to be about that type rather than all types of chronic back pain.

  • 9.

    Crow is a fashion designer who is studying fashion trends across the US to get a sense of what to design next. To get opinions, he asks ten people who attended the last New York fashion show about their opinions. Is there a possible bias in his selection process?

    Answers:

    • Yes, because by asking people who are invested enough in fashion to attend fashion shows, he is excluding people who have less investment in fashion trends.

    • No, there is no evidence of bias in his selection process.

    • Yes. Because he is only asking people who attended a specific fashion show in a specific city, it will bias the answers towards people who are interested in New York fashion rather than fashion in the rest of the United States.

    • Yes, because he asked people in person rather than over the phone or online, so it may bias the answers.

  • 10.

    Crow is a fashion designer who is studying current fashion trends to get a sense of what to design next. To get opinions, he asks 50 people about their opinions on different popular fashion trends. Most of his questions are about female fashion trends, with only a couple questions about male or androgynous fashion trends. Is there a possible bias in Crow's questions?

    Answers:

    • Yes, he is showing a clear bias in his questions towards his own personal preference in fashion trends.

    • Yes, the sample size is not large enough to gain accurate feedback.

    • Yes. By focusing on popular fashion trends, he is neglecting fashion trends that are lesser known and bias people towards a certain fashion style.

    • No, there is no evidence of bias in his questions.

  • 11.

    Julia wants to find out if there's a correlation between how often a student reads and their grades at her school. She conducts a study where she randomly selects 100 students from different grades and gives them a survey about their reading habits. Is there an issue of bias in her selection process?

    Answers:

    • Yes, 100 students is not a large enough sample size to obtain accurate responses.

    • Yes. By selecting only students, there will be no input from people who read but don't attend school.

    • Yes, the random selection process will lend itself to bias based on the people she selected.

    • No, there isn't an issue of bias in her selection process.

  • 12.

    Julia is researching if there's a correlation between how often a student reads and their grades at her school. She conducts a study and asks 100 randomly selected students the question "How many books have you read since the start of the school year that were not assigned for class?". Identify any evidence of bias in this question.

    Answers:

    • By asking about the start of the school year, that will bias the question to be relevant only to people currently enrolled in school.

    • By asking about books they've read since the start of the school year, that will bias students to focus only on recent reading habits.

    • By leaving out books that were assigned for class, that will bias students not to count those books.

    • There is no evidence of bias in this question.

  • 13.

    Mercury City wants to learn more about the car buying habits of their citizens. They decide to conduct a research survey, where they will randomly select 1000 households within city limits to call. While trying to find someone to conduct the survey, the local car dealership offers to help. Identify any potential bias in this method, if any.

    Answers:

    • There is no bias. The car dealership handling the survey is an excellent choice because they are more informed about car sales.

    • Randomly calling 1000 households is not enough for a decent sample size for the population, and may lead to an unintentional bias result. They should aim for 5000 people for a proper sample size.

    • Calling households within city limits would bias the results for people who only live within city limits, limiting the scope of the survey.

    • The car dealership handling the survey could bias the results as they have an invested interest in getting certain results, and that may skew how they administer the survey.

  • 14.

    Mercury City is trying to learn more about the car buying habits of their citizens. They decide to conduct a research survey, where they will randomly select 1000 households within city limits to call. One of the questions on the survey asks "do you prefer cool looking sport cars, or minivans?" Is there an issue with bias with this question? Why or why not?

    Answers:

    • No, as it's okay to have binary questions on surveys that doesn't include other options.

    • Yes, as it leaves out other choices of car in the question, so it may bias people who are indifferent to either type of car.

    • No. It only includes terminology that is truthful to the situation, so it does not cause unfair bias.

    • Yes. It attaches unnecessary adjectives to one of the choices, which may bias respondents towards a certain answer.

  • 15.

    Oscar wants to learn more about the trash habits of people in New York City. He creates a poll to ask people, and randomly selects 50 people who live in New York City to ask questions. Is there a potential issue of bias in this survey?

    Answers:

    • Yes. By focusing answers to New Yorkers, it may show a bias towards people within that city rather than the average population.

    • Yes. With a city as highly populated as New York City, 50 people is not a large enough sample size, which may lead to bias.

    • Yes, With a city as highly populated as New York City, 50 people is too large a sample size, which may lead to bias.

    • No, there is no potential issue of bias.

  • 16.

    Oscar, who loves trash, wants to learn more about the relationship New Yorkers have with their garbage. He creates a poll to ask people, and one of the questions included reads "Do you love your trash, or are you normal?". Does this question introduce a possibility of bias?

    Answers:

    • Yes, because people may be biased towards the first listed answer.

    • Yes, as it's a value judgement of the respondent that implies you are abnormal if you hold a certain opinion.

    • Yes, because it uses unnecessary adjectives that may bias respondents.

    • No, there is no issue of bias in the question.

  • 17.

    Josh is on his quest to find out if hot dogs are sandwiches. To get a sense of how his workplace feels on this matter, he makes a survey to give to all his coworkers. The surveys are left on their desks before they arrive to work, and may be handed in anonymously. Is there an issue of bias in this method?

    Answers:

    • Yes. By giving the option of handing in the survey anonymously, it may introduce bias as they may answer differently if their names are attached.

    • Yes. By asking all his coworkers rather than a sample size, it may cause higher variability that leads to bias.

    • Yes. By not handing out the surveys in person, it may bias people in their answers as they don't know who is running the survey.

    • No. Surprisingly, there is no issue of potential bias in his survey method.

  • 18.

    Josh is on his ongoing quest to find out if hot dogs are sandwiches. To get a sense of how his workplace feels on this matter, he makes a survey to give to all his coworkers. Josh made multiple versions of the survey where any questions with multiple choices have had the order of the answers changed. Will this introduce bias?

    Answers:

    • Yes. Changing the order of the answers on the multiple choice questions makes it a different question for each respondent, which may introduce bias.

    • Yes. Changing the order of the answers of the multiple choice may cause confusion, which may introduce bias.

    • No. While this is a surprising amount of work to put into a survey of this nature, it is a good step to avoid bias from respondents.

    • Yes. Having multiple choice questions on a survey may introduce bias, as it limits what people can answer.

  • 19.

    Elliot is surveying the school band about their past musical experience. He writes up an extensive survey and asks the teacher to distribute it to all the different band members. Is there a potential issue of bias with this method? Why or why not?

    Answers:

    • Yes. By asking the teacher to administer the survey, it may bias the responses as students may interpret the survey as a graded assignment.

    • Yes. By writing out an extensive survey, it may bias the student answers as they don't want to answer that many questions.

    • No, there is no issue of bias with his method.

    • Yes. By handing it out to band members only, they have a bias sample of the population, which is the school at large.

  • 20.

    Elliot, a fellow band member, is surveying the school band about their past musical experience. He writes up an extensive survey, with one of the questions asking "How great is practicing music?". Is this a biased question, why or why not?

    Answers:

    • Yes. This question contains jargon that may confuse students.

    • Yes. It is a leading question that may influence answers.

    • No. This is not a biased question.

    • Yes. This is an unclear question that may influence answers.

  • 21.

    Dante is surveying the most popular sides to serve at a barbeque in Texas. He posts an internet poll that's posted publicly, and gets over 3000 responses. Is there an issue of bias in Dante's method? Why or why not?

    Answers:

    • Yes. The poll did not receive enough responses for a proper sample size to represent the population of Texas.

    • Yes. Due to the poll being posted online publicly, people who are not from Texas may respond and bias the answer.

    • Yes. Asking about Texas barbeque only may bias the answers and leave out other types of barbeque styles.

    • No, there is no issue of bias in Dante's method.

  • 22.

    Dante is researching the most popular sides to serve at a barbeque in Texas. He creates a poll, with one of the questions reading "Select your favorite from the following three: Potato Chips, M&C, P. Salad." Is there a potential issue of bias in this question?

    Answers:

    • Yes. It is a leading question that may influence their answers.

    • Yes. It contains jargon that may not be understood by all people taking the survey, influencing their answer.

    • No, there is no issue of bias in this survey.

    • Yes. It contains unnecessary adjectives that could influence answers.

  • 23.

    Exit polls are being done on election day to get a sense of how a voting precinct voted. One of the pollsters assigned to survey voters is wearing merchandise for the candidate they support. Is there a potential issue of bias in this situation? Why or why not?

    Answers:

    • Yes. Having the survey being done on election day may influence how people respond to their questions as they are thinking about voting.

    • Yes. The pollster is wearing something that may influence the answer of the people they survey, as well as people's willingness to answer them.

    • Yes. The pollster is wearing something that contains jargon and may confuse the voters they question.

    • No. There is no potential issue of bias here.

  • 24.

    Exit polls are being done on election day to get a sense of how a voting precinct voted. One of the questions asked is "Which candidate did you vote for and how would you rate them as a person?" Does this question contain any issue of bias?

    Answers:

    • Yes. It is a leading question that may influence the answers of the respondent.

    • Yes. It's a double barreled question and may lead to confusion. They should be two separate questions.

    • Yes. It contains unnecessary adjectives that may influence answers of the respondent.

    • No. There is no issue of bias in this question.

  • 25.

    Doug wants to know about the TV-watching habits of kindergarteners (typically 5-6 year olds) at Mercury Elementary School. He creates a 2-page written survey and hands them out randomly to 30 different kindergarteners out of the total 100 enrolled at the school. Is there a potential issue of bias with his method? Why or why not?

    Answers:

    • Yes. Due to him handing it out to 30 kindergarteners, it creates too small of a sample size to get an accurate picture.

    • Yes. Him handing out the surveys to the kindergarteners may influence them, as they see who is conducting the survey.

    • Yes. Due to it being a written survey, it may bias the survey responses as many kindergarteners would struggle to read a whole survey.

    • No, there is no potential issue of bias here.

  • 26.

    Doug is learning about the TV-watching habits of kindergarteners at Mercury Elementary School. In one of the questions, he asks "If you have seen it, rate the show Mister Arthur's Neighborhood from 1 to 5". Is there a potential issue of bias with this question?

    Answers:

    • Yes, without clarifying what the rating score means, it may confuse them and bias their answers.

    • Yes. By adding a rating scale at all, it may bias their answers.

    • Yes. By assuming they have seen the show, it may bias the answers.

    • No. There is no potential issue of bias with this question.

  • 27.

    Sweet Times Bakery is revising their menu for the new year, which means they want to know what changes they should make. In addition to looking at sales records, they create a survey that they give to each customer over the course of two months. About 50% of the customers respond to the survey. Is there an issue of bias with their method?

    Answers:

    • Yes. By having the survey go for two months, that is not enough time to get an accurate picture of their customer base.

    • No, there is no issue of bias with their survey method.

    • Yes. By only giving the survey out to customers, they are ignoring the opinions of people who don't shop at the bakery.

    • Yes. By only receiving 50% of the surveys back, it's not a large enough sample size.

  • 28.

    Sweet Times Bakery is revising their menu for the new year, which means they want to know which items they can remove from the menu and what new items should be added. They create a survey to give to their customers, with one of the questions reading "If any, which item on the current menu would you remove, and what would you add to replace it?" Is there a potential issue with this question? Why or why not?

    Answers:

    • No. There is no potential issue with this question.

    • Yes. It's a question that contains jargon and may confuse customers.

    • Yes. It strongly implies that there needs to be items to be removed from the menu, leading customers to a biased response.

    • Yes. It's a double-barreled question that may lead to confusion with the survey takers.

  • 29.

    Kevin, who tends to be very vocal about his opinions, wants to know his family's pizza preferences. He creates a quick survey that he asks each member of his family to their face. Is there an issue of bias with his method?

    Answers:

    • No. There is no issue of bias with this method.

    • Yes. Due to it being a quick survey, the questions may not cover the information that Kevin is looking for.

    • Yes, due to Kevin being known as vocal with his opinions, it may influence the answers of his other family members.

    • Yes. Surveys should never be asked face-to-face, as it may influence the respondents.

  • 30.

    Kevin created a quick survey for his family members so he can learn about their pizza preferences. One of the questions reads "Do you prefer tasty pepperoni, boring cheese, or neither?" Is there a potential issue of bias with this question?

    Answers:

    • Yes. It contains jargon and assumes the family members know what pepperoni is.

    • Yes. By attaching unnecessary adjectives, it could lead to biased responses based on how they are framed.

    • Yes. By only having two real choices in the question, it may bias the response.

    • No. There's no issue of bias with this question.

Support