Biased survey questions: six types, with a rewrite for each

We rewrite leading, loaded, double-barreled, absolute, ambiguous and unbalanced survey questions, and cover the order and social desirability biases.

November 13, 2026

Godefroy

Key takeaways
  • A survey question is biased when its wording, its answer options or its place in the form push people towards an answer other than the one they hold. Six forms come back in most questionnaires: leading, loaded, double-barreled, absolute, ambiguous, and the unbalanced scale.
  • Rewrite a biased question before the survey goes out. Once the answers are in, nothing tells you which ones the wording produced.
  • A few words added to a question can move the result by 25 points, even when every answer is sincere.
  • Some bias comes from around the question rather than from inside it: the question asked just before, and the answer that makes the respondent look good. Order the form from general to specific, and keep sensitive counts in an anonymous format.
  • A form gives everyone the same wording, so a respondent who reads a question differently from you still ticks a box. Once the survey is out, only a follow-up question in an interview catches that misreading.

A biased survey question is one whose wording, answer options or position in the form push respondents towards an answer other than the one they actually hold. Six types turn up in most questionnaires: the leading question, the loaded question, the double-barreled question, the absolute question, the ambiguous question and the unbalanced answer scale. Each one has a rewrite, given below with a before and after example. We also cover three biases that come from around the question rather than from inside it: the order of the questions, the pull towards the answer that looks good, and the habit of agreeing with any statement.

In January 2003, the Pew Research Center, a US polling organisation, asked Americans whether they favoured or opposed “taking military action in Iraq to end Saddam Hussein’s rule”. 68% were in favour. When the question added that US forces “might suffer thousands of casualties”, support fell to 43%, according to Pew’s own guide to writing survey questions. Both numbers are real: the second question simply gave people a fact the first one left out. Neither of them is the opinion of the country, though. Each one is the answer to a particular sentence.

Your satisfaction survey works the same way. The answer depends on the sentence you wrote, yet once the responses come in, nothing in the spreadsheet shows which part of the number came from the wording.

What makes a survey question biased

To spot a biased question, check that someone who disagrees with you can answer it as easily and as honestly as someone who agrees. The test takes ten seconds. Read the question as the customer who is about to cancel, then as the one who loves the product. If one of them has to fight the wording, or cannot find their answer among the options, the question favours one side.

Keep two other problems apart from this one. Sampling bias concerns who answers: a survey sent only to active users says nothing about the ones who left, however neutral the questions are. Nonresponse concerns who skips a question. Both distort a result, but rewriting a question fixes neither.

Six types of biased survey questions, with a rewrite for each

Leading questions

A leading question hands the respondent the answer you expect. It usually sounds like a friendly paraphrase of your own hypothesis.

  • ❌ “How much did you enjoy the new onboarding?”
  • ✅ “How would you rate the new onboarding?” (very poor, poor, neither good nor poor, good, very good)

The first version takes the enjoyment for granted and only asks how much of it there was. Someone who found onboarding tedious has no answer that fits, so they pick the lowest amount of enjoyment on offer, which still counts as enjoyment.

Elizabeth Loftus and John Palmer, psychologists at the University of Washington, measured how much a single word changes an answer. They showed students the same film of a car accident and asked how fast the cars were going when they “contacted” each other, or when they “smashed” into each other. The average estimate was 31.8 miles per hour with “contacted” and 40.8 with “smashed”, as we detail in our bank of open-ended questions. The students saw the same film. They answered the verb.

Often the sentence before the question does the leading. “We have worked hard to improve support. How satisfied are you with it now?” asks for a reward. Cut the sentence that tells the respondent what you hope to hear.

Loaded questions

A loaded question builds in an assumption the respondent may not share, or uses a word that carries a judgment.

  • ❌ “What do you like most about the new dashboard?”
  • ✅ “Have you used the new dashboard?” then, for those who say yes, “What, if anything, would you change about it?”

The first version assumes the person has used the dashboard and likes something in it. Someone who never opened it answers anyway, because the question gives them no way out. Their invented answer is then counted with the real ones. Ask a filter question first, then ask the second question only of those who say yes.

In the other kind of loaded question, a single word passes judgment. Pew reports that support for spending more on “assistance to the poor” is much higher than support for spending more on “welfare”, for the same policy. In 2005, 51% of Americans favoured “making it legal for doctors to give terminally ill patients the means to end their lives”, while only 44% favoured “making it legal for doctors to assist terminally ill patients in committing suicide”. The same thing happens with a pair of verbs. In 1941, the survey researcher Donald Rugg asked half his sample whether the United States should “forbid” public speeches against democracy, and the other half whether it should “allow” them. The two wordings gave results 21 points apart, as we explain with our examples of dichotomous questions.

In a product survey, the loaded words are harder to spot: “bug” for a behaviour the user may consider normal, “only” in “it only takes two minutes”, “despite” in “despite the price increase”. Replace each with the plainest word that describes the thing without rating it.

Double-barreled questions

A double-barreled question asks two things at once and accepts only one answer.

  • ❌ “How satisfied are you with the speed and reliability of the app?”
  • ✅ “How satisfied are you with the speed of the app?” then “How satisfied are you with its reliability?”

A customer with a fast app that crashes every Friday has no honest answer to the first version. They either average the two or answer about the part that bothers them most. You cannot tell which. Pew’s own example is “How much confidence do you have in President Obama to handle domestic and foreign policy?”, which they recommend splitting in two.

The tell is the word “and” between two things that can vary separately. “The tool is easy to use and saves me time” is two statements. So is “Was support quick and helpful?” Some double-barreled questions use “or” instead: “Did you find the price too high or the setup too long?” pushes the person to pick one of your two hypotheses.

Absolute questions

An absolute question puts a word like “always”, “never”, “every” or “all” in a yes or no question.

  • ❌ “Do you always read the release notes?” (yes, no)
  • ✅ “Of the last five releases, how many sets of release notes did you read?” (0, 1, 2, 3, 4, 5)

Almost nobody always does anything. Strict respondents answer no and generous ones answer yes, though both mean “most of the time”. The result measures how literally people read, more than what they do. The fix is to ask for a count over a period that someone can reconstruct.

Ambiguous questions

An ambiguous question contains a word that respondents will each read their own way.

  • ❌ “Do you use the tool regularly?”
  • ✅ “In the last 30 days, on how many days did you open the tool?”

“Regularly” means daily to one person and once a month to another. Tarek Al Baghal, at the Institute for Social and Economic Research of the University of Essex, asked people to turn frequency words into a number of times a year: “very often” averaged 15.5, with individual answers running from 7 to 37. Everyone agrees that “very often” is more than “often”, but the number behind the word changes from one person to the next.

The same goes for jargon your team uses every day. “How satisfied are you with the workspace settings?” assumes the respondent knows which screen you call the workspace. Define the term in the question, or describe the screen.

Unbalanced answer scales

An unbalanced scale offers more room on one side than on the other.

  • ❌ “How would you rate support?” (excellent, very good, good, fair)
  • ✅ “How would you rate support?” (very poor, poor, neither good nor poor, good, very good)

The first scale has three positive options and one middling one, so a dissatisfied customer has nowhere to go but “fair”. The report then shows 100% of answers at fair or above. A balanced scale has the same number of points on each side of a neutral midpoint, with labels that mirror each other.

Answer ranges can bias the result too. Norbert Schwarz, then at ZUMA, the German survey methods centre in Mannheim, and three colleagues asked the same question about how much television people watched a day, on two scales. When the scale topped out at “more than 2½ hours”, 16.2% of people picked that top option. When it ran up to “more than 4½ hours”, 37.5% reported more than 2½ hours. People read the middle of the scale as the normal amount, as we explain in our closed-ended question examples. If you can, ask for the number itself rather than a range.

Six biased survey questions with their rewrites: leading, loaded, double-barreled, absolute, ambiguous and unbalanced scale

Types of biased questions compared

TypeExampleWhat goes wrongRewrite
Leading”How much did you enjoy the new onboarding?”The expected answer is in the questionAsk for a rating on a balanced scale
Loaded”What do you like most about the dashboard?”It assumes use, or a word passes judgmentFilter first, then use neutral words
Double-barreled”Speed and reliability of the app?”One answer for two thingsOne question per thing
Absolute”Do you always read the release notes?""Always” makes strict and generous readers answer differentlyA count over a period
Ambiguous”Do you use the tool regularly?”Each person reads “regularly” their own wayA number over a dated period, terms defined
Unbalanced scaleExcellent, very good, good, fairDissatisfied people have no negative optionEqual points on each side of a midpoint

Biases that come from around the question

A question can be perfectly worded and still collect a skewed answer, because of the question before it or because of the respondent’s own habits.

Question order

The question asked just before changes how people answer the next one. In December 2008, Pew found that 88% of Americans said they were dissatisfied with the way things were going in the country when that question came right after one on George W. Bush’s job approval, against 78% when it came first. After rating the president, people had their grievances fresh in mind.

In a product survey, the same effect shows up when a detailed block on recent bugs comes before the overall satisfaction score. The respondent has just listed everything that went wrong, so the overall rating drops. Put the general question before the specific ones, and randomise the order of options and blocks when your survey tool allows it, so the order effect is spread across respondents instead of always hitting the same answer.

Social desirability bias

People report what makes them look good. In a 2016 analysis of the 2014 US midterm elections, Pew found that 75% of registered voters in its panel said they had voted, while voter records confirmed a vote for 63% of them, according to its comparison of turnout measures. Pew notes that part of the gap comes from errors in the records. The rest is people claiming a vote they did not cast.

The same pull applies to questions about reading the documentation, following a security policy or giving feedback to a manager. Three changes reduce it:

  • Make the less flattering answer normal in the question itself: “Many people don’t get round to the documentation. In the last month, did you open it?”
  • Ask about a behaviour over a precise period rather than about a habit, since people describe their habits as they would like them to be.
  • Keep sensitive counts in an anonymous self-completed form. A respondent alone with a form has no one to impress.

Acquiescence bias

With agree and disagree statements, people agree more often than their actual opinion would predict. Jon Krosnick and Stanley Presser, survey methodologists at Stanford and the University of Maryland, estimate the effect at about 10 percentage points. The fix, backed by the studies we go through with our closed-ended question examples, is to name the options directly rather than ask people to agree.

Three Pew Research Center results showing how wording, question order and social desirability move a survey number

How to ask unbiased survey questions

Go through your draft question by question with this list before anyone else sees it:

  1. Write down what you need to learn from the question. If you cannot, the question has no reason to be in the survey.
  2. Keep one idea per question. Every “and” or “or” between two things that can vary separately becomes two questions.
  3. Remove any sentence before the question that tells the respondent what you hope to hear.
  4. Check what the question assumes. If it assumes a behaviour, ask a filter question first.
  5. Replace every word that rates the thing with the plainest word that describes it.
  6. Replace “always”, “never”, “regularly” and “often” with a count over a dated period.
  7. Give scales the same number of points on each side, keep the midpoint, and make sure the options cover every possible answer.
  8. Order the survey from general to specific, and randomise options and blocks where order does not carry meaning.
  9. Pretest with five people from your audience. Ask each one to read the questions aloud and say what they understand. Survey methodologists call this cognitive interviewing. It brings out the ambiguous words that nobody on your team notices any more.

The pretest catches the most problems for the least effort, yet most teams skip it. Spend ten minutes on the phone with someone outside the project and you often find out that “workspace” means nothing to them, or that they read your scale backwards.

graph TD
  A["Read the question as a critic, then as a fan"] --> B{"Can both answer without fighting the wording?"}
  B -->|"No"| C["Remove the expected answer and any judgment word"]
  B -->|"Yes"| D{"Does it ask about two things?"}
  D -->|"Yes"| E["Split it in two"]
  D -->|"No"| F{"Does it assume something?"}
  F -->|"Yes"| G["Add a filter question first"]
  F -->|"No"| H{"Could two people read a word differently?"}
  H -->|"Yes"| I["Replace it with a count or a definition"]
  H -->|"No"| J{"Are the options balanced and complete?"}
  J -->|"No"| K["Mirror the scale around a midpoint"]
  J -->|"Yes"| L["Keep the question"]

The misreading a form cannot fix

The checklist fixes the wording. It cannot fix a reading you did not anticipate, because a form gives every respondent the same sentence and no way to ask what you meant. Someone who reads “regularly” as “every quarter” ticks yes, a yes that looks exactly like a daily user’s. Nothing in the results flags it.

An interview catches that. When an answer stays vague, the interviewer asks what the person means, so the misreading comes out while they are still talking. That follow-up is also where interviews pick up much of their own bias, because an interviewer with a hypothesis tends to put it in the follow-up: “so it was the price that put you off?”

We built Raconte to run that conversation without the scheduling. You write the brief once and send a link. The person talks for a few minutes, whenever it suits them. As soon as an answer stays vague, the AI follows up. Every respondent gets the same brief, so the questions do not drift from one interview to the next the way they do when five colleagues share the calls. The AI also has no hypothesis it hopes to confirm. Its follow-ups still come from your brief, though, so a brief written as “find out why the price is a problem” produces leading follow-ups. Check the brief against the same checklist as the survey. You get the transcript, the audio, a summary and per-message sentiment for each interview.

A voice interview does not suit every question. On a sensitive count, such as how often someone breaks a security rule, an anonymous form remains the safer format, since the person has nobody to impress. A study by Gale Lucas and Jonathan Gratch at the USC Institute for Creative Technologies found that people disclosed more to a virtual interviewer they believed was automated than to the same one when told a person was operating it. That gap favours an AI over a human interviewer, but it does not make a conversation as anonymous as a blank form. And when all you need is one countable fact from two hundred people, a well-written form is faster for everyone. We go through when to use a survey or a voice interview, and how to combine them.

Where to start

Open the survey you are about to send and run the checklist on each question. The double-barreled ones and the unbalanced scales take a minute each to fix. The ambiguous words are harder to see from inside the team, so do the pretest with five people from outside it.

Then pick the two or three questions where you most need to know why, and ask them in a conversation instead. Prepare them with an interview guide, so every conversation follows the same structure. With Raconte, the first 60 interview minutes each month are free, with no card. Beyond that, you pay by the minute.

Frequently asked questions

What is a biased survey question?

A biased survey question is one whose wording, answer options or position in the survey push respondents towards an answer other than the one they hold. The six common types are the leading question, the loaded question, the double-barreled question, the absolute question, the ambiguous question and the unbalanced answer scale. Question order and social desirability can bias an answer even when the question itself is well written.

What is an example of a leading question?

“How much did you enjoy the new onboarding?” is a leading question, because it takes the enjoyment for granted and only asks how much there was. A neutral version is “How would you rate the new onboarding?”, with a balanced scale from very poor to very good. “Don’t you think the new design is easier?” is another, since the negative form invites a yes.

What is a double-barreled question?

A double-barreled question asks about two things but accepts a single answer, such as “How satisfied are you with the speed and reliability of the app?”. Someone who finds the app fast but unreliable cannot answer it honestly. Split it into one question about speed and one about reliability. The word “and” between two things that can vary separately is the usual tell.

How do you ask unbiased survey questions?

Keep one idea per question, remove words that rate what you are asking about, add a filter question before asking about something the person may not have used, ask for counts over a dated period instead of “always” or “regularly”, balance every scale around a midpoint, and order the survey from general to specific. Then have five people from your audience read it aloud and say what they understood.

What is the difference between a leading and a loaded question?

A leading question points to the answer you expect, as in “How much did you enjoy it?”. A loaded question builds in an assumption or a judgment the respondent may not share, as in “What do you like most about the dashboard?”, which assumes they used it and liked something. Both push the answer in one direction. A leading question needs only rewording, while a loaded one usually also needs a filter question before it.

Can an answer scale be biased?

Yes. A scale with more positive options than negative ones, such as excellent, very good, good and fair, gives a dissatisfied respondent nowhere to go. The range matters too: in an experiment by Norbert Schwarz and colleagues, the share of people reporting more than 2½ hours of television a day went from 16.2% to 37.5% when only the scale range changed. Use the same number of points on each side of a neutral midpoint.

Is social desirability bias caused by the question?

Partly. Social desirability bias comes from the respondent wanting to look good, so it appears even with a neutral question. The wording can make it worse or better. To reduce it, ask about a precise recent period and make the less flattering answer sound normal. An anonymous self-completed form reduces it more than an interview with a person.

Continue reading