The Pitfalls of Self-Report (Or, Why Focus Groups Fail Us)

What is self-report?

Self-report[1] is just what it sounds like: when someone reports or tells you about him or herself. Ask a person what they think, do, or believe, and what you get back is self-report data. 

Self-report data comes from several different methods, including: 

  • Questionnaires 
  • Interviews 
  • Open Questions (short response, qualitative data)
  • Closed Questions (age, weight, favorite sports teams etc)
  • Rating Scales
  • Fixed-Choice Questions (yes or no etc)
  • Focus Groups

These kinds of studies are common, and for good reason: they’re cheap, they don’t require fancy equipment, and participants kind of love them. (Research has found that about ~40% of our speech is about none other than ourselves[2].) After all, if you want to know about me, who better to ask?

But decades of social and cognitive psychology have uncovered very real limitations to self-report. It turns out we are notoriously bad at understanding our own minds.

Here are just a few of the well-known pitfalls of self-report.


Pitfall #1: Our minds are easily manipulated

A famous study[3] showed respondents videos of two cars as they met in a head-on car accident. After the videos were over, researchers asked the respondents to estimate how fast the cars had been going just before the collision. 

But they phrased the question differently each time, substituting in words meant to suggest different speeds: “About how fast were the cars going when they (contacted/hit/bumped/collided/smashed) each other?”

Word choice affected the speed the respondents remembered observing. When asked about cars that “contacted” each other, participants guessed, on average, about 31.8 mph. Those who heard “hit” guessed 34, and the “smashed” group guessed 40.8. Yet they all saw the exact same video.

This is just one of many studies that show how susceptible our memories are to manipulation after the fact. Next time you fill out a survey or participate in a focus group, think about how a slightly different phrasing of the question could impact the answer you give. 


Pitfall #2: We don’t know our reasons

It’s not just our memories — even our current thoughts and beliefs can be toyed with without our knowing. 

A popular study published in the 1970s[4] showed that people are completely unaware of external factors influencing what comes to mind.

Participants were asked to memorize a list of random word pairs, some of which were inserted by the experimenters as “primes.”  For example, “ocean-moon” was expected to make participants think of the brand “Tide.” Sure enough, when asked questions such as “What is your favorite detergent?”, those who saw word pairings like “ocean-moon” said “Tide” twice as often as those who didn’t.

But when asked why, nobody said “Because of the word pairing task”. Instead, they made up other answers. “My mom uses Tide”, “I like Tide packaging”, “Tide is well known”. That could be true —but it doesn’t explain why those who saw the “ocean-moon” pairing were more likely to say so.

This phenomenon is known as confabulation, and it’s a plague on self-report studies. We often don’t really know why we do what we do. But instead of recognizing that, we make up explanations out of thin air.


Pitfall #3: We’re reliably biased

Even when researchers aren’t playing tricks on us and we’re really trying to be honest, we’re affected by systematic cognitive biases that skew our answers. Here are just a few: 

Social Desirability Bias[5]: Participants may respond untruthfully to present a favorable image of themselves. Research shows[6] that social desirability bias skewed results of 45% of papers that tested for it!

  • Picture This: Mark is taking a survey. The experimenters ask him whether he likes Microsoft or Apple more. Based on his purchase habits, the answer is clear: He always buys Microsoft products for their much more affordable price tag. But he’s sensitive about looking cheap so he says Apple, so he’s seen as able to afford expensive and sleek electronics. 

Groupthink[7]: A group may make a “bad” or non-optimal decision in order to maintain harmony within the group. This one especially affects focus groups, when participants are interviewed along with several others. 

  • Picture This: We’ve known about groupthink since the early 70’s, but anyone who has encountered the “hive mind” on Twitter or Facebook knows it’s alive and well today. It’s a known problem affecting decision-making in politics and business, and a major reason for seeking diverse teams and divergent thinking[8].

Demand Characteristics[9]: Participants may subtly change their responses in order to be considered a “good subject”.

  • Picture This: Kristen is a recent hire at a small company, and Human Resources has sent out an “anonymous” employee satisfaction survey. She knows what she really thinks…but she worries that her responses may be traced back to her. Instead of responding honestly, she says everything is fine. The result? Survey answers that look nice, but don’t reflect the real world.

Self-report clearly has its problems, but we don’t want to throw the baby out with the bathwater. What participants say about themselves will always be part of the story — just not the entire thing.

At SPARK Neuro, we study self-reported thoughts and feelings along with biometric measures of  respondents’ nervous systems. Sometimes, self-reported answers are accurate. But when they aren’t, we have the rest of the story

 

Citations

  1. Lavrakas, P. J. (2008). Encyclopedia of survey research methods. Thousand Oaks, CA: Sage Publications, Inc. doi: 10.4135/9781412963947
  2. Tamir, D. I., Zaki, J., & Mitchell, J. P. (2015). Informing others is associated with behavioral and neural signatures of value. Journal of Experimental Psychology: General, 144(6), 1114.
  3. Loftus, E. F., & Palmer, J. C. (1974). Reconstruction of auto-mobile destruction: An example of the interaction between language and memory. Journal of Verbal Learning and Verbal Behavior, 13, 585-589.
  4. Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84(3), 231.
  5. King, M. F., & Bruner, G. C. (2000). Social desirability bias: A neglected aspect of validity testing. Psychology & Marketing, 17(2), 79-103 .
  6. Van de Mortel, T. F. (2008). Faking it: social desirability response bias in self-report research. Australian Journal of Advanced Nursing, The, 25(4), 40.
  7. Janis, I. L. (2008). Groupthink. IEEE Engineering Management Review, 36(1), 36.
  8. Rock, D., & Grant, H. (2019, March 19). Why Diverse Teams Are Smarter. Harvard Business Review. Retrieved from https://hbr.org/2016/11/why-diverse-teams-are-smarter
  9. Allen, M. (2017). The sage encyclopedia of communication research methods (Vols. 1-4). Thousand Oaks, CA: SAGE Publications, Inc. doi: 10.4135/9781483381411