On Surveys, Bias, and Admitting You're Wrong

Daniel Klein is an economics professor at George Mason Univ. and a self-described libertarian. Though some of his libertarian beliefs would be considered "liberal"—ending of all narcotics laws, for example—most are more on the conservative side, such as ending of the income tax. Klein definitely energized conservatives when he published an opinion piece in the Wall Street Journal two years ago declaring American liberals ignorant about economics. "Responding to a set of survey questions that tested people's real-world understanding of basic economic principles, self-identified progressives and liberals did much worse than conservatives and libertarians," he says in a recent follow-up in The Atlantic. The Journal piece quickly spread across the Internet, leading to 10,000 downloads of the study upon which it was based. Klein had co-authored that with Elijka Buturovic, a psychology Ph.D. from Columbia Univ. who works for an opinion poll company Klein doesn't name.

The follow-up's title explains why it was necessary: "I was Wrong, and So are You." The catalyst was a new study the pair did that checked the possibility unintended bias in the first survey's questions caused the one-sided results. Indeed they had. The new study made clear that on average, U.S. conservatives, liberals, and libertarians are equally dumb about economics.

To understand how the results could be so different, read two example statements from the studies:

• "Restrictions on housing development make housing less affordable."
• "A dollar means more to a poor person than it does to a rich person."

The first came from the first study, and if you are a conservative or libertarian, you are more likely to want to agree with it, and therefore will if you do not know the objective facts. If you are liberal, you probably want to agree with the second instead, which came from the second study. The first group would disagree with the second statement, and the liberals with the first. But both statements are factually correct, based on significant research.

"You may have noticed that several of the statements we analyzed (in the first survey) implicitly challenge positions held by the left, while none specifically challenge conservative or libertarian positions," Klein says in the Atlantic article. "A great deal of research shows that people are more likely to heed information that supports their prior positions, and discard or discount contrary information." That means liberals were more likely to discount the right-leaning statements of the first survey, and thus more likely to give wrong answers. Buturovic had been researching this bias effect, which led the pair to do the second survey using statements mostly challenging conservative or libertarian positions. This time, people who identified themselves as liberal were right more often than those in the other camps. Studies show that a dollar is more precious to a poor person, but more than 30 percent of libertarians and 40 percent of conservatives disagreed with that statement, versus only 4 percent of progressives. Klein chides his fellow libertarians: "c'mon, people!" But the bottom line comes in his next sentence. "Consistently, the more a statement challenged a group's position, the worse the group did," he says.

Klein then chides himself. "Shouldn't a college professor have known better? Perhaps. But adjusting for bias and groupthink is not so easy," he says. He points out that education levels had no impact on accuracy, which means it had no impact on confirmation bias.

This article bolsters several of my earlier posts. A recent one warned of the dangers of do-it-yourself surveys. In this case, even "a college professor" who was not an expert on surveys had to be corrected on his conclusions by a specialist. I have also explained my own bias toward science-based team leadership partly because researchers go to great lengths to overcome bias and are more likely than consultants to admit when they are wrong. Klein provides a very public, honorable example of that effort and that willingness. Finally, speaking as an ex-journalist, I have made the point that the public perception of scientific flip-flopping is more the fault of poor journalism than of science. The first article "set off fireworks," Klein reports, and was picked up by major U.S. newspapers. The second received relatively little notice. Everyone who read the first stream of articles and blog posts will go through life believing Klein's first position was his final one.

There are plenty of lessons in this for the workplace, besides the one about surveys:

• If something seems to completely support your position, demand more proof to make sure you are not just viewing it the way you want to. I actively seek studies that differ with my current knowledge.
• If something seems too simple an answer, seek out alternate explanations. I refuse to report study conclusions that support (or refute) my beliefs if I think the evidence from the study is unclear.
• If you find out you were wrong, say you were wrong. Admitting his error to the 450,000 paying customers of The Atlantic could not have been easy for Klein, but doing so gives those readers the chance to open their minds.

To check your understanding of your world, open it to the scrutiny of others. You might know your job or your team better than many people, but you do not know it better than all people. Because you are human, you are biased. Whether you let this fact mislead your organization into wasting time, money and stress, or admit that you are human and seek out the facts wherever they lead, is your choice.

Action Item: The next time you take a strong position in a meeting, ask yourself: "How do I know that?" Within 24 hours, spend 30 minutes searching for factual evidence (not just opinions) that counter your position.

Source: Klein, D. (2011), "I was Wrong, and So are You," The Atlantic 308(5):66.