Winter 2009

Finding the Truth Even When the Consensus is Wrong
In "The Wisdom of Crowds" James Surowiecki wrote about the ability of large groups to answer questions better than any single individual. He used a number of examples to show that a diverse collection of individuals is likely to make certain types of decisions and predictions better than individuals or even experts. But what about situations in which the collective wisdom is wrong?

In many situations, subjective judgment is needed to choose the right course of action. When an IDEALYST® game generates fifty new product ideas, an organization has to choose a limited number of ideas to develop further. Even if all fifty sound good, they can’t all be formally researched and evaluated. Only those that are a good fit with the company’s core competencies, that can be brought to the market in a reasonable time, or that are anticipated to be big winners in the marketplace, will get the detailed planning necessary to make an informed decision about investment and resource commitment.

Four years ago, Dražen Prelec, a Professor at MIT’s Sloan School of Management, published an article in Science entitled "A Bayesian Truth Serum for Subjective Data." The article presented an information scoring formula that could be applied to subjective data in situations where the objective truth is unknowable. The title of a more recent paper is more explicit: "An algorithm that finds truth even if most people are wrong." When this scoring technique is applied to IDEALYST results, the best ideas can be identified, even when the consensus is wrong!

The formula works by posing a question that has two or more possible answers to some number of respondents. The question should have only one right answer, and it should be reasonable to believe that if the collective knowledge of the "crowd" could be appropriately pooled, the right answer could be determined. For example:
  • Chicago is the capital of Illinois (True or False).
  • The price of this painting is ____ (< $1,000, $1,000 - $30,000, $30,000 - $1,000,000 or > $1,000,000).
  • This new product idea will be ____ (a money loser, a breakeven proposition, a small success, a huge win).
Each respondent is asked to pick the answer they believe to be true and to predict the proportion of the sample that will choose each of the possible answers. In other words, what do they think is the right answer and what do they think everyone else will answer?

Those respondents who show they have the best knowledge of how the rest of the world thinks, get the most weight in picking the best answer. Someone who knows that Springfield is the capital of Illinois, but believes that most people will choose Chicago, will get more weight attached to their prediction than someone who chooses Chicago and thinks others will also choose Chicago. Professor Prelec’s recent research provides a validation of this formula using data on the price of paintings as estimated by gallery owners. Using Bayesian Truth Serum, Prelec correctly identifies the masterpieces (price greater than $1M), even when most gallery owners think the price is much lower.

As a follow-on to IDEALYST or other idea generation exercises, members of an innovation team can use the Bayesian Truth Serum to identify the best ideas for follow up. AMS has recently adapted this research into its new REALYST™ idea filtering and evaluation process. Just as IDEALYST eliminates politics from brainstorming, REALYST keeps the most vocal participants from unduly influencing the results. And even if the consensus is wrong, the answer will likely be right.

-Bob Klein
bklein@ams-inc.com

© Copyright 2009 Applied Marketing Science, Inc.