Column: Quant Trumps Pundit; Crowd Trumps Quant
Forget about Obama versus Romney. The real contest in the 2012 election was about analytics.
Politics is just the latest front in a war being fought in businesses and consulting firms around the world, with this round pitting quants such as New York Times blogger Nate Silver against the intuition of pundits like James Carville or Karl Rove. The quants bring data, computers and formal models. The pundits — though they do use data — rely more on gut feelings, industry experience and personal contacts.
In the latest skirmish, the quants won. They predicted the election outcome far more accurately than the pundits did. Therein lies a lesson for executives and policy makers alike: Wisdom and intuition may actually be hurting your firm or organization.
Consider the Federal Reserve, arguably the most powerful economic institution on the planet. The Fed’s staff economists do for economic statistics what Silver does for poll numbers, crunching piles of data into a very accurate forecast. By contrast, the members of the Federal Open Market Committee are more like Carville, wisened by years of experience and equipped with anecdotes from their industry contacts.
A surprising study by economists Christina Romer and David Romer found that adding the opinions of these pundits to the staff forecast actually produces an inferior result. The surprising conclusion is that we would be better served if the members of the FOMC simply withheld their judgment.
This doesn’t mean the quants can declare victory. As it turns out, there is an even better forecaster: crowds. Political prediction markets, such as Intrade, which rely on the wisdom of crowds rather than any individual uber-pundit, have historically done a better job of predicting elections than even very sophisticated statistical models do. In 2008, the prediction markets were more accurate than Silver in forecasting the election results. In 2012, the scorekeepers at PunditTracker.com called it a draw, despite attempts by Romney supporters to manipulate the odds at Intrade.
The implications go far beyond elections. Research has shown that prediction markets can forecast economic, business and sporting outcomes better than relevant experts can. They even helped Hewlett-Packard predict printer sales better than its own analysts could. What these markets do is efficiently aggregate many different types of information pertinent to the forecast at hand — not merely the parts that are quantifiable and thus easy to subject to the quants’ analytical tools. The quant may be smarter than any other person in the room, but he’s not smarter than the room as a whole.
Prediction markets can be difficult for businesses to use. For one, you need to find a lot of people who are willing to bet on the outcome that interests you. That can be particularly problematic in places that have anti-gambling laws.
Luckily, you don’t need a market to draw upon the wisdom of crowds. In recent research, David Rothschild and Justin Wolfers analyzed the results of election polls that asked random samples of Americans who they thought would win — another way to crowd-source insight. They found that polls of voters’ expectations consistently outperformed standard polls, which asked voters whom they plan to vote for. In 345 separate races, the expectation polls correctly predicted 81 percent of all election winners, compared with 69 percent for the regular polls.
The recent election provided an ideal real-world test of the different types of polls. Quants such as Silver based their projections on polls asking people whom they intended to vote for. These projections varied sharply over time, picking up the strong pro-Obama sentiment only in the last few days of the campaign. Some polls predicted a close contest, and a few even pointed to a victory for Republican nominee Mitt Romney.
By contrast, surveys of voter expectations pointed to President Obama’s win much earlier and more consistently, and also correctly suggested that he would win by a healthy margin.
Asking the crowd for its forecast yields useful insights because it treats poll respondents as mini-anthropologists, asking them to report back not just on their own thoughts and feelings, but also on those of their friends, neighbors and co-workers. This boosts a poll’s effective sample size, as each respondent may end up speaking for the preferences of 10 or 20 others.
The same logic that works for political forecasts might work well in other contexts. Think about how most market research works. A clothing manufacturer may convene a dozen people in a focus group, and ask them whether they prefer a red shirt to a blue shirt. Such exercises have only limited value, partly because the opinions of so few people may not represent the broader population. If, however, each of these people is asked to reflect on the preferences of his or her friends and family, the responses may actually mirror hundreds of people from a dozen separate communities.
The approach may be helpful in constructing better economic indicators, as well. For instance, economists are very interested in tracking employment. Rather than just asking a sample of people if they have a job (as the Bureau of Labor Statistics currently does), why not also ask whether the firm where they work is hiring or firing? Gallup Inc. has recently started asking precisely this question, and some preliminary analysis suggests that it can help explain labor market developments. (Disclosure: Justin Wolfers is a senior scientist at Gallup.)
The 2012 election had a clear winner: Analytics beat intuition. This is threatening both to the likes of Carville and Rove and to their intuition-driven counterparts in the corporate world. But the quants also have to respect the crowd. The success of prediction markets and expectations polls tells us something truly humbling — that knowledge doesn’t just reside in the executive suite or in a quantitative model. For executives nimble and humble enough to accept this, it presents a great opportunity.
Betsey Stevenson is an associate professor of public policy at the University of Michigan. Justin Wolfers is an associate professor of business and public policy at the University of Pennsylvania, and a non-resident senior fellow of the Brookings Institution.