Death by data: Use what’s relevant, not what’s available

More data may not mean better decisions.

- DEEP MUKHERJEE

In 1974, Paul Slovic, a psychologist and decision-science guru, performed an experiment on horse handicappers: i.e., professional predictors of horse racing outcomes. The handicappers were tasked with predicting the winner of 40 races. For each horse, 40 attributes were collected. Four rounds of prediction were to be made by these handicappers. In round 1, each handicapper got to choose five attributes to predict the winner. They were also asked to comment on their confidence in their predictions. If they had picked a horse at random, their success rate would have been 10%, since each race had 10 horses. But in round 1, the correct-prediction rate was 17%, implying that the five attributes used by handicappers had given them better-than-random predictive power.

In the second, third and fourth rounds of prediction, respectively, 10, 20 and all 40 attributes were provided. The study found that predictive ability remained around 17%, but the handicappers’ confidence in their predictions increased substantially. While this spike in confidence was only human, there might be several reasons for why their prediction quality did not improve with more data. One reason could be that the human ability to process abundant information is limited. Another could be that the extra data was not relevant to the task. At times, just because data is available, one starts believing that it must be useful—a problem of availability bias.