Statistical Models
Baseball, wine, medical diagnosis, university admissions, and criminal recidivism all represent realms where simple statistical models have outperformed “experts.”
William Grove, et al (”Clinical Versus Mechanical Prediction“) consider 136 studies of simple quant models versus human judgements. The range of studies covered areas as diverse as criminal recidivism to occupational choice, diagnosis of heart attacks to academic performance. Across these studies 64 clearly favoured the model, 64 showed approximately the same result between the model and human judgement, and a mere 8 studies found in favour of human judgements. In all of these 8 the humans had more information than the quant models.
The average specialist in the study got 67% of the cases they were presented with correct. The quant models’ average hit ratio was 73%.
—
There is plenty of evidence to suggest that we tend to overweight our own opinions and experiences against statistical evidence. Ilan Yaniv (”Advice taking in decision making“) has an experiment based on general knowledge questions such as: In which year were the Dead Sea scrolls discovered?
Participants are asked to give a point estimate and a 95% confidence interval. Having done this they are then presented with an advisor’s suggested answer, and asked for their final best estimate and rate of estimates. The final answers are more accurate than the initial guesses.
The most logical way of combining your view with that of the advisor is to give equal weight to each answer. However, participants do not do this (their final answers would be even more accurate if they did). Instead they put a 71% weight on their own answer.