As I was getting ready for a panel discussion on big data a little while ago I tried to think of a concise (i.e. panel-friendly) way to explain why I thought the current data explosion was such a big deal for business world. I realized that were were two separate-but-related points I was trying to make.
The first is, as a statistician will tell you, that more data is better. All other things being equal, you’d rather have 100 people respond to your poll than 50, and 1000 than 100. And even when all other things aren’t equal, you’d still often rather have a lot of ragged data than a small number of high-quality observations. The statistical toolkit is full of ways to extract the signal from the noise, even with pretty messy data.
The second point is that, as I’ve written a bunch of times before (see here and here, for example) data-driven decisions, predictions, and diagnoses are much better than those that come from human intuition and HiPPOs (the ‘Highest-Paid Person’s Opinions’). The research is overwhelming on this point. HiPPOs might be good for some things, but their crystal balls just don’t work very well. The soulless output of a data-driven, mechanistic algorithm is demonstrably and significantly better, in domain after domain. So HiPPOs should become an endangered species.
Above is a cute graphic I’ve drawn to help make these points. It shows that currently many companies make (too) many of their important decisions with small volumes of data and great reliance on HiPPOs. I think that the twin forces of technological progress and competition will cause successful companies to move up and to the right in the future.
Do you agree? If not, why not? Leave a comment, please, and let us know.