Technology Beats a Full House

How big a deal is IT? Is it really a ‘game changer’ for business? And if it is, does it change the game in a way that leaves the players looking more similar, or more differentiated from each other?

Our recent Harvard Business Review article makes the case that IT has in fact changed the game of business in a way that increases the gaps between winners and losers. As I wrote here earlier, though, it’s very important to keep testing this hypothesis. I’ve believed for a while that IT is a game changer for competition, which makes it particularly important for me to find good objective tests —  ones that help keep me from falling in love with my own arguments and theories. Such tests help me avoid what a sharp person called “the triumph of ideology over evidence” (whose phrase is this, does anyone know?).

So let me tell the story about my favorite of the tests we did as we were conducting the research that led to the article.

Of the four of us on the team, Erik Brynjolfsson, Michael Sorell, and I are pretty big baseball fans. Our fourth, Feng Zhu, used to listen politely as we’d start off most of our meetings talking about recent games and giving each other grief. Erik and I are Red Sox fans, while Michael roots for the Yankees. We forgive him this because he’s a great colleague.

Early on, we were gathered around the white board thinking through the question “How would we test if IT mattered for competition or not?” Erik mentioned Steven Jay Gould‘s Full House as a possible source of ideas, and I got my copy from my office bookshelf.

Full House is a true geek’s book. It combines paleontology, evolution, and baseball statistics to advance an elegant argument: that we humans have a counterproductive tendency to focus on averages and trends over time, rather than on variation around the average. For Gould, variation is where the action is.

He brings this argument to life by resolving (to my satisfaction, anyway) the eternal debate in baseball about why there aren’t any more .400 hitters. Ted Williams of the Boston Red Sox hit .406 in 1941, and no one has been over .400 for an entire season since (take that, DiMaggio worshippers). This is odd, because .400 hitters, while never common, were not unheard of prior to that time. There were seven of them in the 1920s, for example.

Explanations for the extinction of the .400 hitter include the advent of night baseball, the ‘invention’ of the specialist late-inning relief pitcher, the dilution of talent brought on by expansion, greater emphasis on hitting for power over hitting for average, etc. etc. The search for the true explanation has taken up countless column inches in print and hours in bars.

Gould cut through all these arguments by constructing a couple graphs and explaining their implications. First, he graphed the average batting average across all major league players every year since the late 19th century. It’s an apples-to-apples comparison over this long period because baseball has been played by an almost completely consistent set of rules for the whole time. Games have always been nine innings long, each half inning has always consisted of three outs, bunting foul with two strikes has always been an out, etc. etc.

Here’s a reproduction of Gould’s graph of average batting average:

Gould wrote in Full House that this graph has all the messy variation we’d expect to see in real-world data. It’s also the graph most of us would draw to understand why there are no more .400 hitters. But it doesn’t shed much light on the question. Looking at it, it’s hard to say that hitters have been getting much better or worse, on average, since 1941, or indeed over the entire history of baseball.

Gould’s great insight was to construct another graph that charted not the average batting average, but instead the amount of variation or spread around this average. He calculated one common measure of spread, called the ‘standard deviation,’ in batting average for all players for all years, graphed the result, and was astonished at what he saw. In sharp contrast to the erratic pattern in the graph of averages shown above, the graph of standard deviations  revealed a single stately trend: a decline over time in variation around the average (whatever the average happened to be):

He wrote:

“… I never dreamed that the decline of variation would be so regular… the decline of standard deviations for batting averages is so regular that the pattern [in the graph] looks like a law of nature…  I can assure you that this pattern represents regularity with a vengeance.”

He seized on the two facts of baseball’s constant rules and the constantly shrinking spread in performance to generate a lovely hypothesis:

“Complex systems improve when the best performers play by the same rules over extended periods of time.  As systems improve, they equilibrate and variation decreases.”

This variation decrease explains why .400 hitters haven’t appeared recently. Early in baseball history a .400 hitter wasn’t that far away from the average performance, when ‘far away’ is measured in terms of the standard deviation of that time. Over time, though, that standard deviation shrank —  in other words, players’ batting averages tended to cluster more tightly around the average. And so hitting .400 meant being even farther away from average, again when ‘far away’ is expressed in terms of the number of standard deviations (which is the smart way to do such measurements).

If this theory is correct, then we should be pessimistic about seeing another .400 hitter any time soon. The standard deviation will only continue to shrink as baseball continues to be played by consistent rules, which means that a player will have to be even more of a freak of nature than Ted Williams was (in other words, even more standard deviations away from the mean than he was) in order to reach this benchmark.

What on Earth does all this have to do with IT’s impact on competition? After talking and drawing on the white board for a while we realized that Gould’s hypothesis above, which we came to call the “Full House hypothesis”, yielded a great test of our beliefs about the importance of IT.

If the combination of the Web and commercial enterprise IT really was a ‘game changer’ for the competitive game of business, then the introduction of these technologies should be accompanied not by a decrease in variation in performance, but instead by the opposite — an increase in performance spread among competitors. It would be as if the rules of baseball suddenly changed hugely: if players had to hop on one leg the whole time, or play with a frisbee instead of a ball, or have a hot dog eating contest before each inning. Rule changes this big would upset the game’s equilibrium, and increase the variation in performance among hitters.

If the Web and enterprise IT similarly upset the existing equilibrium of the game of business, they would also lead to greater spread in performance. This widening of the spread would start in the mid 1990s, when both technologies became available to companies. And the widening would be biggest in industries that spent the most on IT, since they’d be the ones that had their games changed most profoundly.

It’s not too hard to determine if this has in fact been going on. Publicly traded companies publish ratios that indicate how well they’re performing. These ratios can be thought of as batting averages for the company, and so work well for testing the Full House hypothesis. They include profit margin and gross profit margin, EBITDA margin, return on assets, market capitalization per dollar of revenue and per dollar of profit, and Tobin’s Q (the ratio of a firm’s market value to the value of its assets).

We calculated these and other company ‘batting averages’ for all publicly traded companies going as far back as we could, and used data from the Bureau of Economic Analysis to divide all companies into 61 industries based on how much they spent on IT, expressed as a percentage of all the fixed assets they spent money on (details of this work are given in the academic version of our article, which is available here.).

We programmed our statistical software (OK, Michael and Feng did) to test the two hypotheses that 1) spread in company ratios within industries started to increase in the mid 1990s and that 2) this increase was bigger in industries that spent more on IT. We felt that this was a pretty stern test, inspired by Gould’s work, of IT’s impact on the game of business.

Variation in baseball batting averages did nothing but decline over time even in the face of changes like the start of the live ball era in 1920, the lowering of the pitcher’s mound in 1969, and the introduction of the designated hitter in the American League in 1973. None of these game changes was big enough to reverse the decrease in variation in the complex system of baseball. Were the novel corporate technologies of the mid 1990s a big enough deal to increase variation in the complex game of business?

It looks like they were. For virtually all the ratios we considered, variation in high-IT industries started to increase in the mid 1990s and stayed high after that, with some exceptions during the post-2000 economic slump.  The less IT an industry had, the less pronounced this trend was. A couple graphs show these patterns clearly (in these graphs we use the more conservative intraquartile range, rather than standard deviation, as the measure of variation / spread). Here are graphs showing  how the spread changed over time in high-IT industries vs. low-IT industries: