Returns to IT Investments: Noisy Signals

The previous post described the large and steady increases in the IT intensity of the US economy over the past twenty years.  So what have companies received from all the money they’ve spent on computers, software, and networks?

For a long time it appeared that in the most obvious area —  productivity growth —  companies were getting nothing from all their IT.  As the Nobel Prize-winning economist Robert Solow observed in 1987, "We see evidence of the computer age everywhere except in the productivity statistics."

IT vendors and CIOs everywhere started to breathe a bit easier in the mid 1990s when a steady stream of rigorous research found clear, strong, and positive links between companies’ IT investments and their subsequent productivity growth rates.  Erik Brynjolfsson and Lorin Hitt’s 1996 paper "Paradox Lost?  Firm-level Evidence on the Returns to Information Systems Spending"1 was one of the first in this stream.  In it, the authors explained why earlier research had not found a strong IT-productivity relationship:

"… we attribute the different results to fact that our data set that is more current and larger than others explored.  We conclude that the productivity paradox disappeared by 1991, at least for our sample of firms."

Later work found strong and positive links between IT and other critical outcomes like market capitalization .  In recent years the weight of evidence for IT’s positive aggregate impact has become so strong that some formerly skeptical scholars like Northwestern’s Robert Gordon have changed their views. 

This research, however, also revealed a troubling aspect of the returns to IT investments:  on average they were positive, but in aggregate they were all over the map.  The graph below, reproduced from Brynjolfsson and Hitt’s 2000 paper "Beyond Computation: Information Technology, Organizational Transformation and Business Practices"2 shows the relationship between IT stock and productivity for a large sample of US companies; each company is a separate data point.

Of course, there’s no such thing as a guaranteed return to any investment except T-bills so we should expect to see these data points to be somewhat spread out, but this much (keep in mind that the axes on the graph above are logarithmic, not linear, so they make the spread appear much smaller than it actually is.)?  My old engineering professors at MIT would have said that the IT investments shown the figure above are definitely generating a signal (the positive average return) but that there’s also a lot of noise (the large spread of the data points).

The same pattern holds when researchers have looked at performance outcomes other than productivity growth:  the average return is positive, but the variation is so great that the positive average shouldn’t provide much confidence to business leaders.  Furthermore, IT generates noisy signals not only when we look at company performance, but also when we examine a few other important areas:

The organization of work within companies.  In a large company-level study, Brynjolfsson, Hitt, and Stanford’s Tim Bresnahan found that IT investment levels were positively associated with workplace attributes like team-based work organization and individual autonomy.3  Their findings seem to support Tom Malone‘s predictions in The Future of Work —  that decisions will be pushed downward and centralization will decrease.  However, as Daniel Robey and Marie-Claude Boudreau wrote in a review of research on the organizational impact of IT:

"… over many years researchers have discovered information technologies to be associated with seemingly polarized pairs of social outcomes:  empowered employees… and oppressed employees…  extended hierarchy… and reduced hierarchy… organizational rigidity… and organizational flexibility… increases in staff and radical downsizing."4

The organization of work across companies.  In 1987, Malone and his colleagues Joanne Yates and Robert Benjamin made a very clear prediction about the broad impact of IT:  "information technology will lead to an overall shift toward proportionally more use of markets —  rather than hierarchies —  to coordinate economic activity"5  Their argument was that since markets generally have higher coordination costs that hierarchies, and since IT lowers coordination costs, than all other things being equal more IT should lead to more use of markets.

This ‘Electronic Markets Hypothesis’ (EMH) has been widely accepted, and it looks like there’s ample evidence to support it.  A lot of activities and processes that used to be done internally are now being outsourced, both domestically and internationally, with the result that companies are being ‘hollowed out’ to some extent.  Malone, Brynjolffson, and their colleagues found a link between IT and reduced company size in a 1994 study , and Wharton’s Ravi Aron has clearly documented the key roles that IT plays in facilitating outsourcing and offshoring.  As he shows in a recent HBR article, not only does technology reduce communication costs so much that it’s essentially free to send knowledge work all around the globe, but IT also ensures that work can be codified, measured, and monitored.  These capabilities are essential; they give outsourcers confidence that their work can be performed halfway around the world without sacrificing control and oversight. 

But are outsourcing arrangements really examples of electronic markets?  To many, they look like hierarchies split across two firms rather than stereotypical markets — forums in which many parties come and go at will and use the price mechanism to continually match supply and demand.  By this definition, eBay is clearly an electronic market, and a spectacularly successful and important one. 

But in the B2B world, how many eBays are there besides, well, eBay?   The dot-com era saw the birth of hundreds, if not thousands, of B2B exchanges and industrial emarketplaces.  Investors greeted them with enthusiasm (publicly traded emarketplaces had a combined market capitalization of $100 billion at one point), as did many analysts, journalists, and academics.  The only ones who didn’t jump on the emarketplace bandwagon, it turned out, were the industrial buyers and sellers who were supposed to use them and provide their revenue.  Companies stayed away from B2B exchanges in droves, and the overwhelming majority of them have closed down (remember Chemdex, which became Ventro?) or changed into vendors of procurement software (like Covisint and Sciquest).  Only a few scrappy survivors, like Alibris in the book industry and Shipserv in shipping, survive as industry-specific neutral emarketplaces.

A later post will dive into the reasons underlying the emarketplace meltdown and revisit the EMH.  For now, suffice it to say that lots of work is still being carried out within hierarchies instead of across markets, and I believe this is the case not in spite of IT, but in part because of IT.  I have, in other words, an ‘Electronic Hierarchies Hypothesis.’ 

IT adoption efforts.   It’s relatively easy to get some types of IT up and running.  Projects to  adopt email, Office, CAD systems, instant messaging, and calendaring have their headaches, but they rarely fail utterly, cause organizational chaos and infighting, or bring down entire companies.  Efforts to adopt technologies like ERP, SCM, and CRM, however, generate very noisy signals.  Sometimes they go smoothly and deliver powerful new capabilities (as was the case at Otis Elevator, Cisco, and CVS), and sometimes they drive companies out of business (like Foxmeyer Drug) or handicap them severely (as at Nike and Hershey’s).  Studies have found failure rates of 30-75% in efforts to adopt enterprise systems, depending on the technology investigated and the definition of failure. 

ERP is clearly more technically challenging than email, but virtually everyone who’s studied the noisy signal of IT adoption efforts (myself included) has concluded that technical factors don’t explain much of the difference between successes and failures.  As a study by Joe McDonagh concluded, "economic and technical considerations are unlikely to feature prominently when IT fails to deliver."6  In a later post, I’ll provide a non-technical explanation for why some IT projects are so much nastier and failure-prone than others.

Competitive Advantage:  Finally, IT’s signal appears to be noisiest in exactly the area where business leaders most want a clear answer:  competitive impact.  IT academics, myself included, have written many case studies about specific technologies or applications that appeared to give their adopters a real leg up on the competition.  In some cases, however, a return to the scene of the case years later revealed that the former differentiator had turned into just another legacy system.  One study, conducted by William Kettinger, Varun Grover, and Albert Segars, found that of 30 companies adopting systems widely identified as ‘strategic,’ only 7 had higher market share and profits in two later time periods.7

In other cases the technology was clearly competitively valuable —  so valuable, in fact, that all competitors also acquired it as quickly as possible.  Wharton’s Eric Clemons labels IT like this a ‘strategic necessity’ and points to bank ATMs as a prime example.8

Nick Carr looked easy availability of IT and its steadily falling prices, and concluded that "IT Doesn’t Matter" for competition —  that it’s clearly a strategic necessity to have some technology, but rarely if ever a good idea to pursue competitive advantage via IT.  The smart strategy, he argued, was to get in place a cheap, stable, and secure infrastructure for IT (just as for electricity, dial tone service, and other utilities) and then stop thinking about it.  In his words, "IT management should, frankly, become boring." 

At a time when large US corporations are spending close to 5% of revenue each year on IT, it’s vital to get some clarity on the noisy signal of information technology.  My work, and this blog, are aimed at doing just that.  The next post will start to lay out a framework business leaders can use to think through the impact of IT on their companies, to help understand why IT signals are so noisy, and to identify where and how they can intervene to obtain maximum benefit and advantage.


1Brynjolfsson, E. and L. Hitt (1996). "Paradox Lost? Firm-level Evidence on the Returns to Information Systems Spending." Management Science 42(4): 541-558.

 2Brynjolfsson, E. and L. Hitt (2000). "Beyond Computation: Information Technology, Organizational Transformation and Business Practices." Journal of Economic Perspectives 14(4): 23-48.

3Bresnahan, T. F., E. Brynjolfsson, et al. (2002). "Information technology, workplace organization, and the demand for skilled labor: firm-level evidence." The Quarterly Journal of Economics CXVII(1): 339-376.

4Robey, D. and M.-C. Boudreau (1999). "Accounting for the Contradictory Organizational Consequences of Information Technology: Theoretical Directions and Methodological Implications." Information Systems Research 10(2): 167-185.

5Malone, T. W., J. Yates, et al. (1987). "Electronic Markets and Electronic Hierarchies." Communications of the ACM 30(6): 484-497.

6McDonagh, J. (2001). "Not for the faint hearted: social and organizational challenges in IT-enabled change." Organization Development Journal 19(1): 11.

7Kettinger, W. J., V. Grover, et al. (1994). "Strategic information systems revisited: A study in sustainability and performance." MIS Quarterly 18(1): 31-58.

8Clemons, E. K. and M. C. Row (1991). "Sustaining IT Advantage: The Role of Structural Differences." MIS Quarterly 15(3): 275-292.