Why Big Data Isn't Enough

This is an MIT Sloan Management Review Article. As "big data" becomes increasingly integrated into many aspects of our lives, we are hearing more calls for revolutionary changes in how researchers work. To save time in understanding the behavior of complex systems or in predicting outcomes, some analysts say it should now be possible to let the data "tell the story" rather than having to develop a hypothesis and go through painstaking steps to prove it. The success of companies such as Google Inc. and Facebook Inc., which have transformed the advertising and social media worlds by applying data mining and mathematics, has led many to believe that traditional methodologies based on models and theories may no longer be necessary. Among young professionals (and many MBA students), there is almost a blind faith that sophisticated algorithms can be used to explore huge databases and find interesting relationships independent of any theories or prior beliefs. The assumption is: The bigger the data, the more powerful the findings. As appealing as this viewpoint may be, authors Sen Chai and Willy Shih think it's misguided - and potentially risky for businesses that involve scientific research or technological innovation. For example, the data might appear to support a new drug design or a new scientific approach when there isn't actually a causal relationship. Although the authors acknowledge that data mining has enabled tremendous advances in business intelligence and in the understanding of consumer behavior - think of how Amazon.com Inc. figures out what you might want to buy, or how content recommendation engines such as those used by Netflix Inc. work - applying this approach to technical disciplines, they argue, is different. The authors studied several fields where massive amounts of data are available and collected:

• Big data