The Promise and Peril of Big Data

Are leaders paying appropriate attention?

As the amount of data that surrounds our private and professional lives has grown from a trickle to a torrent over the last few decades, the ability to discover correlations through computation and analytics holds significant potential to deliver value.

Want to shave more pennies off the supply chain dollar? Simple. Just pool the data sets and work over the aggregated set to reveal its cost-saving secrets. Want to know what three items are most likely to indicate credit-card fraud? No problem. Just crunch the transaction data the right way, and—presto—Champagne, razors, and diapers are leading indicators of a stolen card. Want to find out how many teenagers there are in Durham who would like to go to a Twenty One Pilots concert? Facebook can tell you.

As the volume, velocity, and variety of data increases within and across the digital business ecosystem, the demand for increasingly high-powered computational crunching and increasingly sophisticated algorithmic alchemy is growing by leaps and bounds. Five out of six of the world’s largest corporations are working feverishly to apply artificial intelligence to help us make sense of the deluge of data we are drowning in today. The era of Big Data Analytics has arrived, and not a moment too soon!


So if demand for big data services is increasing based on sound evidence that it yields predictable and reliable efficiency results, what seems to be the problem? The problem is that—in its first incarnation, at least—big data analytics is focused primarily on the exploit side of the enterprise equation. Data is being crunched and analyzed to exploit a firm’s existing operations much more than it is being applied to explore new opportunities for growth.

With the ever-increasing pressure to deliver short-term results and the high likelihood that increased analysis of growing piles of data will reveal tangible opportunities to improve operational efficiency, leaders may be unconsciously lured into spending too much of their time attending to improving their current business.

Taking this pattern of attention to its logical extension reveals an undesirable outcome: The law of diminishing returns will set in, and everlarger data sets will have to be mined with ever-more sophisticated algorithms to yield increasingly incremental operational gains at the expense of missing a fundamental shift in the business where the opportunity to create new value is squandered or the threat of undermining the existing business goes undetected.

To successfully run an ambidextrous organization that requires one operating system to exploit the present and another one to explore the future, leaders must learn to not only effectively balance their attention between the present and the future, they also must recognize that big data analytics plays a critical dual role of crunching information to optimize productivity for one operating system and connecting people to create value for the other.

On the crunching side of the equation, leaders must recognize that mining the past for efficiencies can only be effective in slowing the inevitable decline of the current business. On the creating side of the equation, leaders must recognize that the same set of technologies being used to crunch data—if perceived and applied differently— could be leveraged to intuit new value horizons for the organization by enabling people to connect, communicate, coordinate, collaborate, and develop collective wisdom around what the future may hold.


In the era of Digital Darwinism, leaders have come to realize that the true sweet spot for organizations that want to survive and thrive in an increasingly complex and connected environment lies in constantly striving to find the right balance between exploiting what we know and exploring what we don’t. The first step in achieving this illusive state of equilibrium lies in avoiding the trap of mining the past at the expense of making the future. 

Tony O’Driscoll is global head of Strategic Leadership Solutions for Duke CE, where he focuses on identifying and implementing cutting-edge learning strategies and methodologies to get leadership ready for what’s next.