The Evolution of Big Data

Takeaway: The fact that so many industries would benefit from a solution to data issues says a lot about big data's future - and perhaps our own as well.

In the early 2000s, it was clear that there was a huge need for innovation in regards to data. Limitations on what firms could do with their data frustrated executives and decreased efficiency tremendously. Many companies stored massive amounts of information, but were simply unable to manage, analyze or manipulate it to their advantage. It is this growing pressure that gave way to the big data frontier.

In 2003, Google created MapReduce, a data application that allowed the firm to process and analyze information about its search queries across thousands of servers in a short span of time. Both scalable and adaptable, the program allowed Google to perform thousands of data tasks in mere minutes, which improved productivity and redefined perceived boundaries about what could be done with data. Almost 10 years later, big data has become a central tenet of information technology. Its far-reaching scope and ability has fundamentally changed data management in the workplace. But what has prompted this evolution and how exactly will big data impact the future? We thought you’d never ask. (For some background reading on big data, check out Big Data: How It's Captured, Crunched and Used to Make Business Decisions.)

Seeking Answers to Big Data Questions

The beauty of MapReduce was the way it simplified highly complex tasks. Communication could be managed across machines, system failures could be addressed and input data could be organized automatically, a process that could be overseen by individuals who no longer needed highly technical skills. By making data processing not only possible but approachable, Google inspired a cultural shift in data management. It wasn’t long before thousands of major firms were using MapReduce for their data.

But there was one problem: MapReduce was simply a programming model. While it facilitated the basics of data processing, it was not itself the answer to existing data shortcomings; it was only a much-needed step in the right direction. Corporations were still in need of a system that could address their unique data needs and go beyond the bare essentials of data management. In short, the technology needed to evolve.

Enter Hadoop

Enter Hadoop, an open-source framework software created by several programmers, including Doug Cutting. Where MapReduce was basic and broad, Hadoop provided a refreshing specificity. Companies could design their own tailor-made applications that addressed data needs in ways that no other software could, and it was generally compatible with other file systems. A firm with talented programmers could design a file system that would achieve unique tasks with data that seemed unreachable before. Possibly the best part about it was that developers would share applications and programs among each other that could be expounded upon and perfected.

By democratizing such an important resource, Hadoop became a trend. After all, that many large corporations, especially search engine firms, felt they had needed it for decades! It wasn't long before search engine giants such as Yahoo were announcing the implementation of large Hadoop applications that generated data used in Web search queries. In what seemed like a wave, several prominent companies announced the adoption of this technology for their massive databases, including Facebook, Amazon, Fox, Apple, eBay and FourSquare. Hadoop set the new standard for data processing.

Big Data, Big Problems

While advancements in data technology have reshaped the way companies treat data, many executives still find them unequipped for the full range of required tasks. In July 2012, Oracle released a survey of more than 300 C-level executives, who revealed that while 36 percent of companies rely on IT to manage and analyze data, 29 percent of them feel that their systems lack sufficient abilities to meet their companies' needs. Possibly the most striking finding of the study was that 93 percent of respondents believed that their firm was losing up to 14 percent of its revenue by not being able to use collected data. That's revenue that could be spent on making better products and hiring more workers. In a time where companies are struggling to stay profitable, improving data so that firms can become more profitable is a necessity. The survey indicates that despite those who believe that big data’s influence on commerce has already passed, the opportunities for growth and advancement it holds have yet to be fully realized.

What the Future Holds for Big Data

The good news is that Hadoop and MapReduce have inspired many other data management tools. Many new companies are creating extensive data platforms that run on Hadoop, but offer a wide array of analytic functions and easier system integration. It seems that corporations have invested a great deal of resources into addressing data concerns and the financial success of data firms has been proof of this. In 2010, data firms made an estimated $3.2 billion in retail sales. Many experts have estimated that this number will grow to a whopping $17 billion by the year 2015 alone. This is a fact that has not been lost on some of the largest technology companies. Both IBM and Oracle have spent billions over the past several months to acquire data firms. Many other firms will make similar moves in the coming years as they continue to vie for a competitive market share.

The Big Data Frontier

The amount of data that is collected continues to grow exponentially, which has some worried and others excited. The upside is that human beings will continue to become more productive and adaptive as we learn new things about our world through the analysis of data. The downside is that there is such a vast amount of data that many fear that we are incapable of properly storing it all, much less properly managing it so that it can be used by all who need it.

That said, advancements in big data can provide unprecedented opportunities for solutions to urgent issues concerning data. For example, experts have suggested that if big data were implemented properly with an emphasis on efficiency and quality, it would have the potential to save around $300 billion per year in health care expenditures alone; retailers would be able to improve their operating margins, the public sector could provide better services and large enterprises would save billions. And so, it seems that solving our data issues is not just needed in company boardrooms, but everywhere. Which says good things about big data's future - and perhaps ours as well.