Big Data: How It's Captured, Crunched and Used to Make Business Decisions

Takeaway: Finding ways to turn the flood of data into useful information for business decisions is a growing challenge to the IT profession and C-level executives.

As of 2012, a staggering 2.5 quintillion bytes of data are created daily; 90 percent of the data in the world today have been generated in the last two years alone. These data come from everywhere: sensors used to gather climate information, social media sites, digital pictures and videos, purchase transaction records, and cell phone GPS signals, to name just a few sources. Finding ways to turn the flood of data into useful information for business decisions is a growing challenge to the IT profession and C-level executives. That's where one of today's top tech buzzwords comes in: big data. And it isn't getting buzz for nothing. Big data has the power to change business in a big way. Here we'll take a look at how it works.

What Is Big Data?

The term "big data" describes data sets that are growing exponentially and that are too large, raw, and unstructured for analysis using traditional database technology and techniques. Whether terabytes or petabytes, the precise amount of data is less the issue than how that data is used.

There are three dimensions to big data: volume, velocity and variety. Companies are awash in the amount of data, data are being created and processed at ever greater rates and the types of data, such as social media and context-aware mobile devices, are proliferating.

So how is any of this information useful? In fact, there are a number of ways that big data can create value for an organization. First, big data can unlock significant value by making information transparent and usable at much higher frequencies. Second, as organizations create and store more transactional data in digital form, they can collect detailed performance data on everything from product inventories to sick days. This is how companies are using data collection and analysis to conduct controlled experiments and make better management decisions. Others are using data for basic forecasting to high-frequency nowcasting to adjust their business levers just in time.

In addition, big data allows narrower segmentation of customers and more precisely tailored products or services. These sophisticated analytics can substantially improve decision making. What's more, big data can also be used to improve the development of the next generation of products and services. For instance, manufacturers are using data obtained from sensors embedded in products to create unique service offerings. (Just how to sort out all this data is a profession in an of itself. Read more in Data Scientists: The New Rock Stars of the Tech World.)

Capturing and Crunching Big Data

To capture and crunch big data, companies have to deploy new storage, computing and analytic technologies and techniques. The range of technology challenges and the priorities for tackling them will differ depending on the data maturity of the firm. However, legacy systems and incompatible standards and formats can prevent the integration of data and impede the more sophisticated analytics that create value. This means that big data also requires big technology.

Several new and enhanced data management and data analysis approaches assist with the effective management of big data and the creation of analytics from that data. The actual approach used will depend on the volume of data, the variety of data, the complexity of the analytical processing workloads involved, and the responsiveness required by the business. It will also depend on the capabilities provided by vendors for managing, administering and governing the big data environment. These capabilities are important selection criteria for product evaluation.

Big data technologies include open source database management systems designed to handle huge amounts of data, including Cassandra and Hadoop, as well as business intelligence software designed to report, analyze and present data.

Making Use of Big Data for Business Decisions

Forrester Research estimates that organizations effectively use only 5 percent of their available information. That leaves a lot of room for optimization and improvement, which is why making use of large digital datasets for business decisions requires the assembly of a technology stack that consists of everything from storage and computing to analytical and visualization software applications. The specific technology requirements and priorities will vary based on the big data levers that are to be implemented and an institution's data maturity.

So is it worth the trouble? In a word, yes. The business benefits of using big data are clear. For example, the McKinsey Global Institute estimates that a retailer using big data effectively could increase its operating margin by more than 60 percent. When it comes to ROI, it just doesn't get much better than that.

To benefit from big data, McKinsey recommends that business leaders take the following steps:

1. Inventory all data assets
2. Identify value creation opportunities and risks
3. Build up internal capabilities to create a data-driven organization
4. Develop an enterprise information strategy to implement technology
5. Address data policy issues, such as privacy, security and intellectual property

Data policy issues are of particular concern when it comes to big data. Large databases often contain highly sensitive information, such as company secrets or data that must be protected by law. Plus, there is often a trade-off between the availability and confidentiality of data. If an organization wants data to be available and useful, there is often less security surrounding that data as a result. To process big data for real-time decision-making, centralization of the data is crucial. But as centralizing increases, the ability to sequester and secure confidential data declines.

In addition, the size of the data set can make implementing security and privacy controls unwieldy. Encrypting all those data for security reasons would be a time-consuming and expensive undertaking and would slow data processing, thus impeding rapid decision-making.

The key to dealing with the privacy and security challenges of big data is the first step identified above: inventory all data assets. Once the organization understands where the big data resides and what kind of data there are, it can take steps, such as investing in security technology capable of handling big data volumes, to secure its confidential information.

Bigger Data on the Way

So what’s next? Well, one thing is certain: Big data is here to stay.

But big data is about more than size; it's about opportunity. In this case, it's an opportunity to find insights in new and emerging types of data and content, to make business more agile, and to answer questions that were previously considered beyond reach.

The key to benefiting from it, then, is to capture and crunch it, and use it effectively to make smart business decisions. Easier said than done, but so far the results are proving well worth big efforts.