What does big data mean for official statistics?

The question of how big data may change official statistics, and the research of federal or government agencies, is a fascinating one, partly because of the nature of modern data mining and collection systems.

In some ways, big data competes with other traditional methods of information gathering. For example, widely aggregated data from the internet can be a more accurate predictor than what is produced by a "community of experts" in a given industry. This means that big data may push different types of official statistics toward more accurate predictions or more efficient methodology.

Another way that the data is likely to change official statistics is that future research methods may integrate big data solutions in order to benefit from this component of a research effort. For example, the U.S. Census Bureau performs many physical audits of people, buildings and infrastructure within the U.S., basing its statistics on real-time observations and responses. Building big data into U.S. Census Bureau efforts might mean aggregating internet data about these same systems, and comparing it to what workers find out in the field. There are any number of ways to integrate big data and field research, which will determine the specific results for any kind of economic, social or industry-specific research project.

In short, big data will change official statistics into a more modern and sophisticated form of reporting, one where careful physical tabulation is enhanced by technical models and algorithms that work on the basis of averaging or projecting from huge volumes of mined or collected data. That’s just one reason that the vast loads of big data collected by governments and businesses are so valuable and carefully guarded.