How do firms collect data "at the edge" of a network?

Businesses may use “edge data collection” methods to capture different kinds of information for business intelligence or other operational use. Collecting data at the edge of a network involves getting data in a decentralized point in the network architecture, for example, from small individual nodes without proximity to a central data warehouse.

In the early days of big data, a consistent philosophy emerged: that best practices, in most cases, involved routing data to a central data warehouse, where it would be stored, retrieved, analyzed and sculpted. This has remained a dominant model until recently, when “edge” data collection started to arise as a practical alternative.

To collect data near the edge of a network, businesses look far afield from the data warehouse and consider how to gather and analyze data “near the end user.” An excellent example is in internet of things (IoT) systems, where it may not be practical to funnel a lot of device or sensor data into the data warehouse. When it makes more sense to collect the data and observe it directly from the mass of disparate IoT devices, businesses collect data at the edge of the network, rather than in the center. In general, companies do this using specific middleware to get the data from individual small network nodes.

By collecting data at the edge, companies can do more analysis without the cost and effort of maintaining all of the data in archives. In other cases, the data collected at the edge does travel to the center: database professionals may take advantage of the journey to normalize and otherwise refine the data in question. One big reason for data collection at the edge is that the data may not be “clean” in its initial capture – doing more at the edge means the companies can determine exactly which data goes into the center, and how it gets internalized. The company can use protocols based on the condition of the data, taking in only clean data, or again, working on the data as it progresses from the edge to the center.

The pursuit of “edge analytics” is gaining ground in IoT architectures and other types of enterprise systems. Because companies can “thin” data or otherwise cull data results, edge data collection and analytics can help with issues such as network congestion and latency.