Are data silos still a problem in 2022? You bet.
But the IT landscape is constantly evolving; the ways silos are perceived and dealt with in the tech world change constantly. (Also read: Breaking Silos: How to Consolidate, Cleanse and Use Your Data for Good.)
That prompts the question: Will data silos ever completely disappear?
This article provides background on data silos and explores emerging solutions that could help you banish them within your organization.
What is a Data Silo?
A data silo is when valuable business data gets stuck somewhere in a network or system where it's less useful than it could otherwise be.
The opposite of a data silo, then, is a system where data always flows freely to wherever it can be best used.
It's not hard to imagine the kinds of scenarios that create data silos in big and complex networks. Vendor lock-in is one common example, but any scenario in which data can't get where it needs to go can create a data silo.
Why Are Data Silos Problematic?
The free flow of information is vital to enterprise systems; if information can't be shared, it may not be as useful.
For example, if a siloed database contained information on customer purchase history and product information, a company's wider network would not be able to leverage that data to inform its strategic direction. (Also read: Destroying Silos With Integrated Data Analytics Platforms.)
In addition to burdening storage capacity, data silos are a problem because they decrease the efficacy of compiled information. The value of data is in its use cases: Data silos prevent that data from moving to where it can do the most good for a business.
Why Are Data Silos So Hard to Get Rid Of?
There are two central reasons why companies often fail to eliminate data silos, according to an article by Krishna Subramanian:
- Computing costs.
- Storage costs.
Both reasons come from the same place: As companies grow and acquire new kinds of data, their data sets get bigger and more complicated — but their data delivery systems don't always grow at the same rate. That means more and more data is put into cold storage to be used "eventually" — only "eventually" doesn't always come.
This costs both computing power and money to maintain storage capacity.
Moreover, data silos can be difficult to get rid of because the longer they're left unmanaged, the bigger they grow. And the bigger they grow, the more perplexing and expensive they can become to the teams assigned to handle them.
In short, data silos can be tough to eliminate — but it's important to try lest they put a damper on an enterprise network's overall success.
Data Silo Solutions
1. Data Lakes
A data lake is formed when companies deposit all their structured and unstructured data into a centralized depository, where it remains for easy retrieval.
However, like a real lake, the data in a data lake is pretty amorphous. If you needed to pull a specific kind of fish out of a physical lake, you’d have to put some work into figuring out where that fish was. The same, in many cases, holds true with data lakes.
To put it differently, Subramanian has written about how a data lake can end up being a “dumping ground” for data without good governance. (Also read: Data Governance Is Everyone's Business.)
2. ETL and ELT
There are two relatively new data processes that compete for business use when it comes to dealing with data silos:
- ETL, which stands for "extract, transform, load."
- ELT, which stands for "extract, load, transform."
In each of these processes, businesses extract data from a legacy system, load it onto a new system and transform it. The only thing differentiating the two processes is whether the business transforms the data before or after the loading process; in ETL, it happens before, and in ELT, it happens after.
Why the distinction? Some analysts have pointed out the value of transformation after loading: Transformed data might take more resources to load –- so you can save that work by loading first and transforming at the end. However, that leaves the transformation work to be done after loading, which isn't convenient for some businesses depending on their staffing and configurations.
In recent cloud-based ETL, the process can be used to collect data from disparate systems, or to port on-premises data into specified cloud environments. Cloud-based data storage warehouses can improve security, efficiency and capacity overall, and many companies are using big vendor services like AWS and Azure to build these types of platforms.
The API, or application programming interface, has received much attention as a connective tissue for digital systems.
In theory, APIs can solve a lot of data silo problems — but only to the extent that engineers and other stakeholders can easily connect APIs as a pipeline. Again, though, this is easier said than done. (Also read: Open API: The Future of Application Programming Interfaces.)
4. AI and ML and Data Integration
When it comes to dealing with data silos, new artificial intelligence (AI) and machine learning (ML) engines are also coming forward on the horizon.
AI and ML have made significant advances in recent years and are now able to classify and move data at levels never before seen. Consequently, AI and ML may represent the future of breaking down data silos.
By using the insights and intelligence that AI and ML produce, companies can build better plans for integrating data across a distributed network. One way to think about it is as a "smarter aggregation system," where the AI and ML applications are the catalysts, and the techniques (like ETL/ELT) are the mechanism for achieving these goals.
Some experts talk about creating a "sharing culture" for data. Others talk specifically about different data governance methods; and others still mention how you can use vendor services like AWS S3, in which data is stored in object buckets for retrieval using metadata to identify what's in there.
But on top of these general tips, one thing remains clear: Data silos, in one form or another, are likely to remain in our corporate networks. As such, developing new tools and strategies to deal with them is paramount.