Big Data Silos: What They Are and How to Deal With Them
Big data silos can be hard to figure out - but new processes are enabling free flow of data.
The age of big data has brought us a prodigious number of new ways to present information and solve problems. In the past few decades, we’ve seen traditional mainframe computing blossom into a scenario where our IT systems can aggregate enormous mountains of data and use it to provide key insights for business or government goals as well as other purposes.
However, there have been challenges and obstacles along the way, and one of the biggest ones is the big data silo.
If you’re confused about what a big data silo is, think of a physical silo: a metal tower full of resources that is securely locked against tampering or unauthorized access.
That’s sort of what a silo is in the tech world, although it’s a little different.
What Is a Big Data Silo?
In the digital world, the free flow of data is critically important. The data is the “lifeblood” of the system – the raison d’etre for any architecture. The data is “served” to users in infinite new and compelling ways – but it all rests on inter-connectedness.
When this free flow of data is impeded for any reason, professionals talk about that as a “data silo.”
Again, the data is being locked away from general use or generalized access. It exists somewhere, but it can’t get to where it can be useful.
Going back to the previous physical silo analogy, most farm silos hold corn or other products for animal or human feed. But nobody’s eating that stuff unless you can move it from the silo out into the world – and that same principle is inherent in the big data silo problem. The solution is to be able to move the data outside of an isolated area of the software infrastructure, so that it can be used for its intended purpose. (For more on data silos, see Breaking Silos: How to Consolidate, Cleanse and Use Your Data for Good.)
The Structural Silo
Just a few years ago, many of us thought of most big data silos as structural in nature.
The idea was that programming and design just weren’t sufficient to allow that free flow of data. Enterprise systems had pieces of data “stuck” in the corners of a greater architecture, and needed middleware and specific solutions to pull it out and get it where it needed to go.
The IT world has made great progress with structural data silos recently, but we've also seen that structural silos are not the only problem.
Take a look at this Harvard Business Review article from December 2016 called “Breaking Down Data Silos.” Writer Edd Wilder-James separates the main types of data silos into four categories: structural, political, growth and vendor lock-in. This is a great way to start talking about how silos happen, and what can be done about them.
Political Data Silos
How can data silos be political, and what does that mean for the tech industry?
We asked Hogne Titlestad, co-founder and Chief Technology Architect at Friend, which is a Norwegian-born internet OS billing itself as “a much-needed alternative to Big Tech silos.”
“One of the main challenges facing computing today is the growing number of silos being created by the ‘Big Tech’ corporations,” Titlestad says. “From these large players, we have seen politics entering into software projects, meaning that a lot of the chaos seen elsewhere in the world has now begun to influence the technology space.”
Titlestad suggests projects will see more of this activity in 2019, including what he calls “charge from emerging software technologies and collaborative platforms which will challenge big tech oligopolies by offering functionality and user experiences equal to the best among them.”
Open source, he says, plays a major role in a more communal takeover of IT resources and the solution to political data silos – opening up data interfaces.
“As the internet generations mature, individuals must begin to take advantage of the opportunities to operate on a global scale, bringing skillful people together,” Titlestad says. “With such a great back-catalogue of fantastic open source projects available on Github, Gitlab, Sourceforge and others, new disruptive applications are certain to emerge in locations around the world – building on top of the foundation of decades passed … One of the most important things that individuals can do is continue to voice their disdain towards big tech and its exploitation of personal data in the pursuit of profit.”
Vendor Lock-In Data Silos
In some ways, vendor lock-in can also go along with the category of “political” data silos.
“Software vendors are among the first to know that access to data is power, and their strategies can frustrate the desire of users to export the data [contained] in applications,” Wilders-James writes. “This is particularly dangerous with software-as-a-service applications, where the vendor wants to keep you within their cloud platform. Vendors have also worked hard to create entire job functions and career paths centered around their software. Any hint of move from that world could threaten the livelihood of a trained and certified software professional.” (Some avoid vendor lock-in by employing multi-could. Learn more in 10 Myths About Multi-Cloud Data Management.)
Growth Data Silos
In addition to these types of obstacles, you also have what Wilders-James and others call the “growth” silo.
“Executive alignment is key to consolidating data silos,” Anjul Bhambhri, VP of Platform Engineering at Adobe, told Techopedia Jan. 12, expounding on some related issues with business planning. “The best way to approach data silo consolidation is to first achieve executive alignment on business outcomes. Tackling the steps of data consolidation before agreeing to business outcomes is putting the cart before the horse. If there isn’t alignment on business goals and outcomes, there’s a good chance of missing the mark on creating that holistic view of the customer and not yielding a ROI for technology investments.”
Dealing with Data Silos
By looking at these above categories of data silos, you can see how to approach each one practically. Companies can plan better for growth and scalability, make sure vendors aren’t holding their data hostage, open up interfaces from an oligarchical tech standpoint, and last but certainly not least, design their systems so that data doesn’t get stuck away in a corner, or an attic or basement. Those are the marching orders that many engineers and design teams have when they try to work toward the future and prevent data silos from impeding business functionality.