Data Crisis on the Horizon – Why We Need to Rethink Data Storage

Data has always played an important role in modern industry, and that role is becoming more prominent with each passing year.

Recently, the rise of ‘big data’ has had a huge impact on industry. Revenues for software and services are predicted to grow to $103bn in 2027, a compound annual growth rate of 11.4%. Data is being constantly churned out everywhere, from our social media accounts to the equipment we use to drill for oil, and it is extraordinarily valuable.

That means one of the biggest issues facing tech in the coming years will be how to store all this data, in a way that’s not only secure and affordable, but also easily accessible. As the amount of available data grows we need to make sure that the storage systems we have in place can handle it, and won’t collapse under the strain.

At the moment, that’s proving tough. The current storage options are already struggling to handle the exploding quantities of data, and as time goes on this will only get harder. We need to re-think the way we view storage and come up with new solutions in a changing world.

Before we explore some solutions to this, let’s first take a closer look at the problems with current centralized storage methods.

Why the Current Approach is Broken

The current methods for data storage are mainly based around large, centralized data centers. Even cloud storage relies heavily on these massive central databases, and there are all kinds of problems with this approach. These include:

• They’re bloated and oversized. This makes them expensive and time-consuming to maintain, not to mention devastating for the environment.
• They’re vulnerable to hacks and cybercrime because they have a central point of failure that can be targeted to bring the whole thing down. On top of this, any centrally controlled system is at risk of corruption from within.
• They’re tough to scale. The amount of data out there is growing exponentially, and these centralized databases will soon find it difficult to operate with so much demand.

It’s not an ideal situation. But what’s the solution? Many believe that blockchain, with its decentralized approach to storage, could be the answer.

Blockchain’s Solution

Blockchain technology made its name as the ledger underpinning Bitcoin and other cryptocurrencies. But the tech works with all kinds of data, and it differs crucially from existing methods by the lack of a central point. This gives it a bunch of advantages:

• It’s secure. The data is encrypted and distributed across multiple nodes, forming a decentralized network with no central point that makes it hard for attackers to do damage.
• Since there’s no central party in charge, it’s very difficult to corrupt a blockchain. This can only be done in some networks by seizing control of a majority of nodes, which is almost impossible. It’s truly democratic and highly resistant to fraud.
• It’s immutable. This means data can’t be manipulated or handled, in other words, users can trust the data and the integrity behind it.
• It’s fast. By drawing on a vast network of nodes instead of central servers, blockchains have the potential to be much quicker than traditional storage methods, making access seamless.

Blockchain seems like a clear winner in lots of areas, and it is. But up to now, the technology has struggled when it comes to large-scale data storage.

Let’s take a look at why that is, and how it could soon change.

Blockchain’s Problems — And how to Solve Them

Traditionally, as blockchains have grown to include more and more data, they’ve struggled to scale. Bitcoin is probably the clearest example of this — in times of increased activity and higher value, its transaction times and cost have both skyrocketed, causing serious problems for users.

This is a major problem; data is growing all the time and blockchains must be able to adapt to this constant growth and storage demand. If blockchains are unable to cope with large volumes, they’re unsuitable for use in mainstream data storage. There are several current approaches and companies try to solving this problem.

Storage Cost

While blockchain has great potential for data storage, the technology first needs to address its issues with storage. When blockchains come under strain, costs tend to spiral out of control. For example, when Bitcoin experienced a price boom last year, transaction costs soared as high as $50.That just isn’t good enough to confront the explosion in data that lies ahead.

Sia and Storj both operate large blockchain-based storage networks aimed at delivering fast and cheap data storage without the need for a central point or controlling third party. Both networks work by fragmenting files into chunks, then encrypting these and distributing them across the network. Sia has already been deployed with more than 130 TB stored across more than 300 contributors. Storj is set to launch soon.

Filecoin works in a similar way, relying on nodes in the powerful IFPS network to distribute its files. The company raised more than $250 million and is currently developing decentralized marketing for data storage.

Data Security and Scalability

For any data storage system, security must be paramount. Current cloud-based models, like those offered by Google and Amazon, prioritize security, but their centralized nature puts them at a disadvantage because they’re built around a central point of failure and leave the users’ data in the hands of the company.

Blockchain technology, often praised for its airtight structure and inherent security, is the clear winner here.

But one issue plaguing the blockchain space is that of scalability. Ethereum and Blockchain, the biggest blockchains, struggle to get into double figures when it comes to transactions per second. Visa, on the other hand, can handle 44,000, and Facebook a whopping 175,000.

Arweave, the new rising startup in this ecosystem, has pioneered a novel way of dealing with data. The company found a way to store the data directly on the blockchain itself, unlike some other approaches to blockchain data storage, which rely on third-party protocols built on top of existing blockchains.

Essentially, as the amount of data stored in the blockchain increases, the hashing required for consensus decreases, making it highly scalable.

In addition to this, the platform uses a new consensus mechanism called Proof of Access, where miners compete for who can replicate the most data. This is far less energy intensive than existing models like Proof of Work, which essentially rewards the miner who can muster the most computing power.

"What we've done is actually solved on-chain data storage, unlike other solutions with P2P file distribution network in the background and then settlement of payment on-chain, we have made a system of cryptoeconomic incentives that allow you to grow the size of the blockchain to massive sizes and then distribute the data across all computers", says Sam Williams, CEO of Arweave. This approach offers a better way of storing our data, and it works on a large scale.

What’s more, the platform stores data permanently and requires only a one-time fee from its users. There’s no need to get tied up in expensive monthly contracts, an unhappy side effect of many cloud storage models.

Arweave also recently announced the launch of the Permaweb — a new immutable and decentralized web that offers low-cost, zero-maintenance storage, forever. It offers convenience, efficiency, and information security free from meddling third parties. It also comes with its own privacy layer, offering communication safe from any external forces.

As data plays an ever more important role in our world, we’ll need to move towards innovations and with pioneers like Arweave, Filecoin, Sia and Storj, 2019 might be the year of where we move away from outdated storage methods.