After 351 days of uptime, Solana users may have been thinking all outages were behind them.
But the blockchain network — known for high-speed transactions and scalability, and a host to external ecosystems such as Jupiter Exchange — experienced a major outage on Tuesday, February 6, with block progression halted for five hours.
With Solana regularly outperforming Bitcoin and Ethereum in terms of daily active users and processing tens of millions of transactions daily, stability needs to be top priority, especially if the ecosystem is to attract non-crypto partners.
The Solana Foundation quickly began to work on fixing the ‘glitch’, with its engineering team patching the problem and validators jumping in to restart the network.
While a crypto enthusiast may understand outages as growing pains, any outage becomes a problem when you have large partners like VISA supporting stablecoin settlement through Solana.
Naturally, shots were fired from other quarters:
Solana's 11th outage in the last 2 years. This is not what decentralization looks like. https://t.co/wszopDo5Jy
— Charles Edwards (@caprioleio) February 6, 2024
— Charles Hoskinson (@IOHK_Charles) February 6, 2024
The outage, and how it was resolved, is a useful insight into the constant centralization vs decentralization debate and a reminder that if crypto is to go mainstream, the public needs to be able to trust that their transactions will go through when they need them to.
Unpacking the Solana Outage: Performance Degradation and Immediate Response
The incident on February 6, first reported by Solana validator Laine on X, indicated that the network’s block progression had been halted due to unspecified issues, causing a ripple effect across its ecosystem.
Solana Mainnet-Beta is experiencing a performance degradatation, block progression is currently halted, core engineers & validators are actively investigating.
— Laine ❤️ stakewiz.com (@laine_sa_) February 6, 2024
The Solana Foundation’s status handle confirmed the problem, stating that core engineers, alongside validators, were actively investigating the cause of the disruption.
Engineers from across the ecosystem are readying a new validator software release, which includes a patch to address an issue which caused the cluster to halt. Validator operators should prepare for an upgrade and restart of the network.
— Solana Status (@SolanaStatus) February 6, 2024
Core engineers at Solana identified a fix and proceeded to build a new version of the validator software.
This update rectified the glitch, a full post-mortem of which has not yet been disclosed, bringing the network back online for users and developers.
One thing that might cause concern is that the restart procedure, detailed in Solana community communications, necessitated a significant portion of validators, specifically those with at least 80% of SOL staked, to be active for the network to resume operations effectively.
If not, the restart would need to be tried again, with non-responsive validators removed.
Core Team and Collaborators Needed for a Restart
Solana retains some control of their network, allowing authoritative bug fixes and restarts, and yet wishes to move to a decentralized state.
A decentralized protocol means users can trust in and personally validate the system and smart contracts without fearing an authoritarian at the top. However, it can cause problems when issues emerge and consensus is needed to solve them.
In this case, Solana Labs and the Solana Foundation could act quickly but still rely on the wider community to complete the restart.
The process was monitored by network validator @mtromp through an X post, who urged validators to prepare for an upgrade and restart of the network.
~16% to go!
The last bit can be the slowest but looking good to restart soon. pic.twitter.com/bmbNJ8ILUX
— marty | superfast (@mtromp) February 6, 2024
Validators and Snapshots
The recovery process involved validators generating snapshots using their local ledger state to facilitate a seamless network restart.
This process underscores the importance of snapshots, which serve as exact copies of blockchain data at a specific point in time, ensuring data integrity and continuity.
The Solana Foundation also urged validator operators to expedite the upgrade process, emphasizing the collaborative effort required to stabilize the network.
Mainnet validator operators: release version v1.17.20 is ready – please upgrade and restart with the following instructions: https://t.co/e03MMyeIYB
— Solana Status (@SolanaStatus) February 6, 2024
Price Impact on SOL
The outage has notably impacted Solana’s SOL tokens, which decreased over 2% to $94 in the aftermath of the news, although SOL price has recovered most losses in the day since.
Despite the urgency and potential concerns raised by the outage, the Solana community, including platforms like Jupiter Exchange, reassured users by offering support and ensuring that measures like stop-loss orders for perpetual contracts would function as intended, thereby maintaining a semblance of normalcy and security for traders amidst the volatility.
The Solana network is currently down, but our whole team is on standby to reply to all your questions if needed – just open a ticket in discord, and we will get to it asap.
In addition, if you have a stop loss for your perps, it will execute exactly at that price.
— Jupiter 🪐 (@JupiterExchange) February 6, 2024
As the Solana ecosystem rallies to overcome this outage, the incident sheds light on blockchain networks’ continuous challenges in maintaining performance and reliability.
The collaborative response by engineers and validators alike demonstrates a committed effort to safeguard the network’s integrity, ensuring that Solana remains a key player in the evolving world of blockchain technology.
But it’s an example of how vital uptime and quick reactions to issues are if the wider public is to incorporate blockchain and crypto into their daily lives.