Solana’s near-finality event and validator hosting concentration

Since the intern running @solana is apparently busy with other things, here is the technical breakdown and everything I could verify about today’s near-finality event on #Solana.

At the worst point of the incident, 28.83% of all staked SOL was delinquent.

Solana needs more than 66.67% of stake participating in consensus to maintain normal finality, which means the critical point sits at roughly 33.34% delinquent stake.

The network was therefore only 4.51 percentage points, roughly 20 million SOL, away from losing finality.

4.51%....

Around 90 validators were affected. Solana did not halt and finality was never actually lost, but this was far closer than most people seem to realise.

The cause was not an exploit, malicious staking, a compromised Solana program or some attacker transaction you can paste into Solscan.

The failure happened at the infrastructure layer, specifically at Teraswitch AS20326, which hosts a very large amount of Solana validator infrastructure.

According to TeraSwitch’s own incident report, a malformed Internet default route, "0.0.0.0/0", originated through its MIA1 infrastructure in Miami.

The route lost attributes that normally determine how routers should treat it and was then propagated through a route reflector at AMS2 in Amsterdam into TeraSwitch infrastructure across Europe and Asia-Pacific.

Edge routers started preferring that malformed route over their legitimate local default route, while the downstream datacenter core rejected it.

In practical terms, the valid path had been displaced and the replacement could not actually carry the traffic. Validators behind the affected infrastructure became unreachable, stopped voting and their stake turned delinquent.

The affected TeraSwitch sites were LON1, AMS1, AMS2, AMS3, DUB1, DUB2, FRA2, SGP1, SGP2, TYO1, TYO2 and TYO3, covering:
London,
Amsterdam,
Dublin,
Frankfurt,
Singapore and Tokyo.

TeraSwitch says its North American sites were not affected by this specific propagation problem.

Engineers identified the malformed route within roughly 10 minutes, isolated MIA1 from the private backbone and restored service by 04:16:15 UTC.

The exact underlying defect still does not appear to be conclusively identified in the public incident material, including whether the original problem sat inside the MIA1 edge routers themselves or somewhere in the route-reflector path.

The far more important part is the concentration behind the failure.

Marinade’s incident analysis put 118.9 million SOL on TeraSwitch AS20326, representing approximately 27.34% of all staked SOL, and roughly 94% of that stake went offline together.

That is the real story here.

➡️ You can have a large validator count and still carry substantial systemic risk if a meaningful share of those validators depend on the same ASN, hosting provider, routing layer or upstream infrastructure.

@SolanaFndn already recognises this problem in its own delegation rules.

Validators participating in the delegation programme are required to operate on an ASN and hosting provider containing less than 25% of total network stake, with a separate 15% datacenter concentration limit.

Those limits are delegation requirements rather than protocol-enforced caps, but their existence makes the point fairly obvious: infrastructure concentration matters because apparently separate validators can still fail together when they share the same underlying network.

The recovery data is not particularly reassuring either, btw.

Marinade found 59 validators representing around 80.2 million SOL returning during roughly the same recovery window.

Of the 74 validators it says it could evaluate, only 3 appeared to perform a clean failover: Laine, Cogent Crypto and Lion3d.

That does not prove the remaining validators had no backup systems, but whatever redundancy existed clearly did not restore their consensus participation before the shared infrastructure recovered.

Helius, one of the largest affected operators, reportedly remained offline for approximately the full 33-minute disruption.

There is another unresolved piece worth watching.

Marinade observed an additional 14.1 million SOL associated with other infrastructure providers disappearing during the same period, including validators attributed to https://t.co/78MQVq5owl, Limestone, Butterfly Research and Allnodes.

There is currently no public evidence proving that TeraSwitch caused those outages, so I would not make that claim atm.

What it does show is how difficult it can be to identify real infrastructure independence from provider labels alone.

Two validators can appear to sit with completely different companies while still sharing an upstream carrier, transit network, routing dependency or physical facility somewhere further down the stack.

Because this was a networking failure, there are also no attacker transactions to trace.

The observable Solana footprint is validator behaviour: consensus votes stop arriving, vote credits stop accumulating, stake becomes delinquent, leader participation is missed and the validators return once connectivity is restored.

@Marinade estimates the affected validators collectively missed roughly 333 SOL in rewards, which is economically minor compared with the consensus risk but still provides another measurable consequence of the event.

The important distinction is that Solana did not go down, and the official status page is therefore correct not to classify this as a mainnet outage.

Enough stake remained online for blocks to continue reaching finality.

But calling it irrelevant would be equally dishonest.

At 28.83% delinquent stake against a roughly 33.34% threshold, another 4.51 percentage points, around 20 million SOL, becoming unavailable would have left the remaining network without enough participating stake to maintain normal finality.

That would not automatically have meant stolen funds, a destroyed ledger or every validator suddenly switching off.

It would have meant that new state could no longer continue reaching normal irreversible finality until enough voting stake returned. For exchanges, bridges, custodians and any infrastructure that depends on finalized settlement, that is a serious operational problem.

What remains unproven is the exact software defect behind the malformed route, why the additional 14.1 million SOL on other providers disappeared at the same time, whether every affected validator actually lacked independent backup infrastructure, whether every validator currently listed behind TeraSwitch was delinquent at the exact incident peak, and whether any malicious actor was involved.

On the last point, there is currently no evidence that there was one.

For a network securing this much value, that is a very real infrastructure concentration problem and a far more useful decentralisation metric than simply counting validator identities.

The numbers I currently consider supported are 28.83% peak delinquent stake, a roughly 33.34% finality boundary, 4.51 percentage points of remaining headroom, around 20 million SOL between the observed peak and that boundary, roughly 90 affected validators, approximately 333 SOL in missed rewards, 118.9 million SOL behind TeraSwitch AS20326, 27.34% of total network stake on that ASN, around 94% of that stake going offline together, 59 validators representing 80.2 million SOL returning in the same recovery window, only 3 of 74 showing an apparent clean failover, another 14.1 million SOL on other providers disappearing during the same period, 12 affected TeraSwitch locations and full service restoration at 04:16:15 UTC.