Why Do Public Blockchains Halt? Truly Understanding Blockchain Consensus Through Cosmos's 25-Hour Downtime

On the evening of September 22, a user initiated an ATOM transfer.

After overnight, the transaction remained stuck in "waiting for confirmation".

No private key was lost, and the wallet showed no signature anomalies. Upon checking again the next day, multiple public RPCs indicated that Cosmos Hub had halted at block height 33,086,740.

Without new blocks being produced, there was naturally nowhere to pack this transaction into.

It wasn't until roughly a day later, when Cosmos Hub resumed block production, that this previously pending ATOM transfer finally succeeded.

For ordinary users, this might be the most intuitive lesson in understanding blockchain consensus.

We are accustomed to saying "no central authority can shut down a public chain," but reality is significantly more complex. A sufficiently decentralized blockchain indeed typically lacks a physical "off switch" in a server room, yet it can still come to a complete halt.

This pause by Cosmos Hub perfectly exposed the underlying mechanisms, usually hidden from view, directly to everyday users.

I. Why Did Cosmos Suddenly "Stop Block Production"?

First, we need to clarify a frequently confused point: Cosmos Hub itself was not directly attacked in this incident.

The event initially occurred on Neutron.

On September 22, a Neutron governance proposal titled "AIATO: AI Agent Takeover" was approved. The attacker exploited loopholes in chain-level governance permissions, utilizing privileged commands natively provided by the wasmd framework to change the contract administrators of applications like Astroport and Drop to addresses controlled by the attacker.

This does not fit the conventional understanding of a "code vulnerability" or "protocol flaw".

To put it simply, while each application has its own "lock", chain-level governance on Neutron holds a higher-authority "master key". When the attacker controlled the governance outcome, they essentially obtained this key, allowing them to reassign administrators, migrate contracts, and further siphon off assets.

What truly pulled Cosmos Hub into the fray was the subsequent cross-chain fund transfer.

A post-mortem by Cosmos Labs revealed that before Neutron went offline, the attacker had already diverted some assets across multiple networks, with approximately 1.7 million ATOM transferred into Cosmos Hub, where they began circulating through cross-chain liquidity pools.

In other words, Cosmos Hub itself suffered no direct attack, and funds belonging to ordinary Hub users were not directly stolen via the Neutron vulnerability.

However, since the stolen ATOM had already entered the Hub, to prevent the remaining ATOM from flowing out, some Cosmos Hub validators began taking their nodes offline.

By around 19:18 SGT on September 22, the offline validators represented over one-third of the total voting power, preventing Cosmos Hub from producing new blocks and causing it to halt at 33,086,740.

This step was crucial.

It signifies that Cosmos Hub lacks a corporate "Pause" button waiting to be clicked, nor did it require an on-chain governance vote beforehand. What actually stopped the network was enough validators choosing to cease participating in consensus formation.

But even more noteworthy is the recovery process that followed.

About four hours after the chain halt, validators received a comprehensive recovery plan: execute a one-time state modification targeting the halted block height, transferring the remaining ATOM from the attacker's address into a multi-signature address jointly managed by community validators.

Subsequently, Cosmos Labs developed a patch for Gaia v28.3.0 based on the agreed-upon plan, tested it, and distributed it to the validators.

This version of Gaia would execute a one-time state transition at the designated recovery height, moving 1,227,121 ATOM from the attacker's address into a 4-of-6 multi-signature address established by six parties: Nansen, Keplr, Enigma, Silknodes, Kiln, and Polkachu.

By the early morning of September 23, validators confirming the installation of v28.3.0 exceeded 67% of the total voting power. Consequently, Cosmos Hub coordinated a restart at 12:00 UTC that day. Approximately six minutes later, this one-time state modification was executed at block height 33,086,741, and the network resumed normal block production.

In essence, throughout the entire process from block production halting to resuming operation on Cosmos, validators first caused the network to lose Liveness, then over two-thirds of the voting power accepted a new set of state transition rules, ultimately making those rules the canonical state post-recovery.

At this point, a seemingly simple question arises: If it is a decentralized public chain, why can verification power exceeding one-third halt it, yet restoring the network requires a sufficient majority of validators to collectively accept and run the same software?

The answer, it turns out, lies precisely within the concept of "consensus".

II. Consensus Was Never Meant to Be "Unstoppable"

The most common misconception surrounding blockchain is equating "decentralization" with "zero downtime".

In reality, what consensus mechanisms truly solve is how numerous nodes reach agreement on transaction order and ledger state without a central ledger keeper.

However, different public chains implement this differently.

In Bitcoin, the most classic approach is PoW (Proof of Work)—miners compete to produce blocks using computational power. When the network temporarily forks into two legitimate branches, nodes select one to continue building upon based on cumulative work.

Thus, Bitcoin lacks a definitive moment where "a block is permanently Finalized after 67% votes", leaning closer to probabilistic finality. The more subsequent blocks are added, the exponentially higher the computational cost required to reorganize and override previous transactions.

This is also why it was traditionally advised to wait for six confirmations for a Bitcoin transaction; regardless of how much computational power you command, you cannot simply bypass the consensus rules actively enforced by the network.

Of course, this doesn't mean Bitcoin's state is "absolutely unchangeable" under any circumstances. Theoretically, if the entire ecosystem agrees to adopt a new client and novel consensus rules via a Hard Fork, it can render previously invalid state changes valid.

But here lies the problem: who has the capacity to compel a sufficient number of miners, Full Nodes, exchanges, wallets, and users to collectively accept such a rule change?

Virtually no one.

The development team cannot dictate consensus rules for the entire Bitcoin network, nor can miners or exchanges easily do so, given the exceptionally high threshold for consensus alignment. Back when Binance had 7,000 BTC stolen, some suggested CZ reach out to major mining pools to intervene, but it ultimately came to nothing.

Ethereum provides another classic case study.

After transitioning to PoS, Ethereum currently employs the Gasper consensus, which combines Casper FFG and LMD-GHOST. Simply put, one mechanism determines "which chain to follow at any given time", while the other ensures blocks achieve true Finality.

Blocks can only progress toward final determination once validators representing at least two-thirds of staked ETH agree on the corresponding checkpoints. Conversely, if over one-third of the stake fails to vote correctly for an extended period, the network may temporarily lose its ability to finalize blocks. However, Ethereum incorporates an inactivity leak mechanism that gradually reduces the effective weight of offline validators during prolonged finalization failures, eventually giving the network a chance to recover Finality.

Truly altering such outcomes similarly requires changing protocol rules and clients.

As seen in the 2016 The DAO incident, the Ethereum community ultimately enacted a Hard Fork, executing a special state modification explicitly termed an irregular state change by the Ethereum Foundation at block 1,920,000, routing the relevant ETH into a recovery contract.

Naturally, a faction of miners and community members who refused to upgrade and continued maintaining the original state eventually spawned Ethereum Classic (ETC), resulting in the well-known ETH and ETC fork, demonstrating that not everyone consents to such rule changes.

Cosmos Hub operates differently, utilizing CometBFT, which aligns closely with typical BFT consensus.

You can view it as a quintessential BFT consensus, meaning a block truly commits only after receiving Commit messages from validators controlling over two-thirds of the voting power.

Its advantage lies in extremely explicit Finality. Once a block is committed with sufficient validator weight, there is no need to keep waiting for successive blocks like in PoW, trading probability for perceived security.

Yet its flip side is equally straightforward: if one-third or more of the voting power ceases to provide the votes necessary for Commit, no amount of effort from the remaining validators can surpass that two-thirds threshold.

In such scenarios, the safest choice for the network is exactly what transpired this time—a "pause on block production"—so viewed through the lens of distributed systems theory, this brief outage of Cosmos Hub is hardly mysterious.

In short, once a group holding sufficient voting power stops participating, the consensus protocol adheres to its own rules, preferring to sacrifice availability rather than continue confirming new blocks in the absence of adequate consensus.

Underlying this corresponds to two concepts in distributed systems often conflated by everyday users:

• Safety: Preventing different nodes from simultaneously confirming two mutually conflicting final states;
• Liveness: Whether the network can continue operating forward and processing new transactions;

For BFT systems, when participating consensus nodes fall below a critical threshold, pausing is sometimes precisely the price paid to maintain Safety. To put it bluntly, this decentralized ledger would rather remain frozen than allow remaining participants to record divergent histories.

Looking back from this angle, one realizes that many seemingly disparate incidents in public chain history actually revolve around a single theme:

What should a network do when distributed nodes can no longer reach an agreement on the "correct state"?

III. From Bitcoin to Solana, Where Exactly Are the True Risk Boundaries of Public Chains?

This is not the first time Cosmos has brought this question to the forefront.

As early as 2013, Bitcoin experienced a highly classic chain split incident.

At the time, Bitcoin core version 0.8 switched its underlying database from Berkeley DB to LevelDB. Subsequently, a block containing numerous transaction inputs emerged. While newer nodes processed it normally, certain legacy nodes deemed the block invalid due to Berkeley DB's lock quantity limits.

An awkward scenario unfolded: although everyone was running Bitcoin, the new and old clients began generating contradictory answers regarding "whether this block was actually valid".

The network consequently split into two chains, with the new 0.8 branch temporarily commanding roughly 60% of hashing power, unable to rely on standard hash rate competition to quickly self-converge.

Ultimately, major mining pools coordinated to revert to the older version, regaining sufficient hashing power under the legacy rules, allowing the network to converge once more. Bitcoin later formally addressed this incident through BIP 50.

By 2016, Ethereum's The DAO incident pushed the issue a step further.

As mentioned earlier, the Ethereum community ultimately executed a Hard Fork, performing a special state modification explicitly labeled an irregular state change by the Ethereum Foundation at block 1,920,000, redirecting the involved ETH into a recovery contract.

Yet not everyone endorsed this intervention. A segment of miners and community members refusing to accept the state modification continued maintaining the original rules, eventually birthing the long-standing Ethereum Classic (ETC).

The DAO Fork stands as another classic event, essentially signaling to everyone that in extreme scenarios, social consensus exists beyond code-based consensus; if sufficient agreement cannot be reached, a single chain can genuinely fracture into two.

The 2021 Solana incident showcased a completely different failure pathway.

In September of that year, a massive influx of bot trades flooded the network, exhausting validator node memory and causing widespread crashes. Ultimately, the entire network lost the ability to form consensus on the current state, halting the confirmation of new blocks for approximately 17 hours before validators collectively coordinated a network recovery.

Comparing these incidents reveals they are fundamentally distinct:

• The 2013 Bitcoin issue stemmed from different clients executing varying validity rules;
• The 2021 Solana issue arose from mass validator nodes failing to participate in consensus normally, causing the network to lose Liveness;
• What Ethereum's DAO faced was closer to a question of whether a community should proactively modify state via new protocol rules;
• Meanwhile, this Cosmos Hub incident carries another layer of uniqueness: the network first voluntarily sacrificed Liveness through validator coordination to halt the movement of compromised assets; subsequently, a sufficiently large proportion of voting power collectively accepted new software and a restored state, enabling the network to regain alignment;

Therefore, rather than naively dismissing these events as proof that "blockchains can actually be shut down" or that "decentralization is a myth," it is more accurate to acknowledge a truer reality:

Consensus mechanisms have never been fault-proof machinery; what they truly offer is a set of decentralized rules, dictating things like: who decides the valid chain during disputes; how many participants are needed to grant a state finality; whether the network chooses to persist or halt upon encountering faults; and what collective actions are permissible to alter operational guidelines under extreme circumstances.

This leaves the recent Cosmos incident with a question far more worthy of reflection for ordinary users than merely "should the chain have halted?"

Final Thoughts

We frequently quote the maxim: not your keys, not your coins.

This principle undeniably remains true, though its primary emphasis is asset custody control—provided your private keys remain in your possession, wallets, exchanges, or third-party services cannot improperly forge your transfers.

This assumes, however, that the blockchain you are using maintains the capacity to process that signature at all times.

On the day Cosmos Hub halted block production, users still retained possession of their private keys, and their assets did not vanish into thin air; it simply meant that even if you correctly signed a transaction, no new blocks existed to receive it.

The recovery process further demonstrates that if a sufficient majority of consensus participants adopt a new set of state rules, the on-chain status of specific accounts can indeed undergo alteration without requiring the original address's private key signature.

This does not invalidate the "Not your keys, not your coins" mantra, but it serves as a reminder that private key sovereignty and fundamental consensus authority are inherently separate matters.

The same logic applies to wallets.

Wallets can ensure private keys and signing authority remain exclusively with users, rapidly detect chain-level anomalies, accurately display transaction statuses, establish RPC and node redundancy, and verify final transaction outcomes once the network recovers.

Yet wallets cannot restore consensus on behalf of a public chain, cannot guarantee the underlying network will never experience interruptions, nor can they assure that on-chain rules and states will never undergo consensus-level modifications.

Thus, what a mature decentralized system truly needs to pursue may never be the absolute dogma that "nothing can ever be changed"; conversely, it should strive to clearly delineate these imperfect boundaries: who possesses the authority to pause consensus? What threshold of voting weight is required? Under what conditions is emergency intervention permissible?

Because genuine decentralization cannot immunize a system against all accidents, the true challenge lies in ensuring that even when crises inevitably occur, we still know precisely who acts, what rules they follow, and what degree of consensus dictates how the ledger shall proceed.

Join the official Coincamps community:

X: https://x.com/coincamps

Telegram: https://t.me/coin_camps