Postmortem: June 25th Block Production Outage

Base is committed to building in the open, which includes sharing detailed public postmortems when issues arise.

Overview

On June 25th and 26th 2026, Base mainnet experienced two block production outages. The first incident began at 11:47am EDT on the 25th, and lasted 116 minutes. The second started at 11:28am EDT on the 26th and lasted 20 minutes. The root cause was the same and they will be discussed together throughout this postmortem.

The integrity of the chain was not compromised and all funds on Base were safe. Block production resumed safely after we mitigated the incident. This postmortem dives into the root cause, the impact, how we mitigated, and what we plan to improve moving forward.

Root Cause

We identified a bug in sequencer block-building logic that allowed stale journal state to persist after a transaction validation failure. An invalid transaction was received by the block builder and failed during execution, as expected, but erroneously did not clear the journal state that contained the accounts and storage slots that had been accessed.

After that failed transaction, a valid transaction was received and executed before the journal was properly cleared. As a result of the inaccurate journal state, the valid transaction had a mismatch in the amount of gas charged. This caused a sequenced block that contained an invalid state transition, leading to a block that could not be accepted by any other nodes. The chain fully halted at this point.

Impact

During the 116 and 20 minute outages, Base experienced the following issues. 

Block Production

Complete halt of new L2 blocks produced. The sequencer and validator nodes could not progress past the invalid block until sequencing was restored.

Transaction Processing

During the outage, new transactions that queued in the mempool were not included onchain. This caused the txpool to grow larger than what it could store, and reject newly submitted transactions. Requests submitting eth_sendRawTransaction transactions to Base were receiving errors throughout this time.

Mitigation

The outage was fixed by applying a patch to the sequencers to ensure the journal state was properly updated during execution. 

The fix can be seen in this PR: https://github.com/base/base/pull/3806

Mitigation took longer than expected due to infrastructure conditions unrelated to the original bug. The sequencers in the active/passive cluster do consensus-layer sync rather than execution sync like validator nodes. A race condition in the engine reset feature prevented the sequencers from catching up to tip after being restarted. This affected only the sequencers, but was a separate bug we had to fix during mitigation.

It is because of this second issue that the incident occurred again the following day. The issue can be seen here: https://github.com/base/base/pull/3805/changes

Going forward

Base is committed to building a platform for global finance, and will continue investing in the reliability and performance of the chain to live up to that mission. We recognize this incident caused disruption for users, builders, and node operators.

We are using this incident to harden protocol fuzz testing practices and load testing harnesses to be able to more easily identify adversarial transaction patterns that could expose bugs like the one we saw here. In addition to operational and monitoring improvements, we expect these changes to improve resilience and will continue validating them in production and through testing.

Additionally, this case required validator nodes to be restarted in order to continue syncing. We intend to implement graceful recovery into base-consensus for faster and easier ecosystem recovery in these situations.

Please subscribe to notifications on https://status.base.org to stay informed going forward.

---

Incident Timeline

Phase 1: The Chain Halt

Phase 2: Rapid Response & Triage Benchmarks

Phase 3: Mitigation & Network Recovery

\

The second chain halt on June 26th was similar in root cause as the first. It started at 11:28am EDT and lasted 20 minutes.