Filecoin outage history

Many Filecoin nodes are not syncing due to "failed to verify beacon". Lotus implementers are investigating the issue with the Drand implementers, and we will provide updates in #fil-network-announcement and https://github.com/filecoin-project/lotus/issues/12467 as soon as we have more information.

We have identified the issue: the recent BLS library dependency update in Lotus was causing BLS signature validation failure. The Lotus team is preparing the patches against the latest stable release for v1.28.x and v1.29.x, and we recommend that all node operators update to the newest release as soon as possible! For more details and ongoing updates, please see https://github.com/filecoin-project/lotus/issues/12467.

We have identified the issue: the recent BLS library dependency update in Lotus was causing BLS signature validation failure. The Lotus team is preparing the patches against the latest stable release for v1.28.x and v1.29.x, and we recommend that all node operators update to the newest release as soon as possible! For more details and ongoing updates, please see https://github.com/filecoin-project/lotus/issues/12467.

Lotus patches are out now (https://github.com/filecoin-project/lotus/releases/tag/v1.28.3; https://github.com/filecoin-project/lotus/releases/tag/v1.29.1) and the impacted nodes are recovering. Implementers are considering this incident as resolved now, and we will keep monitoring the network. Thank you to the Filecoin node operators for all your support for a speedy recovery!

The Filecoin network is halted with most nodes stuck at height 336459 with mismatched state compute error due to indeterminate behavior. We have found the root cause and are testing the fix. Next Step: Miners will need to upgrade to lotus v1.4.0 (https://github.com/filecoin-project/lotus/releases/tag/v1.4.0) with the fix when it is released. Once miners upgrade, the chain will start to catch up.

Lotus v1.4.0 (https://github.com/filecoin-project/lotus/releases/tag/v1.4.0) has been released. Please upgrade ASAP and see the recovery instructions (https://github.com/filecoin-project/lotus/discussions/5232) for more information/troubleshooting.

The network has restarted and progressed to the expected block height (337120 @ 3:20pm PT). As more power joins the upgraded chain, the network will continue to stabilize. Check back for an update here when >50% of power is in sync.

We are continuing to monitor for any further issues.

If you’re just catching up: check the upgrade and recovery instructions (https://github.com/filecoin-project/lotus/discussions/5232) to get back in sync. Miners have upgraded & the chain is now recovering smoothly. 🎉🙏💪 We don’t expect further action is required, but will continue monitoring. Expect degraded chain quality. We should expect the network to be somewhat degraded for the next few hours as miners continue to upgrade and power recovers (currently ~34% upgraded). We recommend pausing all non-critical messages until chain capacity is restored to normal levels - BaseFee may be volatile as the network recovers. Full recovery ETA. Conservatively, we will deem the network to be fully recovered & stable 2x finality from when the chain resumed. This means 1,400 epochs (11.6 hrs) from 337120 (2020-12-19 23:20 UTC). So, conservatively, full recovery at epoch 338520 (2020-12-20 11:00 UTC). For custodians, exchanges, wallets: The full recovery ETA is very conservative, and the chain is expected to be fully functional long before. Our recommendation for you would be to keep transfers out disabled until that time or until you gain comfort with chain status -- at least for larger amounts. Your own internal transfers, trading, and other fully off-chain activity should be fine.

Please see this post on [**Resolving the Dec 19 Chain Halt: Cause, Impact, & Take Aways **](https://filecoin.io/blog/resolving-the-chain-halt/)for cause and impact of this outage. More specific post mortems WIP from the actors & infra teams to describe specific learnings and mitigations for the future.

We are currently investigating this issue.

When your product or service isn’t functioning as expected, let your customers know by creating an incident. Communicate early, even if you don’t know exactly what’s going on.

As you continue to work through the incident, update your customers frequently.

Let your users know once a fix is in place, and keep communication clear and precise.

Empathize with those affected and let them know everything is operating as normal.