Chain halt during v4.0.0 upgrade

Incident Report for dYdX Chain

Resolved

This incident has been resolved.

Posted Apr 08, 2024 - 16:53 UTC

Monitoring

A fix has been implemented and blocks are being produced. We are monitoring the results.

Posted Apr 08, 2024 - 16:23 UTC

Update

FIX AVAILABLE TO SOFTWARE BUG CAUSING CHAIN HALT

Use the new binary v4.0.2 (https://github.com/dydxprotocol/v4-chain/releases/tag/protocol%2Fv4.0.2) to fix the state corruption issue. This fix is not a state-breaking change.

Recommended approach: Use a snapshot taken at post-upgrade height (12791712) and run v4.0.2 . The post-upgrade snapshot has correctly persisted the state for block 12791712 and this snapshot was generated using v4.0.2

Snapshot Link: https://mainnet-v4-post-upgrade-snapshot.s3.ap-northeast-1.amazonaws.com/dydx_mainnet_12791712.tar.lz4

NOTE FOR VALIDATOR NODE OPERATORS

If you are running a validator node, please reach out to dYdX Operations Services at infrastructure@dydxopsservices.com for recovery instructions or to take part in the co-ordinated recovery.

Posted Apr 08, 2024 - 15:32 UTC

Update

The issue continues to be investigated. It's been agreed to reconvene with the validators around 15:00 UTC.

This means that the devs won't suggest a workaround or a fix until then so that the validators won't get jailed for not being online when the chain restarts.

Posted Apr 08, 2024 - 09:21 UTC

Investigating

The issue is being debugged. Updates will appear here.

Posted Apr 08, 2024 - 06:50 UTC

This incident affected: dYdX Chain (Protocol).