Elevated block time due crashing nodes

Incident Report for dYdX Chain

Resolved

The dYdX Chain seems to have been stabilized with dYdX Chain Protocol v4.0.5, we advise all validators to update to it as soon as possible with the steps suggested earlier.

Posted Apr 11, 2024 - 21:41 UTC

Update

There’s a new issue that validators and other node operators ran into, with an error signature of: "invalid memory address or nil pointer dereference"

The root-cause has been identified and fixed in release v4.0.5 which all validators and node operators are advised to update to:
https://github.com/dydxprotocol/v4-chain/releases/tag/protocol%2Fv4.0.5

Paths for nodes on different binaries:
- 4.0.2 → Nodes on v4.0.2 might want to stay on v4.0.2 until they are ready to do the resync on v4.0.4/v4.0.5. Recent snapshots using v4.0.4 are available below. If they stop their node while running on v4.0.2, there’s a chance it won’t restart:
- use a recent snapshot that’s created on v4.0.4, and start running v4.0.5 on top of the snapshot
- or use v4.0.5 to resync from post-upgrade height
- 4.0.4 → just switch over to 4.0.5, because there’s no state change, just a mempool metric change.

v4.0.4 snapshots:
- https://snapshots.bwarelabs.com/dydx/mainnet/dydx20240409.tar.lz4 (block height 12870104)
- https://polkachu.com/tendermint_snapshots/dydx

Posted Apr 09, 2024 - 20:21 UTC

Monitoring

To address the issues with sporadic node crashes, validators should consider updating to v4.0.4:
https://github.com/dydxprotocol/v4-chain/releases/tag/protocol%2Fv4.0.4
This is a rolling update.

To ensure that no malicious errors have snuck in your state between the time of upgrade to v4.0.2 and the time you get to running v4.0.4, we suggest you apply a post-upgrade snapshot and run the latest version on top:
- https://mainnet-v4-post-upgrade-snapshot.s3.ap-northeast-1.amazonaws.com/dydx_mainnet_12791712.tar.lz4
- or any other resource that will safely bring your node back to 12791712

Posted Apr 09, 2024 - 06:11 UTC

Identified

The devs think they've identified the issue. Working on a remedy / recovery steps. Testing internally and a small set of external validators to confirm the fix.

The fix will be more widely available in the next couple of hours.

Posted Apr 08, 2024 - 20:17 UTC

Investigating

We are currently investigating this issue.

Posted Apr 08, 2024 - 17:50 UTC

This incident affected: dYdX Chain (Protocol).