Update - We are continuing to investigate this issue.
Apr 15, 2024 - 23:25 UTC
Investigating - We are investigating an issue impacting withdrawing into non-USDC assets.
USDC withdrawal is fully functional.

Apr 15, 2024 - 23:24 UTC
dYdX Chain (Protocol) Operational
90 days ago
99.55 % uptime
Today
Indexer Operational
90 days ago
99.85 % uptime
Today
Frontend (dydx.trade) Degraded Performance
90 days ago
100.0 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.
Past Incidents
Apr 17, 2024

No incidents reported today.

Apr 16, 2024

No incidents reported.

Apr 15, 2024

Unresolved incident: Issue with Non-USDC Withdrawal.

Apr 14, 2024

No incidents reported.

Apr 13, 2024

No incidents reported.

Apr 12, 2024

No incidents reported.

Apr 11, 2024
Resolved - The dYdX Chain seems to have been stabilized with dYdX Chain Protocol v4.0.5, we advise all validators to update to it as soon as possible with the steps suggested earlier.
Apr 11, 21:41 UTC
Update - There’s a new issue that validators and other node operators ran into, with an error signature of: "invalid memory address or nil pointer dereference"

The root-cause has been identified and fixed in release v4.0.5 which all validators and node operators are advised to update to:
https://github.com/dydxprotocol/v4-chain/releases/tag/protocol%2Fv4.0.5

Paths for nodes on different binaries:
- 4.0.2 → Nodes on v4.0.2 might want to stay on v4.0.2 until they are ready to do the resync on v4.0.4/v4.0.5. Recent snapshots using v4.0.4 are available below. If they stop their node while running on v4.0.2, there’s a chance it won’t restart:
- use a recent snapshot that’s created on v4.0.4, and start running v4.0.5 on top of the snapshot
- or use v4.0.5 to resync from post-upgrade height
- 4.0.4 → just switch over to 4.0.5, because there’s no state change, just a mempool metric change.


v4.0.4 snapshots:
- https://snapshots.bwarelabs.com/dydx/mainnet/dydx20240409.tar.lz4 (block height 12870104)
- https://polkachu.com/tendermint_snapshots/dydx

Apr 9, 20:21 UTC
Monitoring - To address the issues with sporadic node crashes, validators should consider updating to v4.0.4:
https://github.com/dydxprotocol/v4-chain/releases/tag/protocol%2Fv4.0.4
This is a rolling update.


To ensure that no malicious errors have snuck in your state between the time of upgrade to v4.0.2 and the time you get to running v4.0.4, we suggest you apply a post-upgrade snapshot and run the latest version on top:
- https://mainnet-v4-post-upgrade-snapshot.s3.ap-northeast-1.amazonaws.com/dydx_mainnet_12791712.tar.lz4
- or any other resource that will safely bring your node back to 12791712

Apr 9, 06:11 UTC
Identified - The devs think they've identified the issue. Working on a remedy / recovery steps. Testing internally and a small set of external validators to confirm the fix.

The fix will be more widely available in the next couple of hours.

Apr 8, 20:17 UTC
Investigating - We are currently investigating this issue.
Apr 8, 17:50 UTC
Apr 10, 2024

No incidents reported.

Apr 9, 2024
Apr 8, 2024
Resolved - This incident has been resolved.
Apr 8, 16:53 UTC
Monitoring - A fix has been implemented and blocks are being produced. We are monitoring the results.
Apr 8, 16:23 UTC
Update - FIX AVAILABLE TO SOFTWARE BUG CAUSING CHAIN HALT

Use the new binary v4.0.2 (https://github.com/dydxprotocol/v4-chain/releases/tag/protocol%2Fv4.0.2) to fix the state corruption issue. This fix is not a state-breaking change.

Recommended approach: Use a snapshot taken at post-upgrade height (12791712) and run v4.0.2 . The post-upgrade snapshot has correctly persisted the state for block 12791712 and this snapshot was generated using v4.0.2

Snapshot Link: https://mainnet-v4-post-upgrade-snapshot.s3.ap-northeast-1.amazonaws.com/dydx_mainnet_12791712.tar.lz4

NOTE FOR VALIDATOR NODE OPERATORS

If you are running a validator node, please reach out to dYdX Operations Services at infrastructure@dydxopsservices.com for recovery instructions or to take part in the co-ordinated recovery.

Apr 8, 15:32 UTC
Update - The issue continues to be investigated. It's been agreed to reconvene with the validators around 15:00 UTC.

This means that the devs won't suggest a workaround or a fix until then so that the validators won't get jailed for not being online when the chain restarts.

Apr 8, 09:21 UTC
Investigating - The issue is being debugged. Updates will appear here.
Apr 8, 06:50 UTC
Completed - The scheduled maintenance has been completed.
Apr 8, 06:30 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 8, 05:30 UTC
Scheduled - The dYdX Chain will undergo a protocol upgrade that could potentially disrupt functionalities around the time of the upgrade block height: https://www.mintscan.io/dydx/block/12791712

Important upgrade notes:
~~~ !!! ~~~
Validators that use tmkms or Horcrux need to update to the latest version of the tools PRIOR the upgrade height
- tmkms v0.14.0 : https://github.com/iqlusioninc/tmkms/releases/tag/v0.14.0
- Horcrux v3.3.0 : https://github.com/strangelove-ventures/horcrux/releases/tag/v3.3.0
~~~ !!! ~~~

Upgrade proposal: https://www.mintscan.io/dydx/proposals/46

Apr 7, 16:21 UTC
Apr 7, 2024

No incidents reported.

Apr 6, 2024

No incidents reported.

Apr 5, 2024

No incidents reported.

Apr 4, 2024

No incidents reported.

Apr 3, 2024

No incidents reported.