Bitmex Status

The incident is now resolved

No update message available.

No update message available.

No update message available.

No update message available.

No update message available.

REST API queries for trading related data (read-only HTTP GET requests, not order management requests) took longer to return than usual over a period of 23 minutes from 18:13 UTC to 18:36 UTC. The root causes of both issues have been identified and we are working on permanent fixes to prevent a recurrence. We apologise for any inconvenience this may be causing. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

From 06:53 to 07:03 UTC today, September 3rd, 2018, the service of our websocket API was impaired, which impacted the live data updates on the [bitmex.com](http://bitmex.com) website as well as clients connected directly to our websocket API. As part of a scheduled software release on the trading engine, our market data distribution component became saturated with a cascade of updates during the restart process which temporarily prevented downstream data distribution on the websocket API. We apologise for the disruption. The root cause has been identified and mitigated in the production environment. BitMEX engineers have already deployed a fix for the underlying issue to prevent a re-occurrence. We are also improving a number of alerting mechanisms to enable us to recover from potential issues in a more timely manner.

From 14:29 to 15:26 UTC today, August 4, 2018, access to [BitMEX.com](http://BitMEX.com) and the BitMEX API was severely impaired. A large spike in load caused an overload in a critical internal system used for rate limiting and sessions, creating a feedback loop in that system where data was recalculated more often than usual, exacerbating the load. The feedback loop has been fixed and systems have returned back to normal. BitMEX engineers are working to separate these systems entirely, so that these particular non-critical operations cannot affect processing time of critical trade requests and logins. We apologise for the disruption. BitMEX engineers have disabled the cause of the feedback loop and are working the weekend to separate these critical systems so that such an event cannot occur again.

Today, May 17, 2018, the BitMEX trading engine encountered several separate and heretofore unpredictable problems, causing feed latency and downtime in spurts throughout the day. Disks mounted to the main trading engine hardware degraded sharply in performance at roughly 10:00 UTC. This degradation caused feed latency during scheduled archive and reindex jobs, which caused significant backpressure. Disk I/O operations were running at roughly 1/20 of their expected rate. BitMEX runs redundant drives, but in this case, both drives were simultaneously exhibiting this degraded behavior. We had no choice but to schedule a maintenance downtime to replace them. Unfortunately, backpressure reached critical levels faster than we expected and we moved up our timetable. At no point was data integrity compromised by this problem, but restoring the machine to a functional state with nominal disk performance took longer than expected to execute and verify. After this action was complete, we restarted trading. Unfortunately, another problem was uncovered during the next archive, where a reindex job combined with a previously rare request pattern led to unexpected index regeneration and symbol revalidation on specific tables. This led to another backpressure scenario, with similar symptoms. We have identified and fixed multiple contributing factors to the above behavior. The trading engine team will be closely monitoring engine performance throughout the day while continuing root cause analysis for the slowdowns.