Bitmex outage history

We are investigating an issue causing a delay to withdrawals. Deposits are currently unaffected. We ensure users that funds are safe, and withdrawals will be processed as soon as the issue is identified and resolved. If you have any further questions please reach out to customer support.

We have identified the issue and are working on a fix.

We have implemented a fix and are processing withdrawals. We apologise for any inconvenience caused.

The withdrawal backlog has been cleared and systems are operating as expected.

We have identified an issue where some users would have seen incorrect information or balances reflected in their account UI. • **Fund Security:** Please rest assured that **all user funds are completely safe and secure.** This is strictly a frontend display issue. • **Fix deployed:** Our engineering team has identified the root cause and restored the correct UI data.

All systems operating normally - we are continuing to monitor

The incident is now resolved

We are currently experiencing issues with connections via Cloudflare Bypass. We are investigating and will update when the cause of the issue has been identified. Standard connections are unaffected.

We have identified the issue with Cloudflare Bypass connections and have addressed the issue. We will continue to monitor these connections.

Connections are restored. Monitoring complete

Users in certain regions are unable to access the platform due to an issue with our CDN configuration.

The issue has been resolved, all systems are operating normally

We have identified an issue preventing the processing of withdrawals to a completed state, and are actively working on a fix. We would like to assure you that all funds remain safe.

We have cleared the backlog of withdrawals and continue to work on a permanent fix.

The fix has been implemented and withdrawals are now flowing as expected.

The issue is now resolved

We are observing increased error rates for the websocket in Testnet

We have detected increased error rates in our authentication service. Logins and API calls may result in errors. We are investigating and will post an update soon

We have identified the issue and are implementing a fix

We have pushed a fix and have observed error rates returning to normal. We will continue to monitor

This issue is now resolved

Between 04:52:11 UTC and 04:53:47 UTC on 20 December 2019, the trading engine stopped all operations as a result of an infrastructure related issue. No data was lost during this time, and upon resolution 95 seconds later the trading engine continued normal operations. As there was no activity in the trading engine during this time, there were no data updates published via the Websocket API. During this 95 second period, requests made via the REST API \(e.g. new/amend/cancel order requests and queries for public or private data\) returned an HTTP status 500 response \(server side error\). Similarly, new subscription requests via the Websocket API during this time returned an error. The infrastructure issue was identified shortly before the incident via our internal alerting and resolved by 04:53:47 UTC. Root cause of the issue has been identified and steps implemented to prevent a similar issue from recurring. We have improved our infrastructure monitoring to detect and alert about potential issues much sooner.

The issue has been resolved. We apologise for any inconvenience this may be causing. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

We are observing the interruption of the trading engine. We apologise for any inconvenience this may be causing. We are looking into the cause of this issue, and will be sharing an update as soon as possible. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

The issue has been resolved. We apologise for any inconvenience this may be causing. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

Between 21:30 UTC 25 November 2019 and 07:05 UTC 26 November 2019 the Websocket API was running with degraded capacity. This resulted in slightly more latent feeds during this time and some isolated occurrences of substantial lag during traffic spikes correlated with large market moves. The issue was identified when latency thresholds were breached in our automated monitoring systems around 06:55 UTC. From 07:05 the issue was resolved and full capacity was restored. We continue to monitor the impacted services closely. The issue was caused by incorrect CPU pinning following a release of a market data distribution component at 21:30 UTC 25 November 2019. The impact of this was only observed during large traffic spikes which occurred several hours later and so was not identified during the post release checks at the time. The configuration of this service has been corrected and the deployment tested to prevent this from happening again.

We are observing a slight increase in latency in the feeds data transiting through the websocket API. We apologise for any inconvenience this may be causing. We are looking into the cause of this issue, and will be sharing an update as soon as possible. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

We are observing an interruption in several feeds of websocket API. We apologise for any inconvenience this may be causing. We are looking into the cause of this issue, and will be sharing an update as soon as possible. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

Between 10:44:13 and 10:52:29 UTC on 24 July 2019, the realtime data on the following websocket API feeds was interrupted: account, affiliate, execution, funds, instrument, margin, order, position, trade, transact, wallet Users of the BitMEX website may have noticed that some data was not updating during this period; e.g. in the Recent Trades panel, Open Orders panel, Fills panel, and Position panel. The following public feeds were unaffected during this period: funding, insurance, liquidation, settlement, impactQuote, impactQuoteBin1m, quote, quoteBin1m, quoteBin5m, quoteBin1h, quoteBin1d, tradeBin1m, tradeBin5m, tradeBin1h, tradeBin1d, orderBookL2\_25, orderBook10, orderBookL2 The REST API was also unaffected and client-side data loss due to missed feed updates during this period can be resolved using the REST API. During this period we continued to process all order instructions and the trading engine was unaffected. The outage was due to a slow consuming internal market data service which became degraded, putting pressure on our market data infrastructure. This internal market data service has been suspended whilst the root cause is being investigated. Additionally, several safety mechanisms are currently being deployed to prevent a potential recurrence from impacting user-facing feeds.

The issue has been resolved. We apologise for any inconvenience this may be causing. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

Between 07:50 and 07:58 UTC on 27 June 2019, the following websocket API feeds were interrupted due to a complication during a planned upgrade of our market data distribution services: Account, affiliate, execution, funds, instrument, margin, order, position, trade, transact, wallet Users of the BitMEX website may have noticed some data not updating during this period e.g. in the Recent Trades panel, Open Orders panel, Fills panel, and Position panel. The following public feeds were unaffected during this period: Funding, insurance, liquidation, settlement, impactQuote, impactQuoteBin1m, quote, quoteBin1m, quoteBin5m, quoteBin1h, quoteBin1d, tradeBin1m, tradeBin5m, tradeBin1h, tradeBin1d, orderBookL2\_25, orderBook10, orderBookL2 During this period we continued to process order instructions and the trading engine was unaffected. Due to this issue, data in a subset of data mirrors which service user REST API requests was left in an incomplete state. A side-effect of this was that some users observed stale open orders on the BitMEX website for orders which were already cancelled for a period of 90 minutes whilst data was being restored. Any API users that may be missing updates for this period can now backfill data via the REST API.

The issue has been resolved. We apologise for any inconvenience this may be causing. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

We are observing an interruption in several feeds of websocket API. We apologise for any inconvenience this may be causing. We are looking into the cause of this issue, and will be sharing an update as soon as possible. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

At 21:09:00 UTC 25 June, we released an update to our API layer that inadvertently started to count WebSocket subscriptions to certain tables against the request rate limit that had otherwise been exempt. This update may have impacted customers who heavily utilise the WebSocket API. Once the issue was identified at 00:19 UTC 26 June, we immediately rolled back the update to bring systems back to normal. We apologise for any inconvenience this may have caused. To read more about which subscriptions are exempt from the request rate limiter, see our previous blog post \[[https://blog.bitmex.com/update-to-our-realtime-apis-image-delivery/](https://blog.bitmex.com/update-to-our-realtime-apis-image-delivery/)\] for details.

The issue has been resolved. We apologise for any inconvenience this may be causing. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

We have identified an issue where certain users of the websocket API issue may incorrectly have been rate-limited. We apologise for any inconvenience this may be causing. We are looking into the cause of this issue, and will be sharing an update as soon as possible. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

We have identified that the orderbook for the ETHUSD product is an incorrect state. We apologise for any inconvenience this may be causing. We are looking into the cause of this issue, and will be sharing an update as soon as possible. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

The issue has been resolved. We apologise for any inconvenience this may be causing. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

Between 09:25:54 UTC and 09:44:30 UTC 24 June 2019 the orderBookL2, orderBookL2\_25, orderBook10, and quote realtime websocket feeds for ETHUSD were in a degraded state. During this period, the state of the ETHUSD orderbook on these feeds was incorrect. We were able to identify and resolve the root cause of the issue within a minute of detection. The issue was caused by a rare sequence of order events that triggered a bug in an optimisation of the orderBookL2 calculation which had been deployed to the production environment several hours earlier. This change has since been reverted. There was no impact to orders in the trading engine itself – just the presentation of the calculated orderbook for ETHUSD downstream of the trading engine. We have deployed additional automated feed validators to detect potential similar issues in the future and to alert us earlier.

We have identified an unusually high rate of auto-deleveraging events. We apologise for any inconvenience this may be causing. We are looking into the cause of this issue, and will be sharing an update as soon as possible. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

The issue has been resolved. We apologise for any inconvenience this may be causing. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

On 2 April 2019 between 04:44:34 UTC and 05:22:08 UTC, less than 200 positions were auto-deleveraged due to the sharp price movements of the underlying mark price on XBTU19 and ETHM19. At the time of these auto-deleveraging events, the Insurance Fund allocated to these contracts was minimal. The Insurance Fund is allocated individually to each contract according to how many liquidations contribute to that specific contract \(System Gains and Losses\). In the case of expired contracts, BitMEX has a process in place to roll over the Insurance Fund allocated to these contracts into the next front month contract. With the recent expiry on the 29th March 2019, this process failed and front month contracts did not receive their reallocation, and the funds remained unallocated. As a result, a handful of users were auto-deleveraged upon large liquidations within these affected contracts. BitMEX receives auto-deleveraging reports and was made aware of the unusually high rate of auto-deleveraging events, at which point we investigated the matter. We identified the root cause, corrected the allocation and put further controls in place to ensure that reallocation failures are automatically flagged internally. For users that were affected, BitMEX will be reaching out to you personally to explain the situation and document your compensation. We compensated users based on the maximum potential profit that they would have made over the timeframe of these auto-deleveraging events. We exited these users at the best price of each contract: longs at 5,079.5 on XBTU19 and shorts at 0.03103 on ETHM19. BitMEX did not profit from these auto-deleveraged positions.

We have identified a minor outage for approximately 15 seconds. We are looking into the cause of this issue, and will be sharing an update as soon as possible. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

On 29 March at 12:00 UTC, BitMEX experienced a minor outage for approximately 15 seconds whereby all requests would have been load shed as the engine was blocked during settlement operations. The platform was back to normal after the 15 second outage.

The issue has been resolved. We apologise for any inconvenience this may be causing. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

The issue has been resolved. We apologise for any inconvenience this may be causing. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

We have identified a disfunction of the BitMEX website, preventing a small group of users from placing orders via the website. The API is left unaffected. We are looking into the cause of this issue, and will be sharing an update as soon as possible. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

The issue has been resolved. We apologise for any inconvenience this may be causing. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

We have observed an interruption of the trading engine. We apologise for any inconvenience this may be causing. We are looking into the cause of this issue, and will be sharing an update as soon as possible. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

On 19 February at 05:31 UTC, BitMEX experienced a minor outage for approximately 1 minute whereby all trading engine operations were suspended. This issue occurred due to a sustained period of data transfers between the internal market data distribution components. This was part of a regularly scheduled update to improve the overall resiliency of the platform. The root cause has been identified and a fix via internal processes has been put in place to prevent a recurrence. Additionally, we are continuing to re-work our market data distribution architecture to eliminate any potential impact to the trading engine in such a scenario.

Between 05:40 and 07:11 UTC today, a subset of the requests to the BitMEX REST API experienced slow API responses and eventual API timeouts due to resource contention at the API layer. Upon detection via our internal alerting mechanisms we identified the cause and mitigated the immediate impact within a few minutes. There is currently no ongoing issue and there was no impact to the trading engine or user data during this time. Fixes for the underlying root cause of the issue have been identified and are being worked on as a priority. We will follow up with another announcement once these are live. We have also increased the sensitivity of our system monitoring to detect and resolve potential similar issues much sooner. We apologise for any inconvenience this may have caused.

The issue has been resolved. We apologise for any inconvenience this may be causing. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

We have identified that a subset of the requests to the BitMEX REST API experienced slow API responses. We apologise for any inconvenience this may be causing. We are looking into the cause of this issue, and will be sharing an update as soon as possible. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

Earlier today, the [testnet.bitmex.com](http://testnet.bitmex.com) website and the API were not available for approximately 16 minutes from 19:42 to 19:57 UTC. We identified the cause, made adjustments and service has resumed immediately. We apologise for any inconvenience this may have caused

The issue has been resolved. We apologise for any inconvenience this may be causing. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

We have identified an interruption of service in Testnet. We apologise for any inconvenience this may be causing. We are looking into the cause of this issue, and will be sharing an update as soon as possible. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

The issue has been resolved. We apologise for any inconvenience this may be causing. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.

At 05:48:10 UTC and 06:10:10 UTC, BitMEX experienced minor outages for approximately 30 seconds whereby requests to the trading engine were load-shed as the engine was busy. During these times, clients would have observed a lack of updates over the WebSocket API for the same reason. The outages were due to data replay complications during a regularly scheduled market data distribution component restart. There was no data loss during these events and an additional safety mechanism to prevent a similar situation from impacting the trading engine has already been deployed. The root causes have also been identified and we are currently working on permanent fixes to prevent a recurrence. Updates will follow in the future. On 15 Jan 2019, we successfully released an enhancement to our internal market data distribution component's re-subscription logic. This addresses the root cause of the previous week's issue and along with the additional safety mechanism to prevent impact to the trading engine deployed at the time, we don't anticipate a reoccurrence of last week's issue.

We are observing a series of short trading engine interruptions. We apologise for any inconvenience this may be causing. We are looking into the cause of this issue, and will be sharing an update as soon as possible. If you have any further questions, please contact us via https://www.bitmex.com/app/support/contact.