On Thursday 23 March 2023, our buying and selling engine was suspended from 12:55 UTC to 13:20 UTC, with orders executed in cancel solely mode throughout this time.
This suspension was triggered by our audit course of detecting a doable inconsistency within the alternate state and halting the alternate as a precautionary measure.
At BitMEX our positions and margins are checked a number of instances a minute. With balances cross-checked towards on-chain data, each 10 minutes. Bugs, flaws or intrusions, inflicting positions to not match, instantly halt our alternate.
We program our alternate to halt, as defending the property of our customers is central to our operations. We don’t compromise safety for comfort, which is why we’ve misplaced zero cryptocurrency. And as communicated throughout the downtime through our Status web page and social media channels, all of our customers’ funds remained secure always.
Following a complete overview of the downtime, we’re offering you, our customers, with a extra detailed abstract of the incident as set out under. Ought to you might have any extra questions, as all the time you possibly can attain us by contacting Support.
What occurred on 23 March?
As shared above this suspension was triggered by our audit course of detecting a doable inconsistency within the alternate state and halting the alternate as a precautionary measure.
A race situation surrounding inside transfers prompted a momentary state inconsistency. The audit course of detected this and took motion – as supposed.
Our engineering workforce had been capable of assess the scenario, determine the basis trigger and deem it secure for buying and selling to proceed.
What was the consumer affect?
The alternate operated in cancel-only mode whereas buying and selling and matching was suspended for 25 minutes. There was some market disruption when the alternate resumed buying and selling.
At present, our assist workforce is responding to these prospects who had been impacted. No prospects had been liquidated because of the market disruption.
What’s being executed to make sure this doesn’t occur once more?
As a 24/7 alternate, we try to supply good uptime and uninterrupted companies, nonetheless this can’t be assured. We don’t make compromises in the case of the protection of our customers’ funds. You possibly can learn extra about how we preserve funds secure here.
A working group has been fashioned to discover how we, at BitMEX, can higher resume buying and selling transferring ahead. Moreover, we’re reviewing the behaviour of the audit course of, to see if full suspension will be prevented below sure strict circumstances.
How can I defend myself towards the dangers related to downtime occasions?
We offer a lot of refined instruments and settings by which it’s doable for merchants to regulate the danger posed to them by downtime occasions.
For merchants who use our API, we provide a “Lifeless Man’s Change” function, which permits merchants to set a timeout for cancellation of orders in case they’re unable to achieve the alternate. For implementation particulars and an instance of setting a timeout, please see the documentation for this function here.
For enhanced safety merchants can set the set off worth of a cease order to reference Final Value, Mark Value or Index Value. When the alternate is reopened after a downtime occasion, dramatic swings within the Final Value can happen. To keep away from cease orders being triggered by †any dramatic swings in Final Value, merchants can set their stops to set off based mostly on the Mark Price or Index Value, quite than the Final Value.
These are simply two instance options of the BitMEX platform that enable our merchants to regulate threat and cut back the affect of downtime. Our Support team can present additional details about these (and another) threat administration instruments on the platform.
Conclusion
We hope this weblog submit offers all customers with a transparent description of this downtime incident and the strong ongoing steps we’re taking to boost platform stability and our general buying and selling expertise.
As all the time, we encourage any involved customers and people impacted by this downtime to contact Support.