Update #1.1
from leena
More Info
The community have collected data of the events, as well as suggesting proposals.
The Data
<https://www.notion.so/THORChain-Incident-07-16-e08a7be6cf4c4bce95ba2636759709e2#c5f8922b241c47888767012c89a8af1d>
The Proposals
<https://www.notion.so/Proposals-6eb7f21707f647fe8dbbd5ae5867a044>
These are being reviewed. The current plan allows for flexibility, but still a period of time where the network can be brought online for UTXO chains, with ETH halted until stabilised and solvency addressed.
On the haltTrading Actions by mimir.
Originally halted on blockheight #1480200 (<https://viewblock.io/thorchain/tx/F43F6B3D34E787132C51A64784E8C32ED6F167DF44E91796C9780D9700E14A98>)
Momentarily re-enabled on #1482145
<https://viewblock.io/thorchain/tx/01153E28C6A3B706D8CDD41B259600E02290E50C2FA2713189B6CFE83C29B330>
Halted again #1482401 (25 mins later)
<https://viewblock.io/thorchain/tx/5D215660B98071913E0037913BE9428410BDA72FA400A01701801FABFD310F4B>
On review of the data, this was the incorrect thing to have done, and mimir-admins are reviewing their decision-making processes wrt this incident. mimir-admin is a temporary measure until mimir-node is enabled.
Update #1.0
from leena:
Update and Plan Forward
Background
1) V0.60.0 was released
2) halted=true was done prior, so no swaps were ever made
3) more than 2/3rds updated to V0.60.0, which had a store migration to fix the ETH insolvency
4) 3 nodes were slow to update, which meant the store migration never happened
5) Some LPs began withdrawing (asymmetrically to ETH), so they had a claim on 10k ETH, not 500 ETH
6) This quickly drained the vaults, nodes halted the network again.
How?
THORChain has a policy of never blocking withdrawals in order to ensure that LP funds are always available, AND not updating the store until 100% of nodes have upgraded (to stop consensus failures). This is why (5) was allowed to happen, since there is a window of opportunity from 67% to 100% updated that the old store values exist.
These policies may need to be re-visited, this will be a discussion after return-to-trading. THORChain must be able to secure capital and not allow unfair exploitation by a narrow few (protect the majority from the minority).
Plan
Staged release:
V0.60.1
- ETH Bifrost will not sign any outbounds (the queue will build, LPs may attempt to withdraw - but they will not receive funds)
- Mimir/noops to partially restore state/solvency
V0.61.0
- Chain-specific halting: ETH will be halted for ins and outs
- The Outbound Queue will be purged, LPs who attempted to withdraw will be re-credited their redeemed liquidity_units (put back in the pool)
- Trading will be opened for UTXO chains
- store migration as-required to fix discrepancies
V0.62.0
- Auto-solvency checker
- store migration as-required to fix discrepancies
ETH chain re-enabled for trading once the state is assessed to be correct