In Bitcoin Unlimited, on line 1185, in graphene.cpp, at https://github.com/BitcoinUnlimited/BitcoinUnlimited/pull/1511/files, there is a seemingly insignificant constant, nTimeToWait. On the morning of December 3, the value of this constant caused the Bitcoin Cash network to reorganize and discard a valid block - but upon further examination, the consequences could have been much worse.
The origin of this issue originates from a decision made by us in June 2015, when Michael Marquadt (/u/theymos), owner of the bitcointalk.org forums, many reddit "subreddits," and several cryptocurrency news websites, decided to split the community by banning discussion of larger blocksizes on his platforms. We were banned from his forums for ignoring his rules. When the Bitcoin Core developers failed to denounce his tactics, we feared that Bitcoin Core could allow code to be slipped in to do something like drop blocks with Coinbase transactions containing "censored text," and looked for an alternate mining client. The first one to become widely available when we began implementing SHA-256 mining was Bitcoin Unlimited.
Over time, we developed software that is tested to work with Bitcoin Unlimited, and therefore it became a risk to change to any other client. For example, a geth upgrade caused us to be unable to mine Ethereum blocks for a few hours, so we had to revert to an older version and still don't understand what caused the problem. That's despite the geth developers claiming that the code was a "minor update." Since Bitcoin Unlimited has worked for years, we don't want to mess with it, and continue to mine several SHA-256 coins with it despite many Bitcoin Cash miners apparently mining with Bitcoin ABC.
As it turns out, Bitcoin Unlimited Cash does not support all of the "compact block" transfer protocols that other Bitcoin Cash clients do. To get around the problem, a hack was inserted: Bitcoin Unlimited will wait an average of 10s to see if a bandwidth-saving transfer protocol that it supports has a block available from another node. If not, then the original protocol that re-transmits all the transactions in the block is used instead. That 10s doubles because two nodes need to communicate.
On December 3, however, this simple line caused a major issue with Bitcoin Cash that could have affected almost every business that used the protocol. Another pool, btc.top, found Bitcoin Cash block 559307 about 8 seconds before we also found a block at the same height, as indicated at https://www.blocktrail.com/BCC/blocks/orphans/1. The download timer caused notification of that block to be received after our block was found, which triggered our "orphan mining" code. This code doubles the expected value of mining a coin after we identify that one of our blocks will be orphaned, because if we find the next block, we will confirm the previous block as well as the subsequent one. The function caused all SHA-256 hashrate to be diverted to mining Bitcoin Cash. In addition, once our published profitability rose, bots that monitor it noticed and diverted a large amount of hashrate to the pool, likely from cloud mining rental services. It was now profitable to pay even wildly inflated prices on places like NiceHash and earn profit by directing hashrate here. The increased hashrate succeeded in orphaning btc.top's block and reorganizing the Bitcoin Cash blockchain across the network.
In this case, the only loser was btc.top, because they ceased mining their chain at that point. Our "orphan mining" code ensures that the two blocks we mine include all of the transactions in the orphaned block, so exchanges depending on confirmation would simply see that their transactions possibly moved up in height by one block.
Originally, I believed that this issue was a one-off incident that would have been resolved with one pool or the other losing two blocks in the worst case. But after further examination, I discovered that there are many more bots monitoring our expected payouts than I had thought. Had a second block been found by some other pool following btc.top's chain and our now hugely expanded pool during the 20s window caused by the hardcoded delay, then the system would have tripled the expected profitability and continued on. It's likely that even more hashrate would have been sent to the pool to mine Bitcoin Cash. Earlier that week, 200PH/s of additional SHA-256 hashrate suddenly appeared when Bitcoin SV became extremely profitable, so it's reasonable to expect that all available rentable cloud miners in the world could have been pointed to the pool if we were paying 3x bitcoin profitability.
Given that nobody was present to monitor the system at that time, humans would not have known to resolve the issue. We have notifications to notify of downtime, but why would anyone think to write a notification for when profit was high? With 20s delays, the odds of the problem escalating to a two-block race between two networks with similar hashrate were as high as 3.3%, and the odds of a three-block crisis were 1 in 900. After three blocks, the debt owed to customers would have grown so high that it would no longer have been legally possible for us to simply "give up," just as it is not legally possible for us to decide to refund money accidentally sent with ridiculously high transaction fees (like the $100,000 Verge transaction fee last Christmas Day - Congratulations to superminer dmdllc2017) because that money is already owed to customers. A fork of Bitcoin Cash could have occurred as a result.
Bitcoin forked due to a different issue in early 2013, and the solution then was for miners to agree on one of the chains as valid. Even if some kind of agreement to compensate miners on one side of the Bitcoin Cash chain were reached, commerce on sites like Purse.io would have been disrupted until a solution was reached.
The surprising part is that if you take the math even further, the odds were just 1 in 27,000 that a three-block fork would have occurred due to this constant in Bitcoin Unlimited. Bitcoin Cash blocks occur every 10 minutes, so a critical situation was likely to occur twice per year. Given that Bitcoin Cash has been around for 1.3 years, the only reasons that the network did not fork is because of good luck, because pools have been willing to give up against their economic interests in situations like this, or because they never bothered to implement the code to improve their profitability by building upon their orphaned blocks.
Note that the odds are not affected by how many pools run Bitcoin Unlimited, because there is so much cloud mining hashrate available that it will all go to any pool that promises large payouts after the race begins. Furthermore, our algorithm ensures that it never reverses normal transactions; it only assigns the mining rewards to different people. It's much easier to implement code that ignores the transactions on both forks and just mines whatever - or, to purposely use the delay to engage in criminal activity.
While most people in cryptocurrency state that "competing implementations" are an important goal of client development, this occurrence should demonstrate that the consensus rules are not the only important code to adhere to in new clients. This delay was a far more serious problem than is widely recognized. Fortunately, user Jonathan Toomim contributed a pull request to Bitcoin Unlimited that reduced the timer to an average of 1s in the latest version, just 10% of what it was in the previous version, which will make this scenario far more unlikely in the future.
Comments (3)
Archived discussion. New comments and replies are closed.