The most common type of support ticket we receive involves "Internet connectivity problems." For years, we performed investigations into these issues under the assumption that there was either a customer connectivity problem, or that a bug was present in the mining servers that only emerged with that customer's specific mining configuration. An example of the type of bug that might cause such problems would be the use of a new type of mining equipment, as occurred when the Antminer S17 was first introduced - the S17 didn't follow the stratum protocol as defined. We've also determined that some customers report standard mining server reboots for bugfixes or upgrades as "connectivity issues."
As our new staff members have become more trained and capable, we've started to investigate whether what this general cause of problems can be addressed by looking into other causes rather than specific configurations. Through this research, we've decided to spend significant resources in the coming months to move as close as possible to the goal of 100% uptime, even though reaching perfection is unattainable.
Why does mining uptime have to be close to 100%?
"Uptime" in this discussion refers exclusively to mining server uptime. The consequences of the website, trading services, forums, E-Mail, API, and other services being offline are much less severe than a mining server outage. One reason - that customers don't earn money when they aren't mining - is obvious, but there are other more subtle issues created when the mining servers disconnect for even a brief period. For one, some ASICs switch to a "backup pool" and then don't switch back to the primary pool until the backup pool disconnects, so a one-minute outage can result in the customer mining at the backup pool until the backup pool restarts its servers. More seriously, there are now very poorly designed ethash ASICs that require more than 35 minutes to restart after a mining server outage of any duration.
The system doesn't have to be "offline" to be considered as downtime, either. Coins can go into error because an exchange's API stops providing price data, which affects miners statically mining one coin or dynamically or statically mining that algorithm. The only solution in this case is to disconnect customers who are mining that coin. Another case when a coin can go into error is when the daemon itself fails, which is the situation addressed by the coming release.
"Backup daemons"
The next release of PROHASHING's mining servers will support "backup daemons." Currently, when a call, like "getconnectioncount," is made to a coin, it is sent to the one daemon running on the coins server. If the call succeeds, the action dependent upon it is taken; if not, then the most likely outcome is that the coin is placed into error and workers are assigned to another coin or disconnected. Reasons why such calls could fail include having too few peers, a lack of confidence the system is mining the correct blockchain, disk corruption (as was silently occurring for years due to a failed RAID controller), or upgrades and maintenance.
The last reason - upgrades and maintenance - is particularly problematic because upgrades to coins often include rescanning the blockchain or catching up from the blocks that occurred while the upgrade occurred. For Ethereum, it's rare that an upgrade requires fewer than five minutes. PROHASHING has no control over when these upgrades occur, so without the "backup daemon" solution we could never approach perfect uptime even if we had no other problems. Upgrades can also be required to Debian, and these operating system issues can't be skipped if they are security vulnerabilities.
How it works
During the week of July 11, Chris will install duplicate copies of the most critical coins like Bitcoin, Litecoin, Ethereum, ZCash, Ethereum Classic, and others on a different coin server. Once finished, exact copies of each of the most critical coins will exist on two servers, although the coin servers will not have all of the same coins on them.
Once finished, a new release of the mining server will be issued that sends the same command to all duplicate daemons for that coin when more than one daemon is installed. The first successful response will be used, and the remaining calls will be discarded. Likewise, for the blocks coming from the "blocknotify" command, the template response with the highest transaction fees for a coin at a given height will be sent to workers, and all others will be ignored so as not to cause a work restart (this feature already exists and required no additional changes.) In the case that one daemon returns an error or times out but one other succeeds, that error will be ignored. If all daemons return an error, then the error is acted upon as before.
If "getblocktemplate" is called as above, and one daemon errors "daemon is still synchronizing" while the other returns a template, then that (coin, algorithm) combination can now continue operating with no interruption even though one of the daemons is in an error state.
Upgrading systems and coins
The upgrade path for a new coin version becomes more complex with "backup daemons." First, one instance of the coin is prevented from communicating with WAMP or the mining servers. Then the upgrade process is performed and tested. If the upgrade occurs before the coin's hard fork, the output will be identical to the older version. Once tested, the daemon is allowed to communicate with the servers, and then the other daemon is upgraded using the same procedure. Zero downtime occurs during this process, compared to a disconnect and several minutes of lost mining in the previous version.
Operating system upgrades that require a reboot usually mandate a downtime of at least 30 minutes because of the massive computational power and disk usage required for coins to get and validate the latest blocks after a restart. Now, an entire server can be rebooted with no impact to mining, regardless of how long the server takes to restart.
Future expansions
If we were designing a new mining server from scratch, we would not have designed "backup daemons" in the way we did - we could have tracked which daemons were online after one coin failed and saved significant bandwidth rather than sending duplicate calls for every operation. However, given that we are continually improving a 9-year-old system, implementing the fewest changes possible is always our target, even if doing so results in less-than-optimal performance or code quality.
Therefore, the entire process of "backup daemons" was designed to be backwards-compatible. Once the initial version of "backup daemons" exclusive to the mining servers is released, the website, trader, and other services will continue to exclusively use the primary daemon for their operations. Later, Michael and Miguel can modify the website to call the backup daemons in the case they receive no response from the primaries, but website reliability is of a lower priority than mining reliability and the uptime focus will remain on the mining servers. An additional new feature planned during the next few weeks is monitoring for when a daemon for an algorithm's "anchor coin" goes offline. That way, we can be notified in real-time when the backup daemons, not only the primary daemons, are not responding, and investigate before a simultaneous failure occurs.
Another feature planned for 2023 where the backwards-compatibility is actually a feature rather than a hinderance is the ability to deploy daemons to cloud servers in other regions of the world. Because the payout system only knows about the primary daemon, the backup daemons on the cloud servers will always have encrypted wallets and by default the password to decrypt them will never be transmitted outside the local network.
Conclusion
Over the coming months, PROHASHING plans to focus on eliminating as many sources of worker disconnects as possible. We already eliminated a memory leak that had been present since 2018 and which caused the mining servers to crash every six days.
Our near-term plans involve the ability to hot swap code, so that mining server restarts that do not involve adding or removing fields in objects in memory can be issued without a restart. Eventually, we will separate the codebase so that a "connection manager" process is created; the only purpose of this "connection manager" is to ensure that workers continue mining the current block template for up to 30 seconds in the case that the rest of the mining server falls offline. These "connection managers" will be so simple so as to be able to remain online for years because the code will be well-understood and the risk of bugs will be low.
Comments (0)
Archived discussion. New comments and replies are closed.
No archived comments on this article.