PROHASHING's upcoming worldwide architecture

Introduction

On Monday, November 14, 2022, PROHASHING will be taking a dramatic step forward to improve our services for customers worldwide.  The company's new architecture will provide faster load times for the website, better API data, fewer rejected shares, and increased profitability.

However, significant changes to PROHASHING's configuration, and to the configuration of miners that connect to PROHASHING, will be necessary.  This article explains the technical aspects of the upcoming changes in detail, why the changes are necessary, what benefits they will provide, and what you need to do to prepare for this upcoming release.

 

Action Items

If you are simply looking for the minimum summary of how these changes affect you:

• Before Monday, November 14, 2022, you MUST change the connection URL of all workers connected to PROHASHING to a regional mining server.  You can change the URL right now to stratum+tcp://us.mining.prohashing.com:[algorithm specific port], and your workers will automatically be reconfigured on November 14 if you want to "set and forget."
• On Monday, November 14, 2022, all PROHASHING services except for E-Mail, support, and the forums will be offline for the deployment of the architecture and for the yearly database maintenance.  The exact time of day and outage length are not yet known.
• On Monday, November 14, 2022, PROHASHING will cease to offer resets of two-factor authentication keys for customers who are not members of the PROHASHING Trusted Mining Program.  You MUST use Authy to automatically back up your keys, manually back up the keys using other authentication apps, or make sure that whatever authentication app you choose allows you to keep a backup key.  After this date, failure to back up keys will result in permanent loss of your account and unpaid earnings.

 

Adding a Content Delivery Network (CDN) to the website

Improving PageSpeed

One of the major focuses of PROHASHING's work during the second quarter was improvement of website performance, towards which three man-months of work was dedicated.  The work included improvements such as eliminating unused code, enabling features such as HTTP2, adding a new in-memory cache for Javascript files, recompressing images, lazy loading components as they come into view, and even replacing the background rasterized starfield image with a randomly generated SVG that is 10% of its size.

According to Google PageSpeed, PROHASHING's website now has a "largest contentful paint" time of 3.6s on mobile devices, which Google considers acceptable but not exceptional.  This time declined from 21s in June, after months of performance improvements.  The equivalent desktop time has also improved from 12s to 0.9s.  PROHASHING's website load time is now faster than all of the major pools and cloud services we tested, surpassing Slush, NiceHash, Ethermine, and others.

Time to first byte

However, Google's research indicates that website visitors are much more likely to leave websites if the load time exceeds three seconds.  Because the Angular framework itself requires a minimum number of bundled libraries, we do not believe it is possible to achieve the required reduction of 0.6s on mobile devices solely by reducing the amount of code downloaded by the phone.  There is one major area of improvement that was overlooked until now: the "time to first byte" remains at 1.1s, far higher than some of the other pools' sites.  We determined through research that those pools were using a Content Delivery Network (CDN) to serve some webpages.

A CDN reduces the time required to load a website by directing requests to a server closer to the customer's geographic location.  Even if there were no routers in the way to slow things down, light would still require more than a tenth of a second to travel around the world if a fiber-optic cable were somehow stretched around it.  A more realistic latency for a connection from Australia to the United States would be 250ms, so receiving any data at all would require - at minimum - 0.5s, because two-way communication is required.  In reality, multiple round trips are required to set up a truly secure encrypted connection, and multiple files are downloaded, so that latency for an Australian customer could exceed 2.5s before the phone could even begin to execute the site's code.

The 1.1s average of "time to first byte" is high because while US customers can get the first data in 200ms or less, countries outside North America currently need to send data across the oceans and have a less friendly experience that pushes the average up.  When the CDN is implemented, foreign customers will be able to connect to local servers like US customers do, and they will have the same fast page load experience.

How a CDN works

A CDN works by using the Internet's Domain Name System (DNS.)  DNS works by pointing domains, like prohashing.com, to IP addresses.  For most sites, all DNS servers in the world copy the same data, such as how every DNS server in the world pointed prohashing.com to 50.220.121.209 at the time this article was first posted.  When the CDN is released, the CDN provider will work with Internet Service Provider (ISP)s so that every country, and sometimes every region within a country, resolves a different IP address for prohashing.com.  The webservers at those IP addresses will have copies of content that doesn't change, like PROHASHING's background stylesheets.

For data that does change, and for data that needs to be secure, like current profitabilities and payout data, the CDN passes the requests through to the dedicated PROHASHING servers in State College, Pennsylvania.  Therefore, data that cannot cause a loss of money, like source code and stylesheets, can be distributed worldwide, and wallets, passwords, and customer information never leaves our own servers that have not been hacked in 9 years.

Need to change URLs

Unfortunately, using a CDN comes with a drawback: it will no longer be possible to mine on the prohashing.com domain.  The reason is a fundamental limitation in the design of the Internet: IP addresses are stored in the DNS system and ports are not.  If miners continued to mine on the prohashing.com domain, we would need to be able to install mining servers on every single one of the CDN provider's servers.  Even if the CDN provider allowed that, the providers sometimes have more than 300 regions, maintaining so many mining servers would cost so much that fees would have to be doubled or tripled.

Unfortunately, PROHASHING did not foresee the need for a CDN when the company was launched on December 23, 2013.  At the time, we thought that using a different domain for mining like many other pools did was needlessly complex and simplification would bring more customers.  Unfortunately, that simplicity brought limitations.

To allow there to be one location for mining servers and a CDN for the website, mining needs to be moved to a different domain, and we have decided upon us.mining.prohashing.com.  However, as you'll see in the next section,  although there will not be 300 locations, we will indeed be introducing multiple locations for mining.

 

Expanding mining to worldwide servers

The current architecture

Currently, all customers connect their workers to the main prohashing.com servers in State College, Pennsylvania.  The single location for mining presents some of the same issues that a single location for hosting a website does - latency is high in locations far away from the main servers.  For mining, being close to a server is even more advantageous, for the following reasons:

• Individual workers receive fewer rejected shares from closer servers because a new block is less likely to be discovered by the mining servers during the time it is waiting for the worker's share to arrive
• The pool can offer higher profitability, because its orphan rate is lower if found blocks are transmitted to other peers more quickly.

The shortcomings of high latency can be addressed similarly to the way the CDN works for the website, by adding additional servers around the world.  For mining, however, adding servers is more complex because of the need to accurately assign workers to the most profitable coin for each algorithm.

Regional mining servers

PROHASHING faces unique challenges that are not present for single-coin pools.  A pool that mines bitcoins and only bitcoins can make copies of itself on multiple servers around the world, and the only significant technical problem is when and how to centralize all the found shares and blocks between all the copies.  In a simple form, the shares could simply be stored in a text file, and then the text files on all servers downloaded by a simple program once per hour that inserts it into a database connected to a website.

PROHASHING needs to be aware of how many customers are assigned to all of the coins on an algorithm, so that the system can determine what the expected reward is for customers mining that algorithm.  A simple solution to this problem is to have all of the regional servers communicate with the primary mining servers at the State College location, which is the strategy currently used by the EU server that PROHASHING operates.  Unfortunately, this solution isn't complete because share data and found block data also needs to be sent to the main servers and a response received before coins can be reassigned, leading to stale shares.

The ideal solution is to do as much computation as possible on the new regional mining servers and to only communicate with the primary servers for data that is not latency-critical.  PROHASHING has achieved that with the upcoming release.  In the initial regional mining servers release, the following time-sensitive data will be handled locally within each region:

• Hashing and checking shares to determine whether they are valid
• Responding to clients immediately with whether the shares are valid without contacting the primary servers
• Receiving block data from coin daemons installed locally and connected to both the primary servers' daemons and also to daemons geographically close to the regional server
• Sending the signal to submit found blocks to daemons on all regional servers and primary servers simultaneously, regardless of the region where the worker that found the block is connected

Meanwhile, the following data that is not time-sensitive will be processed by communicating with the primary servers:

• Computing the value of shares and crediting balances to customers
• Computing what price PROHASHING can expect to receive for the next coin it would mine for each block, and sending it to the regional servers every few minutes
• Sending proportions of customers assigned to each coin back to the primary servers for share crediting when the next shares are credited
• Sending data about profitability and number of connected workers to the website's REST API

Security

The reason it took 9 years to release the worldwide architecture was largely because until now, we were not able to figure out how to design the worldwide system to maintain the same level of security as currently exists.  In particular, there was no easy way to prevent private keys and debts owed from being hosted on servers we don't physically control.  Changing or copying either of these values could result in a loss of money.

The worldwide distributed architecture cracked the security problem by figuring out how to minimize the data actually located on the regional servers.  One key insight is that mining can be conducted with encrypted wallets on the regional servers, while the encryption key never leaves the primary servers.  Another is that almost all of the share processing information only needs to go in one direction - from the regional servers to the primary servers - after which it can be removed from memory.  Therefore, crooked employees of the hosting company could, at worst, modify the source code to impact future earnings, which would likely trigger existing failsafes and which could not impact existing data.

Expansion plans

As stated in the "summary" section, the primary servers are now connectable at us.mining.prohashing.com in parallel with the existing connection methods.  During the week of October 31, we will begin a phased rollout of a new eu.mining.prohashing.com regional mining server located in Poland.

On November 14, the current eu.prohashing.com server will be permanently retired, as it will then be obsolete due to the new low-latency design.  The DNS record for eu.prohashing.com will be deleted and those workers will need to change to eu.mining.prohashing.com.  At the same time, the website CDN will make prohashing.com inaccessible for mining (but not to website visitors,) and customers who wish to use the primary mining servers will need to use us.mining.prohashing.com.

From the customer's point of view, selection of a mining server will not impact profitability or service quality for any reason other than latency.  You should test the latency of each regional server to determine which one has the lowest latency and use that one.  If you want to use the existing primary servers, or if you just want to make changes now so you don't forget about it before November 14, you can change your workers to us.mining.prohashing.com right now.

Future worldwide expansion will be explored depending upon the uptake of the EU regional server.  If uptake is high, the current schedule is for a third regional mining server to be brought online in Singapore during the week of December 11.

 

Yearly system maintenance

Consolidating the yearly server work

Historically, PROHASHING has conducted server maintenance during the US Thanksgiving week.  For four out of the eight years this work has been performed, it has been done on the Tuesday of US Thanksgiving week.  In 2022, we will be advancing the maintenance by one week so that all of the tasks can be performed at once.  A secondary benefit to moving up the tasks by a week is that when the CDN is released, the DNS system will require time to propagate the changes throughout the world, and customers living in regions where the changes had not yet been propagated would be unable to connect anyway.  This propagation time can occur while the system is offline.

Tasks to be performed

During the 2022 downtime, as occurred in previous years, Chris will be upgrading PostgreSQL, this time to version 15.  PostgreSQL has been focusing recently on performance improvements, and this year's query performance improvements will make a significant difference in our 400-line queries, like the one used to compute active rules for advanced payout proportions.

Additionally, Chris will upgrade all server hypervisors to Debian 11, which will allow for cost-cutting due to its lower electricity usage.  In the current environment of high wage inflation, cost cutting is critical to be able to keep up with market pay, and we expect to save at least $500/year in electricity costs simply by running the latest version of the Linux kernel that is included in Debian 11.  Unlike the virtual machines, which were already upgraded, we cannot upgrade the primary daemons, and then the backup daemons, while keeping one virtual machine online at all times, as the hypervisor takes all virtual machines offline, and therefore the hypervisor upgrade will add to the downtime.

Impact on customers

The exact time and duration of the downtime is not yet able to be estimated because we have not yet evaluated whether there are any additional hardware-related tasks, like swapping disks or memory modules, that can also be performed at the same time.  Once a task schedule is identified, we will update customers as to the downtime length.

We do not currently anticipate the forums, E-Mail, and support system being offline during this downtime, although these services may also be added to the upgrade list if it turns out that the hardware hosting them needs to be upgraded.

 

Two-factor authentication changes

End of support

While not at the same impact level as the other changes occurring on November 14, another cost-cutting measure that PROHASHING has unfortunately identified is the need to cease support for many two-factor authentication reset requests.  

During an examination of cost-cutting and time-saving measures, Chris identified that approximately 90% of two-factor authentication reset requests are currently unable to be completed successfully, despite this type of ticket requiring as many as 20 minutes per incident to research and resolve.  The most common reason for the tickets remaining incomplete is simply that the customer stops responding during the reset process for an unknown reason.  There are also other cases where PROHASHING simply does not possess identity information to actually validate against what the customer is providing, which is why only Trusted Mining Program members will be allowed to submit these requests in the future.  Finally, PROHASHING stores its identity documents on an offline SD card so that hackers cannot gain access to the data. Reading the card on an offline computer is a time-consuming process, and disclosure is risked if the computer reading the data were to later be connected to the Internet.  Connecting the card in a "write-only" mode except during the January tax season is much more secure.

Impact on customers

Beginning on November 14, 2022, PROHASHING will no longer assist customers who lose access to two-factor authentication keys and who are not members of the PROHASHING Trusted Mining Program.  If you lose access to your two-factor authentication keys, you will permanently lose access to your account, and may not be able to be paid any remaining balances. This is similar to how, if you lose the keys to your cryptocurrency cold wallet, you permanently lose access to the wallet, which is why it will be so important for customers to keep track of backup codes for their 2 factor authentication apps.

Preventative measures

Customers can easily prevent any impact by this policy change by doing one of the following:

• Recommended:  Use Authy to store your two-factor authentication keys.  Authy performs automatic backups, so that the destruction of a disk or of a phone will not cause any data loss.
• If you choose an authentication app other than Authy, make sure that the app allows you to write down a backup key for a case where your device becomes lost, damaged, or stolen.
• Manually copy the files storing the keys from other two-factor authentication programs to a backup device.  This procedure may be insecure and is out of the scope of this article.
• Not recommended:  Disable two-factor authentication on your account, which is insecure but which will prevent you from losing access due to a lost key.

 

Conclusion

PROHASHING is moving forward with dramatic service improvements for our customers around the world.  While some short-term configuration changes are required by all customers to support these improvements, the end result will be a faster website and more profitable mining pool.  If you aren't clear what you need to do, or if you see a technical reason why your workers are unable to make the required changes, Sarah Manter will be happy to support you; just E-Mail help@prohashing.com, or click on the support button in the footer of the website to ask her a question.

We look forward to serving you over the next month as these exciting changes are rolled out to everyone!