Last weekend, just after we had expanded from another round of performance constraints, PROHASHING was targeted by denial-of-service (DoS) attacks. This post will focus on two areas: how we eliminated the attacks, and how the success or failure of a business often comes down to customer service.
In the past, I wrote that DoS attacks, particularly distributed DoS attacks (DDoS), are one of the critical problems that will face society in the next decade. Currently, attacks mostly cause websites to go down. In a decade, attacks will probably cause your augmented reality goggles to cut out while you are in the middle of a party with people across the globe, or cause you to think more slowly when your neural interface can't obtain the data it needs. In 20 years, the attacks will cut off the communications between your nanobots while they are in the process of repairing your heart attack, causing you to die.
The major issue with DDoS attacks is that the cost of mitigating the attacks is greater than that of executing them. Businesses can afford to mitigate attacks, but the common person will have little defense when life without the Internet becomes unimaginable.
The attacks on our system began on Thursday, June 16, when a huge amount of traffic was directed at Verizon's network. Some sort of automatic protection must have kicked in after 10 minutes, allowing no traffic to reach the system. Chris called Verizon's support line, and during the interactive voice response (IVR) system's checks, he noticed that the system was accessible again. Everything returned to normal, and at the time, it was not clear that an attack had occurred. The connectivity issues continued for several hours, and because Chris didn't know that an attack was occurring, he spent time investigating other potential problems with the system. Every time the connection cut out, he found that the Verizon system's automated checks would resolve the problem without his needing to talk to a customer service representative.
On Friday night at the same time, the same condition recurred, but this time Verizon's routers at the edge of their network blocked traffic to the system and did not restore access when Chris called customer service. It was at this point he was able to determine that an attack might have occurred, when Chris came up with the theory that a large amount of traffic had caused the networking equipment at Verizon to fail. He was able to get a customer service representative on the line, but the representative was surprisingly incompetent about the primary service they provide (i.e. Internet networking.) Chris found, however, that the other IP addresses had not been deactivated, so he switched one of the servers and its DNS record to that IP address and the system returned to normal.
On Saturday night, the attacks occurred again, and this time they took out another IP address. Chris ran out of energy and fell asleep, simply being unable to continue working due to exhaustion. When he woke up, he called Verizon and was able to talk to an agent who actually got in contact with a technician. They troubleshot the blocked IP addresses for two hours and could not identify the cause of the problem. Chris was told that they would have to escalate the case to their highest level of support and that it would take two days to resolve.
Chris then asked whether he could simply purchase additional IP addresses so that he could figure out a mitigation technique, and ignore this problem for now. Verizon said that the department that handled sales could only take requests on Mondays through Fridays from 8:30 to 5:00. It was at this point that a third address failed due to whatever flaw was in Verizon's equipment, and Chris took the system offline because it could not function without at least three of the five IP addresses we possessed being available.
Chris then contacted Cloudflare, an attack protection provider. Their website was extremely confusing and written in "business speak," with few technical details about how what services they actually provide. He called them to try to get someone on the phone to actually learn about what they did, and they took two days to return his call. While he gave up with Cloudflare long before then, he eventually found out that Cloudflare is only able to protect websites because its limited number of IP addresses relies on HTTP request hostnames to forward the requests.
Later that Sunday he settled upon Javapipe, a provider that actually detailed what services they provided on their site. Our first thought was that we needed their standard DDoS protection server, which simply retransmits good packets it receives with the same source and destination IP addresses. We would directly reply as if the packets coming from us actually originated from them. However, we found after modifying our servers that some router upstream was blocking the packets we rewrote to appear to be coming from the Javapipe server. A support ticket to Javapipe suggested that we set up a virtual private network (VPN), which would send traffic in both directions through their servers so that the packets coming from us weren't blocked by the upstream router.
Having talked with Javapipe, Chris ordered a virtual private server (VPS) and configured it with OpenVPN to route all traffic to our servers and back out again. When Verizon's business office returned on Monday, he was finally able to get their sales department to allocate new IP addresses (they never did determine why those other IPs still don't work.) But when he started the system and as miners started to connect, he found that OpenVPN was singlethreaded and the system's core was overloaded. The core were overloaded because Javapipe's "latest processors" were actually 1st generation Core machines, and hardware encryption didn't become available until the 2nd generation machines were released.
Therefore, he had to order three more cores and set up three more OpenVPN tunnels, and then balance the load through all of them on Tuesday. Once he accomplished that, CPU load on each core dropped to just 25%, eliminating the random dropouts that had occurred as packets were unable to be processed.
Next, we found that the WAMP server that provides live profitability data was overloading the now CPU-limited connection because a small number of customers were making many connections. Additionally, we discovered that the messages being sent were inefficient, with it being possible to reduce their size by 95% through sending differences since the last message, rather than the entire message again. I began work on a "difference engine" to only send the differences between the last piece of data and the current piece, but in the meantime Chris discovered that Crossbar has an option for websocket compression. When he enabled it, traffic coming out of the Crossbar WAMP server was reduced from 60Mbps to 2Mbps, a 30x decrease. I scrapped the "difference engine" because it was believed that the risk of introducing new bugs would outweigh a further reduction in bandwidth.
The attacks have continued over the past few nights, but they are now ineffective. After about 2s of downtime, Javapipe's services begin to filter the traffic and everything returns to normal. As with the x11 mining release, we found that when one goes looking for bugs, one tends to find them. The bugs that one finds are often issues that have been around for a while that nobody had known about. The WAMP server bandwidth overload explained many issues that previously made little sense.
In the end, it took about one week to get the system back to normal operation. The downtime was about 4 days in total. The opportunity cost in lost revenue from money not earned during the downtime, customers who will take a long time to return, customers who will never return, the cost of all the protection services, the errors and bugs introduced by hasty introduction of untested code into a production system, the delay of SHA-256 mining by two weeks to deal with the attacks, thefts due to the delay in implementing E-Mail notification on payout address changes on the website, and the increased number of orphaned blocks caused by the VPN latency is estimated to be about half a million dollars. The attacks are easy to execute because the attackers likely spent a trivial amount in comparison, perhaps a thousand dollars. The problems could have been worse, however: the attacks happened at night on a weekend, when we were available to respond for as long as necessary to solve the problem. Even though Verizon business would have been open if the attacks had occurred at, say, 10:00am on a Monday when Chris was sleeping and I was unavailable, they might have been missed and we would still be thinking that the only problem was an upstream router issue.
While Chris learned a lot about how to prevent and respond to attacks, the most interesting lessons are those relating to business.
First, DDoS attacks are particularly troubling to cryptocurrency businesses, because they can't host their services on high-bandwidth shared servers. We have 370 coin wallets with a hundred thousand dollars in them at times, and there are too many horror stories of insider jobs to take any risk of someone at a hosting facility pulling a drive and copying keys, or even looking into the memory of a virtual machine. Additionally, social engineering is ineffective when you are the only person who has access to the computer. Our only choice is to have every physical server in our possession, where nobody else can modify configurations or trick someone else into modifying configurations.
But because our systems are on our own servers, and the only thing on our servers is our systems, they need to be hosted nearby. Not only do we need to be able to make sure they are still there, we need to be able to go to the location so that they can be upgraded when necessary. Last-mile bandwidth is expensive. Thus, a VPN is necessary to connect from a high-bandwidth "DDoS protected" location to a low-bandwidth one. While the arrangement drives up orphan rates (a loss of about $80 per day at current estimates), every pool should suffer from this same problem, leaving an equal playing field. If your pool is using a shared server in Romania to store its data, then the increased profits from the reduced orphan rates are being paid for in the expected value of your money being lost due to poor security.
A additional lesson is that companies that try to hide what they do behind nice pictures and simple catchphrases on their websites lose business. Chris and I have taken a simple approach to our dealings lately: if we can't understand exactly how what somebody does works, then we aren't going to buy it from them. We shouldn't have to look through hidden pages of simple text to find basic numbers and metrics. Cloudflare and Coinbase's GDAX are examples of companies that do this. Never require a customer to click "view demo" to see what services are actually offered!
More importantly, Chris and I are more convinced than ever that the quality of customer service is the most important factor in a business's success or failure. We specifically avoided Cloudflare because we figured that if their customer service representatives know nothing more than what they read from a script, then their hiring practices are likely to be to pay just enough to get a minimally qualified person in every position. We knew that it would be impossible to contact someone knowledgeable when an attack actually occurs. Furthermore, a company in their field should know that attacks need to be mitigated quickly, not by a sales agent returning a Saturday evening call on Monday night.
Verizon's support was also unacceptable for a company that wants to retain customers. Telecommunications is a field where Sprint will buy out an AT&T phone contract just to get a customer to switch at a loss, because they know that customers are likely to remain with a provider for years. In our case, Verizon dug up the ground to the office where the servers are located and paid technicians to run equipment through a concrete wall. This investment will be lost when we cancel our service with them, as Chris has already called Comcast Enterprise Solutions, who promised him that they actually have people available on the weekends who will respond within 15 minutes. Comcast has scheduled a conference call on the afternoon of June 26 to negotiate a price. We figure that if we can avoid just one day of downtime over the next year, it will be worth paying a thousand dollars per month more. If you want to put a number to performing 5 minutes of maintenance on a Sunday, it's worth $12,000 per year.
The amount of money these companies are losing is what strikes me most. If a large customer that was willing to pay thousands of dollars to get back online immediately called me, I would be doing whatever is necessary to help him. But when Chris called a nearby datacenter knowing that they charged that much and asking what services they provided, he never received a call back.
Javapipe's customer service is unbelievable, and the fact that it is always possible to contact them within 15 minutes through their ticket system means that the only reason we would ever consider switching from them would be if their services are insufficient for potential larger attacks in the future. At one point, Chris said that his business was going to go offline if they did not get to his ticket within the hour, and they actually escalated the ticket and activated the new services he requested.
I believe we are still in a better position than many of the companies in the cryptocurrency industry. Besides the obvious benefits of having resolved yet another reliability issue with our system, we discovered how easy it is to gain customers in general. Even if Prohashing becomes too unprofitable to continue when the current cryptocurrency bubble crashes, all you have to do in any business is to answer the phone, know what you're talking about, and be honest. The number of companies who can't do these basic things is astounding. If you can respond to tickets quickly, your product can be inferior to what the competition provides and you will get a lot of customers. Javapipe provides less bandwidth than Cloudflare, but we would never consider Cloudflare because of our experience with them.
Looking around the cryptocurrency industry, one can see businesses and industries that are succeeding, and those that aren't, and it's obvious why. Ethereum is succeeding because its founder treats people respectfully, is knowledgeable about what he's doing, and tells the truth. Bitcoin is failing because most Core members lie and attack people who would otherwise contribute. Likewise, Novaexchange is growing exponentially and we are their largest customer. Meanwhile, I closed my Coinbase account because Coinbase doesn't reply to support tickets - plain and simple. Clevermining never listed their names or contact information their website, and even threatened us if we didn't pay 42 bitcoins to acquire them, and they ended up folding because people respect those who are willing to show their faces and stand behind their actions.
In conclusion, we successfully mitigated a number of denial of service attacks. In the process, we learned that companies live and die not only by the quality of their products, but also by the quality of their customer service. We plan to set up a ticketing system to better track customer service requests, and will be hiring somebody to respond to tickets more quickly during the Monday-Friday, 8-4 timeframe. If you get hit with attacks, we highly recommend Javapipe. If you prefer to use a different provider, then whether you can get in touch with someone when something goes wrong is more important than the raw numbers. Finally, if you're buying into ICOs or investing in companies, pay close attention to how the owners treat people, as organizations that ignore or belittle their customers fail quickly.
In the past, I wrote that DoS attacks, particularly distributed DoS attacks (DDoS), are one of the critical problems that will face society in the next decade. Currently, attacks mostly cause websites to go down. In a decade, attacks will probably cause your augmented reality goggles to cut out while you are in the middle of a party with people across the globe, or cause you to think more slowly when your neural interface can't obtain the data it needs. In 20 years, the attacks will cut off the communications between your nanobots while they are in the process of repairing your heart attack, causing you to die.
The major issue with DDoS attacks is that the cost of mitigating the attacks is greater than that of executing them. Businesses can afford to mitigate attacks, but the common person will have little defense when life without the Internet becomes unimaginable.
The attacks on our system began on Thursday, June 16, when a huge amount of traffic was directed at Verizon's network. Some sort of automatic protection must have kicked in after 10 minutes, allowing no traffic to reach the system. Chris called Verizon's support line, and during the interactive voice response (IVR) system's checks, he noticed that the system was accessible again. Everything returned to normal, and at the time, it was not clear that an attack had occurred. The connectivity issues continued for several hours, and because Chris didn't know that an attack was occurring, he spent time investigating other potential problems with the system. Every time the connection cut out, he found that the Verizon system's automated checks would resolve the problem without his needing to talk to a customer service representative.
On Friday night at the same time, the same condition recurred, but this time Verizon's routers at the edge of their network blocked traffic to the system and did not restore access when Chris called customer service. It was at this point he was able to determine that an attack might have occurred, when Chris came up with the theory that a large amount of traffic had caused the networking equipment at Verizon to fail. He was able to get a customer service representative on the line, but the representative was surprisingly incompetent about the primary service they provide (i.e. Internet networking.) Chris found, however, that the other IP addresses had not been deactivated, so he switched one of the servers and its DNS record to that IP address and the system returned to normal.
On Saturday night, the attacks occurred again, and this time they took out another IP address. Chris ran out of energy and fell asleep, simply being unable to continue working due to exhaustion. When he woke up, he called Verizon and was able to talk to an agent who actually got in contact with a technician. They troubleshot the blocked IP addresses for two hours and could not identify the cause of the problem. Chris was told that they would have to escalate the case to their highest level of support and that it would take two days to resolve.
Chris then asked whether he could simply purchase additional IP addresses so that he could figure out a mitigation technique, and ignore this problem for now. Verizon said that the department that handled sales could only take requests on Mondays through Fridays from 8:30 to 5:00. It was at this point that a third address failed due to whatever flaw was in Verizon's equipment, and Chris took the system offline because it could not function without at least three of the five IP addresses we possessed being available.
Chris then contacted Cloudflare, an attack protection provider. Their website was extremely confusing and written in "business speak," with few technical details about how what services they actually provide. He called them to try to get someone on the phone to actually learn about what they did, and they took two days to return his call. While he gave up with Cloudflare long before then, he eventually found out that Cloudflare is only able to protect websites because its limited number of IP addresses relies on HTTP request hostnames to forward the requests.
Later that Sunday he settled upon Javapipe, a provider that actually detailed what services they provided on their site. Our first thought was that we needed their standard DDoS protection server, which simply retransmits good packets it receives with the same source and destination IP addresses. We would directly reply as if the packets coming from us actually originated from them. However, we found after modifying our servers that some router upstream was blocking the packets we rewrote to appear to be coming from the Javapipe server. A support ticket to Javapipe suggested that we set up a virtual private network (VPN), which would send traffic in both directions through their servers so that the packets coming from us weren't blocked by the upstream router.
Having talked with Javapipe, Chris ordered a virtual private server (VPS) and configured it with OpenVPN to route all traffic to our servers and back out again. When Verizon's business office returned on Monday, he was finally able to get their sales department to allocate new IP addresses (they never did determine why those other IPs still don't work.) But when he started the system and as miners started to connect, he found that OpenVPN was singlethreaded and the system's core was overloaded. The core were overloaded because Javapipe's "latest processors" were actually 1st generation Core machines, and hardware encryption didn't become available until the 2nd generation machines were released.
Therefore, he had to order three more cores and set up three more OpenVPN tunnels, and then balance the load through all of them on Tuesday. Once he accomplished that, CPU load on each core dropped to just 25%, eliminating the random dropouts that had occurred as packets were unable to be processed.
Next, we found that the WAMP server that provides live profitability data was overloading the now CPU-limited connection because a small number of customers were making many connections. Additionally, we discovered that the messages being sent were inefficient, with it being possible to reduce their size by 95% through sending differences since the last message, rather than the entire message again. I began work on a "difference engine" to only send the differences between the last piece of data and the current piece, but in the meantime Chris discovered that Crossbar has an option for websocket compression. When he enabled it, traffic coming out of the Crossbar WAMP server was reduced from 60Mbps to 2Mbps, a 30x decrease. I scrapped the "difference engine" because it was believed that the risk of introducing new bugs would outweigh a further reduction in bandwidth.
The attacks have continued over the past few nights, but they are now ineffective. After about 2s of downtime, Javapipe's services begin to filter the traffic and everything returns to normal. As with the x11 mining release, we found that when one goes looking for bugs, one tends to find them. The bugs that one finds are often issues that have been around for a while that nobody had known about. The WAMP server bandwidth overload explained many issues that previously made little sense.
In the end, it took about one week to get the system back to normal operation. The downtime was about 4 days in total. The opportunity cost in lost revenue from money not earned during the downtime, customers who will take a long time to return, customers who will never return, the cost of all the protection services, the errors and bugs introduced by hasty introduction of untested code into a production system, the delay of SHA-256 mining by two weeks to deal with the attacks, thefts due to the delay in implementing E-Mail notification on payout address changes on the website, and the increased number of orphaned blocks caused by the VPN latency is estimated to be about half a million dollars. The attacks are easy to execute because the attackers likely spent a trivial amount in comparison, perhaps a thousand dollars. The problems could have been worse, however: the attacks happened at night on a weekend, when we were available to respond for as long as necessary to solve the problem. Even though Verizon business would have been open if the attacks had occurred at, say, 10:00am on a Monday when Chris was sleeping and I was unavailable, they might have been missed and we would still be thinking that the only problem was an upstream router issue.
While Chris learned a lot about how to prevent and respond to attacks, the most interesting lessons are those relating to business.
First, DDoS attacks are particularly troubling to cryptocurrency businesses, because they can't host their services on high-bandwidth shared servers. We have 370 coin wallets with a hundred thousand dollars in them at times, and there are too many horror stories of insider jobs to take any risk of someone at a hosting facility pulling a drive and copying keys, or even looking into the memory of a virtual machine. Additionally, social engineering is ineffective when you are the only person who has access to the computer. Our only choice is to have every physical server in our possession, where nobody else can modify configurations or trick someone else into modifying configurations.
But because our systems are on our own servers, and the only thing on our servers is our systems, they need to be hosted nearby. Not only do we need to be able to make sure they are still there, we need to be able to go to the location so that they can be upgraded when necessary. Last-mile bandwidth is expensive. Thus, a VPN is necessary to connect from a high-bandwidth "DDoS protected" location to a low-bandwidth one. While the arrangement drives up orphan rates (a loss of about $80 per day at current estimates), every pool should suffer from this same problem, leaving an equal playing field. If your pool is using a shared server in Romania to store its data, then the increased profits from the reduced orphan rates are being paid for in the expected value of your money being lost due to poor security.
A additional lesson is that companies that try to hide what they do behind nice pictures and simple catchphrases on their websites lose business. Chris and I have taken a simple approach to our dealings lately: if we can't understand exactly how what somebody does works, then we aren't going to buy it from them. We shouldn't have to look through hidden pages of simple text to find basic numbers and metrics. Cloudflare and Coinbase's GDAX are examples of companies that do this. Never require a customer to click "view demo" to see what services are actually offered!
More importantly, Chris and I are more convinced than ever that the quality of customer service is the most important factor in a business's success or failure. We specifically avoided Cloudflare because we figured that if their customer service representatives know nothing more than what they read from a script, then their hiring practices are likely to be to pay just enough to get a minimally qualified person in every position. We knew that it would be impossible to contact someone knowledgeable when an attack actually occurs. Furthermore, a company in their field should know that attacks need to be mitigated quickly, not by a sales agent returning a Saturday evening call on Monday night.
Verizon's support was also unacceptable for a company that wants to retain customers. Telecommunications is a field where Sprint will buy out an AT&T phone contract just to get a customer to switch at a loss, because they know that customers are likely to remain with a provider for years. In our case, Verizon dug up the ground to the office where the servers are located and paid technicians to run equipment through a concrete wall. This investment will be lost when we cancel our service with them, as Chris has already called Comcast Enterprise Solutions, who promised him that they actually have people available on the weekends who will respond within 15 minutes. Comcast has scheduled a conference call on the afternoon of June 26 to negotiate a price. We figure that if we can avoid just one day of downtime over the next year, it will be worth paying a thousand dollars per month more. If you want to put a number to performing 5 minutes of maintenance on a Sunday, it's worth $12,000 per year.
The amount of money these companies are losing is what strikes me most. If a large customer that was willing to pay thousands of dollars to get back online immediately called me, I would be doing whatever is necessary to help him. But when Chris called a nearby datacenter knowing that they charged that much and asking what services they provided, he never received a call back.
Javapipe's customer service is unbelievable, and the fact that it is always possible to contact them within 15 minutes through their ticket system means that the only reason we would ever consider switching from them would be if their services are insufficient for potential larger attacks in the future. At one point, Chris said that his business was going to go offline if they did not get to his ticket within the hour, and they actually escalated the ticket and activated the new services he requested.
I believe we are still in a better position than many of the companies in the cryptocurrency industry. Besides the obvious benefits of having resolved yet another reliability issue with our system, we discovered how easy it is to gain customers in general. Even if PROHASHING becomes too unprofitable to continue when the current cryptocurrency bubble crashes, all you have to do in any business is to answer the phone, know what you're talking about, and be honest. The number of companies who can't do these basic things is astounding. If you can respond to tickets quickly, your product can be inferior to what the competition provides and you will get a lot of customers. Javapipe provides less bandwidth than Cloudflare, but we would never consider Cloudflare because of our experience with them.
Looking around the cryptocurrency industry, one can see businesses and industries that are succeeding, and those that aren't, and it's obvious why. Ethereum is succeeding because its founder treats people respectfully, is knowledgeable about what he's doing, and tells the truth. Bitcoin is failing because most Core members lie and attack people who would otherwise contribute. Likewise, Novaexchange is growing exponentially and we are their largest customer. Meanwhile, I closed my Coinbase account because Coinbase doesn't reply to support tickets - plain and simple. Clevermining never listed their names or contact information their website, and even threatened us if we didn't pay 42 bitcoins to acquire them, and they ended up folding because people respect those who are willing to show their faces and stand behind their actions.
In conclusion, we successfully mitigated a number of denial of service attacks. In the process, we learned that companies live and die not only by the quality of their products, but also by the quality of their customer service. We plan to set up a ticketing system to better track customer service requests, and will be hiring somebody to respond to tickets more quickly during the Monday-Friday, 8-4 timeframe. If you get hit with attacks, we highly recommend Javapipe. If you prefer to use a different provider, then whether you can get in touch with someone when something goes wrong is more important than the raw numbers. Finally, if you're buying into ICOs or investing in companies, pay close attention to how the owners treat people, as organizations that ignore or belittle their customers fail quickly.
Comments (9)
Archived discussion. New comments and replies are closed.