Updating the xx network to protect against DDoS attacks (and an urgent xx network MainNet update)

Updating the xx network to protect against DDoS attacks

The team needs to deploy an update to the xx network to protect it from DDoS attacks before the xx messenger launch, and as a result we will be releasing an update on Tuesday, December 21st.

The Problem

Most privacy systems have a fairly poor time against DDoS attacks, specifically attacks that send an overwhelming amount of ostensibly valid traffic into the system. The underlying issue is that anonymizing traffic is very expensive (both computation and bandwidth). In most efficient privacy systems, it is much more expensive for the system than the submitter. This asymmetry can be exploited to easily overwhelm the system with traffic, ensuring it is not available to those who want to use it.

Overall, the xx network is uniquely situated to solve this issue. Due to the integration of a blockchain with the cMix mix network, the xx network can use economic incentivization (i.e. paying for sending messages with xx coins) to ensure cost for usage of the xx network is homologous with the work the network puts in. You can read more about this system in the xx network white paper in the section, “Postage”. 

This solution has 2 issues:

  • The systems to pay for usage of the mixnet are not yet in place
  • The system needs to have a free tier to allow general availability of untraceable communication.

The second issue ensures that even when deployed, economic incentivization is not sufficient. The moral imperative of universal access to privacy (for example through the xx messenger) demands a significant level of free usage be publicly available. As a result, another solution must be found.

The Solution

This is a particularly difficult problem, and solutions like mini proofs of work on send seem to work, but only create burdens in the thousands of USD a day to overcome and saturate the network.  

After spending significant time on this problem, we have settled on a solution based upon the scarcity of IPv4 addresses.  This is not a fool proof solution, and would likely be overcomable by nation-state level adversaries at the scale of the xx network currently, but we believe it to be effective against most adversaries the network will face in the near future. 

Essentially, the solution is to rate-limit, across the entire network, all IP addresses. Gateways will share which IPs sent messages through them, and as a group aggregating them to ensure that no IP can send more than 4000 mix-packets a day (more on this later).  Given the system currently processes 216,000,000 mix packets per day, this means one would need 54,000 separate IPv4 addresses to saturate the system. And, given the network is expected to grow, this should be at 108,000 IPv4 addresses shortly (at 550 nodes).

At this scale, some companies, ISPs, and nation-states would have the capacity to overwhelm the system, as well as the larger botnets, but all of which would expend significant cost. It is also possible to permanently ban IPs (via the tech committee)  which are engaging in malicious behavior, so attacks are somewhat of a one and done affair.  

The network will almost certainly grow. For example, with a full implementation of the pipelining feature, as well as growth of the network, a requirement of millions or more IPv4 addresses at scale is expected.  

How?

After every round is submitted to a team for processing, the gateway of the first node in the team will gossip a list of sender IDs and IP addresses to the network. These will propagate through, and every gateway will add those senders to structures called “leaky buckets” which track usage of the network. When a client attempts to send a message, these “leaky buckets” will be checked to ensure they are allowed to send, and reject them if they have exceeded their limits.

This solution runs into a problem pretty quickly, at full tilt the gossip payloads are large and the system subtends ~50 Mib/s of bandwidth per our simulation. A pretty large sum when considering the 100Mib/s target for gateways.  

Gossip protocols are generally tuned to distribute information quickly, and it turns out that when tuned to be slower, they use much less bandwidth. Given that DDoS attacks are longer than a second or two, such tuning is actually fine for this application. 

With this slower approach to gossip, bandwidth can be kept below 5MiB/s, an acceptable amount in the current network.

Wait, rate limiting of clients?

Yes, clients (and IP addresses) will be rate limited within the xx network. The current target is 4000 messages per day (About 4 megabytes of private traffic). This is a somewhat soft limit. All clients and IP addresses will have a 28,000 pool of messages worth of traffic available to them, refilling 4,000 messages a day. Furthermore, the team has been hard at work ensuring that picture and voice sending is properly compressed within the xx messenger. This means that users will be able to send dozens of pictures and 10+ minutes of voice messages a day without eating into their pool. All well beyond the average usage of messaging clients.

Future work

It is possible to reduce the bandwidth used by this rate limit gossip even further. Gossip protocols are designed to distribute data to the entire network, but what if they werent? 

Normally gossip protocols have each node in the network retransmit a message multiple times, but if instead the payloads themselves track how often they are retransmitted, you can build a “bad gossip”, which generally only delivers a message to a fraction of the network. This structure also uses a fraction of the bandwidth, and is in fact perfect for our application. 

Because of the thousands of messages available to send, and the unpredictableness of the gossip protocol, it is possible to only receive, say 1/3rd of rate limiting info and then assume the rest. So on each gateway, the limit would for example be 1333 messages, but clients wouldn’t generally hit that limit until they send 4,000 messages. 

This solution drastically reduces the bandwidth used, down to less than 1MiB/s. We have not implemented this solution at this stage because it would require modifying the gossip protocol’s internals, and the team is on a tight timeline to release the xx messenger.  

Network Update

We will be updating the network to add this rate limit on Tuesday, December 21st. This has been tested fully on internal test nets.

As is the norm, please keep the discussion to the MainNet Forum so it is easier for the team and community to track.

The MRs can be found here:

https://git.xx.network/xx_network/primitives/-/merge_requests/21

https://git.xx.network/elixxir/comms/-/merge_requests/32

https://git.xx.network/elixxir/gateway/-/merge_requests/37

https://git.xx.network/elixxir/client/-/merge_requests/83

https://git.xx.network/elixxir/client/-/merge_requests/85

Share on facebook
Share on twitter
Share on linkedin
Share on reddit
Popular