Deribit called upon the cloud networking knowledge of CloudNation to help extend MultiCast market data to AWS. CloudNation consultants Hugo van Duijn and Joost Wolfsen share their knowledge in this blog how to deploy this solution based on the newest Amazon technology.

The partnership of Deribit & CloudNation

After the initial stretch of Deribit trading services over a private low latency solution to AWS, the crypto platform now also enabled market data through multicast feeds to its customers in AWS. Until recently Deribit customers that wanted to receive the Deribit Multicast service needed to be co-located or cross-connected in Equinix LD4. Because of hard work, the service is now also available for customers in the AWS London and Tokyo regions.

Due to recent developments in the Multicast support within the AWS platform, it became a logical move to extend Multicast service to AWS. In order to do this, we used a model similar as the current TCP based trading service, which had been extended to AWS by Advanced AWS Consulting Partner CloudNation in 2021. Deribit called upon the cloud networking knowledge of CloudNation again to deploy this solution based on the newest Amazon technology.

The Multicast challenge

Because the Multicast service is a great success for customers that are cross-connected or co-located, Deribit wanted to extend this service to their customers that are located in the AWS cloud. Preferably in the form of a low latency solution that is stable, doesn’t require maintenance often and is simple to manage. And above all: it should be able to get the Multicast feeds from the Deribit datacenter to the customers.

The solution should explicitly not be using the PIM (Protocol Independent Multicast) routing protocol for connections to customers, because it causes a lot of management overhead. Instead, it should rely on IGMPv2 to handle Multicast group memberships. Furthermore it should be possible to have control over which AWS accounts are able to receive Multicast traffic and which ones shouldn’t, preferably administrated with the Terraform infrastructure as code solution used for the existing AWS deployment.

Another requirement was added when the first tests showed a successful proof of concept in the eu-west-2 (London) AWS region. This was the extension of the service to the AP-Northeast-1 (Tokyo) AWS region where Deribit already has a presence delivering a different service to their customers.

Tried solution with Juniper

One of the major limitations was that DirectConnect does not support Multicast traffic. AWS released a document with several reference architectures for Multicast networking with AWS Transit Gateway on May 5th 2022 to work around this, which was the trigger to start this project. One of the reference architectures describes how Multicast traffic can be sent to an on-premise data center, by using GRE tunneling and PIM multicast routing.

The other relevant architecture describes how the AWS Transit gateway can be used to send Multicast traffic from a VPC to a VPC in another AWS account by using shared Transit Gateway attachments.

Based on these architectures a start was made on getting multicast traffic from Deribit in Equinix LD4 to AWS. This was deemed the hardest step and failure to achieve this goal might jeopardize the whole project. Since it was not yet sure if all requirements could be met, it was decided to build a proof of concept for transporting Multicast traffic into AWS via a GRE tunnel.

To setup GRE tunneling between LD4 and AWS you will need third party networking solutions in AWS which are found on the AWS marketplace. Well known vendors such as Fortigate, Cisco and Juniper have offerings on the marketplace usually in the form of a virtual appliance which is then deployed as an EC2 instance in your AWS account.

First choice for this proof of concept was Juniper, which showed favorable pricing when compared to other providers of GRE capable marketplace appliances. Furthermore the Juniper documentation on GRE tunneling and multicast traffic was quite extensive, as well as being a platform that the CloudNation engineers had experience with.

After deploying a Juniper appliance in the Deribit AWS VPC it was quite simple to build a GRE tunnel which connected to the Deribit hardware in LD4 over the existing Direct Connect between AWS and Deribit LD4. Once the tunnel was up, we unfortunately discovered that the Deribit hardware in LD4 did not have the functionality needed to route Multicast traffic over the GRE tunnel. So this reference architecture was discarded.

That meant going back to the drawing board for the part of the architecture that is responsible for getting Multicast Traffic into AWS. The second reference architecture for sharing multicast traffic with other accounts was still useable though.

Docker containers and TCP encapsulated Multicast

Since getting Multicast Traffic inside the VPC using DirectConnect and GRE was not an option, the Multicast had to originate from within AWS. By having both Deribit and CloudNation engineers brainstorm together and really think out of the box, a new direction was discovered. Deribit had already created a container using an MCAST-RELAY application which could output Multicast and consume encapsulated Multicast in TCP packets, which were sent from datacenter and received through DirectConnect and Transit Gateway.

We chose to run the container in Amazon Elastic Container service (ECS) based on a design which is described in Blog: Running multicast-enabled containers on AWS.

Some very important takeaways were :

• Fargate does not support multicast, EC2 Linux deployment mode must be used;
• The ECS containers need to use the awsvpc networking mode;
• IGMPv3 is default but not supported so ECS EC2 instances had to be bootstrapped with the code below: