While there are other WebRTC solutions on the market, Subspace stands out from the rest. At Subspace, we realized the public internet was never intended for today’s real-time interactions. In seeking to solve this problem, we brought together seasoned experts from across many disciplines—distributed systems, network engineering and design, Internet peering and policy, and TCP—to create a better, more performant internet designed for WebRTC applications.
Why the Internet Cannot Work for Real-time Applications
The internet is fundamentally broken for real-time applications. The internet was created as a reliable way to transmit information. Designed as a distributed system, each network operator directly decides through Border Gateway Protocol (BGP) policy the best way to route traffic through its network. Not all operators are aligned. Some may send outside traffic over longer, higher-latency paths while routing its own traffic through better performing paths. In some cases, there is a financial benefit for the operator to route outside traffic on a less optimal path, often charging a premium for the optimal paths.
Internet-Grade Routers and Problems with Soft-failure Detection
Even if you don’t consider the financial incentives that some operators might have, response times and the resolution of faults across the public internet is poor in our opinion. While modern optical networks can move traffic off a failed fiber path in milliseconds, there is a significant gap in the detection and response for soft failures. A soft failure is one in which a component in a hardware device has begun to fail but has not fully failed. Soft failures result in an undesirable and even non-deterministic impact to the traffic traversing the affected device. A key characteristic of a soft failure is that it typically goes unnoticed until it completely fails.
In terms of complexity, internet-grade routers differ dramatically from the average home router. They usually occupy a sizable physical footprint, consume a significant amount of power, and can weigh several hundred pounds.
Examples of these internet-grade routers include:
Regardless of the manufacturer, internet-grade routers follow a similar general architecture—a chassis containing multiple line cards, one or more control plane modules, and a backplane that interconnects the line cards and control modules. Line cards are the modules where network circuits, such as copper or fiber-based Ethernet or Wavelength Division Multiplexing (WDM) connect. At a high level, the public internet is a large web of connections between devices like this all over the world.
Line cards commonly contain a number of optical transceivers, such as Quad Small Form-factor Pluggable (QSFP) 28s that operate at 100 Gigabits per second (Gbps). QSFP28s use a laser to transmit data onto a fiber-optic strand.
Internet-grade routers rarely connect a single QSFP or circuit. Instead, there are hundreds of these transceivers on a single chassis spaced closely together. Each transceiver contains a laser and generates a non-trivial amount of heat. As with any electronic device, heat is a factor to a component’s lifespan, so transceivers are often one of the first components to fail.
However, the transceiver usually fails slowly over time, beginning by causing small levels of sustained packet loss and gradually increasing over time until it fails completely. This can take days, weeks, or even months.
Even high-end internet-grade routers do not detect soft failures until it is far too late. In the most extreme case, the issues go unnoticed until the transceiver completely fails and the link goes down, often after months of sustained low-level packet loss.
Detecting Failures
Bidirectional Forwarding Detection (BFD) greatly improves the situation caused by soft failures, but both parties must enable BFD for each peering circuit. Even when both parties enable BFD, it still does not react until there is significant loss.
In cases where either party has not enabled BFD, severe packet loss would cause so much corruption that the routing protocol (BGP, for example) will reach a point where it is no longer able to send or receive KEEPALIVE messages. The inability to send or receive KEEPALIVE messages results in the removal of that routing adjacency and routes learned from that peer.
The internet uses BGP version 4. BGPv4 suggests suggests a default HoldTime interval of 90 seconds and a default KeepaliveTime of one-third of the configured HoldTime. This means that one KEEPALIVE message will be sent every 30s. After 90s of not receiving one, the neighbor is deemed dead and the link logically removed from routing decisions. Several common routing vendors have adopted a more conservative default of 60/180s. On most internet-grade routers, this can be tuned as low as 5/15s.
And this is why the public internet simply cannot work for real-time applications. In WebRTC apps, 180 seconds—or even the most aggressive 15s—is simply too slow for a reaction time. By that point, traffic crossing the link has massive amounts of packet loss. Traffic crossing the faulty link may incur small amounts of packet loss for days, weeks, or even months leading up to the ultimate failure. This is the most common threat to your real-time application and one that is extremely difficult to troubleshoot and isolate. Even worse, it’s a problem on someone else's network that you have to prove exists and motivate them to correct.
Routing and Pathfinding on Subspace’s Network
At Subspace, we understand that internet-scale routing is complex, difficult, and broken. We are experts in internet routing, networks, and in the Transmission Control Protocol (TCP). We understand that 0.1% of sustained packet loss across a given link is unacceptable for real-time computing.
The Subspace approach is unique. Our global network encompasses 150+ Points-of-Presence (POP) across the world, and is one of the top 10 most-connected Internet Exchange (IX) participants. These IX locations are where the world’s network operators connect their routers. Our PoPs interconnect over a number of mediums, including dedicated fiber paths. While Subspace uses the BGP routing protocol at our edge to announce our Anycast prefixes, we use no standard routing protocols inside our network. Instead, traffic is routed across the best possible path using our own custom-built routing stack.
Subspace network map can be found here.
Real-time Precision Measurements
To detect soft-failures, Subspace built a custom monitoring solution that sends probes between PoPs across every possible combination of ingress and egress links in real-time, all the time. This system processes more than one million measurements per second.
Routing and Pathfinding
While others also monitor networks to detect and alert on failures, there is a time delay between alerting on a problem and mitigating it. At Subspace, we created a better way to measure. We built a custom routing platform based on a hybrid of centralized and decentralized Pathfinding. This patented routing platform is tightly coupled with our real-time precision measurements. While others use real-time monitoring for detection alone, we use it for real-time routing decisions. This combination allows us to select the most optimal path in real-time based on real-time conditions such as latency, loss, and jitter.
One challenge we faced early on is that a purely centralized approach cannot respond in real time. With that early centralized approach, our software continuously consumed precision measurements and used that to compute a graph of the current-best paths between all PoPs. The cycle time of remotely processing precision measurements, computing the best paths, and then programming those paths back down to all PoPs was simply too long. We tuned many aspects but ultimately realized that the order of magnitude of improvement demanded by real-time computing demanded a different approach.
This led to distributed Pathfinding. With distributed Pathfinding, we run custom software on every PoP that ingests a local subset of real-time precision measurements. By using a local view of measurements, we can process them at a much faster rate.
However, immediately changing from centralized to decentralized Pathfinding does not solve all of the problems. In routing, the lowest latency loss-free path is always preferable. Any amount of sustained loss is harmful. When we observe sustained loss, we may opt for a slightly higher-latency but loss-free path. The best path may not always be direct from one PoP to another; instead the best path may traverse one or more intermediary PoPs.
Performing routing decisions from a local view of measurements does not provide enough data. For this reason, Subspace operates using a hybrid of centralized and decentralized pathfinding. In this distributed Pathfinding procedure, the centralized side dictates the end-to-end path while the decentralized side determines which individual link(s) to take.
Further Refining Subspace
But Subspace was not satisfied with “good enough.” We recently completed a significant improvement to our routing platform that will unlock the next order of magnitude of improvements.
As I explained above, reaction time with centralized pathfinding remained too slow. Unexpected PoP failures require swift reaction to avoid introducing packet loss.
To speed up centralized Pathfinding’s reaction time, Subspace changed how we process real-time precision measurements. After we introduced decentralized Pathfinding, we processed real-time precision measurements in duplicate—at both the centralized and decentralized levels.
Decentralized Pathfinding can process measurements on a much faster cycle because it only needs to consider a local view of real-time precision measurements. Instead of processing measurements in duplicate, we modified our decentralized software to report the post-calculated “best edge” to our centralized Pathfinding system and entirely removed the centralized processing.
This change unlocked that next order of magnitude of improvement. Previously, our largest bottleneck was the rate at which we could process real-time precision measurements on the centralized level. The rate at which we performed real-time precision measurements was constrained by a one-second time budget for processing those measurements at the centralized level. As our network continues to grow, we face scaling challenges while trying to remain within that one-second budget. Exceeding that budget would cause us to fall behind, resulting in Pathfinding decisions made based on past data instead of current data.
While this change dramatically improved performance, we are already developing the next major improvement to this routing design.
Subspace Solves Pathfinding Challenges
As an engineer, I enjoy building software systems that solve difficult problems without unnecessary complexity. At Subspace, we provide a superior network experience for your real-time application. We are the world’s first dedicated network for WebRTC. With WebRTC-CDN, you can get your real-time application running on Subspace with a simple REST API in under 30 minutes.
Eliminate those customer support cases related to poor network conditions. When you use the world’s only managed WebRTC service running on the premier private network created for real-time, you immediately improve your customer experience and you can focus on what matters most—building the best WebRTC application.
David Swafford is a Software Engineering Lead at Subspace and lifelong network engineer, dating back to 1997 when he installed his first home network using Cat5 cabling. Follow him on LinkedIn to learn more about his network engineering journey.
