The 2020 pandemic exposed a fundamental flaw in the way the Internet is built.
Online games were blamed for clogged networks and slowed speeds, but in reality, the Internet is structurally broken when it comes to real-time interactivity. You’ve heard the line:
Kids, get off your consoles; I’m on a video call!
It’s 4 PM on Wednesday. You’re on an important Zoom call and right in the middle of presenting when your face freezes. Those on the call can only catch half of what you say and are left staring at an unflattering image of you mid-word.
How We Got Here: The Evolving Internet
The Internet is built for the reliable delivery of content, such as e-mail and file transfers, but not built for performance. Border Gateway Protocol (BGP)—the standardized routing protocol for the Internet—has no idea what traffic it is routing or in which direction it is sending data. It simply routes data using the cheapest path possible depending on various factors, including politics, commercial agreements, and cost. As a result, certain types of data, namely real-time communications, suffer as the Internet favors servicing volumetric traffic.
At the start of the millennium, content delivery networks employed global networks of servers to deliver static content faster, leading the way to make OTT (Over-the-top, services offered over the top of traditional content distributors) possible with the infrastructure built by companies such as Netflix, YouTube, and Akamai. More recently, companies like Amazon Web Services have democratized cloud computing, bringing connectivity to more users in more regions of the world. The next evolution of the Internet should be one in which the network itself is democratized, enabling speed-of-light communication for the applications that require high-velocity delivery, such as real-time communications. This means delivering voice over internet protocol calls (VoIP, e.g., Skype, Clubhouse), online multiplayer video games, and more, regardless of the compute or data server owner.
Complex Issues with a Complex Network
The Internet is a complicated web of networks and billions of computers and devices that connect to each other and work together to send and receive information. The greater the distance between the sending and receiving computers, the longer it takes the data to arrive at its destination. Information can also take different paths, which may increase or decrease the speed of delivery.
A common analogy used to explain the way the Internet works is the postal service. In the same way that your letter travels from you to the recipient through multiple facilities (carrier, sorting, post office), data travels through numerous routers on its way to the recipient computer. Just as your letter can arrive faster depending on which mail service you use, so too can data transmit faster depending on which path it takes, though the distance between the sender and receiver has the most influence on travel time.
There are a couple of significant differences in this analogy, however.
One, there are generally more “stops” along the way for data, and two, the data is broken up into multiple smaller “packets” of information rather than sent as one whole mail piece.
Because there are different paths along which individual packets of information can travel, each entailing a variable number of stops along the way, packets of data can easily get lost.
Packet loss, as it’s called, results in missing information and a degraded player experience. In a Skype call, packet loss is experienced as gaps in audio. In video games, packet loss results in strange behavior like teleportation, in which a game freezes, then suddenly moves a player’s character to another place when the game resumes.
Another result of these inconsistent, differing routes is variation in delivery time. Some routes are faster than others, so some packets may arrive more quickly than others since the Internet is not conscious of which path it takes in sending data from one point to another. This variation in the time it takes for data to travel across the network from one endpoint to another is called latency, and the result of latency is jitter. On a Skype call, jitter manifests as jumbled speech that creates an indecipherable conversation. In video games, jitter manifests in behavior like errant shots, in which a player’s aim may be correct, but the shot doesn’t register in time to hit the target where they are.
While lag (high levels of latency), jitter, and packet loss may impact any online service or application, certain applications depend upon real-time packet delivery and minimal jitter and packet loss to function correctly and maintain a reasonable quality of experience. Packet delivery, consistent latency, and minimal lag are critical for services and applications like VoIP and online gaming and in numerous other industries such as high-frequency trading, telemedicine, and the Internet of Things.
For these applications, milliseconds saved in delivering data can lead to a real or virtual life-and-death situation and mean the difference between winning and losing millions of dollars or between a quality and a subpar gaming experience.
How Subspace Improves Real-Time Application Delivery
A millisecond’s delay in delivery does not matter in sending an email or a text message, but it does matter in real-time communications. So why should these applications travel on the same network, in the same way?
Subspace has created an Internet byway through which latency-critical web traffic such as video games, VoIP, and any real-time traffic can consistently pass through at the speed of light. Like Waze for the Internet, Subspace gives the Internet a GPS of sorts, finding the absolute fastest and most secure and consistent path for real-time data to travel from endpoint to endpoint. As a result, latency and lag are reduced, and real-time applications reach their full potential.
While other companies have attempted to provide solutions for expediting real-time communications on the Internet, none have addressed the issue across the entire system. Many introduce other issues in their attempts. Commercial networking solutions like Cisco and Juniper were not built for global coordination or to understand and control game traffic. Standalone solutions that require SDKs installed in clients and servers can create significant security and stability risks. And services provided by CDNs like Cloudflare and Akamai utilize the same principles and infrastructure as their primary businesses—that is, volumetric traffic—making the possibility of optimizing real-time traffic at scale a costly and difficult endeavor.
In the case of video games, lag, packet loss, and jitter are such a concern that some game companies have endeavored to build their own custom solutions. This “do it yourself” approach requires massive internal coordination, investment, and attention, which many publishers cannot afford to prioritize. And still, a DIY approach limits the publisher to its existing commercial relationships and capabilities.
The Subspace network optimizes every component of network performance, from the infrastructure stack to the networking stack, including the control plane, and the data plane. With a global Internet architected specifically for real-time traffic at every point, we are achieving speed-of-light communication and democratizing the network. Regardless of the data’s location, Subspace uses a vast system of Internet quality metrics and a proprietary algorithm to direct real-time interactive traffic onto our platform and across the public and our private Internet to and from servers. The algorithm can balance multiple variables to suit each application’s needs—variables such as absolute latency, any latency above a threshold, jitter & loss. The optimal delivery platform via the Subspace onramp drives better, more consistent experiences that match each application’s needs.
The network can be accessed via PacketAccelerator: a Global IP-level proxy that lets you access all the advantages of a network purpose-built for video games and other real-time applications and via GlobalTURN: a service that enables video games with WebRTC-enabled voice and video elements to run TURN globally, without deploying or managing servers.
Users often traverse lag-ridden routes to servers.
We start with purpose-built points of presence (PoPs), deploying our hardware in data centers worldwide. Each PoP is capable of handling millions of servers, millions of simultaneous player sessions, and enough capacity for all of the traffic while also providing room for absorbing DDOS attacks, with the expectation that traffic by its nature is mostly small UDP packets.
The PoPs are distributed globally and specifically network-engineered to reduce the distance needed for players to reach our onramp, minimizing the risk of congestion and misrouting. Players are given a Subspace IP and UDP port as a proxy for a server.
The Subspace PoP uses a proprietary algorithm to determine the optimal path through our private network before ultimately reaching the game server and back. We do this both ways, packets can take a different route back from the server if Subspace finds a better path for the application needs, another differentiator to CDNs, which are optimized for one-way traffic. Continuous telemetry measures the latency, jitter, and loss between Subspace PoPs, feeding into an algorithm that determines the optimum route between the edge and application servers. On the network’s server-side, we have PoPs directly connected with major providers to minimize latency and lag upon exiting our network.
User to Subspace on a local network to AWS-peered Subspace PoP.
Subspace’s global network, which has already launched in multiple regions and is being aggressively deployed towards a global optimized network, already supports some of the world’s most popular online games. And it’s not just about performance. The Subspace platform leverages all of these PoPs to provide a tremendous platform for stopping DDoS attacks against application servers at the edge, before bot networks can aggregate into attacks that get too big to handle or have to be sent off to latency-inducing scrubbing centers. Troublesome attack vectors, replay attacks, and other volumetric attacks can be easily stopped at the Subspace edge, while the game server infrastructure remains protected and hidden. Custom security protocols have also been accommodated, further protecting the application servers.
Today, many of us use the Internet for conferencing, and many of us experience lag during our calls—delays between callers that cause disruptions or the audio/video periodically clipping out—but accept the mediocre quality as if it's an immovable and inalienable facet of Internet life. However, in the world of online gaming, millisecond delays in the delivery of data create a virtual life-and-death situation in which the lightning-fast reflexes of an experienced, dedicated gamer cannot beat laggy Internet connectivity.
What causes lag, and what can be done about it?
At Subspace, we do not accept the status quo and are building a global network to address the Internet’s shortcomings for multiplayer online games. The Internet needs a dedicated network that prioritizes real-time interactivity and multiplayer online gaming data.
Edge compute (or many servers running in many retro areas) technical approach does not work for players who want to play with people who are not close by.
Also, the fewer players in a server region you have, the slower the matchmaking throughput; competitive players end up in boring matches, playing with the same few players with similar skill levels.
Why the Internet Doesn’t Work for Your Online Game
There are four primary reasons why the Internet doesn’t work for your online game and why it will remain that way:
- The cables that comprise the Internet are not wired for your games. Because a large part of the original Internet’s content comes from US-based entities, you can see that many undersea cables are directed to connect countries to the US rather than countries to each other in a mesh pattern. As the map below shows, if your game servers are in Singapore or Hong Kong, it’s quite likely that your Australian players would route through the US. Direct paths are more in demand today but come at a higher cost. No reasonable ISP would pay that for static content.
Cable map. Source: http://www.submarinecablemap.com/
- The Internet is an amalgamation of different service providers (xSPs) and Autonomous System Number/Networks (ASNs) joined together through Border Gateway Protocol (BGP) routing policies. While it is in the interest of an xSP to minimize the number of hops and cables that traffic must flow through, when it comes to ASN-to-ASN-to-ASN traffic, xSPs use BGP to facilitate traffic delivery, prioritizing the lowest cost possible through various peering and transit arrangements.
That’s why your gamers may not use the cable with the shortest path you see on the map above. On a less macro level, sub-optimal routing also happens inside a country and even inside cities. For example, it’s common to see traffic from one provider in the Philippines connect to another provider through a peering link in the US!
A real-world example, though certainly not isolated, is shown in the diagram below. The route from Atlanta to Washington D.C. was direct, but the return path went through California.
-
The BGP protocol isn’t responsive to outages and congestion the way you think it should be. Unless the link is broken entirely, BGP doesn’t care. That means your player could see a lot of data loss and jitter, and BGP doesn’t care because the vast majority of Internet traffic isn’t sensitive to either.
-
Builders of infrastructure look to where the capacity needs are—in video and CDN. With the explosive growth in OTT and software downloads, including for gaming, SPs focus on where to get the cheapest capacity and, in many cases, actively throttle network quality. Their “best effort” connectivity guarantee should really be called “least effort.” As long as it works, as long as users get their files or watch a video with a few seconds of pre-buffering, it’s good enough for them.
The end result of those factors is:
LAG, a general term used to describe long and/or unstable latencies, i.e., jitter, and packet loss, resulting in uneven application behavior and gameplay, e.g., few more disadvantages caused by high ping (lag), such as the inability to dodge projectiles, damage feedback being delayed and the game just feeling sh*t. * a Reddit quote
The latencies between your players and your game servers may be permanently slower than what is ideal and necessary. It may be so for periods that last from a few hours to several days, or players may experience transient periods of congestion, loss, and jitter that affect gameplay.
It goes without saying that a player experiencing lag does not enjoy playing the game.
Responses are unpredictable, and shots are often missed as the target has already moved by the time a shot is taken. This not only negatively affects the quality of experience for the player with a laggy connection, but it also affects the perceived experience of other players as well.
How Subspace Is Fixing the Internet for the Future
Subspace has been supporting online games for players throughout the Middle East with game servers in Mumbai and Bahrain. As such, we have been able to analyze latencies from hundreds of thousands of real game clients, PCs, and consoles, through both normal Internet routing and via the Subspace onramp. These thousands of measurements per second validate the Subspace advantage. Although the distances between Saudi Arabia, Kuwait, the UAE, and Bahrain are quite small, sub-optimal BGP routing is abundant. Historically, we have seen 70-90% of games played over Subspace when gamers are able to choose the faster path. These graphs illustrate cases in which networks could have surprisingly or unexpectedly poor BGP connectivity as well as good networks that degrade periodically for hours or days.
To maximize network performance, one must look not just at the country level but also at the city and ASN levels. Just as whole countries don’t interconnect with other countries in a uniform way, some parts of an ASN do not connect uniformly. For illustrative purposes, though, we will look at two ASNs in Saudi Arabia connecting to data centers in their neighbor, Bahrain.
The first graph shows how many players are in each latency bucket from one particular network. The latencies through BGP are shown in yellow and those through Subspace in purple. Faster latencies are on the left, slower on the right, with the height showing the number of players at each latency from 0 to 400 msec. For a few weeks in May and June, normal Internet routing was elevated to 200 to 400 milliseconds round trip to Bahrain. While this configuration may have been appropriate for everyday volumetric traffic, which probably mostly came from Europe or the U.S., the latency makes games unplayable.
Fortunately, this game provider is using Subspace. With Subspace, the vast majority of players are within 40 to 80 milliseconds to the Bahrain game servers, on par with players’ latencies on other better-connected networks. The band is tighter as well, with very few players above 100 msec. Having players with more comparable latencies makes the game more fair and competitive.
Illustrative distribution of latencies via normal Internet routing and via Subspace, 24 hours.
The second graph below illustrates that there are also good networks that deliver adequate performance most of the time but can degrade for just a few hours or days at a time. In these cases, Subspace transport can act as an “insurance” policy, with steady network quality when the normal Internet route degrades. We frequently observed BGP latencies varying between weekends and weekdays and daytime and nighttime. More recently, post-COVID-19, more congestion seems to happen during the week.
Insurance in motion. Steady latency over Subspace versus unstable Internet routing. Modeled after Subspace data - Saudi Arabia.
This graph shows a week of latency measurements over time from a different network in Saudi Arabia to Bahrain for both Subspace and normal BGP routing. The 25th percentile, which represents the latencies that the fastest 25% of players each hour experienced, and the 75th percentiles are shown. The normal Internet route, in yellow, is not generally unplayable but degraded during weekdays by 15 to 20 milliseconds. The latency with Subspace remained consistent within a few milliseconds over the entire week.
Good hours and bad hours, good days and bad days.
With Subspace, there are only good hours and good days.
What’s Next for the Internet and Gaming?
The Internet is not built to support real-time multiplayer gaming traffic.
With service providers focusing on the 95% of high volume traffic that is not very latency-sensitive, lag is not a problem that is going to solve itself.
Lag in a multiplayer game is a virtual life-and-death scenario. As games increase in complexity and tactility and utilize faster frame rates, solving lag becomes even more critical. Subspace has a proven architecture to do this by making the Internet better and more usable, as the most demanding gamers expect.
Gamers are known to invest significant resources to optimize game performance, and the last thing they want to see is lag when there is a solution out there to stomp it out.
That’s where Subspace comes in. We’ve built a network platform that is truly optimized for real-time gaming. You make gains almost immediately by plugging into a super-highway made for predictable, low-latency performance.
Subspace PacketAccelerator reduces latency and accelerates packets, helping to increase your players’ performance and decrease their stress. Subspace GlobalTURN allows you to run TURN globally without having to maintain servers around the world. And with precision measurements and continuous optimization, we’re democratizing access to quality, high-speed networks for players around the globe.
Want to start building on Subspace today? Sign up here.
