SIGCOMM 2021 Best Paper: Internet Hypergiants Expand into End-User Networks

Home >

SIGCOMM 2021 Best Paper: Internet Hypergiants Expand into End-User Networks

SIGCOMM 2021 Best Paper: Internet Hypergiants Expand into End-User Networks

BIFOLD Fellow Prof. Dr. Georgios Smaragdakis and his colleagues received the prestigious ACM SIGCOMM 2021 Best Paper Award for their research into the expansion of Hypergiant’s off-nets. They developed a methodology to measure how a few extremely large internet content providers deploy more and more servers in end-user networks over the last years. Their findings indicate changes in the structure of the internet, potentially impacting network end-user experience and neutrality regulations.

Percentages of internet users that can be served by Hypergiants’ off-nets in their networks.
(Copyright: Petros Gigis et al.)

An increasing amount of the digital content delivered to Internet users originates from a few very large providers, like Google, Facebook or Netflix, the so-called Hypergiants (HGs). In 2007 thousands of autonomous networks (AS) – e.g. networks of an Internet service provider or university – were necessary to provide 50% of all content. In 2019 only five Hypergiants managed to originate half of the total Internet traffic alone. To cope with the unprecedented demand, most of these Hypergiants increased their network capacities in their own networks, but also installed and operated their servers, called off-nets, inside other networks. Such off-nets operate closer to the end user and, thus, accelerate content delivery to end-users as well as support applications, e.g., video streaming and edge computing (machine learning, artificial intelligence, and 5G).

At least three of the current HGs increased their off-net footprint significantly between 2013 and 2021 (note: y-axis scales differ).
(Copyright: Petros Gigis et al.)

In their paper “Seven years in the life of Hypergiants’ off-nets,” Georgios Smaragdakis, Professor of Cybersecurity at TU Delft, and his colleagues from University College of London, Microsoft, Columbia University and FORTH-ICS present a methodology to measure the increase of such off-nets footprints by analyzing massive public data sets that include active scans and server digital certificates (TLS) that span over seven years (2013-2021). By analyzing the ownership of the certificates over time, they were able to track the deployment of Hypergiants’ off-nets around the globe. These Internet analytics are important to understand how the structure and operation of the Internet and the data flow has changed. For this work the researchers received the prestigious Best Paper Award of the 2021 ACM Special Interest Group on Data Communication (SIGCOMM 2021) conference. SIGCOMM is the flagship conference of the Association of Computing Machinery (ACM) on the topics of internet architecture and networking.

Prof. Dr. Georgios Smaragdakis
(Copyright: Georgios Smaragdakis)

“Internet infrastructures are the backbone of contemporary communication. Understanding developments in this sector is a key prerequisite for improving end-user experience, security, and privacy. We are very pleased that our efforts to monitor and explain changes in the Internet architectures are internationally recognized.”

“This is the first generic and scalable method to survey this development in the wild. We make publicly available the only extensive collection of data and visualizations that describe such Hypergiant off-net developments over seven years, from 2013 to 2021”, explains Georgios Smaragdakis. He and his colleagues found that large Hypergiants can serve large fractions of the world’s internet users directly from within the users’ networks. “While the deployment of off-nets can improve end-user performance and the introduction of encryption improve user privacy, our study shows that information about these deployments is leaked and can be potentially misused by adversaries or to gain business intelligence. In our work we suggest ways to address such issues”, says Georgios Smaragdakis. Prof. Smaragdakis and his colleagues believe that the insights by their data analysis and the release of public data can inform studies in other fields, including economics, political science, and regulation.

The publication in detail:

Petros Gigis, Matt Calder, Lefteris Manassakis, George Nomikos, Vasileios Kotronis, Xenofontas A. Dimitropoulos, Ethan Katz-Bassett, Georgios Smaragdakis: Seven years in the life of Hypergiants’ off-nets. SIGCOMM 2021: 516-533


Content Hypergiants deliver the vast majority of Internet traffic to end users. In recent years, some have invested heavily in deploying services and servers inside end-user networks. With several dozen Hypergiants and thousands of servers deployed inside networks, these off-net (meaning outside the Hypergiant networks) deployments change the structure of the Internet. Previous efforts to study them have relied on proprietary data or specialized per-Hypergiant measurement techniques that neither scale nor generalize, providing a limited view of content delivery on today’s Internet.
In this paper, we develop a generic and easy to implement methodology to measure the expansion of Hypergiants’ off-nets. Our key observation is that Hypergiants increasingly encrypt their traffic to protect their customers’ privacy. Thus, we can analyze publicly available Internet-wide scans of port 443 and retrieve TLS certificates to discover which IP addresses host Hypergiant certificates in order to infer the networks hosting off-nets for the corresponding Hypergiants. Our results show that the number of networks hosting Hypergiant off-nets has tripled from 2013 to 2021, reaching 4.5k networks. The largest Hypergiants dominate these deployments, with almost all of these networks hosting an off-net for at least one — and increasingly two or more — of Google, Netflix, Facebook, or Akamai. These four Hypergiants have off-nets within networks that provide access to a significant fraction of end user population.

More information is available from:

Prof. Dr. Georgios Smaragdakis

TU Delft – Cybersecurity
Van Mourik Broekmanweg 6
2628 XE Delft
The Netherlands


COVID-19: A Stresstest for the Internet

Home >

COVID-19: A Stresstest for the Internet

COVID-19: A Stress Test for the Internet

25 percent increase in Internet traffic within only a few days

On 11 March 2020, the day the WHO declared the coronavirus a global pandemic, the impact of SARS-CoV-2 also spread to the World Wide Web. Following this announcement, governments around the world began enacting stay-at-home orders and other regulations for working from home and homeschooling. Within a single week, Internet traffic volume increased by 25 percent – an increase which under normal circumstances is usually observed over the course of a year. Taking account of increased use during the second lockdown in fall 2020, the overall use of Internet services in 2020 increased between 35 and 50 percent, depending on the network. An international, interdisciplinary group of researchers led by Professor Dr. Georgios Smaragdakis, professor of Internet measurement and analysis at TU Berlin and Fellow of the Berlin Institute for the Foundations of Learning and Data (BIFOLD), has published these figures and other findings in a paper in Communications of the Association for Computing Machinery (ACM). The leading professional association recently named the paper a research highlight.

Despite the worldwide restrictions necessitated by COVID-19, life continued with the Internet playing an important role.

From essentially one day to the next, almost nothing was possible without a stable Internet connection. Since March of last year, team meetings, school lessons, and even private celebrations have primarily been held via digital screens. Those without a broadband connection or sufficient electronic devices have missed out. Despite the worldwide restrictions necessitated by COVID-19, life continued with the Internet playing an important role for companies, the education sector, entertainment, retail, and social interactions. “In the spring of 2020, no one could say with certainty whether the Internet would be able to withstand this rush demand,” explains Georgios Smaragdakis. “No one had previously expected a sudden surge in Internet traffic of such proportions.” In their project, financed in part by BIFOLD, the researchers investigated Internet data streams from different Internet providers across Europe. “Together they provide us with a good understanding of the impacts that the COVID-19 waves and lockdown measures had on Internet traffic,” continues Georgios Smaragdakis.

Within a year of the implementation of the first lockdown measures, the aggregate volume of Internet data traffic increased by approximately 40 percent, significantly more than the expected annual growth. At the same time, mobile data traffic first slightly decreased and then only grew moderately, as people were out and about less, thus using less mobile data. “Our calculations show that the use of services such as video conferencing and VPNs increased by up to 300 percent. Gaming applications also significantly increased. After moderate growth during the spring lockdown, use increased by about 300 percent during the fall lockdown. And while these applications were primarily used in the evening or on the weekend pre-pandemic, gaming usage increases were evenly distributed across each day of the week during the second lockdown, mainly in the mornings,” remarks Georgios Smaragdakis.

Overall traffic patterns in Internet usage have clearly changed: While peak times before the pandemic were on the weekend and in the evening, the sudden growth in Internet usage primarily occurred on weekdays during working hours. This asynchronous growth is precisely one reason why researchers believe the Internet was able to handle the increased traffic relatively well. Smaragdakis believes the good structure and overprovisioning of the network operators also helped.

“In terms of digitalization, the last months have been a tremendous success,” says Smaragdakis. “In just a matter of weeks, German universities and government authorities adopted developments that they had previously failed to implement in years. These days, a broadband connection is not just something that is nice to have, but rather an essential requirement to be able to work. This level of digitalization is the new normal. It will not be possible to return to previous practices.”

The researchers’ study also shows that overprovisioning, proactive network management, and automatization were key to providing resistant networks which could cope with the drastic and unexpected fluctuations in demand like those experienced during the COVID-19 pandemic. “Many, but not all, network providers succeeded in doing this. With the pandemic set to continue for some time, it is important that we continue to examine data traffic to understand how usage changes during these unprecedented times,” he concludes.

The Publication in Detail

Anja Feldmann, Oliver Gasser, Franziska Lichtblau, Enric Pujol, Ingmar Poese, Christoph Dietzel, Daniel Wagner, Matthias Wichtlhuber, Juan Tapiador, Narseo Vallina-Rodriguez, Oliver Hohlfeld, Georgios Smaragdakis
In March 2020, the World Health Organization declared the Corona Virus 2019 (COVID-19) outbreak a global pandemic. As a result, billions of people were either encouraged or forced by their governments to stay home to reduce the spread of the virus. This caused many to turn to the Internet for work, education, social interaction, and entertainment. With the Internet demand rising at an unprecedented rate, the question of whether the Internet could sustain this additional load emerged. To answer this question, this paper will review the impact of the first year of the COVID-19 pandemic on Internet traffic in order to analyze its performance. In order to keep our study broad, we collect and analyze Internet traffic data from multiple locations at the core and edge of the Internet. From this, we characterize how traffic and application demands change, to describe the “new normal,” and explain how the Internet reacted during these unprecedented times.
Communications of the ACM, July 2021, Vol. 64 No. 7, Pages 101-108

For further information please contact:

Prof. Dr. Georgios Smaragdakis
TU Berlin
Tel.: 030 314-75169