DataCenterNews Asia logo
Specialist data center news for Asia
Story image

Top internet outages of 2019 - ThousandEyes

By Newsdesk
Wed 15 Jan 2020
FYI, this story is more than a year old

ThousandEyes has announced last year’s most disruptive outages and what the industry can learn from them.

When Internet outages happen, they can be extremely disruptive to a business.

By preventing users from reaching applications and services, outages can cause major revenue and reputation damage.

While application delivery is dependent on many Internet Service Providers (ISPs), it also increasingly relies on a large and complex ecosystem of Internet-facing services — such CDN, DNS, DDoS mitigation, and public cloud.

These services work together to provide exceptional digital experiences to users and even brief disruptions can have a significant impact.

“At the same time, enterprises are increasingly relying on Internet transport to connect their sites and reach business critical applications and services. Gone are the days in which applications are solely hosted in private data centers and office locations are connected primarily by MPLS circuits,” says ThousandEyes product marketing director Angelique Medina.

“The Internet is replacing or supplementing services like MPLS as enterprises embrace SD-WAN technologies. As a result, the Internet is now effectively the enterprise backbone, which as a “best effort” transport can have significant yet unforeseen consequences for businesses.”

Over the last year, ThousandEyes has reported on a number of large-scale outages that had ripple effects across the global Internet, impacting enterprises and consumers alike.

The most significant of these outages took place over the northern hemisphere summer and disrupted nearly every top tech company in some fashion.

May 13, 2019 — China telecom outage reveals its global reach

While not the most disruptive outage of 2019, a global and fairly long lasting outage in China Telecom’s network proved to be a harbinger of incidents to come, while also revealing a lesson in how China Telecom’s reach extends far beyond mainland China.

For nearly five hours on May 13, 2019, China Telecom experience substantial packet loss across its backbone, primarily impacting network infrastructure in mainland China, but also affecting China Telecom’s network in Singapore and multiple points in the US, including Los Angeles. Over one hundred services were disrupted. Though not exclusively impacting western sites and services, many users of major US brands such as Apple, Amazon, Microsoft, Slack, Workday, SAP, would have experienced disruptions over the course of the outage window.

This incident illustrated some important realities about China and its impact on the global Internet that many folks aren’t aware of. Specifically, it highlighted that many of the censorship policies that apply to Chinese Internet users may actually be implemented far beyond China’s borders and in countries that have very different attitudes and policies related to Internet use.

June 2, 2019 — Summer of outages begins with Google Cloud

On June 2, 2019, Google Cloud Platform experienced a significant network outage that impacted services hosted in parts of us-west, us-east and us-central regions. This outage impacted Google’s own applications, including GSuite and YouTube. The outage lasted more than four hours, which becomes notable given the criticality of its service to business customers.

Google issued an official report of the incident several days later. ThousandEyes vantage points were able to see the outage as it unfolded in real time, revealing its characteristics and scale ahead of more detailed information becoming publicly available.

Beginning at approximately 9am ET in the US, ThousandEyes observed 100% packet loss from global monitors attempting to connect to a service hosted in GCP us-west2-a. Similar losses were seen for sites hosted in several portions of GCP US East, including us-east4-c.

The complete unavailability of parts of Google’s network, as seen by ThousandEyes, turned out to be due to Google’s network control plane inadvertently getting taken offline. Google later revealed that during the outage period, a set of automated policies determined which services were or were not reachable through the unaffected parts of its network.

One of the most important takeaways from cloud outages is that it’s vitally important to ensure your cloud architecture has sufficient resiliency measures, whether on a multi-region basis or even multi-cloud basis, to protect from future recurrence of outages. It’s reasonable to expect that IT infrastructure and services will sometimes have outages, even in the cloud.

June 6, 2019 — An unfortunate series of events takes down WhatsApp for many users

On June 6, 2019, a large number of users around the globe attempting to access the WhatsApp service experienced connectivity issues. ThousandEyes was able to immediately see that 100% packet loss was preventing the service’s reachability.

Upon further analysis, ThousandEyes determined the root cause of this packet loss was a massive route leak that steered traffic to China Telecom — a service provider that does not forward any Facebook-related traffic.

The incident was triggered when a Swiss colocation company called Safe Host announced to the Internet that the best way to reach WhatsApp and thousands of IP prefixes was through its network, AS 21217. When Safe Host advertised these routes, they were accepted by China Telecom and further propagated through other ISPs such as Cogent.

Users whose traffic was routed to Cogent and ultimately handed off to China Telecom would have been completely unable to reach the service.

It’s unclear why China Telecom would accept routes to a service that it censors, but what is clear is the lesson of this outage. BGP route leaks are not uncommon on the Internet. When you rely on the Internet, an ecosystem that is deeply interconnected and vulnerable, you must understand how it works and expect that a glitch in one service provider can have cascading effects on another.

The unfortunate reality is that business risks associated with BGP route leaks and other Internet flaws are greater given the modern enterprise and service delivery landscape.

June 24, 2019 — Cloudflare users fall victim to routing mishap

Just a couple of weeks after the massive route leak that impacted WhatsApp users, the Internet experienced yet another route-related incident - this one far more damaging.

On June 24, 2019, for nearly two hours, a significant BGP routing error impacted users trying to access services fronted by CDN provider Cloudflare, including gaming platforms Discord and Nintendo Life.

ThousandEyes analysis found that a significant BGP route leak affected a variety of prefixes from multiple providers. DQE, a transit provider, was the original source of the route leak, which was propagated through Allegheny Technologies, a customer of both DQE and Verizon. Unfortunately, Verizon further propagated the route leak, magnifying the impact.

Sites served through the CloudFlare CDN were impacted for nearly two hours.

This major Internet disruption affected about 15% of Cloudflare’s global traffic and impacted services like Discord, Facebook and Reddit. The route leak also affected access to some AWS services.

The root cause of the incident was eventually traced to DQE’s use of a BGP optimiser software that created routes to Cloudflare services that were only meant to be used within DQE’s internal network. When these routes were accidentally leaked to one of its customers, mayhem ensued.

This incident was yet another reminder of how incredibly easy it is to dramatically alter the Internet service delivery landscape. In a cloud-centric world, enterprises must have visibility into the Internet if they’re going to be successful in delivering services to their users.

July 4, 2019 — Apple Services impacted on fourth of July

On July 4, 2019, just before 9am PT, users connecting to Apple’s website and some of its services, such as Apple Pay, began experiencing significant packet loss for a period of over 90 minutes. This issue prevented many users from successfully connecting to Apple. ThousandEyes route visibility demonstrated that the packet loss was caused by a BGP route flap. A BGP route flap is caused when a routing announcement is made and withdrawn in quick succession, often repeatedly.

While Apple services are certainly important for many Internet users, the fact that the incident occurred early on a holiday seems to have prevented the incident from sparking more than a few user complaints. The lesson from this incident is that outages don’t happen in a vacuum. Sometimes even significant outages may go unnoticed (or conversely create significant uproar) simply based on their timing and context.

September 6, 2019 — DDoS attackers target the Internet’s knowledge base

On September 6, 2019, access to Wikipedia sites from around the world was disrupted for close to nine hours, the result of a massive and sustained Distributed Denial of Service (DDoS) attack. DDoS attacks can overwhelm their target’s web infrastructure and also create congestion within service provider networks that can lead to packet loss. These effects are exactly what ThousandEyes observed when Wikipedia came under attack.

During the course of the incident, ThousandEyes saw a significant drop in HTTP server availability from around the world, as well as a dramatic increase in HTTP response times. As a result, users across many regions were unable to establish an Internet connection for ongoing communication with Wikipedia servers. ThousandEyes also measured packet loss of up to 60% from our global vantage points, a condition that would have further prevented access to Wikipedia sites.

While DDoS events are an unfortunate reality of operating on the Internet, organisations should have visibility into the scope, impact and behaviour of these events and be able to validate that DDoS mitigation steps are effective.

Related stories
Top stories
Story image
Colocation
Digital Edge chooses Nortek’s StatePoint for new data center
Digital Edge will use Nortek's StatePoint liquid cooling technology in its new data center, the first commercial colocation operator in Asia to do so.
Story image
Cybersecurity
The 'A-B-C' of effective application security
Software applications have been a key tool for businesses for decades, but the way they are designed and operated has changed during the past few years.
Story image
Microsoft
SAS Viya on Microsoft Azure to deliver 204% return - study
The Forrester Total Economic Impact study finds SAS Viya on Microsoft Azure brings a 204% return on investment over three years.
Story image
Talend
Talend introduces new data health solutions for businesses
Talend has announced its latest version of Talend Data Fabric, with the release of Talend Trust Score enabling data teams to establish a foundation for data health.
Story image
Infrastructure
Report - Data investment the key to better business growth
New research from Digital Realty has revealed that almost half (47%) of IT leaders globally believe their business investment in data systems and infrastructure is a key obstacle or concern.
Story image
Power / Energy
Keysight Technologies introduces new next-gen DPT solution
Keysight Technologies has announced its new next-generation Double-Pulse Tester (DPT) with the PD1550A Advanced Dynamic Power Device Analyser.
Story image
Akamai
Akamai announces new products across security, computing
Akamai has announced a series of new products and updates to existing products across its security and compute product lines, including its entry into the infrastructure as a service (IaaS) market.
Story image
APAC
Odaseva expands in APAC and UK with more security features
Odaseva, a data platform for Salesforce, is establishing new headquarters in London as well as a new data center in India.
Story image
Data Center
Equinix enters Africa, closing US$320 million acquisition of MainOne
The completion of the acquisition augments Equinix's long-term strategy to become a leading African carrier-neutral digital infrastructure company.
Exabeam
Find out how a behavioural analytics-driven approach can transform security operations with the new Exabeam commissioned Forrester study.
Link image
Story image
Surveillance
Genetec launches new enclosure management system for data centers
Genetec has released a new enclosure management solution that will give data centers the ability to secure, monitor and manage access to racks and cabinets remotely.
Story image
Data Center
Fujitsu AU signs PPA to offset 40% of NSW data centre load
The agreement marks a key step for Fujitsu Australia in decarbonising its operations and providing lower-emissions services to its customers.
Story image
Hyperscale
SpaceDC partners with Aofei for data center sales in Asia
SpaceDC has partnered with Aofei Data International to sell Aofei's data centers, CDN and SDN in China.
Story image
Sustainability
ABB unlocks sustainable energy upgrades for data centers
ABB says its new microgrid solutions will get data centers ready for the green revolution and use their own energy sources with a reduced carbon footprint.
Story image
BitTitan
Why tenant consolidation is critical to cloud success
Consolidating tenants can improve cost management, security and engagement after a flurry of reactive activity following the widespread shift to remote operations.
Story image
Microsoft
Microsoft unveils adaptive accessories for disability access
Microsoft is introducing an expansive Inclusive Tech Lab to give people with disabilities greater access to technology through new software features and adaptive accessories.
Story image
Storage
Energy storage demand momentum continues, says BYD
BYD has announced an expansion of its production capacities and will deliver 250,000 units of its energy storage system, BYD Battery-Box Premium.
Story image
Sustainability
RDA and MVGX partner for sustainable data center development
Red Dot Analytics (RDA) and MetaVerse Green Exchange (MVGX) have entered a strategic partnership to make Singapore's data center development and operations more sustainable.
Story image
Cybersecurity
A10 Networks finds over 15 million DDoS weapons in 2021
A10 Networks notes that in the 2H 2021 reporting period, its security research team tracked more than 15.4 million Distributed Denial-of-Service (DDoS) weapons.
Story image
Data Center
Sime Darby Berhad to use Equinix APAC data centers
Equinix has expanded its digital infrastructure services, including its International Business Exchange data centers, to Sime Darby Berhad
Story image
Digital Transformation
Multiplex, NEXTDC making strong progress on S3 data centre
Multiplex has made a significant achievement on Stage 1 of NEXTDC’s S3 data centre, ‘topping out’ the structure in the Artarmon on Sydney’s lower North Shore.
Story image
Sustainability
Grasping the opportunity to rethink the metrics of a sustainable data centre
A data centre traditionally has two distinct operations teams: the Facility Operations team, and the IT Operations team. Collaboration between them is the key to defining, measuring, and delivering long-term efficiency and sustainability improvements.
Story image
Sustainability
Siemens showcases new automated solutions for data centers
Siemens has implemented new automated solutions and AI in the Baltic region's largest data center, providing insight into the future of data center management.
Story image
Power / Energy
DigitalBridge makes $30 million equity investment in LEDC
Leading Edge Data Centres (LEDC) has announced it has secured an AUD$30 million equity investment in its regional edge network from an affiliate of DigitalBridge Group, DigitalBridge.
Story image
Disaster Recovery
Kacific launches emergency connectivity offering, CommsBox
Kacific has announced the release of a new emergency connectivity offering designed to rapidly provide broadband service in emergency or disaster zones.
Story image
Sustainability
Video: 10 Minute IT Jams - SoftIron CMO on Data Center Sustainability
In a special Power/Energy feature week presentation, we are joined by SoftIron CMO Andrew Moloney.
Story image
Data Center
CBRE finds record levels of investment in APAC data centers
CBRE's new report finds direct investment in the sector more than doubled in 2021, surpassing investment volumes for the past four years combined
Story image
Telstra
Telstra expands business offerings in the Philippines
The expansion aims to offer more choice for customers and enhance connectivity into the Philippines, and within the country.
Softiron
For every 10PB of storage run on HyperDrive vs. comparable alternatives, an estimated 6,656 tonnes of CO₂ are saved by reduced energy consumption alone over its lifespan. That’s the equivalent of taking nearly 1,500 cars off the road for a year.
Link image
Story image
Red Hat
Red Hat expands capabilities to provide streamlined application development in cloud
"Application development is undergoing significant change and developers need tools to support this transformation."
Story image
Data Center
Preventing downtime costs and damage with Distributed Infrastructure Management
Distributed Infrastructure Management (DIM) can often be a lifeline for many enterprises that work with highly critical ICT infrastructure and power sources.
Story image
Sustainability
AyalaLand and FLOW partner for data center development
AyalaLand Logistics Holdings Corp (ALLHC) and FLOW Digital Infrastructure have entered into a framework agreement to bolster the development of carrier-neutral data centers in the Philippines.
Story image
Data Center
Digital Edge to build South Korea's largest commercial data center
The project will be the largest commercial data center project in South Korea with total IT power of 120MW and a capital investment of more than KWR$1 trillion.
Story image
Cloud
Colt connectivity with AWS increases services in Asia
Colt Technology Services expands cloud connectivity to AWS Direct Connect Hosted services, with speeds of up to 10 Gbps in Asia.
Story image
Cable
New high-performance cable in the works for Asia
A new high-performance submarine cable is being built to enhance connectivity between Hong Kong, China and Southeast Asia.
Story image
Sustainability
AirTrunk boosts Japan presence with West Tokyo data center
AirTrunk is planning to build TOK2, a new hyperscale data center in Japan which will strengthen the company’s presence in the country.
Story image
Sustainability
YTL unveils development of solar-powered data center campus
YTL Power (YTL) has announced the development of a 500MW data center campus in Johor, the first data center park in Malaysia to be powered by solar energy.
Story image
Sustainability
Power at the edge: the role of data centers in sustainability
The Singaporean moratorium on new data center projects was recently lifted, with one of the conditions being an increased focus on power efficiency and sustainability.
Story image
Hyperscale
Seagate, Phison partnership to improve data center offering
Seagate and Phison have announced plans to expand their SSD portfolio to assist data center customers in reducing total cost ownership (TCO).
Story image
SD-WAN
Orange moves Siemens AG’s entire operations to a SD-WAN
Orange Business Services has migrated Siemens AG's entire global operations, 1168 sites across 94 countries, to a SD-WAN
Story image
Tech Data
Tech Data to use Pluribus Networks’ cloud solutions in APAC
Tech Data says using Pluribus Networks' Unified Cloud Fabric solution will be a "game-changer" for its data center infrastructure customers and partners.
Story image
Infrastructure
Kyndryl and CDC Data Centres enter multi-year agreement
Kyndryl and CDC Data Centres have announced an agreement, under which Kyndryl will design and manage a customised environment within CDC's facilities.
Story image
Sustainability
NTT launches IoT Services for Sustainability offering
"We know what actions are needed to build a more sustainable future and have a robust suite of technologies available to help deliver this impact."
Story image
Digital Transformation
EdgeConneX enters Indonesia, plans for data center campus
EdgeConnex has announced it is expanding its presence in Asia with the acquisition of GTN Data Center in Indonesia.