How A Fastly Customer “Triggered” Yesterday’s “Broad And Severe” Global Internet Outage

Fastly, a major content delivery network, triggered a major internet blackout on Tuesday morning has blamed a software bug. 

We first noticed the problem a little after 0600 ET Tuesday when countless websites, including Reddit, Financial Times, PayPal, and other websites, went down. 

“We experienced a global outage due to an undiscovered software bug that surfaced on June 8 when it was triggered by a valid customer configuration change,” Nick Rockwell, Fastly’s senior vice president of engineering and infrastructure, wrote in a blog post late Tuesday. 

Rockwell said, “the outage was broad and severe, and we’re truly sorry for the impact to our customers and everyone who relies on them.” He said the company “detected the disruption within one minute, then identified and isolated the cause, and disabled the configuration,” adding that “within 49 minutes, 95% of our network was operating as normal.” 

“Even though there were specific conditions that triggered this outage, we should have anticipated it,” Rockwell said.

The company operates servers at strategic points worldwide to support customers moving and storing content.

But when a customer switched their settings, they had accidentally exposed a bug in a software update issued in May, which triggered widespread outages. 

Here’s a timeline of yesterday’s disruption (all times are in UTC): 

09:47 Initial onset of global disruption

09:48 Global disruption identified by Fastly monitoring

09:58 Status post is published

10:27 Fastly Engineering identified the customer configuration

10:36 Impacted services began to recover

11:00 Majority of services recovered

12:35 Incident mitigated

12:44 Status post resolved

17:25 Bug fix deployment began

We noted yesterday that incidents like this underline the fragility of the internet and how the global internet is dependent on a handful of companies like Fastly and Cloudflare for vital infrastructure. On the flip side, the incident also highlights how quickly an outrage of this magnitude can be resolved.