Was your internet down or spotty last Sunday (Aug 30)?

I had several friends asking me about their Internet access Sunday morning. I was able to find the initial Tweet from CenturyLink at the time. I was hoping there’d be more details available later. [Added 5pm – someone sent me a great description and I’ve added it in the comments below. Thanks David Farmer!] It looks like we have confirmation but not much in terms of how it happened or could be prevented. Gizmodo reports…

Widespread internet outages knocked down Cloudflare, the PlayStation Network, Xbox Live, Amazon, Hulu, and a slew of other sites on Sunday morning, and it’s apparently all because of a single internet service provider: CenturyLink.

CenturyLink Tweeted about the problem…

CenturyLink confirmed on Twitter that its technicians were working to fix an IP outage, which was resolved shortly before noon.

“We are able to confirm that all services impacted by today’s IP outage have been restored. We understand how important these services are to our customers, and we sincerely apologize for the impact this outage caused,” the company tweeted.

This entry was posted in Security, Vendors by Ann Treacy. Bookmark the permalink.

About Ann Treacy

Librarian who follows rural broadband in MN and good uses of new technology (blandinonbroadband.org), hosts a radio show on MN music (mostlyminnesota.com), supports people experiencing homelessness in Minnesota (elimstrongtowershelters.org) and helps with social justice issues through Women’s March MN.

1 thought on “Was your internet down or spotty last Sunday (Aug 30)?

  1. It’s awesome having smarter friends. David Farmer offfered an explanation on LinkedIn and gave me permission to share:
    I can help explain what happened. There is this protocol, the Border Gateway Protocol (BGP), it is how carrier routers (not the little box in your house that provides you WiFi, but refrigerator-sized boxes that cost many millions of $) tell each other how to get to different parts of the Internet. The routing table they create using BGP has nearly a million entries for different ranges of IP addresses all over the Internet.

    Besides IP addresses, BGP can also be used to distribute firewall rules as well, between these big routers, this is called Flowspec. The reports are that a Flowspec rule was applied by Centurylink that blocked BGP itself. The rule was deployed across their network, but since it Blocked BGP itself, the rule couldn’t be easily countermanded. This disruption to BGP had wide-ranging effects, at a minimum preventing other carriers from know how to get to Centurylink’s customers.

    Most large Internet customers have multiple service providers, this is called being multi-homed. However, one of the disruptions that occurred was many of the routes for Centurylink’s customers didn’t cleanly get removed because BGP itself was impacted by the Flowspec rule above. So, while some of these multi-homed customers could have received traffic through their other provider(s), many parts of the Internet still believed they could get to these customers through Centurylink, even though they couldn’t.

    While Centurylink confirmed they had an issue with Flowspec, much of the above is conjecture on my part.

    Hope that helps

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s