What happened?
Today at 10:46 AM PDT our monitoring system detected a hijack of two of our six IPv4 prefixes. This would have affected some traffic to/from networks who are not yet RPKI filtering invalids. (hint: for those who have no idea what we are talking about, some great resources are listed at the bottom of this page)
Shortly after the automated alert, we started our investigation and determined that Telstra Wholesale Australia (AS1221) had hijacked the two prefixes. Only customers who have IPs assigned from those two prefixes (as in above screenshot) would have been affected.
In the cropped BGP map below, you can see some internet traffic being re-routed toward the hijacked routes to AS1221, instead of us on the far right.
AS1221 Hijacked BGP Map
Normal BGP Map
BGPStream also picked up the hijacks and the interactive BGP map below.
BGPlay Visualization
View the visualization yourself: http://bgpstream.com/event/251757 and http://bgpstream.com/event/251758
How it was resolved
Our team tried numerous methods to reach out to AS1221 (Telstra Wholesale Australia), and AS4637 (Telstra Global) which seems to be the parent network. We had difficulties reaching the NOC by phone, since it was still early morning in Australia/Hong Kong. We did manage to reach someone by phone in their “IP Faults” department, but unfortunately they were unaware how to deal with the issue or who to put us in touch with. We managed to find an email for the NOC contact and forwarded our email to them. By approximately 1:41 PM PDT, Telstra NOC replied confirming they fixed the (what we assume) mis-configuration on their side.
Screenshot of attempts to communicate with Telstra
From our own internal investigation it seems like AS1221 issue originated in their Sydney POP using their own looking glass tool.
How could have this been prevented?
The current standards in place is to implement RPKI signing and filtering . We did this ourselves back in January 2016 by adding Route Origin Authorizations (ROA) to prefixes we announce. And most recently in April 2020, we started validating and filtering invalid routes. Unfortunately until all network operators around the world start doing the same, they are vulnerable to accepting and re-announcing the hijacked or invalid routes to others.
Without rewriting some already great articles, we recommend visiting some of the below resources and articles for more information on BGP hijacking and RPKI:
- https://www.cloudflare.com/learning/security/glossary/bgp-hijacking/
- https://isbgpsafeyet.com/
- https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-today/
- https://www.manrs.org/2020/09/what-is-bgp-prefix-hijacking-part-1/
- https://www.manrs.org/2020/09/mitigating-prefix-hijacks-with-rpki-part-2/
Despite this being caused by something outside our control, we would like to apologize for the disruption. We hope more network operators start adopting better routing security through standards such as RPKI, and ultimately make the internet a safer place, routing wise ;).
Update September 30 – New information
Via an iTnews Australia article and social media posts, it seems 102 networks around the world were affected including Quad9 DNS and Packet’s Sydney facility that hosts Google’s local region.
Per a tweet, Telstra reported the cause to be:
“due to a technical error overnight, a number of internet prefixes were incorrectly advertised as Telstra’s. This meant some internet traffic may have been routed to Telstra incorrectly…”
Update October 3 – Official statement from Telstra
Telstra has posted an official statement on their webite.
Update October 7
Added new BGPplay diagram and links
Thank You Andrew … we appreciate the effort, and the education.