Category

CloudSingularity

Sep 29
Love0

Post-Mortem on Tuesdays BGP Hijack

By CloudSingularity, Datacentre One Comment

What happened?

Today at 10:46 AM PDT our monitoring system detected a hijack of two of our six IPv4 prefixes. This would have affected some traffic to/from networks who are not yet RPKI filtering invalids. (hint: for those who have no idea what we are talking about, some great resources are listed at the bottom of this page)

Shortly after the automated alert, we started our investigation and determined that Telstra Wholesale Australia (AS1221) had hijacked the two prefixes. Only customers who have IPs assigned from those two prefixes (as in above screenshot) would have been affected. Read More

Apr 13
Love0

Infrastructure Upgrades Roadmap

By CloudSingularity, Datacentre No Comments

We are exited to share our roadmap for new upgrades across our entire infrastructure. These will take place over the next 3 to 4 weeks. These upgrades will help us deliver new solutions and make existing ones faster and more resilient. Check back to this post later for updates.

Phase 1 (planned mid April, 2017) – Completed

  • replacement of switch modules in our BladeCenter Chassis to dedicated pass-thru modules
  • upgrade of our current Cisco out-of-band management switch to Juniper EX series

Phase 2 (planned end of April, 2017) – In-Progress 60% complete

  • upgrade of existing Juniper EX series core switches to Juniper QFX5100 core switches across our YVR1 datacentre and Harbour Centre point of presence (PoP)

Phase 3 (planned  June, 2017)

  • upgrade of Fortinet FortiGate appliance with new higher performance 800 series FortiGate’s

Phase 4 (planned July, 2017)

  • upgrade to Juniper vMX core routing

Existing customers rest assured we will do everything we can to minimize the impact on any existing services which may be affected.

This post will be continuously updated as we progress through the roadmap. For current maintenance notifications please visit and subscribe to our status page located at https://status.cloudsingularity.net

Aug 17
Love1

Post-Mortem on Tuesday Evenings Network Incident

By CloudSingularity, Datacentre No Comments

We would like to share some details on the incident that occurred yesterday evening at our Vancouver Datacentre YVR1.

At approximately 21:00 PST on Monday, August 15th there was a brief 4 minute spike in traffic that caused some packet loss.

On Tuesday, August 16th between 16:30 and 18:00 PST we detected intermittent packet loss until around 18:00 where there was a complete loss in traffic.

Traffic GraphWe have concluded both incidents were related to a Distributed Denial of Service (DDoS) attack. Tuesday’s attack was a larger scale DDoS attack at over 3Gbps/300Kpps. The attack saturated our port which resulted in dropped traffic. We worked with our upstream providers to mitigate the attack by null-routing the IP being attacked. At around 18:24 after the null-route, all traffic resumed to normal.

We understand how critical it is to our customers that the network is up 100% of the time. This was unfortunately something out of our control, as is the nature of DDoS attacks. We strive to minimize disruptions and outages regardless of the source of the issue; and promise to remain transparent on these kind of attacks and incidents of any nature. Finally, as we continuously make improvements to the network we will eventually be able to absorb these type of attacks without incident.