Summary

On Sunday 7th October, 2018 a customer owned IP on the UK datacenter supplier IP Network was subject to a Distributed Denial of Service (DDOS) attack. This attack was first observed at 18:49 (CEST) by the datacenter vendor and was immediately escalated to their on-call engineers. After an initial investigation it was determined that this was a DDOS attack. With the cause of the incident confirmed the investigation continued in an effort to identify the target. The large scale of the attack (approximately 60gbps) caused legitimate traffic on some paths to be disrupted. Upon identifying the DDOS target, the engineer null-routed this IP address, effectively mitigating the attack, allowing legitimate traffic to reach its destination. IP announcements to peering points were additionally modified post-incident to further mitigate inbound traffic and to reduce any congestion / latency as a result.

The following report describes additional details around that event.

Duration of Incident: 118 minutes

Timeline

2018-10-07

18:45 CEST: Attack Started

18:49 CEST: Datacenter vendor support are made aware of an incident with the network.

18:50 CEST: Episerver receives multiple site down alerts

18:53 CEST: Escalated to on-call supplier network engineer.

19:11 CEST: Engineer confirms the cause of the incident is a DDOS attack and continues to investigate.

19:17 CEST: Episerver on-call engineer receives confirmation from Datacenter supplier that they are aware of the situation and working on mitigation.

19:50 CEST: Possible targets determined by the engineer and network team.

20:01 CEST: Continuing investigations reveal the attack is much larger than previously experienced.

20:29 CEST: Target of the DDOS confirmed, engineer null routes IP address.

20:43 CEST: Network engineer reports network stabilized after DDOS, access restored for customers.

23:33 CEST: Minor packet loss reported by customers on certain routes via peering.

23:46 CEST: IP announcements at key peering points updated to not announce the attacked prefix. This forces traffic for that prefix via transits with the null route in place.

2018-10-08

00:07 CEST: IP network confirmed as stable.

2018-10-09

10:17 CEST: Received RCA from datacenter supplier and Episerver close incident ticket.

Root cause

A DDOS attack of approximately 60gbps towards a customer address on the UK datacenter’s IP network, exceeding available capacity, caused disruption to all services on the datacenter’s network for a period of 1 hour 58 minutes.

Resolution and Recovery

The supplier network engineers were able to identify the target of the DDOS and null route its IP address. With this countermeasure in place, the attack was mitigated, and services restored.

Corrective and Preventative Measures

Episerver has been informed by the datacenter provider that the following corrective and preventative actions will be undertaken.

1. Extend and improve remote connectivity to back end systems used for diagnosis for remote engineers. This will enable engineers to better analyze DDOS attacks in the future, ensuring the right systems are available in future similar situations for faster diagnosis.
2. Review current tools used for detecting DDOS attacks.

Final Words

We apologize for the impact to affected customers. We are proud of the reliable availability we are able to provide, along with our proactive alerting system that ensures accurate and timely resolutions to service disruptions. For us, availability is the most important feature and we will do everything we can to learn from the event and to avoid a recurrence in the future.