Issue Summary

On Saturday September 29th, 2018 there was a failure relating to a load balancer in the SE2 Datacenter which resulted in a traffic interruption thereby triggering multiple site down alerts. Our load balancers operate with a active / passive HA configuration and this error was experienced on one of the nodes in that HA configuration.

 The following report describes additional details around that event. 

Timeline

2018-09-29 16:10 CEST: Multiple site down alerts were being created.
2018-09-29 16:14 CEST: Troubleshooting efforts initiated.
2018-09-29 17:15 CEST: The second node in the HA configuration is disabled and traffic returns to normal
2018-09-29 20:45 CEST: Additional troubleshooting and actions are performed to restore the second node which impacts site availability again. The decision to leave the second node disabled is made.
2018-09-30 10:34 CEST: Supplier support is contacted that helps to re-compile the configuration and restore redundancy without impacting site availability.
2018-09-30 11:00 CEST: Redundancy is verified and that traffic is back to normal.

Root cause

In our load balancer HA configuration, one of the load balancers failed to compile its configuration. This failed configuration triggered an active/active split brain scenario which resulted in interruptions to production traffic. After a complete recompiling of the configuration it returned to the preferred active/passive state and functionality was fully restored.

Resolution and Recovery

Immediately upon receiving the alerts troubleshooting efforts were started and a work around was implemented. This work around was to disable one of the load balancer nodes, which restored the traffic flow but with reduced redundancy. This was left in place until the following day (Sunday), when contact was made with the supplier support team. They assisted in recompiling the configuration on the disabled node and once this was completed services were back to full redundancy.

Corrective and Preventative Measures

After a thorough investigation, this was determined to be a rare and unique circumstance that will have a very minimal likelihood of occurring again. As such, no additional preventative measures were taken.

Final Words

We apologize for the impact to affected customers. We are proud of the reliable availability we are able to provide, along with our proactive alerting system that ensures accurate and timely resolutions to service disruptions. For us, availability is the most important feature and we will do everything we can to learn from the event and to avoid a recurrence in the future.