Summary

On Tuesday 2018-02-27 we had an event impacting Episerver FIND. The following report describes additional details around that event.

Incident total duration: 6 hour.

Details

On Tuesday 2018-02-27 16:00 CET, the Find Operations team finalized the release of a new major version of Find. During the release there were three separate errors that occured.

1. During the finalizing steps of the release alerts were triggered on several search clusters in the EU region. Troubleshooting revealed that the release unfortunately included a major bug. A new version was quickly developed and released to mitigate this bug, but a restart of all clusters in all regions was required for the fix to take effect. Between the time of the release (16:00 CET) and the time of the restart (17:20 CET) customers will have experienced errors while indexing. During the restart the service was completely unavailable and customers would have seen HTTP 5XX errors. Restart was completed 18:30 CET.
2. After the restart a minor number of cluster nodes in EU region were still experiencing intermittent errors. It was concluded that there was a separate and isolated issue with one of the load balancers. Nodes were removed from the load balancer and then re-added and the issue was mitigated at 20:50 CET
3. One cluster in the EU regions was still triggering alerts and an error was found in a configuration file coupled to the earlier release. The configuration for the specific cluster was corrected and released again. All clusters were now fully operational 22:00 CET.

Timeline

16:00 CET – The final steps of the deploy took place.

16:03 CET – First alarms came from the EU region.

16:20 CET – The deploy is done but the errors persist. Continuing to investigate.

16:50 CET – The team realize the incorrect part of the deploy and fix it.

17:17 CET – All clusters in all regions are restarted to make the change active and fix the indexing issue.

18:30 CET – The last cluster is fully back after restart. There is still intermittent errors from a few EU clusters.

20:50 CET – The cause for the errors are found in a load balancer and fixed. There is still one cluster in EU not responding.

21:58 CET – The error for the last cluster in EU is found in a configuration file. The configuration file is changed and the last cluster is now working as normal again.

22:00 CET – All monitors report normal operation for all regions of Find.

Root Cause

During the timeframe of the outage there was three separate issues with each a separate root cause:

1. The root cause for the first issue was due to a major bug not discovered during QA and testing.
2. The root cause for the second issue was due to a malfunctioning load balancer, possibly due to the size of the restart operation.
3. The root cause for the third issue was that the wrong configuration file was pushed to one of the clusters due to a bug in the code that push configuration.

Corrective and Preventative Measures

We will be reviewing our internal processes regarding QA and and code review to mitigate that these type of errors seen in issue 1 and 3 can get into production going forward. The second issue was a one time occurence that we will not be investigating further.

Final Words

We apologize for the impact to affected customers. While we are proud of the availability we have on FIND we know how critical this service is to customers. For us, availability is the most important feature and we will do everything we can to learn from the event and to avoid a recurrence in the future.