Summary

EPiServer FIND is a platform service that extends the search features for Episerver, allowing you to build advanced filtering and faceted navigation based on the behavior of website visitors. Both on-premise hosted applications and Digital Experience Cloud hosted applications use the service.

On Wednesday the 23rd of May 2018 we had an event impacting Episerver FIND. The following report describes additional details around that event.

Incident total duration: 42 minutes.

Details

On Monday 2018-05-23 15:27 CEST, the Engineering Operations team received an alert from the global monitoring system on one of the US Find Clusters. The team began troubleshooting the issue immediately, and found it to be isolated to a specific Elasticsearch cluster. The initial investigation revealed that several nodes in the cluster were unresponsive due to problems with the garbage collection. This resulted in a so called "Split Brain" scenario where nodes fall out of the cluster configuration. The affected nodes were restarted, and the cluster was functional again. After approximately 20 minutes we started seeing the same error again and a decision was made to perform a full restart of the cluster. Although the issue was identified, it took additional time for the cluster to be fully functional and required two full cluster restarts until service was fully functional at 16:59 CEST.

Timeline

15:27 CEST- First alarm is triggered by global monitoring system that one of the US Find cluster is not working properly.

15:30 CEST- Second alarm is triggered and the engineering team has identified that it´s due to the garbage collection, which has resulted in Split Brain scenario.

15:33 CEST- Nodes in cluster are restarted due to a split brain.

15:52 CEST- Cluster is functional.

16:15 CEST- Cluster is having issues again and the whole cluster is restarted.

16:27 CEST- Cluster is functional.

16:43 CEST- Cluster is having problems again.

16:44 CEST- Full cluster restart.

16:59 CEST- The cluster is back to normal operation and global monitoring system is reporting full functionality again.

Root Cause

The root cause of this issue was a peak of requests that affected the cluster in a way that it stopped responding and couldn't recuperate by itself.

Impact on other services

During the event, applications using this specific cluster of FIND would have seen network timeouts or slow response times trying to connect to the service.

Corrective and Preventative Measures

This issue is related to the Elasticsearch component of Episerver FIND. Elasticsearch will be upgraded to a newer version during this year, and this issue will be fixed as a part of that upgrade. This issue is also linked to periods of high load on the cluster. An ongoing task is to move indices from this cluster to others to spread the load.

Final Words

We apologize for the impact to affected customers. While we are proud of the availability we have on FIND we know how critical this service is to customers. For us, availability is the most important feature and we will do everything we can to learn from the event and to avoid a recurrence in the future.