Summary

During Monday we had a FIND event in the EMEA region for one of our clusters. The following report describes additional details around that event.

EPiServer FIND is a platform service that extends the search features for Episerver, allowing you to build advanced filtering and faceted navigation based on the behaviour of website visitors. The service is used by both on-premise hosted applications and Digital Experience Cloud hosted applications.

Details

The first alert was triggered at 2016-10-31 10:52 CEST by our monitoring system. The alert was sent through our automated alert triage system to the technical team who takes action on these alerts.

The technical team started troubleshooting the issue at 10:53 CEST and found the issues to be isolated to a specific elasticsearch cluster. After some initial investigation it was discovered that several nodes in the cluster were unresponsive due to problems with garbage collection. This resulted in a so called "Split Brain" scenario where nodes fall out of the cluster configuration. The decision was made to do a restart of the affected nodes in the cluster to minimize the impact of the issue. The restart was executed at 11:13 CEST. The cluster came back to normal function quickly with just minor errors showing until 12:03 CEST.

10:52 CEST - First alarm is trigged.

10:53 CEST - The technical team starts investigation of the issue.

11:05 CEST - It is found that the cluster had several nodes that were unresponsive and this resulted in a split brain scenario.

11:13 CEST - The affected nodes are restarted and the cluster comes back to normal functionality.

12:03 CEST - The global monitoring system is reporting that the cluster is functional again.

Impact on other services

During the event, applications using this specific cluster of FIND would have seen network timeouts or slow response times trying to connect to the service.

Corrective and Preventative Measures

This issue is related to the elasticsearch component of Episerver FIND. Elasticsearch will be upgraded to a newer version during next year, this issue will be fixed as a part of that upgrade. This issue is also linked to periods of high load on the cluster. An ongoing work is to move indices from this cluster to others to spread the load.

Final Words

We apologize for the impact to affected customers. While we are proud of the availability we have on FIND we know how critical this service is to customers. For us, availability is the most important feature and we will do everything we can to learn from the event and to avoid a recurrence in the future.