Summary

Early Saturday morning we experiencded a FIND event in the EMEA region for one of our clusters. The following reports describes additional details around that event.

EPiServer FIND is a platform service that extends the search features for Episerver, allowing you to build advanced filtering and faceted navigation based on the behavior of website visitors. The service is used by both on-premise hosted applications and Digital Experience Cloud hosted applications.

Details

The first alert was fired at 01:05 by our global monitoring system that every minute queries the service and measures response time. The alert was sent through our automated alert triage system to on-duty staff in the technical team who recieves the alert through an SMS and voice call. We started troubleshooting the issue at 01:15 and found the issues to be isolated to a specific elasticsearch cluster and a specific node within that cluster. After some initial investigation we discovered that the node had problems performing a garbage collect and was showing memory errors in it's logs. The decision was made to gracefully restart the node and the services was restored at 01:46. After about 20 minutes we started seeing the same errors again and we could also see other nodes starting to have the same issues. After additional troubleshooting we decided to perform a cold restart of the entire cluster to restore services. The operation to perform a restart of an entire cluster is time consuming due to the number of nodes to be restarted, and it requires planning under the specific circumstances to avoid data loss. When planning activities was completed the restart was initiated at 05:00. After the restart was completed services were restored and we closed the incident at 06:40.

Timeline

2016-08-06 01:05 - First alert sent to technical teams.

2016-08-06 01:15 - Troubleshooting is started.

2016-08-06 01:46 - Services restored.

2016-08-06 02:06 - Second alert sent to technical teams.

2016-08-06 05:00 - Cluster restart commences.

2016-08-06 06:40 - All services restored.

Impact on other services

During the event applications using this specific cluster of FIND would have seen network timeouts or slow response times trying to connect to the service.

Corrective and Preventative Measures

This issue is a rarely seen problem, but it is related to the elasticsearch component of Episerver FIND. Elasticsearch will be upgraded to a newer version during the fall, this issue will be fixed as a part of that upgrade.

Final Words

We apologize for the impact to affected customers. While we are proud of the availability we have on FIND we know how critical this service is to customers. For us, availability is the most important feature and we will do everything we can to learn from the event and to avoid a recurrence in the future.