Summary

Episerver Find is a Cloud based Enterprise Search solution that delivers both enhanced relevance and powerful search functionality to websites. On Wednesday the 27th of February 2019 we had an event impacting the functionality of the service in the EU Digital Experience Cloud region and the following report describes additional details around the event.

Total incident duration: 60 minutes.

Details

On Wednesday 2019-02-27 02:51 AM CET, the Reliability Engineering team received an alert regarding one of the EU Digital Experience Cloud regions. Immediately upon receiving the alert troubleshooting started and the investigation discovered nodes in a specific cluster had problems with JVM Garbage Collection and this caused slow response times. Graceful restarts were performed on the identified nodes to clear memory and service was fully operational at 01:35 PM CET.

Timeline

February 27th, 2019

02:51 AM CET: First alert is triggered by monitoring systems.

02:51 AM CET: Alert acknowledged and troubleshooting initiated. Nodes are restarted.

03:20 AM CET Problem solved for this incident.

10:20 AM CET Second alert is triggered by monitoring system.

10:21 AM CET Nodes are restarted.

11:00 AM CET Issue still exists and escalates to more nodes.

11:50 AM CET Problem solved.

12:34 PM CET System is unstable again. Full restart.

01:35 PM CET Service restored.

02:30 PM CET Expansion of data nodes in Find cluster.

Root cause

The root cause of this incidents was caused by the proxy queue's being exhausted and search queries being evicted from the queue due to long running queries in the back end. The cause for the long running queries in the back end was failed garbage collects that caused JVM memory heap to reach 100% usage on some of the cluster nodes.

Impact on other services

During the event, a subset of applications using this specific cluster of FIND would have seen network timeouts (5xx-errors) or slow response times trying to connect to the service.

Corrective and Preventative Measures

After incident resolution, a post-mortem meeting was held where the decision was made to deploy extra nodes to reduce memory consumption.

Engineers will continue to investigate the root cause of failing gen3 memory garbage collects in order to mitigate any deeper root causes.

Final Words

We apologize for the impact to affected customers. Availability is of high priority for us and we have a strong commitment to delivering high availability for our Find service. We will continue to prioritize our efforts in proving to overcome these recent difficulties and will do everything we can to learn from the event to avoid a recurrence in the future.