Summary

Episerver Find is a Cloud based Enterprise Search solution that delivers both enhanced relevance and powerful search functionality to websites. On Monday the 25th of March 2019 and continued in the morning of the 26th March we had an event impacting the functionality of the service in the EU Digital Experience Cloud region and the following report describes additional details around the event.

Total incident duration: 58 minutes.

Details

Between the time period 2019-03-25 and 2019-03-26, the Reliability Engineering team received alerts regarding one of the EU Digital Experience Cloud regions. Immediately upon receiving the first alert, troubleshooting started, and the issue was identified that nodes in the isolated cluster had problems with JVM Garbage Collection not clearing memory or taking to long time to perform. This results in memory heap exceeding 80% usage and causes Elasticsearch not to execute queries correctly and customers experience high response times or 5xx errors logged on the client side. Continued intermittent alerts received from March 25th 12:10 PM CET until 26th of March 10:25 AM CET and Reliability Engineering performed rolling restarts of the failing nodes to clear memory to resolve the immediate incidents but not the root cause. Decision was made to expand data nodes and to implement a workaround that automatically clear the ES cache to lower the memory consumption to avoid further re-occurrences and service was restored on the March 26th.

Timeline

March 25th, 2019

~ 12:10 PM CET: First alert is triggered by monitoring systems and troubleshooting commenced.
~12:30 PM CET: Second alert is triggered by monitoring systems.
~12:43 PM CET Rolling restarts are performed on failing nodes to mitigate high JVM Heap usage
~13:44 PM CET: Service restored.

March 26th, 2019

~06:06 AM CET: New alert triggered and mitigated by node restart.
~07:25 AM CET: Issue still exists and escalates to more nodes and graceful node restarts are performed.
~09:00 AM CET: Retrospect meeting performed for post mortem.
~09:30 AM CET: Development of fix commenced.
~15:08 PM CET: Fix implemented, performance monitored and incident closure.

Root cause

The root cause for this incident is failing or long running JVM garbage collects that manifests itself in high response times or 5xx errors for the .NET Client

Impact on other services

During the event, applications using this specific cluster of FIND would have experienced network timeouts or slow response times trying to connect to the service.

Corrective and Preventative Measures

Immediate preventative measures to reduce memory consumption was to provision fifteen (15) new data nodes to lower the load on each data node and to implement a clear cache function in Elasticsearch to reduce issues with garbage collection in Java. Long term solution is to implement a more effective cache eviction policy to remove the oldest data and therefore enable memory limiting.

Final Words

We place the utmost importance and pride on achieving and sustaining the highest level of availability for our customers and we regret any disruption in service you have experienced. We continue to work tirelessly to ensure any and all service disruptions are prevented and or mitigated and we will use this incident to further these efforts to help ensure you receive a reliable and positive experience.