SUMMARY

Episerver Search & Navigation (A.K.A. Find) is a Cloud-based Enterprise Search solution that delivers enhanced relevance and powerful search functionality to websites. On Tuesday the 28th of January and Wednesday 29th of January 2020 we had events impacting the functionality of the service in the EU Digital Experience Cloud region and following report describes additional details around the event.

DETAILS

Starting at 17:08 UTC on 28 Jan 2020, the Engineering team received an alert for FIND cluster EMEA05 and troubleshooting initiated immediately. Rolling restart of each node in the cluster was performed as an initial mitigation action and service was recovered at 17:31 UTC on Tuesday the 28th of January, 2020.

On Wednesday 29 Jan 2020 at 12:13 UTC the issue reoccurred and it was identified that the cause was due to an extensive amount of failing or long running JVM garbage collects and by increasing merging segment, less memory can be used and reduce the amount of Garbage collection. A hotfix was implemented to resolve the immediate issue and service was restored at 17:56 UTC on January 29th, 2020.

TIMELINE

January 28th, 2020

17:08 UTC – First alert is triggered by monitoring system.

17:09 UTC – Alert acknowledged and troubleshooting initiated.

17:10 UTC – Rolling restarts are performed to mitigate high memory utilization.

17:31 UTC – Alert closed, and service recovered.

‌

January 29th, 2020

12:13 UTC – Second alert is triggered and immediately acknowledged.

12:20 UTC – Engineering starts working on a fix.

13:05 UTC – STATUSPAGE updated.

16:01 UTC – Third alert is triggered.

16:30 UTC – Engineering starts to implement a fix.

17:56 UTC – Global monitoring system is reporting full functionality restored.

ANALYSIS

The cause of this incident was due to an extensive increase of failing or long running JVM garbage collects.

IMPACT

During the event, requests to the specific FIND cluster would have experienced network timeouts (5xx-errors) or slow response time trying to connect to the service.

CORRECTIVE MEASURES

Short-term mitigation

· Performed node restarts as short-term mitigation to recover service as soon as possible.

· Engineering deployed a fix.

· An additional cluster has been provisioned to reduce the customer load on EMEA05.

Long-term mitigation

· Develop and release a fix to increase threads for merge segment operations in ES to reduce amount of GC.

FINAL WORDS

We sincerely apologize for the impact to affected customers. Customer experience is of high priority for us, and we have a strong commitment to delivering high availability for our services. We will do everything we can to learn from the event to avoid a recurrence in the future.