SUMMARY

Episerver Search & Navigation (formerly Find) is a cloud-based enterprise search solution that delivers enhanced relevance and powerful search functionality to websites. On Friday April 2, 2021 we experienced an event which impacted the functionality of the service in the US Digital Experience Cloud region. Details of the incident are described below.

DETAILS

Between  April 2, 12:05 PM UTC and April 2,16:26 PM UTC on April 2 the Search & Navigation cluster USPROD08 experienced an outage.

What triggered the issue was a significant increase of expensive queries causing spikes in JAVA heap memory consumption. Restarts were performed to clear memory and service was fully operational at 16:26 PM UTC.

TIMELINE

April 2, 2021

12:05 UTC – First alert and automation restarts were triggered.

12:42 UTC – Critical alert acknowledged and troubleshooting initiated.

13:05 UTC –  An unhealthy node was identified and restarted.

13:11 UTC – Service started recovering.

13:40 UTC – Service is restored and monitored.

‌

15:10 UTC – Nodes got evicted from cluster, resulting in a so called "Split brain" scenario.

15:57 UTC – Status Page updated.

16:00 UTC – Nodes were restarted and reconnected to cluster.

16:26 UTC – Critical alert resolved and service fully operational.

ANALYSIS

The cause of this incident was a peak in JAVA heap memory consumption. Analysis identified that the cluster was overwhelmed due to a sudden increase of expensive requests. This caused over-allocation of the heap on the nodes, and failing "garbage" collects. This in turn created failed queries on the node, with high heap usage. When that node was restarted, the Elasticsearch primary shard being indexed to, moved to another node. The scenario replayed itself until the cluster's internal shards had been re-balanced, and the service was fully operational.

IMPACT

During the events, requests to this Search & Navigation cluster may have experienced network timeouts (5xx-errors), or slow response times when trying to connect.

CORRECTIVE MEASURES

Short-term mitigation

• Restart of unhealthy nodes to recover the service as soon as possible.

Long-term mitigation

• This issue is linked to periods of high load on the cluster, and our recommendation is to move to our new Search & Navigation Service platform which consists of the latest cloud version with improved architecture. Migration can be requested by contacting Support.

FINAL WORDS

We apologize for the impact to affected customers. We have a strong commitment to deliver high availability for our Search & Navigation service. We will continue to prioritize our efforts in proving to overcome these recent difficulties, and will do everything we can to learn from the event to avoid a recurrence in the future.