SUMMARY

Episerver Search & Navigation (formerly Find) is a cloud-based enterprise search solution that delivers enhanced relevance and powerful search functionality to websites. On Wednesday March 24, 2021 we experienced an event which impacted the functionality of the service in the EU Digital Experience Cloud region. Details of the incident are described below.

DETAILS

Between 10:17 UTC and 21:20 UTC on March 24, 2021 the Search & Navigation cluster AZNEUPROD01 experienced an outage.

What triggered the issue was a significant increase of the average query latency, causing spikes in the JAVA heap memory consumption. The Engineering team worked diligently to identify the cause of the increased latency, and apply mitigation steps to reduce the impact. Rolling restarts were performed to resolve the incident, and the the service was fully operational at 21:20 UTC.

TIMELINE

March 24, 2021

10:17 UTC – First alert and automation restarts were triggered. 

13:59 UTC – Second alert acknowledged and troubleshooting started.

16:15 UTC – Mitigation steps were initiated. 

16:15 UTC – Status page updated.

16:49 UTC – Proxy nodes were restarted.

18:00 UTC – Rolling restarts of data nodes. 

19:20 UTC – Service started recovering. 

21:20 UTC – Critical alert resolved and service fully operational.

ANALYSIS

The cause of this issue was a significant increase of average query latency, caused by a number of tenants on the shared resources. This resulted in high JAVA Heap memory consumption, and failing "garbage" collects. When this happens, the cluster master evicts those nodes from the cluster, and mark them as failed. The default behavior is then to redistribute the data over the remaining nodes, which caused the memory issue to be moved to another node, and the same scenario was repeated on the other nodes. 

IMPACT

During the events, requests to this Search & Navigation cluster may have experienced network timeouts (5xx-errors), or slow response times when trying to connect. 

CORRECTIVE MEASURES

Short-term mitigation

• Reduced number of search threads during the ongoing incident.
• Adjusted queue capacity to optimize the processing of incoming requests. 
• Restart of proxy nodes and rolling restarts of data nodes to recover service as soon as possible. 

Long-term mitigation

• This issue is linked to periods of high load on the cluster, and our recommendation is to move into the new Search & Navigation Service platform. This consists of the latest cloud-based version with improved architecture. Migration can be requested by contacting support.

FINAL WORDS

We apologize for the impact to affected customers. We have a strong commitment to deliver high availability for our Search & Navigation service. We will continue to prioritize our efforts in proving to overcome these recent difficulties, and will do everything we can to learn from the event to avoid a recurrence in the future.