SUMMARY

Episerver Search & Navigation (formerly Find) is a cloud-based enterprise search solution that delivers enhanced relevance and powerful search functionality to websites. On Friday February 5, 2021 and Saturday February 6, 2021 we experienced events which impacted the functionality of the service in the Canada East Digital Experience Cloud region. Details of the incident are described below.

DETAILS

Between February 5, 17:30 PM UTC and February 6, 03:04 AM UTC the Search & Navigation cluster CAEA01 experienced intermittent outages.

Retrospective analysis concluded the issue started at 17:30 PM UTC on February 5, 2021, but as health checks were reporting a healthy state, troubleshooting was not initiated until 18:51 PM UTC, after clients reported issues. Mitigation of the incident was prolonged due to the fact that nothing was alerting, and the cluster continued to serve requests. Once the issue was identified, a node was disconnected from the cluster but still considered as active, and an expansion of nodes was triggered to spread the load. The service started to recover, but the issue resurfaced until the faulty node was deallocated. The service was fully restored at 03:04 UTC on February 6, 2021.  

TIMELINE

February 5, 2021

18:49 UTC – Client reported issues with request towards cluster CAEA01. 

18:51 UTC – Issue escalated to engineering and initiated investigation. 

_23:26 UTC – Status page updated.
_

February 6, 2021

00:57 UTC – Issue identified and workaround initiated. 

01:10 UTC – Workaround implemented and service started recovering. 

02:21 UTC – Alert is triggered by monitoring system and investigation continued. 

02:50 UTC – Fix implemented.

03:04 UTC – Critical alert resolved and service fully operational. 

ANALYSIS

The investigation identified that a client node went into a bad state, and had been disconnected from the cluster, but continued to receive requests. This scenario did not trigger any alerts, and the cluster was considered to be in a healthy state, as a subset of requests continued to be served from other available client nodes. 

IMPACT

During the events, a subset of requests to this Search & Navigation cluster may have experienced network timeouts (5xx-errors), or slow response times when trying to connect. 

CORRECTIVE MEASURES

Short-term mitigation

• Graceful node restarts.
• Expansion of client node count was performed to properly spread the workload and avoid request timeouts. 
• Degraded node were forced to be deallocated.

Long-term mitigation

• Improve health-checks to exclude nodes that are not connected to the cluster.

FINAL WORDS

We apologize for the impact to affected customers. We have a strong commitment to deliver high availability for our Search & Navigation service. We will continue to prioritize our efforts in proving to overcome these recent difficulties, and will do everything we can to learn from the events to avoid a recurrence in the future.