SUMMARY 

Optimizely Search & Navigation (formerly Find) is a Cloud based Enterprise Search solution that delivers both enhanced relevance and powerful search functionality to websites. On Thursday August 25, 2022, we experienced an event which impacted service functionality in the EU Digital Experience Cloud region. The following report describes additional details around the events.

DETAILS 

Between August 25, 2022 08:23 and 08:55 UTC, the Search & Navigation cluster EMEA15 experienced service degradation. The investigation indicated that the master node was overloaded by a long-running garbage collection process and concurrent connection spike in a few data nodes. Nodes expansion and restart of the master node were performed to distribute the unexpected spike of connections more evenly and mitigate the incident. Service was fully operational by 08:55 UTC. 

TIMELINE 

August 25, 2022

08:23 UTC – First alert triggered and was acknowledged immediately

08:33 UTC – Further investigation revealed a long-pending garbage collection process and mitigation action started 

08:40 UTC – Second alert triggered. Root cause was identified, and a fix was applied

08:42 UTC – STATUSPAGE updated

08:50 UTC – Service started recovering

08:55 UTC – Service fully operational

ANALYSIS

Investigation discovered an unexpected high number of update mapping requests which overloaded the master node, degrading its ability to handle other management tasks. 

IMPACT 

During this series of events, a subset of requests to the EMEA15 Search & Navigation cluster may have experienced network timeouts (5xx-errors), or slower than normal response times while attempting to connect.

CORRECTIVE MEASURES 

Short-term mitigation

• Restarting unhealthy master node and adding more cluster resources. 
• Indices identified with improper setting were contacted for mitigation

Long-term mitigation

• To minimize potential customer impact and stabilize the cluster, we continue to plan and monitor the infrastructure capacity to ensure a healthy and highly reliable performance. 
• Identify the indices that have grown or are on the verge of exceeding the recommended configuration to migrate them to an alternate cluster.

FINAL WORDS

We are sorry! We recognize the negative impact to affected customers and regret the disruption that you sustained. We have a strong commitment to our customers, and to deliver high availability services including Search & Navigation. We will continue to prioritize our efforts to overcome these recent difficulties, and diligently apply lessons-learned to avoid future events of similar nature to ensure that we continue to develop Raving Fans!