SUMMARY

Optimizely Search & Navigation (formerly Find) is a Cloud based Enterprise Search solution that delivers both enhanced relevance and powerful search functionality to websites. During mid-March we experienced intermittent episodes which impacted service functionality in the EU Digital Experience Cloud region. The following report describes additional details around the events.

DETAILS

On March 16, 2022, at 06:55 UTC the Engineering team received an alert related to the Search & Navigation EMEA01 cluster within the EU Digital Experience Cloud region. Immediately upon receiving the alert, troubleshooting began, and the team identified a condition with high JVM Heap Memory usage which did not release on their own.  The affected nodes were gracefully restarted to clear the memory hooks and service was fully operational by 07:16 UTC.

On March 17, 2022, at 12:55 UTC, a similar issue presented itself and it was determined to be the same issue reoccurring.  Additional mitigation actions were implemented immediately, and service was restored by 13:22 UTC. 

On March 18, 2022 at 12:08 UTC and March 24, 2022 at 11:08 UTC, the service was degraded intermittently and recovered itself.  The Engineering team has been working diligently to identify the root cause of the degradation, and apply mitigation steps to reduce the impact. As the Search & Navigation service contains a multitude of components, including but not limited to, API calls towards the cluster, mitigation efforts required a complex and exhaustive investigation, thereby requiring additional time to stabilize and identify the resolution. 

TIMELINE

March 16, 2022

06:55 UTC – First alert and automation restarts were triggered. 

07:10 UTC – Critical alert triggered. Alert acknowledged and troubleshooting initiated.

07:16 UTC – Unhealthy nodes were identified and restarted.

07:16 UTC – Service operation recovered. 

‌

March 17, 2022

12:55 UTC – Critical alert triggered. Investigation immediately started. 

13:03 UTC – Nodes restarting was performed. 

13:09 UTC – Second alert triggered. Additional mitigation was applied. 

13:11 UTC – STATUSPAGE updated 

13:22 UTC – Service was fully recovered. Root cause identified and the Engineering team started working on long-term mitigation actions. 

‌

March 18, 2022

12:08 UTC – Alert triggered by monitoring system. Service degradation occurred intermittently.

12:13 UTC – The service recovered by itself.  

‌

March 24, 2022

11:08 UTC – Critical alert triggered. 

11:09 UTC – Node restarting was automatically performed. 

11:13 UTC – Service fully operational. 

ANALYSIS

According to the results of the investigation, the cluster was overwhelmed by several large customer indices which had grown beyond the recommended size configuration. When an index exceeds the recommended size allocation, it will cause one or more nodes to expend the available Heap Memory with the potential for additional cluster failure. 

IMPACT

During this series of events, a subset of requests to the EMEA11 Search & Navigation cluster may have experienced network timeouts (5xx-errors), or slower than normal response times while attempting to connect. 

CORRECTIVE MEASURES

Short-term mitigation

• Restarting of unhealthy node(s) and reallocation of shards.
• Indices identified as the source of the incident were contacted for immediate mitigation.

Long-term mitigation

• To minimize potential customer impact, indices that have grown or are on the verge of exceeding the recommended configuration will be migrated to an alternate cluster.
• To manage growing accounts, a new tooling will be built that enables faster internal migrations within a cluster. 

FINAL WORDS

We are sorry! We recognize the negative impact to affected customers and regret the disruption that you sustained. We have a strong commitment to our customers, and to deliver high availability services including Search & Navigation. We will continue to prioritize our efforts to overcome these recent difficulties, and diligently apply lessons-learned to avoid future events of similar nature to ensure that we continue to develop Raving Fans!