SUMMARY

Optimizely Search & Navigation (formerly Find) is a Cloud based Enterprise Search solution that delivers both enhanced relevance and powerful search functionality to websites. Between September 12, 2022, and September 13, 2022, we experienced events which impacted service functionality in the EU Digital Experience Cloud region. The following report describes additional details around the events.

DETAILS

On September 12, at 11:03 UTC, the Search & Navigation cluster EMEA15 experienced service degradation. As an initial mitigation effort, automation restarts were performed for the nodes with high heap usage which triggered the shard reassignment process. Troubleshooting further, the engineering team identified an unused tenant of shared resources which was consuming resources. The removal of the tenant, along with the completion of shard reallocation, recovered the cluster. Service was fully operational by 12:35 UTC. 

On September 13, at 08:14 UTC, the service experienced an outage caused by heavy bulk requests from a tenant of shared resources. This additional load on the cluster led a few data nodes to expend their available Heap Memory which in turn evicted these nodes from the cluster. Mitigation efforts including automation node restart and rate limit adjustment brought the cluster back to healthy state. Service was fully operational by 09:25 UTC. 

TIMELINE

September 12, 2022

11:03 UTC – First alert triggered, acknowledged and troubleshooting initiated. 

11:06 UTC – Critical alert triggered. Troubleshooting continues

11:19 UTC – Unhealthy nodes were identified and restarted. 

11:21 UTC – STATUSPAGE updated.

11:22 ~ 12:22 UTC – Issue was identified, and mitigation action was immediately applied. 

12:30 UTC – Service started recovering.

12:35 UTC – Service fully operational.

‌

September 13, 2022

08:14 UTC – First alert triggered, acknowledged and troubleshooting initiated. 

08:27 UTC – Unhealthy nodes were identified, and automation restart was triggered.  

08:38 UTC – STATUSPAGE updated.

08:40 ~ 09:10 UTC – Issue was identified, and a workaround was implemented.

09:15 UTC – Service started recovering and was able to handle the requests. 

09:25 UTC – Service fully operational.

ANALYSIS

The performance was affected by the high workload of the data nodes, which was triggered by excessive bulky queries and the extra resource consumption of an idle tenant in shared resources. 

IMPACT

During this series of events, a subset of requests to the EMEA15 Search & Navigation cluster may have experienced network timeouts (5xx-errors), or slower than normal response times while attempting to connect. 

CORRECTIVE MEASURES

Short-term mitigation

• Restart of unhealthy nodes and removal of the unused tenant.
• Indices identified with improper setting were contacted for immediate mitigation

Long-term mitigation

• To minimize potential customer impact and stabilize the cluster, we continue to plan and monitor the infrastructure capacity to ensure a healthy and highly reliable performance. 
• Identify the indices that have grown or are on the verge of exceeding the recommended configuration to migrate them to an alternate cluster.

FINAL WORDS

We are sorry! We recognize the negative impact to affected customers and regret the disruption that you sustained. We have a strong commitment to our customers, and to deliver high availability services including Search & Navigation. We will continue to prioritize our efforts to overcome these recent difficulties, and diligently apply lessons-learned to avoid future events of similar nature to ensure that we continue to develop Raving Fans!