SUMMARY

Episerver Search & Navigation (A.K.A. Find) is a Cloud-based Enterprise Search solution that delivers enhanced relevance and powerful search functionality to websites. On Thursday the 20th of August 2020, we had an event which impacted the functionality of the service in the EU Digital Experience Cloud region and the following report describes additional details around the event. 

DETAILS

Starting at 18:55 UTC on 20 August 2020, the Engineering team received an alert for FIND cluster EMEA07. Troubleshooting initiated immediately and the investigation discovered a tenant of the shared resource was stuck in the state of deletion progress. Those remnants caused red condition in cluster and resulted in requests to be failed towards the specific index. Once the tenant was completely removed, the service was back in green state and fully operational at 22:44 UTC on 20 August 2020. 

TIMELINE

August 20th, 2020

18:55 UTC – First alert triggered, acknowledged and investigation initiated. 

20:09 UTC – Mitigation steps were applied. 

20:47 UTC – STATUSPAGE updated 

22:38 UTC – The issue was identified and fix was applied.

22:44 UTC – Critical alert resolved and service fully operational.

ANALYSIS

The investigation identified that the cluster turned into red state due to a termination process where the deletion operation for a specific index never completed. Cluster red state indicates that at least one index has lost its primary which is normal during a deletion but as the process got hang it kept the cluster to be in red state the whole time until faulty index was removed. 

IMPACT

During the event, the impact was limited to requests towards the specific index which was in the process of deletion. 

CORRECTIVE MEASURES

Short-term mitigation

• Enable rate limiting on bulk requests to recover service as soon as possible.
• A fix was applied to complete the termination and removing the faulty index.

Long-term mitigation 

• This is the first time we have seen this error and no deeper investigation will be performed. If the issue appears again, a deeper investigation on the cause will be initiated. 

FINAL WORDS

We apologize for the impact to affected customers. We have a strong commitment to delivering high availability for our Find service. We will continue to prioritize our efforts in proving to overcome these recent difficulties and will do everything we can to learn from the event to avoid a recurrence in the future.