SUMMARY

Episerver Search & Navigation (A.K.A. Find) is a Cloud-based Enterprise Search solution that delivers enhanced relevance and powerful search functionality to websites. On Wednesday the 12th of August 2020, we had an event which impacted the functionality of the service in the APAC Digital Experience Cloud region and the following report describes additional details around the event. 

DETAILS

Between 2020-08-12 04:13 UTC and 2020-08-12 11:22 UTC FIND Cluster APAC01 experienced an outage.

The issue was caused by a sudden rise in incoming connections to the client nodes in this specific FIND cluster. Upon detecting the increased load, engineers deployed an initial mitigation. After continued monitoring of the mitigation it was retracted as it didn't meet the required results. Expansion of client node was performed and service was restored at 10:20 UTC on August,12th 2020.

At 02:00 UTC on 13th August 2020, the issue reoccurred briefly when the faulty hardware was removed. A limited subset of clients would have experienced 5xx-errors or longer response times during this time. 

TIMELINE

August 12th, 2020

04:13 UTC – First alert triggered, acknowledged and investigation initiated. 

05:41 UTC – STATUSPAGE updated 

06:42 UTC – Mitigation steps were applied.  

09:50 UTC – The issue was identified and fix started being implemented.

10:02 UTC – Client node scale-out triggered.

10:20 UTC – Service is recovering.

11:22 UTC – Critical alert resolved and service fully operational.**
**

August 13th, 2020

02:00 UTC – Alert triggered and acknowledged. 

02:15 UTC – Full service functionality restored. 

ANALYSIS

The investigation identified one of the client nodes in this specific FIND cluster could no longer connect to the cluster due to an unknown hardware issue which prevented incoming connections from closing and thereby piled up. When the scale-out operation was performed the new node could once again handle the incoming steady sum of connections and service was restored. When the faulty client node was removed from the cluster another short disruption was experienced which confirmed that this issue was isolated to one specific node.

IMPACT

During the event, a subset of requests to this FIND cluster would have seen network timeouts (5xx-errors), or slow response times when trying to connect. 

CORRECTIVE MEASURES

Short-term mitigation

• Engineering team performed restarts to recover service as soon as possible.
• Expansion of client node count was performed to more evenly distribute the unexpected spike of connections. 

Long-term mitigation

• After a thorough investigation, the cause was concluded to be due to an unexpected hardware failure that will have a minimal likelihood of re-occurrence and therefore no further analysis will be taken at this time. If the issue appears again, a deeper investigation on the cause will be initiated. 

FINAL WORDS

We apologize for the impact to affected customers. We have a strong commitment to delivering high availability for our Find service. We will continue to prioritize our efforts in proving to overcome these recent difficulties and will do everything we can to learn from the event to avoid a recurrence in the future.