SUMMARY

Episerver Search & Navigation (A.K.A. Find) is a Cloud-based Enterprise Search solution that delivers enhanced relevance and powerful search functionality to websites. On Tuesday the 9th of June 2020 we experienced an event which impacted the functionality of the service in the US Digital Experience Cloud region. Details of the incident are described below.

DETAILS

Between 2020-06-09 04:06 UTC and 2020-06-09 04:46 UTC US FIND Proxy Layer es-us-api01.episerver.com experienced an outage.

Investigation discovered that a Redis master node in a specific cluster was having an issue post maintenance which affected the Redis slave nodes and caused the proxy service to stop running. A restart was performed and service was fully operational at 04:46 AM UTC.

TIMELINE

June 9th, 2020

~ 04:06 UTC – First alert was triggered by monitoring system.

~ 04:10 UTC – Alert acknowledged and troubleshooting initiated.

~ 04:23 UTC – Mitigation steps were applied.

~ 04:33 UTC – Service started to recover.

~ 04:54 UTC – StatusPage updated.

~ 05:07 UTC – Root cause identified & Incident Closed.

ANALYSIS

The event was triggered due to a scheduled maintenance where the risk evaluation had been deemed to be low and normally a non-impacting event. During a maintenance a node is excluded from the load balancer and service continue to function when maintenance is carried out on the removed node. This was also the case during this window but due to a human error the node in maintenance was still acting as a master and when the node was rebooted the slave nodes were impacted and the proxy service. This resulted in that the nodes could not continue to process any requests.

IMPACT

During the events, requests to this FIND service would have seen network timeouts.

CORRECTIVE MEASURES

Short-term mitigation

The slave mode was disabled on the Redis master node to make them work independently and a restart performed to ensure the change was implemented.

Long-term mitigation

We will be reviewing our internal processes to mitigate that these types of errors can arise in production going forward.

FINAL WORDS

We apologize for the impact to affected customers. We have a strong commitment to delivering high availability for our Find service. We will continue to prioritize our efforts in proving to overcome these recent difficulties and will do everything we can to learn from the event to avoid a recurrence in the future.