Summary

On Wednesday 2017-09-06 we had an event impacting Episerver FIND. The following report describes additional details around that event.

Details

On Wednesday 2017-09-06 12:04 CEST, the Find Operations team got an alert from the global monitoring system on slow response and time outs on one of our EMEA Find Clusters. Where several customers have their Find solution hosted. The operations team started right away to solve the issue. The issue was quickly found and the operations team fixed it straight away. Although the issue was fixed quickly it took time for the cluster to start again. At 14:58 CEST the alarm was reset and full functionality was restored.

Root Cause

The root cause of this issue was a peak of traffic that affected the cluster in that way it stopped responding and couldn't recuperate by itself.

Impact on other services

During the time of the issue customers using the service would have experienced slow response or "HTTP 503 Service Unavailable" responses to their requests.

Corrective and Preventative Measures

This issue is related to the elasticsearch component of Episerver FIND. It is planned for a newer version of elasticsearch, this issue will be fixed as a part of that upgrade. This issue is also linked to periods of high load on the cluster. An ongoing work is to move indices from this cluster to others to spread the load.

Final Words

We apologize for the impact to affected customers. While we are proud of the availability we have on FIND we know how critical this service is to customers. For us, availability is the most important feature and we will do everything we can to learn from the event and to avoid a recurrence in the future.