Summary

Episerver Find is a Cloud based Enterprise Search solution that delivers both enhanced relevance and powerful search functionality to websites. On Thursday the 5th of July 2018 we had an event impacting the functionality of the service and the following report describes additional details around that event.

Details

On Wednesday the 4th of July 2018 1:00 PM CEST a release deployment was initiated of a minor update to the FIND backend. FIND leverages a continuous deployment model where updates normally are rolled out without interruption to services, and the risk of a deployment is considered to be very low. The update was rolled out in stages and all stages was completed without issues. After several hours a latency alert was triggered and engineers started to investigate to identify the root cause of the increased latency. 
As the investigations proceeded, more alerts were triggered and the likely cause was found to be the new backend version. The decision was quickly made to rollback to the older version and that was completed on the 6th of July 2018 at 09:55 AM CEST.

Timeline

2018-07-04 1:00 PM CEST Episerver Find backend release starts.

2018-07-05 11:15 AM CEST - First alarm is triggered by global monitoring system that one of the US Find cluster is not functioning properly and investigation begins.

2018-07-05 11:46 AM CEST - The US Find Cluster is restarted and is fully operational at 11:57 AM CEST. At this point in time there is no suspicion of correlation to the new release.

2018-07-05 3:19 PM CEST - Episerver Find backend releases continues. 

2018-07-05 17:40 PM CEST - Second alarm is triggered by global monitoring system that one of the US Find cluster is not functioning properly and delivering high response time and a cluster restart is performed. 

2018-07-06 0:40 AM CEST  - 2018-07-06 4:20 CEST - Increased latency alerts and external availability checks failing indicates issues with multiple clusters in multiple regions.

2018-07-06 4:20 AM CEST - Decision made to rollback.

2018-07-06 4:24 AM CEST - First cluster rolled back and operational. Team was monitoring that the rollback had the desired outcome.

2018-07-06 4:50 - 6:00 AM CEST - All clusters triggered to rollback.

2018-07-06 09:55 AM CEST - All clusters in all regions are back to normal operations and global monitoring system is reporting full functionality.

Root cause

The root cause was a performance bug in this new backend version. This bug caused query queues to build up and response times to slowly increase over time. 

Corrective and Preventative Measures

We are constantly reviewing our internal process in an effort to identify and drive opportunities for continuous improvement. To improve our processes and to mitigate the risk of this happening again, we will be working with the global team on how to correlate these events in a quicker manor to be able to perform a rollback immediately in case of such an event. We are also reviewing our change peer-review process to further mitigate that bugs are introduced. 

Final Words 

We apologize for the impact to affected customers. While we are proud of the availability we have on FIND we know how critical this service is to customers. For us, functionality and availability is the most important feature and we will do everything we can to learn from the event and to avoid a recurrence in the future.