Summary

On Monday 2017-10-02 we had an event impacting Episerver FIND. The following report describes additional details around that event.

Details

On Monday 2017-10-02 13:41 CEST, the Find Operations team got several alerts from the global monitoring system on the Find services located in Sweden. Where several customers have their Find solution hosted. After some initial investigation it was discovered that all customers that are hosted in Sweden were affected. The issue was located to a database which holds the licence keys. The calls to the Find service could not be checked against the database due to it was corrupt. A restoration of the database was started straight away and after the restoration was complete the service was back in order again. Continuous investigation into why the database went corrupt was performed and it was found that it was caused by a memory issue.

Timeline

13:41 CEST - Several alarms are triggered in the SE region of Find

13:41 CEST - The operations team starts investigation of the issue.

14:05 CEST - The issue is found to be a corrupt database and measures to solve it are put in place.

15:10 CEST - Full functionality is back.

Root Cause

The root cause of this issue was the licence database that became corrupt after a memory issue. This resulted in "HTTP 401 Unauthorized" errors when calls were made to the Find service.

Impact on services

During the time of the issue customers using the service in the Sweden region would have experienced "HTTP 401 Unauthorized" responses to their requests. Corrective and Preventative Measures

This issue was confined to a specific database that is used in the Find service. The operations team are in a first stage making sure a better monitoring is put in place. To make sure they detect the root cause quicker. They are also making sure to have a quicker way of restoring the database if this was to ever happen again. In a second stage the operations team together with the developers are looking into a solution where this kind of issue wouldn't affect the calls to the service so the normal functionality is secured.

Final Words

We apologize for the impact to affected customers. While we are proud of the availability we have on FIND we know how critical this service is to customers. For us, availability is the most important feature and we will do everything we can to learn from the event and to avoid a recurrence in the future.