SUMMARY

Episerver Find is a Cloud-based Enterprise Search solution that delivers enhanced relevance and powerful search functionality to websites. On Thursday the 12th of December 2019 we experienced an event which impacted the functionality of the service in the APAC Digital Experience Cloud region. Details of the incident are described below.

DETAILS

Between 2019-12-12 5:30 UTC to 2019-12-12 07:18 UTC Find cluster APAC01 experienced an outage.

Episerver FIND leverages a CI/CD deployment workflow, meaning that software is deployed to environments automatically as soon as it is ready and received a green from our QA team. This is normally a non-impacting event, meaning deployment are done in stages not to impact the delivery of the service. However, during this deployment there was an issue that had not been caught in testing. Troubleshooting revealed that the release unfortunately included a bug and an updated fixed version was quickly released to mitigate the issue. Service was fully functional at 07:18 UTC December 12th, 2019.

TIMELINE

December 12th, 2019

05:30 UTC – First alert triggered and investigation initiated.

06:21 UTC – STATUSPAGE updated

06:42 UTC – Root caused identified. Engineering started working on the fix.

07:18 UTC – Fix was successfully implemented and service restored.

07:21 UTC - Incident closed.

ANALYSIS

The event was caused by the latest release being deployed to the cluster which contains an upgrade of Node.js.

The root cause was a breaking change in Node.js, forcing base64-encoding which caused crypto hashing differences, leading to an authentication failure.

As all our production credentials are kept inaccessible due to security constraints, this single production cluster suffered from these effects.

IMPACT

During the event, requests to the FIND cluster would have returned with 401 (unauthorized) error message.

CORRECTIVE MEASURES

Short-term mitigation

A hotfix was released and deployed to production to mitigate the issue.

Long-term mitigation

We will be reviewing our internal processes on QA to mitigate that these types of errors can arise in production going forward.

FINAL WORDS

We sincerely apologize for the impact to affected customers. Customer experience is of high priority for us, and we have a strong commitment to delivering high availability for our services. We will do everything we can to learn from the event to avoid a recurrence in the future.