SUMMARY

Episerver Digital Experience Cloud™ Service (DXC Service) is the cloud-based offer from Episerver based on Microsoft cloud technology. A solution that delivers high availability and performance, easy connectivity with other cloud services and existing systems, ability to manage spikes in customer demand, and a platform that is ready to seamlessly adopt the latest technology updates.

On October 22th and 23rd, 2019, DXC Service customers were unable to run deployments and the following root cause analysis was provided by Microsoft.

DETAILS

Between 21:15 UTC on 22 Oct 2019 and 00:23 UTC on 23 Oct 2019, a subset of customers using Storage in West Europe may have experienced service availability issues. In addition, resources with dependencies on the impacted storage unit may have experienced downstream impact in the form of availability issues or high latency. On 21 Oct 2019, a different storage scale unit in the region was first impacted by this bug. Some customers may have been impacted by both incidents if their workloads were spread across both of the storage scale units. As part of mitigating the first incident, we had blocked the backend operation that triggered the bug until the fix was deployed. However, we did not identify a second entry point which could trigger the bug resulting in the incident on 22 Oct 2019.

ANALYSIS

Azure Storage service uses an automatic load balancing system to partition and balance customer workloads across different servers within a storage scale unit. A partition master role maintains the map of how the partitions are distributed across the different partition servers. A routine maintenance operation on one of the partitions caused an inconsistency in the partition map due to a bug. This caused the servers handling the inconsistent state to crash and resulted in a single storage scale unit in West Europe becoming unhealthy. This also resulted in downstream availability impact to several dependent services in the region.

IMPACT

Clients were not able to complete deployments through the self-service portal, PaaS-portal.

CORRECTIVE MEASURES

Episerver was working closely with Microsoft on this issue and has been informed that the following corrective and preventive actions have or will be undertaken by Microsoft.

1. Engineers developed a tool to correct the inconsistency in the partition map on the impacted scale unit. The time taken to develop and test this tool resulted in a higher than expected recovery time. Engineers also blocked the backend operation that triggered the inconsistent state until the underlying bug is fixed. Once service health was restored to the backend Storage scale unit, all dependent services automatically recovered.
2. Engineers understand the root cause of the bug which triggered the inconsistent system state and have rolled out a fix across production to help ensure the triggering functionality has been disabled from all entry-points.
3. Engineers are hardening the load-balancing system to detect and reject inconsistencies in the partition map in a safe manner to prevent impact on system health.

FINAL WORDS

We sincerely apologizes for the impact to affected customers. We are continuously with Microsoft taking steps to improve the Microsoft Azure Platform and the processes to help ensure such incidents do not occur in the future.