PAAS Portal - Deployment Completion Status

Incident Report for Optimizely Service

Postmortem

SUMMARY

Optimizely Customer-Centric Digital Experience Platform (DXP; formerly Digital Experience Cloud™ Service - DXC Service) is the cloud-based offer from Optimizely (formerly Episerver) based on Microsoft cloud technology. A solution that delivers high availability and performance, easy connectivity with other cloud services and existing systems, ability to manage spikes in customer demand, and a platform that is ready to seamlessly adopt the latest technology updates.

Between May 14, 2021 and May 18, 2021, DXP customers experienced issues with deployments and the following information provides details around this incident.

DETAILS

A increasing number of customers experienced issues when trying to complete/reset deployments starting from May 14, 2021 up to May 18, 2021. The DXP Service Management portal did not issue any errors, but the old code remained active even though a "complete" step was performed for the deployment.

TIMELINE

May 13, 2021

06:01 UTC: First report of a deployment completion that did not enable the new code.

19:28 UTC: Incident is escalated to the Optimizely engineering team.

19:36 UTC: Incident is acknowledged by the Optimizely engineering team.

23:53 UTC: Behavior is verified by the Optimizely engineering team.

‌

May 14, 2021

05:49 UTC: A support ticket was opened with our third-party vendor.

09:19 UTC: A few more reports were escalated, and the status page was updated stating that a deployment issue affecting a subset of customers had been identified.

10:41 UTC: A manual procedure was put in place for completing deployments for affected customers.

12:27 UTC: No reported case of this issue had been escalated or seen after 2021-05-14 03:25:00 (UTC), and some customers that had the issue earlier could be seen having successful deployments.

18:46 UTC: A limited number of reported problems was handled with the manual workaround, and troubleshooting to determine the root cause continued with our third-party vendor.

‌

May 17, 2021

10:10 UTC: More reports of issues with completing deployments started to come in, and multiple customers were requesting a manual completion of their deployments.

20:49 UTC: During the development of a temporary workaround, it was discovered that the scope of the issue was bigger than previously known, and therefore required further development. Root cause was identified.

May 18, 2021

09:14 UTC: The number of affected deployments increased, and a decision was made to close down the deployment feature of the DXP Service Management portal, to avoid possible outages due to failed deployments.

14:30 UTC: A workaround was implemented and deployments were enabled again

May 19, 2021

08:19 UTC: After monitoring new deployments since the workaround was implemented without finding new issues, the incident was closed

ANALYSIS

The root cause for the problem was identified to be caused by an update in Azure, which triggered a bug in the PowerShell module that is used to update application settings during deployments. While the bug remains open, the workaround implemented resolved the issue so that updating the application settings worked reliably again. The rollout in Azure was increasing the number of affected deployments as it progressed globally.

IMPACT

A increasing number of customers experienced issues when trying to complete/reset deployments starting from May 14, 2021 up to May 18, 2021. It eventually seemed to affect all deployments and the decision was made to instead close down the deployment feature of the DXP Service Management portal completely.

CORRECTIVE MEASURES

A workaround was implemented to prevent the bug error in the third-party tooling from happening. Our third-party vendor is working on patching the issue on their side.

FINAL WORDS

We place the utmost importance and pride on achieving and sustaining the highest level of availability for our customers and we regret any disruption in service you have experienced. We continue to work tirelessly to ensure any and all service disruptions are prevented and or mitigated and we will use this incident to further these efforts to help ensure you receive a reliable and positive experience.

Posted May 21, 2021 - 12:36 UTC

Resolved

This incident has been resolved.

We will continue our investigation with Microsoft to establish permanent resolution. A post mortem will be published as soon as it becomes available.

Posted May 19, 2021 - 08:19 UTC

Update

The results of the workaround that was implemented earlier have been positive thus far. We will continue to closely monitor this and will provide additional updates going forward.

Posted May 18, 2021 - 20:17 UTC

Update

We are continuing to monitor for any further issues.

Posted May 18, 2021 - 14:30 UTC

Update

A workaround has been implemented and we are monitoring the results.

Posted May 18, 2021 - 14:20 UTC

Update

As we have continued to work on short term mitigation we have received more reported issues due to a recent update by Microsoft Azure. We have therefore taken the decision to disable the deployment capabilities both through user interface and API in DXP to avoid possible outages due to failed deployments until we have a functioning workaround available or a long term fix has been released by our supplier.

We apologize for the inconvenience and will share updates regularly.

Posted May 18, 2021 - 09:29 UTC

Update

A root cause of the issue has been identified to be a bug reported in third party vendors. While the Engineering team is actively coordinating towards permanent solution, the hotfix will be continue rolling out as soon as it becomes ready to mitigate the active impact.
We highly recommend to not start any new deployments at this time or contact our support if you urgently need to schedule a deployment https://support.episerver.com/hc/en-us
We apologize for the inconvenience and we will continue provide new updates.

Posted May 18, 2021 - 07:12 UTC

Update

The testing for a hotfix for this issue remains ongoing and will require a bit more time to complete. While the fix is unlikely to be rolled out this evening, we are continuing to work diligently to get this rolled out as soon as possible. We apologize for the inconvenience and we will continue to provide updates as to its progress.

Posted May 17, 2021 - 19:44 UTC

Update

We are currently in the process of testing a hotfix for this issue and we will provide an update here once the fix has been implemented.

Posted May 17, 2021 - 15:01 UTC

Update

We are continuing to work with third party to establish the resolution.
If the situation changes we will provide a new update.

We sincerely apologize for the inconvenience!

Posted May 17, 2021 - 08:26 UTC

Update

While we are continuing to monitor for further issues, it is possible this issue may reoccur for a subset of clients. For those impacted, we ask that you submit a support case to have the deployment manually performed until we are able to identify and implement a permanent fix. We sincerely apologize for this inconvenience and will continue to provide updates as they become available.

Posted May 14, 2021 - 18:56 UTC

Monitoring

A workaround has been implemented and we are monitoring the results.

Posted May 14, 2021 - 13:51 UTC

Update

The Engineering team is continuing to work closely on the issue with our third party vendors on getting the underlying platform issues resolved.

Posted May 14, 2021 - 12:48 UTC

Identified

We are having an issue with PaaS Portal deployment completion. A subset of clients will be experiencing the old codes remain active while the Portal report successful results.
The root cause has been identified and the work-around are in place. The Engineering team is actively working with our third parties on getting the underlying platform issues resolved.
Please contact support to manually complete your upcoming deployments.
We apologize for the inconvenience and will share an update once we have more information.

Posted May 14, 2021 - 09:19 UTC