SUMMARY

Optimizely Customer-Centric Digital Experience Platform (DXP; formerly Digital Experience Cloud™ Service - DXC Service) is the cloud-based offer from Optimizely (formerly Episerver) based on Microsoft cloud technology. A solution that delivers high availability and performance, easy connectivity with other cloud services and existing systems, ability to manage spikes in customer demand, and a platform that is ready to seamlessly adopt the latest technology updates. 

On Tuesday June 29, 2021 we experienced an event which impacted the functionality of the DXP Management Portal. The following information provides details around this incident.

DETAILS

Between June 29, 2021 3:29 UTC and June 29, 2021 7:12 UTC, the DXP Management Portal experienced an outage.

The DXP Management Portal had ceased to operate, projects were no longer visible, and new deployments could not be launched. This had no effect on live environments, albeit managing them in the DXP Management Portal was not feasible at the time. Investigation identified that the cause of the malfunction was a human error, and as soon as the issue was identified, appropriate actions were taken to revert the process. The service was fully operational at June 29, 2021 7:12 UTC. 

TIMELINE

June 29, 2021

03:29 UTC – First alert triggered and troubleshooting started.

03:40 UTC – Root cause was quickly identified and mitigation actions were initiated. 

03:57 UTC – STATUSPAGE updated. 

07:12 UTC – The DXP Management Portal was recovered and monitored.

07:25 UTC – Service operational and critical alert resolved.

ANALYSIS

The incident was caused by human error which resulted in data inconsistency on the platform.

IMPACT

During the event, users were unable to access their projects in the DXP Management Portal, and could not deploy new versions of their sites.

CORRECTIVE MEASURES

Short-term mitigation

• A fix was applied to restore data to original state.

Long-term mitigation

• We are constantly reviewing our internal process in an effort to identify and drive opportunities for continuous improvement. To improve our processes and to mitigate the risk of this happening again, we will be working with the team on how to avoid similar events in the future. 

FINAL WORDS

We place the utmost importance and pride on achieving and sustaining the highest level of availability for our customers and we regret any disruption in service you have experienced. We continue to work tirelessly to ensure any and all service disruptions are prevented and or mitigated and we will use this incident to further these efforts to help ensure you receive a reliable and positive experience.