SUMMARY

Optimizely Search & Navigation (formerly Find) is a Cloud based Enterprise Search solution that delivers both enhanced relevance and powerful search functionality to websites. On Wednesday, January 4, 2023, we experienced an event which impacted service functionality in the APAC Digital Experience Cloud region. The following report describes additional details around the event.

DETAILS

Between January 4, 2023, 09:40 and 15:08 UTC, a subset of customer using the Search & Navigation cluster APAC04 experienced an intermittent service degradation.

The investigation revealed a coincidental timing with an outage reported by Microsoft which impacts Virtual Machines instances in Australia East resulting in connection failures when trying to access some Virtual Machines hosted in the region.  

TIMELINE

January 4, 2023

09:57 UTC – First alert triggered and troubleshooting initiated. 

11:28 UTC – The issue was escalated to Microsoft.

11:37 UTC – STATUSPAGE updated. 

11:55 UTC - Microsoft confirmed the issue is linked to their outage and was escalated to their Product Team. 

14:45 UTC – Issue was mitigated, and the request were properly handled. 

15:04 UTC – Service was fully operational. 

ANALYSIS

The cause of the incident was confirmed to be caused by the platform change at Microsoft. While building out new capacity for the Azure Ultra Disk Storage service in the Australia East region, a virtual IP address that was already in use by a Premium Storage instance was incorrectly assigned to a new instance of the Ultra Disk storage service. Due to the incorrect assignment of the virtual IP address, traffic to the Azure Premium Storage service was impacted in this region, causing service availability issues for services relying on Storage. 

IMPACT

During this series of events, a subset of requests to the Search & Navigation cluster may have experienced network timeouts (5xx-errors), or slower than normal response times while attempting to connect. 

CORRECTIVE MEASURES

Short-term mitigation

•  Microsoft restored the VIP to the correct configuration to restore Premium Storage service traffic, mitigating the issue. 

Long-term mitigation

• Microsoft has scanned all existing IP allocations to existing storage capacity and new storage capacity being built, to check for any similar issues. 
• Microsoft improve buildout process by adding automated validations to eliminate incorrect virtual IP address allocation under any circumstances.
• Microsoft enhanced monitoring including continuously checking virtual IP address allocation consistency across storage services,

FINAL WORDS

We are sorry! We recognize the negative impact to affected customers and regret the disruption that you sustained. We have a strong commitment to our customers, and to deliver high availability services including Search & Navigation. We will continue to prioritize our efforts to overcome these recent difficulties, and diligently apply lessons-learned to avoid future events of similar nature to ensure that we continue to develop Raving Fans!