Intermittent issue with SSL handshake
Postmortem
SUMMARY
There was an issue with SSL handshakes between Azure App services and Cloudflare CDN that affected a subset of customer in the EMEA Digital Experience Cloud region.
DETAILS
On January 20th 2020 at 19:54 UTC, Episerver began to receive alerts for a subset of EMEA clients sites being down. Initial investigations revealed there was an SSL handshake issue between the Azure App services and the Cloudflare CDN that caused a small percentage of traffic for a subset of EMEA customers to have a 525 error. The issue resolved on its own within 1 hour for each of the affected clients.
TIMELINE
2020-01-20 19:54 UTC- Initial reports of client sites being down. Episerver initiate investigation.
2020-01-20 21:47 UTC- Issue escalated to Engineering for further investigation.
2020-01-20 22:11 UTC- Support ticket raised with Microsoft.
2020-01-21 09:18 UTC– STATUSPAGE updated. _
_
2020-01-22 08:36 UTC– All alerts cleared.
2020-01-23 09:47 UTC- Support ticket raised with Cloudflare.
2020-01-23 13:33 UTC– INCIDENT Closed.
2020-02-03 20:43 UTC- Microsoft and Cloudflare both respond that a root cause could not be identified and there is nothing further for them to investigate.
ANALYSIS
After thorough investigations together with our third party suppliers a root cause has not been established.
IMPACT
A limited subset of Digital Experience Cloud customers experienced 525 errors while accessing the site.
CORRECTIVE MEASURES
There were no immediate corrective measures put into place as the issue was showing to resolve itself after a brief duration of impact.
As this was an unique, rare circumstance it was not possible to determine the cause of the issue but would this occur again we will provide our third-party providers a packet capture during the time of the incident för further analysis.
FINAL WORDS
We place the utmost importance and pride on achieving and sustaining the highest level of availability for our customers and we regret any disruption in service you have experienced. We continue to work tirelessly to ensure any and all service disruptions are prevented and or mitigated and we will use this incident to further these efforts to help ensure you receive a reliable and positive experience.
Resolved
We will continue our investigation to establish the root cause. A RCA will be published as soon as it becomes available.
Update
Monitoring
We will provide more updates as soon as they become available.
Update
We will continue to provide updates in a timely manner as to the progress/status of this investigation.
Update
We apologize for the inconvenience and will provide another status update today at 21:00 UTC or sooner if more information becomes available.
Update
We apologize for the inconvenience, and will post an update as soon as we've confirmed the issue is fully resolved.
Investigating
We are actively working with our third parties to find a root cause.
We apologize for the inconvenience and will share an update once we have more information.