Increased error rates due to AWS outage

Incident Report for Optimizely.com

Resolved

The system has recovered as of midnight (EST) last night. Data is populating in accounts as quickly as we can process it, but for larger accounts this may take some time.

Posted Dec 23, 2021 - 03:21 PST

Monitoring

Optimizely continues to experience ongoing issues as a result of the AWS power loss early on December 22nd. The team is still working to return the system to full health.

Our current understanding is that no incoming data was lost, but that it may still be many hours before all of the data is processed and the system is fully caught up.

Posted Dec 22, 2021 - 16:30 PST

Identified

Due to an AWS power outage and associated issues (https://status.aws.amazon.com/), the Experimentation product is experiencing a variety of errors throughout its infrastructure. A repair process is in place, but service will be degraded until the issue is fully resolved.

Posted Dec 22, 2021 - 05:54 PST

This incident affected: Experimentation Core APIs (Targeting API, Results API, Event API, Optimizely REST API).