Optimizely Data Platform - APIs - NA Region
Incident
Report for Optimizely Service
Resolved
This incident has been resolved.
Monitoring
We are currently addressing two issues affecting the performance of our services. Please find the details of these issues and their current status below:
Issue 1: Increased GraphQL Usage.
We observed an unexpected surge in GraphQL usage that resulted in service performance degradation across multiple customers. To address this, we have implemented enhanced rate-limiting controls to better distribute system resources and prevent a single source from impacting overall service availability. These actions have successfully stabilized performance, allowing all customers to use GraphQL without errors. We are continuing to optimize our infrastructure to better handle variable load patterns and support growing usage demands across our customer base.
Issue 2: Elevated GraphQL Request Latency.
Since around 5 am UTC, there has been an increase in GraphQL request latency, leading to slower segment evaluations for customers. While requests are completing, they are taking longer than usual. We are currently investigating the cause, which appears to be related to routine maintenance. We expect this maintenance to be completed shortly, with improvements in latency anticipated soon after.
Issue 1: Increased GraphQL Usage.
We observed an unexpected surge in GraphQL usage that resulted in service performance degradation across multiple customers. To address this, we have implemented enhanced rate-limiting controls to better distribute system resources and prevent a single source from impacting overall service availability. These actions have successfully stabilized performance, allowing all customers to use GraphQL without errors. We are continuing to optimize our infrastructure to better handle variable load patterns and support growing usage demands across our customer base.
Issue 2: Elevated GraphQL Request Latency.
Since around 5 am UTC, there has been an increase in GraphQL request latency, leading to slower segment evaluations for customers. While requests are completing, they are taking longer than usual. We are currently investigating the cause, which appears to be related to routine maintenance. We expect this maintenance to be completed shortly, with improvements in latency anticipated soon after.
Update
We are continuing to work on the fix for the issue and additional updates will be provided as they become available.
Thank you for your patience!
Thank you for your patience!
Update
We are continuing to work on the fix for the issue identified earlier, and additional updates will be provided as they become available.
We thank you for your patience as teams work toward a solution.
We thank you for your patience as teams work toward a solution.
Update
We are continuing to work on a fix for this issue. Teams have taken all steps available to mitigate the issue, though some customers may still experience intermittent symptoms. We thank you for your patience as teams work toward a solution.
Update
We thank you for your patience as our efforts on a fix remain ongoing. Additional updates will be provided as they become available.
Update
We are continuing to work on a fix for this issue. Thank you for the patience!
Identified
The issue has been identified and a fix is being implemented for the event collection and other APIs.
We are still investigating the issue with GraphQL API. Customers may experience high request latency and errors when using GraphQL.
We are still investigating the issue with GraphQL API. Customers may experience high request latency and errors when using GraphQL.
Investigating
We are investigating the issue with ODP APIs that can lead to high request latency and errors.
This incident affected: Optimizely Data Platform - NA Region (Event Collection, Data Ingestion and Processing, Core App Experiences).