Search & Navigation Performance Issue - US Region (USEA05)
Incident
Report for Optimizely Service
Resolved
This incident has been resolved, and the service is currently operational. The queue that we previously activated on the service is disabled, and we will continue our investigation to determine the cause.
Update
We have multiple engineering teams working diligently to stabilize the service back to expected behaviour. While we are working to resolve the underlying issue we have identified a set of compute intense write-operations contributing to the instability of the cluster. To reduce stress on the service, we have enabled queuing. This will cause a delay in ingestion of new data.
We appreciate your patience and will continue to update the progress in regular intervals.
We appreciate your patience and will continue to update the progress in regular intervals.
Monitoring
Mitigation steps have been applied, and service is currently operational.
We continue to put effort to make cluster stable, further updates will be provided regularly.
Thank you for your patience.
We continue to put effort to make cluster stable, further updates will be provided regularly.
Thank you for your patience.
Investigating
The performance on the Search & Navigation (FIND) service in the US region, USEA05 is currently running in a degraded state. A subset of clients will be intermittently experiencing high latency or 5xx-errors.
The investigation is ongoing. We appreciate your patience and will continue to update with progress.
The investigation is ongoing. We appreciate your patience and will continue to update with progress.
Monitoring
We have applied additional mitigation efforts and the service is now stabilized. We will continue to monitor the health of the service and will post additional updates as the are available.
Update
We are continuing to monitor the health of the service and apply mitigation efforts to improve service stability
Update
The cluster got into an unstable state due to high amount of write-operations from multiple tenants at the same time. We are continuing to investigate the issue and will inform updates accordingly.
Update
We continue work in multiple regions for stabilization of the service. We appreciate your patience and will continue to update with progress.
Update
We are working on stabilizing the service and will continue to update with progress.
Update
We are continuing implementing fixes to make service stabile, further updates will be provided regularly.
Update
We are continuing to implement fixes to make service stabile.
Update
We are continuing investigate the issue updates will be provided regularly.
Investigating
Service is currently running in a degraded state. The investigation is ongoing, further updates will be provided regularly.
Monitoring
A workaround has been implemented, and we are continuing to monitor for any further issues.
Update
We are continuing to implement and monitor mitigation efforts and the service is steadily improving
Identified
Mitigation efforts remain ongoing and we are continuing to work on stabilizing the service
Monitoring
Mitigation efforts have been applied and the service health is restored. We are continuing to monitor the health of the service and will make continued updates accordingly.
Update
We are continuing to investigate the issue and taking remediation steps to recover the service.
We appreciate your patience and will continue to update with progress
We appreciate your patience and will continue to update with progress
Investigating
We are currently investigating an event that is impacting the performance on the Search & Navigation (FIND) service in the US region, USEA05.
A subset of clients will be intermittently experiencing high latency or 5xx-errors.
A subset of clients will be intermittently experiencing high latency or 5xx-errors.
This incident affected: Search and Navigation - NA Region (US East).