Degraded availability for backend web targeting products
Incident Report for Optimizely.com
Resolved
Availability has remained high been increasing for the last 2 hours. Users of personalization campaigns using DCP, ListAttributes or recommendations have all functionality restored.
Posted Oct 01, 2020 - 16:19 PDT
Monitoring
The engineering team has now validated their theory with a change to the CDN and application caching layers and availability has been increasing for the last 30 minutes. We will continue to monitor systems availability and latency.
Posted Oct 01, 2020 - 14:48 PDT
Identified
The replacement containers allowed for a brief recovery period before availability dropped again. The engineering team has now identified a likely root cause of the repeated failures and is validating that theory with a change to the CDN and application caching layers.
Posted Oct 01, 2020 - 12:45 PDT
Update
The engineering team has identified some possible root causes while deploying replacement compute containers now.
Posted Oct 01, 2020 - 11:25 PDT
Investigating
We are currently investigating an issue causing a spike in errors to our backend web targeting systems (tapi.optimizely.com) beginning at 07:07 PDT. Users of personalization campaigns using DCP, ListAttributes or recommendations may see 5xx responses to this API until the issue is resolved.
Posted Oct 01, 2020 - 09:43 PDT
This incident affected: Core APIs (Targeting API).