52 Courier incidents · giugno 2022 — official updates, affected components, duration and resolution details.
I messaggi sono in coda con conseguente invio ritardato.
Abbiamo trovato il problema. Crediamo che sia un problema con il nostro provider cloud. Fissare e' stato eliminato.
I messaggi stanno scorrendo di nuovo. Continueremo a monitorare.
Tutti i messaggi in coda sono stati elaborati.
Tradotto automaticamente dall'aggiornamento ufficiale dell'incidente.
Stiamo attualmente indagando su un problema con messaggi ritardati.
Il problema è stato identificato e stanno lavorando per risolvere un problema. Tutti i messaggi che sono stati in coda saranno inviati.
La correzione come è stata implementata e i messaggi in coda stanno cominciando a essere elaborati di nuovo
Tutti i messaggi in coda sono stati elaborati. Continueremo a monitorare.
Questo incidente è stato risolto.
Tradotto automaticamente dall'aggiornamento ufficiale dell'incidente.
The team has identified a regression causing send requests to be stuck in a queued state. We have initiated a rollback to quickly resolve the issue. More updates to follow
The rollback deploy will land in about 30 minutes.
The team has identified that the most recent deployment was not the cause of the regression. The team is still working on identifying the root cause. Messages will not be dropped, and they will be processed once the fix is out.
Messages are being processed while our pipeline continues to process enqueued messages.
Send pipeline has resolved and enqueued messages are now sending properly.
Slack has reported messaging, and app degredation on their platform that could be impacting deliverability of Slack messages. This is a downstream provider issue and can be monitored on Slack's statuspage: https://slack-status.com/
Please continue to monitor Slack's statuspage for any updates related to message deliverability: https://slack-status.com/
The Courier team has encountered an issue on our platform impacting several services. The team has rolled out a redeploy to roll back changes.
Due to GH Actions experiencing an outage(https://www.githubstatus.com/), the redeploy took longer than expected and are now cutting a release.
We are continuing to work on a fix for this issue. The following services are still impacted: - Login - Webapp
The current outage still impacts the current services: - webhooks - login - API endpoints - FE access. Our team is actively working on the redeployment.
The Courier team has a redeploy ready and waiting on GH Actions to recover from their degradation to publish the release.
The team is exploring alternate methods to redeploy the release until GH Actions recovers.
The team has rolled back the deployment manually bypassing GH Actions outage. We're waiting for the deployment to land and confirm that all core functionality has been restored.
The manual rollback has successfully landed, and all services are operational. The team is diligently monitoring and testing other services.
EU instance redeployment is around 10 minutes out to land production.
EU Courier instance deployment has landed and services operational.
All services operational. A public-facing RFO will be available in the coming days. Thank you for your understanding and continued partnership.
# RFO: February 2, 2026 — Service Interruption ## Executive Summary On February 2, 2026, a deployment introduced a configuration change that referenced a file not present in our production build artifacts, causing backend services to become unavailable. API endpoints, automations, and tracking functionality were impacted for 2 hours and 17 minutes in the US region and 2 hours and 42 minutes in the Ireland region. Recovery was prolonged by a concurrent outage at GitHub Actions, our CI/CD provider, which prevented our standard automated rollback. Our team identified the root cause, executed a manual rollback independent of the affected provider, and restored full service across all regions. We have defined targeted action items to prevent recurrence. ## Incident Overview * **Affected services:** API endpoints \(including message sending\), automations, webhooks, tracking links, and authentication * **Impact:** Requests to backend services returned errors for the duration of the incident. Users were unable to send messages, trigger automations, or access tracking data. No data was lost — requests were rejected before ingestion, so no messages were partially processed or left in an inconsistent state. * **Detection:** Our monitoring systems flagged elevated error rates within minutes of the issue beginning. * **Contributing factor:** GitHub Actions, our CI/CD provider, experienced a complete outage from 10:35 to 16:30 PST. This overlapped with our incident window and prevented our standard automated rollback from executing, extending the time to resolution. ## Timeline of Events All times in PST. | **Time** | **Event** | | --- | --- | | 10:59 | Deployment of latest release initiated through standard CI/CD pipeline | | 11:44 | Deployment completed and went live; services immediately began experiencing errors due to a missing configuration dependency | | 11:49 | GitHub Actions, our CI/CD provider, experienced a complete outage, preventing standard rollback procedures | | 11:52 | Monitoring alerts triggered; engineering team engaged | | 12:00 | Incident declared; rollback initiated; engineering team assembled | | 12:07 | Status page updated — issue identified and rollback in progress. Rollback ends up being blocked by GitHub actions outage. | | 12:39 | Team pivoted to an alternative manual rollback approach. This required testing and validating the new approach. | | 13:50 | Team executed the alternative manual rollback independent of GitHub Actions after they were fully satisfied that the new approach was safe. | | 14:01 | Manual rollback completed in US region; services confirmed operational | | 14:26 | Ireland region deployment completed; services confirmed operational | | 15:13 | All services verified stable across all regions; incident resolved | ## Root Cause Analysis The disruption was traced to a configuration change included in the latest release. The change introduced a startup dependency on a utility file that was intended to be bundled with the deployment package. However, the file was not included in the production build artifacts. When backend services attempted to initialize, they were unable to locate the required file and could not start, resulting in all incoming requests being rejected. This discrepancy was not caught prior to production deployment because the file was present and functioning correctly in the development environment. The difference in how build artifacts are assembled between development and production environments meant the issue only manifested in production. ## Mitigation and Resolution 1. Upon identifying the root cause, the team immediately initiated a rollback to the prior known-good release through our standard CI/CD pipeline. 2. A complete outage at GitHub Actions, our CI/CD provider, prevented the automated rollback from completing. The team identified this external dependency and pivoted to an alternative approach. 3. The team executed a manual rollback by retrieving prior deployment artifacts from our backup storage and deploying them directly, bypassing the affected CI/CD pipeline entirely. 4. Services were restored region by region, with the US region confirmed operational at 14:01 PST and the Ireland region at 14:26 PST. 5. Extended monitoring was conducted across all services before declaring the incident fully resolved at 15:13 PST after we were satisfied everything had been stable for over 45 minutes. ## Action Items | **#** | **Action Item** | **Owner** | **Priority** | **Status** | | --- | --- | --- | --- | --- | | 1 | Require successful staging deployment and smoke tests before any production deployment | Engineering | P1 | In Progress | | 2 | Improve the reliability of automated smoke tests | Engineering | P1 | In Progress | | 3 | Add build-time validation to confirm all referenced startup dependencies are present in deployment packages | Engineering | P2 | Open | | 4 | Adopt an expedited rollback process as the standard emergency procedure, independent of CI/CD provider availability, reducing recovery time by approximately 35 minutes | Engineering | P2 | In Progress |
The Courier team identified an issue affecting observability metrics between 21:38 UTC and 22:13 UTC, during which metrics briefly experienced an outage. A fix was released and metrics are stabilized through observability channels. Metrics received during the outage window will not be accounted for in observability dashboards.
The Courier team was alerted of a small degredation in the Courier web application. We have identified the issues and have rolled out a fix to mitigate. The current features will encounter some latency: - Audience Calculation - Audit Trail logging We will follow up with updates.
Fix is out and application is stable
An issue with our service provider, AWS, has impacted all Courier services. The engineering team is working on resolution.
We are continuing to work on a fix for this issue.
Service has been restored. Engineering is working through system review and will continue to monitor.
The incident has been resolved
The Courier team was alerted of an issue with MS Teams requests experiencing errors sending. We have identified the issue and published a fix that will land in production shortly
Release is in production and Microsoft Teams messages are passing successfully.
Courier experienced a slowdown in message delivery with Mailgun providers around 2AM PST. Our monitoring systems caught the spike in the bottleneck, which eventually stabilized. Our infrastructure team is actively monitoring.
Courier's delivery platform experienced significant latency around 1am PST. Our platform team managed to resolve the issue and is actively monitoring for any potential issues.
An issue with logging in to the Courier platform and overall app performance was identified by our team and a revert of the offending release has been deployed. The estimated time for the release to land is 45 minutes.
Revert has landed and webapp is operational.
Cloudflare has reported an outage that will impact SSO features provided by WorkOS. Customers with Okta SSO might experience some degradation when signing in. Our team is monitoring the status for both Cloudflare and WorkOS. WorkOS status: https://status.workos.com/incidents/k9s870cktcsf Cloudflare Status: https://www.cloudflarestatus.com/incidents/25r9t0vz99rp
WorkOS systems are back to normal.
The Courier team has identified an issue where there are significant delays of up to 10 minutes for messages. We are currently investigating and looking for a root cause to mitigate. Updates will follow.
Backed-up messages were caused by a bottleneck in our system, which has started to catch up. Messages will stabilize as the bottleneck clears.
Message bottleneck has cleared and the queue has stabilized. We will continue to monitor.
Messages have stabilized but we will continue to monitor.
Messages have recovered and pipeline is stable again.
We are currently investigating an issue where EU workspace notifications appear to remain in the “queued” state.
The issue has been identified and a fix is being implemented.
The issue has been resolved. The root cause was purely a logging issue, notifications were still sent but showing up in logs as queued. All logs have been rehydrated
The Courier team has identified an issue with a recent deployment that is impacting AWS SES integrations. We have found the root cause and have reverted the changes. The team is gathering impact.
The reverted changes will land in production in around one hour. The team will investigate the impact and handle message retries accordingly.
Fix has landed and the team is monitoring progress. Messages sent to AWS SES are sending normally. The team will reprocess impacted messages.
The team is still working through the reprocessing for impacted messages. Confirmation of all impacted messages reprocessed will be updated here.
All impacted messages have been reprocessed.
The Courier team has identified an issue where messages fail due to an "Internal Courier Error" in the test environment. Our team found the root cause and are working to mitigate the issue.
We have tested the fix for "Internal Courier Error" and will be releasing it to production soon.
A fix has been merged and will fully deploy within the next hour. Updates to follow.
The fix has landed in production and we will be monitoring for any more "Internal Courier Errors". We will be working on reprocessing any dropped messages caused by this issue.
Messages with tags are no longer impacted, and error has been resolved. The team is working on reprocessing dropped messages.
The team is still working on reprocessing dropped messages caused by this issue. Send pipeline is operational.
All impacted messages have been reprocessed.
The Courier team has identified a slowdown in Segment event processing. There will be a slight delay for messages triggered by incoming Segment triggers. We estimate the queue to normalize soon and are monitoring.
Segment event processing stream has resolved.
The Courier team identified a delay in message processing and event updating. The pipeline queue for message delivery has normalized, however significant slowdowns in message events are still processing.
We're monitoring our processing pipeline as messages are flowing normally. There is a slight delay in event processing which will also impact outbound webhooks.
Message sends are flowing normally, and events are caught up without delays.
The Courier team has identified a slowdown in sending notifications. This issue has been identified by our team and is closely monitoring. We do not expect any messages to drop. We will continue to monitor and update accordingly.
Our team has identified a bottleneck in our system caused by timeouts on our sendworker. Our exponential backoff has caught up and messages are starting to go through.
The team has identified the issue related to network connections on AWS which resulted in a backup of messages on our pipeline. Messages are flowing through as expected, and the stream has caught up.
