56 Braze incidents · november 2024 — official updates, affected components, duration and resolution details.
Vi undersøger intermitterende fejl og latency påvirker US07 Dashboard.
Dette var en gentagelse af et tidligere problem påvirker Dashboard. Et permanent fix er blevet implementeret og Dashboard har været stabil siden. Vi er sikre på, at spørgsmålet nu er helt løst. Vi undskylder for ethvert sammenstød.
Automatisk oversat fra den officielle hændelsesopdatering.
Vores Engineering team undersøger intermitterende latency og fejl på US07 Dashboard.
Vi har identificeret problemet, der fører til intermitterende dashboard fejl og latency og ingeniører arbejder på at løse problemet.
Ingeniører har implementeret et fix og Dashboard har været stabil i over 30 minutter. Vi fortsætter med at overvåge spørgsmålet for at sikre en fuldstændig løsning.
Dashboardet er fortsat stabilt, og spørgsmålet er nu løst.
Automatisk oversat fra den officielle hændelsesopdatering.
We are currently investigating an issue with the US 01 and US 03 clusters.
Between 11:30 ET and 12:55 ET, some customers experienced elevated error rates. Braze engineers identified the issue and resolved it by rolling back a recent change and applying database optimizations. Error rates returned to normal shortly after each fix was applied, and they have remained stable for the past hour. This incident is now resolved.
Vi er i øjeblikket ved at undersøge forringet ydeevne i vores EU02 klynge, hvilket forårsager forsinkelser i Currents behandling for en lille undergruppe af kunder. Ingen data er gået tabt, og størstedelen af Currents data fortsætter med at flyde normalt. Vores team arbejder aktivt mod en løsning og vil levere opdateringer, efterhånden som de bliver tilgængelige.
Vi fortsætter med at undersøge forringet ydeevne i EU02. Vi har identificeret den grundlæggende årsag til den underliggende latency, som stammede fra flere Currents-konnektorer ikke elegant genstarte efter rutinemæssig vedligeholdelse, men optræder i en sund tilstand i vores overvågningssystem. Disse specifikke servere bygge betydelige backlogs mens i denne tilstand. Vi har afsluttet en yndefuld genstart, og har spundet op yderligere ressourcer til at behandle gennem de backgated poster hurtigere. På dette punkt ~ 95% af data flyder normalt, mens op til ~ 5% af vores stik, påvirker en delmængde af kunder, fortsætte med at opleve latency. Vi ser gennemløb betydeligt forbedret på grund af config ændringer, og vil give en opdatering om 1 time, eller når vi har en opdateret tidslinje.
Vi fortsætter med at gøre fremskridt med at rydde den resterende mangel på EU02, forbedre vores resultater og mindske den samlede latency. Vi forventer at blive indhentet inden for den næste time, selv om et lille antal begivenheder kan tage lidt længere tid at behandle fuldt ud. Den resterende effekt er begrænset til mindre end en halv procent af vores samlede trafik. Vi giver en ny opdatering, når vi er fuldt opsnappet.
Denne hændelse er løst. Fra 15: 18 ET / 19: 18 UTC, vi har fuldt ud behandlet tilbage af job, og vores EU02 Currents stik er alle igen behandling inden for normale latency intervaller. Ingen data blev tabt, og alle Currents begivenheder er blevet sendt til deres destinationer.
Automatisk oversat fra den officielle hændelsesopdatering.
Ved ca. 7: 30 ET, blev vi klar over en AWS netværk udfald i US- WEST-2 regionen påvirker netværksforbindelse mellem 6: 55 AM ET og 8: 00 AM ET. AWS har siden løst det underliggende problem. Under slagvinduet, kan nogle kunder have oplevet intermitterende forsinkelser og latency i e-mail levering. Mail leveres nu normalt. Efterhånden som kødybden steg, kan nogle kunder have oplevet forsinkelser i leveringen af beskeder. Disse forsinkelser er blevet løst fra kl. 14.40 ET i dag. Denne afbrydelse opstod inden for AWS infrastruktur og var ikke forårsaget af nogen komponent i Braze platform. For yderligere oplysninger om hændelsen AWS, se venligst på statussiden for AWS: https: / / healt.aws.amazon.com / health / status
Automatisk oversat fra den officielle hændelsesopdatering.
Braze engineers have identified an issue affecting the Braze AI Operator, where users may see the following error message: "Connection unstable - Check your Wi-Fi or internet signal and try again". We are actively investigating the issue and will provide updates as more information becomes available.
A fix is being deployed for the issue affecting the Braze AI Operator, where users were seeing the error message: "Connection unstable - Check your Wi-Fi or internet signal and try again." We are seeing error rates improve and are continuing to monitor closely to confirm full resolution. We will provide a further update once we can confirm the issue is fully resolved.
The issue impacting the Braze AI Operator, where users were seeing the error message: "Connection unstable - Check your Wi-Fi or internet signal and try again," has been fully resolved. The fix has been deployed and error rates have returned to normal levels.
Meta, a third-party, is currently experiencing an issue impacting WhatsApp services. During this time, WhatsApp messages may experience latency. Campaign and Canvas creation or modification may experience errors and latency when utilizing WhatsApp templates. Audience sync to Facebook may also experience errors and latency during this period.
Meta's services appear to be stabilizing. We are continuing to monitor the situation and will provide a further update once full service has been restored.
Meta has recovered from the earlier issue impacting WhatsApp services, and full service has now been restored. WhatsApp message delivery, Campaign and Canvas creation or modification using WhatsApp templates, and Audience sync to Facebook are all operating normally.
Starting at 1:00 PM ET, customers on our US-01 and EU-01 clusters are experiencing elevated latency affecting campaign and Canvas processing, data ingestion, and message sending. Our engineering team is actively investigating, and we will provide updates as the situation develops.
Our engineering team continues to investigate the elevated latency. We will provide another update in 30 min or sooner if there are any changes.
Our engineering team has implemented a change, and we are beginning to see improvements in latency across campaign and Canvas processing, data ingestion, and message sending on US-01 and EU-01. We are monitoring the situation closely.
Latency and error rates have remained at normal levels across all services following the implementation of our fix. This incident is resolved.
Starting at 11:45 PM ET, Braze customers on our US05 have been experiencing latency across campaign/canvas processing, data processing, and message sending due to an ongoing issue with an upstream provider (AWS). We are working on a fix. Please see https://health.aws.amazon.com/health/status for more details.
We are continuing to experience delays in Campaign & Canvas Processing, Data Processing, and Message Sending. Additionally, Dashboard navigation is intermittently erroring. We are continuing to work on a fix while we await updates from AWS. Please see https://health.aws.amazon.com/health/status for more details.
We have implemented a fix to mitigate the impact of the ongoing AWS incident in Availability Zone (use1-az4) in the US-EAST-1 region. At this point, latency and error rates have returned to normal levels for all services. Additionally, AWS is reporting early signs of recovery. Please see https://health.aws.amazon.com/health/status for more details.
Latency and error rates have remained at normal levels for all services, post-implementation of our fix, and this incident is resolved. We will continue to monitor the overall system health until AWS reports full recovery. Please see https://health.aws.amazon.com/health/status for more details.
We have identified a third party issue causing Agent Console auto models to error. Our engineering team is actively working to remediate the issue.
Our engineering team have released a fix and Agent Console auto models are now recovering. We are continuing to monitor the situation.
Following continued monitoring by our engineers, Agent Console auto models have remained stable. This incident is now resolved.
Braze Engineers have identified an issue causing SMS delivery delays across networks in multiple European countries. Braze Engineers confirmed this is related to an ongoing Twilio incident: https://status.twilio.com/incidents/shhrhbqxflt1 We expect to provide another update as soon as more information becomes available.
Braze Engineers continue to monitor the ongoing SMS delivery delays affecting multiple networks across European countries. This remains related to an active Twilio incident, and our team is closely tracking their progress. Twilio has identified the cause and is actively working to resolve the issue. We will provide another update as soon as the situation changes. For real-time updates from Twilio, please refer to: https://status.twilio.com/incidents/shhrhbqxflt1
Braze Engineers continue to monitor the ongoing SMS delivery delays affecting multiple networks across European countries. This remains an active Twilio incident, and our team is closely tracking their progress. Twilio has identified the cause and is actively working to resolve the issue. They state that delivery delays to multiple networks continue in Romania, Serbia, Poland, Bosnia and Herzegovina, and Cabo Verde. The incident has been resolved, and latency has returned to normal in all other countries. Braze's next update on the issue will be 10 UTC on 9 May 2026. For real-time updates from Twilio, please refer to: https://status.twilio.com/incidents/shhrhbqxflt1
SMS sending delays have been resolved for the majority of countries and sending has returned to normal. SMS delivery delays continue to impact Romania, Serbia, Poland, Bosnia and Herzegovina, and Cabo Verde. For real-time updates from Twilio, please refer to: https://status.twilio.com/incidents/shhrhbqxflt1
Latency has returned to normal operational bounds and this incident is now resolved. Twilio's Statuspage will continue to provide updates on the remaining impact to Romania, Serbia, Poland, Bosnia and Herzegovina, and Cabo Verde: https://status.twilio.com/incidents/shhrhbqxflt1
We are currently investigating an issue with the EU01 cluster.
Braze engineers are actively investigating the issue and working to restore services.
Braze engineers have identified the issue. Services remain degraded but are improving.
Braze engineers are processing the remaining backlog of messages. Data processing remains in a degraded state due to backlog and latency issues.
Braze engineers are continuing to work through the remaining messages in the backlog. We are now seeing improvements in processing speeds, though data processing remains in a degraded performance state while the backlog is fully cleared.
The message backlog has been cleared, and data processing has returned to normal operational bounds. All services continue to remain stable, and this incident is considered resolved.
Between 16:00 and 18:14 ET, Braze Engineers identified and resolved an issue causing intermittent SDK errors on the US01 cluster. As of 18:19 ET, services remain stable, and any failed SDK requests during the impact window were automatically retried.
We are currently experiencing an issue where cases sent to [email protected] may not be created as expected. Our team is actively investigating the issue. In the meantime, if you require assistance, please submit your request via the Braze Support Portal or reach out to your Customer Success Manager. We will provide an update as soon as more information becomes available.
We have identified the root cause of the issue is Google Workspace incident (https://www.google.com/appsstatus/dashboard/incidents/224ozRqzW4sFBDK8hLnT). This is causing delays in emails being sent and received.
We are continuing to investigate this incident. Both Google Workspace and Salesforce's Email-to-Case functionality are reporting issues related to the processing of email. New cases created by emailing support@braze as well as responses on existing cases are not being processed. In the meantime please submit your request by clicking "Get Help" in the top-right corner of your Braze workspace. Alternatively, your CSM may also be able to support.
We have begun receiving responses in cases that were delayed by this incident. Case creation via email is now being processed. Support is actively working through the backlog of requests and follow-ups created during the disruption.
The issue affecting email processing for support cases has been resolved. New cases sent to [email protected] and replies to existing cases are now being processed as expected. Our Support team is actively working through the backlog of requests created during the disruption. Thank you for your patience while we worked through this.
Between 17:00 and 18:35 EST, Braze Engineers identified and resolved an issue causing intermittent SDK errors on the US01 cluster. As of 18:40 EST, services remain stable, and any failed SDK requests during the impact window were automatically retried.
Starting around 16:45 UTC, Braze Engineers began observing an issue where EU02 end users may experience errors when accessing or attempting to unsubscribe via the email preference center.
As of 17:47 UTC, Braze Engineers have implemented a fix and are monitoring service health for improvement.
No further issues have been experienced, and all services are operating normally. The incident is now resolved.
Braze Engineers are investigating an issue affecting users' ability to reach the Braze Dashboard for the US07 Cluster, starting around 13:41 EST. At this time, no other services are impacted, and messaging sending is not impacted. Engineers are actively working to restore service.
As of 15:12 ET, Braze engineers have implemented a fix and are continuing to monitor service health for improvement.
No further issues have been experienced, and all services are operating normally. The incident is now resolved.
Braze Engineers are investigating an issue affecting users' ability to reach the Braze Dashboard for the US05 Cluster, starting around 16:15 EST. At this time, no other services are impacted, and messaging sending is not impacted. Engineers are actively working to restore service.
As of 16:47 ET, Braze engineers have implemented a fix and are continuing to monitor service health for improvement.
No further issues have been experienced, and all services are operating normally. The incident is now resolved.
Braze Engineers detected an increase in latency on email outbound messaging for customers on Bird (Sparkpost) non-EU Braze clusters. This latency build up started at 13:53 ET. Braze has engaged Sparkpost, who have declared an incident and are actively investigating, you can follow their Statuspage for updates, and we'll update here in 30 minutes, or when we learn more. https://status.sparkpost.com/incidents/ndjn00g5vwm3
As of 14:34 ET, Sparkpost has identified the root cause to be a networking issue on their US cluster. We are monitoring performance for improvement.
As of 14:41 ET, Sparkpost has declared this incident resolved after implementing a fix. Braze Engineers have confirmed that real time outbound email latency has returned within normal operational bounds as of 14:34 ET. Any messages that failed during this window, were automatically retried. This incident is considered resolved.
We have received reports of errors when navigating the EU01 and EU02 Dashboards. Braze engineers are observing a small increase in errors and are actively investigating the issue.
We are continuing to investigate the issue.
We are continuing to investigate this issue with urgency and have escalated to our network service provider.
We've identified an issue isolated impacting our edge network impacting customers accessing the dashboard from Europe. We have escalated the issue to our network provider and are investigating with urgency.
The errors have stabilised to normal operating levels. We are continuing to monitor the situation closely and are actively investigating with our networking provider.
We can confirm services have continued to remain stable and the issue is now considered resolved.