56 Braze incidents · Kasım 2024 — official updates, affected components, duration and resolution details.
ABD07 Dashboard'ı etkileyen geçici hataları ve gecikmeleri araştırıyoruz.
Bu, Dashboard'ı etkileyen daha önceki bir konunun yeniden canlandırılmasıydı. Kalıcı bir düzeltme uygulandı ve Dashboard o zamandan beri stabil kaldı. Sorunun tamamen çözüleceğine inanıyoruz. Herhangi bir etki için özür dileriz.
Resmî olay güncellemesinden otomatik olarak çevrilmiştir.
Mühendislik ekibimiz ABD07 Dashboard'daki gecikme ve hataları araştırıyor.
Geçici pano hatalarına ve geç saatlere yol açan sorunu tespit ettik ve mühendisler sorunu çözmek için çalışıyorlar.
Mühendisler bir düzeltme uyguladılar ve Dashboard 30 dakikadan fazla stabildi. Tam çözümü sağlamak için konuyu izlemeye devam ediyoruz.
Dashboard stabil olmaya devam ediyor ve konu şimdi çözüldü.
Resmî olay güncellemesinden otomatik olarak çevrilmiştir.
We are currently investigating an issue with the US 01 and US 03 clusters.
Between 11:30 ET and 12:55 ET, some customers experienced elevated error rates. Braze engineers identified the issue and resolved it by rolling back a recent change and applying database optimizations. Error rates returned to normal shortly after each fix was applied, and they have remained stable for the past hour. This incident is now resolved.
Şu anda AB02 grubumuzda bozulan performansı araştırıyoruz, bu da küçük bir alt müşteri için Currents işlemedeki gecikmelere neden oluyor. Hiçbir veri kaybedilmedi ve mevcut verilerin çoğu normal olarak akışa devam ediyor. Ekibimiz aktif olarak bir karara doğru çalışıyor ve mevcut oldukları gibi güncelleştirmeler sağlayacaktır.
EU02'de bozulan performansı incelemeye devam ediyoruz. Altta yatan latency’nin kök nedenini tespit ettik, bu da birden fazla Currents konektörünün rutin bakımdan sonra tekrar tekrar başlamadığı, ancak izleme sistemimizde sağlıklı bir durumda ortaya çıkıyor. Bu özel sunucular bu eyaletteyken önemli geriloglar inşa ederler. Bir lütuf verici bir yeniden inşa ettik ve geri kalan eşyalarla daha hızlı bir şekilde işlemek için ek kaynaklara sahibiz. Bu noktada - verilerin% 95'i normalde akılırken, bağlantılarımızın% 5'ine kadar, bir alt müşteriyi etkilemeye devam ediyor, latency deneyimlemeye devam ediyor. Oluşturulan değişiklikler nedeniyle önemli ölçüde gelişmiş durumdayız ve 1 saat içinde bir güncelleme sağlayacaktır veya güncel bir zamanımız olduğunda.
Kalan backlog'u EU02'de netleştirmeye devam ediyoruz, performansımızı geliştiriyor ve genel gecikmeyi azaltıyoruz. Önümüzdeki saat içinde yakalanmayı bekliyoruz, ancak küçük bir dizi olay tamamen süreç için biraz daha uzun sürebilir. Kalan etki, genel trafiğimizin yüzde yarısı ile sınırlıdır. Tamamen yakaladığımızda başka bir güncelleme sağlayacağız.
Bu olay çözüldü. 15:18 ET / 19:18 UTC olarak, işlerin arkasını tamamen işledik ve AB02 Currents konektörlerimiz normal latency aralıkları içinde tekrar işlemektedir. Hiçbir veri kaybedilmedi ve tüm Currents olayları destinasyonlarına gönderildi.
Resmî olay güncellemesinden otomatik olarak çevrilmiştir.
Yaklaşık 7:30'da, 6:55 AM ET ve 8:00 AM ET arasındaki ağ bağlantılarını etkileyen ABD-WEST-2 bölgesinde bir AWS ağının farkında olduk. AWS, alt konuyu çözdükten bu yana. Etki penceresi sırasında, bazı müşteriler e-posta teslimatında geçici gecikmeler ve gecikmeler yaşayabilir. Mail artık normal olarak teslim edilmektedir. kuyruk derinliği arttıkça, bazı müşteriler mesaj tesliminde canlı gecikmeler yaşayabilir. Bu gecikmeler bugün 2:40 PM ET olarak çözüldü. Bu kesinti AWS altyapısında ortaya çıktı ve Braze platformunun herhangi bir bileşeni tarafından kaynaklanmadı. AWS olayında daha fazla ayrıntı için lütfen AWS durumu sayfasına bakınız: https://health.aws.amazon.com/health/status
Resmî olay güncellemesinden otomatik olarak çevrilmiştir.
Braze engineers have identified an issue affecting the Braze AI Operator, where users may see the following error message: "Connection unstable - Check your Wi-Fi or internet signal and try again". We are actively investigating the issue and will provide updates as more information becomes available.
A fix is being deployed for the issue affecting the Braze AI Operator, where users were seeing the error message: "Connection unstable - Check your Wi-Fi or internet signal and try again." We are seeing error rates improve and are continuing to monitor closely to confirm full resolution. We will provide a further update once we can confirm the issue is fully resolved.
The issue impacting the Braze AI Operator, where users were seeing the error message: "Connection unstable - Check your Wi-Fi or internet signal and try again," has been fully resolved. The fix has been deployed and error rates have returned to normal levels.
Meta, a third-party, is currently experiencing an issue impacting WhatsApp services. During this time, WhatsApp messages may experience latency. Campaign and Canvas creation or modification may experience errors and latency when utilizing WhatsApp templates. Audience sync to Facebook may also experience errors and latency during this period.
Meta's services appear to be stabilizing. We are continuing to monitor the situation and will provide a further update once full service has been restored.
Meta has recovered from the earlier issue impacting WhatsApp services, and full service has now been restored. WhatsApp message delivery, Campaign and Canvas creation or modification using WhatsApp templates, and Audience sync to Facebook are all operating normally.
Starting at 1:00 PM ET, customers on our US-01 and EU-01 clusters are experiencing elevated latency affecting campaign and Canvas processing, data ingestion, and message sending. Our engineering team is actively investigating, and we will provide updates as the situation develops.
Our engineering team continues to investigate the elevated latency. We will provide another update in 30 min or sooner if there are any changes.
Our engineering team has implemented a change, and we are beginning to see improvements in latency across campaign and Canvas processing, data ingestion, and message sending on US-01 and EU-01. We are monitoring the situation closely.
Latency and error rates have remained at normal levels across all services following the implementation of our fix. This incident is resolved.
Starting at 11:45 PM ET, Braze customers on our US05 have been experiencing latency across campaign/canvas processing, data processing, and message sending due to an ongoing issue with an upstream provider (AWS). We are working on a fix. Please see https://health.aws.amazon.com/health/status for more details.
We are continuing to experience delays in Campaign & Canvas Processing, Data Processing, and Message Sending. Additionally, Dashboard navigation is intermittently erroring. We are continuing to work on a fix while we await updates from AWS. Please see https://health.aws.amazon.com/health/status for more details.
We have implemented a fix to mitigate the impact of the ongoing AWS incident in Availability Zone (use1-az4) in the US-EAST-1 region. At this point, latency and error rates have returned to normal levels for all services. Additionally, AWS is reporting early signs of recovery. Please see https://health.aws.amazon.com/health/status for more details.
Latency and error rates have remained at normal levels for all services, post-implementation of our fix, and this incident is resolved. We will continue to monitor the overall system health until AWS reports full recovery. Please see https://health.aws.amazon.com/health/status for more details.
We have identified a third party issue causing Agent Console auto models to error. Our engineering team is actively working to remediate the issue.
Our engineering team have released a fix and Agent Console auto models are now recovering. We are continuing to monitor the situation.
Following continued monitoring by our engineers, Agent Console auto models have remained stable. This incident is now resolved.
Braze Engineers have identified an issue causing SMS delivery delays across networks in multiple European countries. Braze Engineers confirmed this is related to an ongoing Twilio incident: https://status.twilio.com/incidents/shhrhbqxflt1 We expect to provide another update as soon as more information becomes available.
Braze Engineers continue to monitor the ongoing SMS delivery delays affecting multiple networks across European countries. This remains related to an active Twilio incident, and our team is closely tracking their progress. Twilio has identified the cause and is actively working to resolve the issue. We will provide another update as soon as the situation changes. For real-time updates from Twilio, please refer to: https://status.twilio.com/incidents/shhrhbqxflt1
Braze Engineers continue to monitor the ongoing SMS delivery delays affecting multiple networks across European countries. This remains an active Twilio incident, and our team is closely tracking their progress. Twilio has identified the cause and is actively working to resolve the issue. They state that delivery delays to multiple networks continue in Romania, Serbia, Poland, Bosnia and Herzegovina, and Cabo Verde. The incident has been resolved, and latency has returned to normal in all other countries. Braze's next update on the issue will be 10 UTC on 9 May 2026. For real-time updates from Twilio, please refer to: https://status.twilio.com/incidents/shhrhbqxflt1
SMS sending delays have been resolved for the majority of countries and sending has returned to normal. SMS delivery delays continue to impact Romania, Serbia, Poland, Bosnia and Herzegovina, and Cabo Verde. For real-time updates from Twilio, please refer to: https://status.twilio.com/incidents/shhrhbqxflt1
Latency has returned to normal operational bounds and this incident is now resolved. Twilio's Statuspage will continue to provide updates on the remaining impact to Romania, Serbia, Poland, Bosnia and Herzegovina, and Cabo Verde: https://status.twilio.com/incidents/shhrhbqxflt1
We are currently investigating an issue with the EU01 cluster.
Braze engineers are actively investigating the issue and working to restore services.
Braze engineers have identified the issue. Services remain degraded but are improving.
Braze engineers are processing the remaining backlog of messages. Data processing remains in a degraded state due to backlog and latency issues.
Braze engineers are continuing to work through the remaining messages in the backlog. We are now seeing improvements in processing speeds, though data processing remains in a degraded performance state while the backlog is fully cleared.
The message backlog has been cleared, and data processing has returned to normal operational bounds. All services continue to remain stable, and this incident is considered resolved.
Between 16:00 and 18:14 ET, Braze Engineers identified and resolved an issue causing intermittent SDK errors on the US01 cluster. As of 18:19 ET, services remain stable, and any failed SDK requests during the impact window were automatically retried.
We are currently experiencing an issue where cases sent to [email protected] may not be created as expected. Our team is actively investigating the issue. In the meantime, if you require assistance, please submit your request via the Braze Support Portal or reach out to your Customer Success Manager. We will provide an update as soon as more information becomes available.
We have identified the root cause of the issue is Google Workspace incident (https://www.google.com/appsstatus/dashboard/incidents/224ozRqzW4sFBDK8hLnT). This is causing delays in emails being sent and received.
We are continuing to investigate this incident. Both Google Workspace and Salesforce's Email-to-Case functionality are reporting issues related to the processing of email. New cases created by emailing support@braze as well as responses on existing cases are not being processed. In the meantime please submit your request by clicking "Get Help" in the top-right corner of your Braze workspace. Alternatively, your CSM may also be able to support.
We have begun receiving responses in cases that were delayed by this incident. Case creation via email is now being processed. Support is actively working through the backlog of requests and follow-ups created during the disruption.
The issue affecting email processing for support cases has been resolved. New cases sent to [email protected] and replies to existing cases are now being processed as expected. Our Support team is actively working through the backlog of requests created during the disruption. Thank you for your patience while we worked through this.
Between 17:00 and 18:35 EST, Braze Engineers identified and resolved an issue causing intermittent SDK errors on the US01 cluster. As of 18:40 EST, services remain stable, and any failed SDK requests during the impact window were automatically retried.
Starting around 16:45 UTC, Braze Engineers began observing an issue where EU02 end users may experience errors when accessing or attempting to unsubscribe via the email preference center.
As of 17:47 UTC, Braze Engineers have implemented a fix and are monitoring service health for improvement.
No further issues have been experienced, and all services are operating normally. The incident is now resolved.
Braze Engineers are investigating an issue affecting users' ability to reach the Braze Dashboard for the US07 Cluster, starting around 13:41 EST. At this time, no other services are impacted, and messaging sending is not impacted. Engineers are actively working to restore service.
As of 15:12 ET, Braze engineers have implemented a fix and are continuing to monitor service health for improvement.
No further issues have been experienced, and all services are operating normally. The incident is now resolved.
Braze Engineers are investigating an issue affecting users' ability to reach the Braze Dashboard for the US05 Cluster, starting around 16:15 EST. At this time, no other services are impacted, and messaging sending is not impacted. Engineers are actively working to restore service.
As of 16:47 ET, Braze engineers have implemented a fix and are continuing to monitor service health for improvement.
No further issues have been experienced, and all services are operating normally. The incident is now resolved.
Braze Engineers detected an increase in latency on email outbound messaging for customers on Bird (Sparkpost) non-EU Braze clusters. This latency build up started at 13:53 ET. Braze has engaged Sparkpost, who have declared an incident and are actively investigating, you can follow their Statuspage for updates, and we'll update here in 30 minutes, or when we learn more. https://status.sparkpost.com/incidents/ndjn00g5vwm3
As of 14:34 ET, Sparkpost has identified the root cause to be a networking issue on their US cluster. We are monitoring performance for improvement.
As of 14:41 ET, Sparkpost has declared this incident resolved after implementing a fix. Braze Engineers have confirmed that real time outbound email latency has returned within normal operational bounds as of 14:34 ET. Any messages that failed during this window, were automatically retried. This incident is considered resolved.
We have received reports of errors when navigating the EU01 and EU02 Dashboards. Braze engineers are observing a small increase in errors and are actively investigating the issue.
We are continuing to investigate the issue.
We are continuing to investigate this issue with urgency and have escalated to our network service provider.
We've identified an issue isolated impacting our edge network impacting customers accessing the dashboard from Europe. We have escalated the issue to our network provider and are investigating with urgency.
The errors have stabilised to normal operating levels. We are continuing to monitor the situation closely and are actively investigating with our networking provider.
We can confirm services have continued to remain stable and the issue is now considered resolved.