Vero 장애 기록

1. 2026년 7월 21일 AM 8:18 UTCresolved

   사이 09:30-10:30 UTC 20 7월 2026, Vero는 우리의 추적 API에 영향을 미치는 가용성 사건을 경험했습니다. 일부 사용자 속성 및 이벤트 추적 API 요청이 성공적으로 처리되지 않았습니다. 결과적으로, 사건 창 도중 Vero에 보내진 몇몇 자료는 성공적으로 저장되고 가공되지 않을지도 모릅니다. ## 뿌리 원인 사고는 정상적인 서비스에 대한 중단으로 인한 인프라 문제로 발생했습니다. 일단 확인되면, 문제는 구제되고 서비스는 신속하게 복원되었습니다. 불행히도, 우리의 상태 페이지는 우리의 체계의 이 성분을 위한 상태 페이지를 새롭게 하는 과정에 있는 고장 때문에 이 시간에 이었습니다. ## 팔로우 작업 우리는 우리의 모니터링 및 운영 절차와 함께 중단에 주도 된 상황에 대한 검토, 반복 사고의 likelihood를 줄이기 위해. 이 웹 사이트는 귀하가 웹 사이트를 탐색하는 동안 귀하의 경험을 향상시키기 위해 쿠키를 사용합니다. 이 쿠키들 중에서 필요에 따라 분류 된 쿠키는 웹 사이트의 기본적인 기능을 수행하는 데 필수적이므로 브라우저에 저장됩니다. 이러한 쿠키는 귀하의 동의하에 만 브라우저에 저장됩니다. 이러한 쿠키 중 일부를 선택 해제하면 검색 환경에 영향을 미칠 수 있습니다. 우리는이 사건이 발생하는 붕괴에 대한 사과.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2025년 11월 26일 AM 12:53 UTCinvestigating

   We're investigating minor delays in message sending and reports that certain elements of the UI (Logs, Profiles) are not loading correctly.

2. 2025년 11월 26일 AM 1:07 UTCresolved

   We have confirmed the source of the issue and deployed a fix. This issue is now fully resolved and all message are real time.

1. 2025년 10월 20일 PM 3:56 UTCinvestigating

   We are seeing multiple delays across the application. This is due to a widespread AWS outage impacting both our services, as well as tools and services we rely on. See https://health.aws.amazon.com/health/status . Segment refreshes, workflow processing and other key Vero components are all impacted. We apologize for any inconvenience and will be continuing to monitor.

2. 2025년 10월 20일 PM 7:35 UTCmonitoring

   We are seeing infrastructure come back online and many of our backed up queues are slowly coming down. Workflows are still experiencing significant delays, as are some newsletters. We will continue to monitor the situation and update as conditions improve.

3. 2025년 10월 20일 PM 10:54 UTCmonitoring

   We are currently processing all remaining backlogs, and performance should be back to normal for most aspects of Vero. Some workflows and newsletters may still continue to experience delays.

4. 2025년 10월 21일 AM 12:01 UTCresolved

   RESOLVED: Our upstream provider has marked the issue resolved over an hour ago, and we have seen a resumption of normal processing and all services should now be operational. Thank you for your patience.

1. 2025년 8월 5일 PM 12:34 UTCidentified

   We are experiencing delays processing API calls and downstream workflows. We know this is frustrating. We have identified the issue and are working to restore processing as soon as possible.

2. 2025년 8월 5일 PM 1:43 UTCmonitoring

   We've made a fix and things are trending back to normal. We're keeping the status as "Degraded performance" for the time being but are monitoring and will mark as resolved as soon as we can. Thanks for your patience.

3. 2025년 8월 5일 PM 11:05 UTCresolved

   Everything is operating as normal at this time and has been for the last couple of hours.

1. 2025년 7월 30일 PM 2:57 UTCidentified

   We are currently experiencing delays with our API and message processing systems. Our team has implemented a temporary fix and performance is improving, but you may still encounter slower response times or failed requests to Vero's Track API. Message delivery across all channels—including workflows and newsletters—is currently delayed. We're working to fully resolve these issues and will update you as soon as normal service is restored.

2. 2025년 7월 30일 PM 6:25 UTCmonitoring

   The team is continuing to monitor system stability and performance.

3. 2025년 7월 31일 AM 2:43 UTCresolved

   Note: we are trying a new format for our post-mortems. *What was the impact* API processing, workflows (including transactional messages and workflows) and newsletters were intermittently offline for a brief period (15:00-15:30) and then delayed for several hours between 15:00 and 23:30 UTC as we caught up processing. *What caused the impact* A datastore used by our queueing system ran out of memory. *Why'd it happen* After QA testing, we deployed a configuration change to our queueing system yesterday to improve the way processing API and workflow jobs are balanced between customers. This change led to unexpected growth in memory in our queueing system, causing it to fail. Whilst we were alerted the memory growth rate was unprecedently large. *What changes have we made* - Investigated the cause of the memory growth and patched it. - Adjusted our alerting on memory growth so we are alerted earlier to give us more time to fix this issue, in the unlikely event it occurs again. *Any other information* This particular part of our infrastructure has been frustratingly brittle over the last year or so, due to inefficient data storage. As a result we elected to migrate to a data store better-suited for these workloads (DynamoDB) at the start of 2025. That work is nearing completion. It will improve queue processing by 10x at least, giving us much-needed headroom. Questions: please email us at [email protected].

1. 2025년 7월 28일 PM 11:40 UTCidentified

   At approximately 20:45 UTC, we began noticing errors in our mailer service indicating emails were not going out. Shortly afterwards, we were contacted by our upstream email delivery partner indicating they were experiencing infrastructure issues. This is now impacting **all emails** sent via Vero-managed sending domains and is a major outage. If you send via your own account with an ESP (Sendgrid, Mailjet, etc), you are **not impacted**. SMS and push messages are **not impacted**. We are in communication with our upstream partner, and they are working to restore service. While we do not yet have a definite timeline, we will update as soon we do. We are working to get this back online as quickly as possible. We apologise for the inconvenience. We understand how critical email delivery is to your business.

2. 2025년 7월 29일 AM 1:43 UTCmonitoring

   Our downstream provider has resolved the issue and the system is operating as normal. We are now processing the backlog of delayed messages and will mark this issue as resolved once that is complete. Please email us with any questions via [email protected].

3. 2025년 7월 29일 AM 7:35 UTCresolved

   We are confident this issue has been completely resolved. All emails have been processing as normal since approximately 04:00 UTC 29 July. We are working with our downstream provider to review what has occurred and improve our resiliency in response to this incident. Please email us with any questions at [email protected]. Thanks for your patience.

1. 2025년 5월 30일 PM 8:06 UTCinvestigating

   We are experiencing significant delays with Newsletter sending. The issue began overnight, and we are taking steps to mitigate but service will take some time to return to normal. Once we are positive we have addressed the root cause of the issue, we will provide an update and monitor the issue.

2. 2025년 5월 30일 PM 9:21 UTCmonitoring

   We have identified the root cause of the issue and have applied a fix. We will continue monitoring throughout the day until sending returns to normal operations.

3. 2025년 5월 31일 AM 12:39 UTCresolved

   Newsletter sending has resumed normal operations. However, we will continue to monitor through the weekend. Please reach out to [email protected] if you experience additional delays or have any questions.

4. 2025년 6월 2일 AM 5:58 UTCpostmortem

   On Friday, May 30, customers experienced significant delays in the delivery of one-to-many newsletter campaigns. We know how critical timely delivery is and are providing a post mortem. **What happened** Between 0700 30 May - 0130 31 May UTC, scheduled newsletter campaigns were delayed due to a combination of factors: 1. **Downstream service disruption**. A configuration issue at one of our external partners impacted message throughput. 2. **Authentication issues.** A component of our infrastructure experienced a disruption related to service authentication with another internal service, which contributed to the backlog. These issues together resulted in prolonged delivery times for a subset of campaign traffic. **Resolution** We worked with our external partner to address the configuration issue and restored full functionality to the affected internal service. Message queues were cleared and delivery returned to normal by 0130 31 May. **Preventative and detective measures** To improve reliability and reduce risk going forward, we are: * Enhancing proactive monitoring and alerting for system-level authentication and connectivity health. * with our third-party partner to strengthen deployment safeguards. -- We’re committed to ensuring the dependability of your campaigns. While this incident was complex and multi-faceted, we take full responsibility and apologise for the inconvenience. Thank you for your trust and patience. Please email us at [[email protected]](mailto:[email protected]) if you have any questions.

1. 2025년 5월 28일 AM 9:40 UTCidentified

   We are receiving multiple reports that the "Delivery" metric in the Vero UI and integrations via webhooks are currently under-reporting. We have identified the cause of this issue. At this time it appears it relates to reporting only: messages are still being successfully delivered. We will provide an update as soon as we have deployed a fix and are monitoring the solution. Please email [email protected] with any questions.

2. 2025년 5월 28일 PM 10:29 UTCmonitoring

   This issue has been identified and patched. We are currently monitoring to ensure the issue has been fully addressed. In the meanwhile, delivery stats are being backfilled but will take some hours before they are fully accurate.

3. 2025년 5월 29일 PM 7:00 UTCresolved

   All stats should now be caught up. We have monitored overnight and observed no additional issues.

1. 2025년 3월 19일 PM 2:50 UTCmonitoring

   We have identified an issue introduced during a recent upgrade that affected data processing for newsletters sent through Vero 2.0. On 16 March 06:26 UTC we made a change to increase the send rate of newsletters sent through Vero 2.0. Due to an issue in this deploy, some email profiles have since been incorrectly identified, leading to: 1. Missing report data for some campaigns. 2. Emails sent to previously-unsubscribed users. We’ve deployed a fix and are currently back-filling the data. From this time, unsubscribed users *are* being correctly respected. Additionally, we’re working to recover the missing reporting stats and correct any previously-impacted profiles. This process is ongoing, with full restoration expected within the next 24 hours. We’ll provide further updates as the process progresses. We understand the impact of this bug and apologise for the inconvenience caused. Please email via [email protected] with any questions.

2. 2025년 4월 2일 PM 9:39 UTCresolved

   This incident has been resolved.

1. 2024년 8월 6일 AM 9:38 UTCinvestigating

   We are currently investigating an issue in which the main Customers page is not displaying all customer data.

2. 2024년 8월 6일 AM 10:26 UTCresolved

   This incident has been resolved.

1. 2024년 7월 29일 PM 7:14 UTCinvestigating

   We are currently experiencing an API outage. Our platform team is investigating but has yet to identify a root cause.

2. 2024년 7월 29일 PM 9:02 UTCresolved

   The root cause of the issue has been identified, and an appropriate configuration change has been made. Service appears to be restored, and we are marking this issue as resolved. We will continue to monitor but do not anticipate additional issues. Please reach out to [email protected] with any questions or concerns.

1. 2024년 7월 10일 AM 7:51 UTCinvestigating

   We are currently investigating this issue.

2. 2024년 7월 10일 PM 12:24 UTCidentified

   We have identified the cause of the issue.

3. 2024년 7월 10일 PM 10:14 UTCmonitoring

   We have been monitoring the situation since identifying and addressing the issue earlier today. We are still working through the backlog of webhooks and will continue to monitor until we are back in real time.

4. 2024년 7월 11일 AM 1:23 UTCmonitoring

   We have now processed 98% of the backlog and are working through the final 2%. We will leave this issue as "Monitoring" until we've fully cleared the backlog.

5. 2024년 7월 11일 PM 11:33 UTCresolved

   This incident has been resolved.

1. 2024년 7월 11일 AM 7:34 UTCresolved

   The incident related to delays in webhook processing has resulted in delays to campaign reporting in Vero’s UI. No data has been lost and all reports will be backfilled.

1. 2024년 6월 20일 AM 9:08 UTCinvestigating

   We're investigating reports of delays processing API events and triggering automated campaigns.

2. 2024년 6월 20일 AM 9:49 UTCinvestigating

   We are still investigating. We apologise for the delay. We will provide further updates as soon as we can.

3. 2024년 6월 20일 AM 11:43 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2024년 6월 20일 PM 1:05 UTCresolved

   This incident has been resolved.

1. 2024년 6월 19일 AM 6:28 UTCinvestigating

   We are currently investigating this issue.

2. 2024년 6월 19일 AM 6:56 UTCresolved

   This incident has been resolved.

1. 2024년 6월 10일 PM 11:30 UTCidentified

   One of our email delivery processing clusters is experiencing delays, affecting a portion of our customers. This is impacting automated (transactional, behavioural, workflow) campaigns, resulting in significant delays. We have found the root cause and are working on a fix now.

2. 2024년 6월 11일 AM 1:41 UTCresolved

   We've now resolved this issue. We will reach out to the affected customers to discuss any follow-up questions. If you have any questions, please email us at [email protected].

1. 2024년 5월 2일 AM 9:10 UTCmonitoring

   We are currently monitoring delays in newsletter sending.

2. 2024년 5월 2일 AM 11:20 UTCresolved

   This incident has been resolved.

1. 2024년 4월 2일 AM 12:21 UTCresolved

   At 11:17 UTC (1 April), our team was alerted to delays in automated campaign processing. At this time we marked behavioural, transactional and workflow processing as degraded. This affected all customers. By 11:54 UTC, our team had isolated the root cause: delayed filter evaluations on one of our data sources. At 12:30 UTC, processing performance had returned to normal but there was a large backlog of automated campaign jobs to work through. We continued to monitor as the backlog reduced and began work on a fix to the root cause. At 18:49 UTC, we noticed a spike in processing again. This was quickly resolved (as of 19:30). We continued to monitor the backlog. As of 21:30 UTC, the backlog has cleared and workflow processing is 100% realtime for all but handful of customers (with larger backlogs). These remaining backlogs should be fully cleared by 00:00 UTC. At 21:56 we deployed a patch for the root cause. We believe we have fixed the underlying cause and this will not occur again. We will monitor extra closely over the coming 48 hours to ensure this is the case. If you have any questions, please email [email protected]. Thank you. We also want to note that this incident was unrelated to the database upgrades/maintenance conducted over the weekend. These changes have gone smoothly. -- Processing has returned to normal and we can report the following metrics for the key time windows. These are the P50 to P90 reports for workflow node processing. This represents how long it takes to process a node vs. when the node was queued for evaluation. We expect a p90 of 5 minutes across all nodes. Note that this include **all node types** across all workflow types (transactional, non-transactional, etc.). For example, A node with 50 conditions looking back across a large date range will take longer to evaluate than a simple email node. 10:00 - 21:30 UTC 1 April (catching up on backlog) • p50 = 1 hour • p75 = 2.45 hours • p90 = 4.6 hours 21:30 - 00:00 UTC 1 April (tail of backlog, with majority of customers in realtime) • p50 = 48 minutes • p75 = 1.8 hours • p90 = 3.1 hours 00:00 - 01:00 UTC 2 April (backlog fully processed and processing in realtime) • p50 = 31 seconds • p75 = 3 minutes • p90 = 5.4 minutes

1. 2023년 10월 20일 PM 9:24 UTCmonitoring

   We apologize for the service degradation experienced intermittently throughout this last week. These have been caused by some large, necessary infrastructure improvements we are making to our system. The primary changes have now been made and we expect to see improvements to any remaining performance issues over the next several hours. We will continue to monitor the situation over the weekend, and provide a more detailed post-mortem afterwards. Thanks for your patience. If you have any questions, please emails us via [email protected].

2. 2023년 10월 24일 AM 12:58 UTCmonitoring

   We are continuing to monitor the situation and make adjustments to improve performance. We will share a post mortem when we are satisfied that the situation is resolved

3. 2023년 10월 28일 AM 12:19 UTCresolved

   We believe we have addressed the underlying configuration issues when led to last week's major service degradation. Over the past few months we've been working on some important optimizations and security patches to several of our key data stores and infrastructure. These changes were meant to be rolling changes with no production impact and many of the changes have been. Unfortunately, some of the changes we made led to unanticipated issues, causing degraded performance across the majority of Vero's services. One of the database shards on which we store and read user properties has been falling out of sync with the primary data store. Unfortunately, the services that rely on this particular data source include those that personalize email content and evaluate automated campaigns and workflows. These services will wait for eventual consistency but have a low tolerance for delays in order to deliver the speed our customers expect. This has caused periodic, significant delays. We've been able to uncover the source of these issues and have made configuration changes that have resolved the synchronization lag. We are monitoring to ensure there are no further issues. Separately, we recently made some important optimizations to our event storage to improve performance. Despite significant planning, these changes led to unexpected degredation in some of the queries related to automated email delays. Our platform team has been hard at work optimizing these queries and we've seen major improvements. Processing is back to BAU at this time. We are monitoring and are continuing to make further optimizations until we've seen the net performance improvements we're targeting. It has taken us longer than we would like to roll forward and adjust the configuration to return performance to the needed levels. We have been monitoring and have seen all services operating as expected. We will continue monitoring until we are certain we have restored the systems to their full capacity.

1. 2023년 8월 21일 PM 8:26 UTCinvestigating

   We've encountered performance issues resulting in degraded performance across the product. API processing, segment calculation, email sending, and the UI have all been impacted. No data has been lost. At 7:15AM Monday UTC, we were alerted to an unusually high number of unprocessed API jobs. These jobs eventually came back down after intervention by our platform team. Throughout the morning, the API queue has seen multiple additional spikes, and we've seen degraded performance across the entirety of the application. We are continuing to investigate the cause and will update when we have determined a course of action.

2. 2023년 8월 22일 AM 7:30 UTCmonitoring

   Between ~7am UTC Monday and ~10pm UTC Monday the Vero system experienced periodic outages/delays affecting: • Reports in the UI • Automated/workflow campaign evaluation speed. • API ingestion. • The UI itself. We know these outages have a major impact on your end-user experience and we apologise for the inconvenience caused. -- For those interested, at ~10pm UTC Sunday, we conducted some unplanned maintenance to one of our core data indexes. Whilst unplanned, we made the required changes after planning the alternatives and thinking through the risks. The new configuration is designed to be more performant, running on more modern architecture. This should have been a very run-of-the-mill, invisible upgrade. It took some time to become apparent but the new configuration was not performing to specification. By 7am UTC Monday this was leading to delays in several services. Throughout Monday (UTC) our team actively managed the situation to prioritise API and automated email processing was prioritised (as these are the most critical systems). As of ~10pm UTC Monday, all services expect reports were returned to normal processing speeds. As of ~1am UTC Tuesday, reports have also returned to normal processing. We will be conducting a post-mortem internally to learn and better plan for future changes to this specific service. -- At this time we are continuing to monitor the situation. We will mark this issue as resolved once we are comfortable things have been operating as normal for ~24 hours. -- We work hard to ensure 99.99%+ uptime on all core campaign processing. All changes we have been making to our infrastructure recently are in the service of: • Faster automated/workflow campaign processing. • Delivery of new channels such as SMS (and beyond). Thank you for your support. If you have any questions please email us at [email protected].

3. 2023년 8월 23일 PM 1:54 UTCresolved

   This incident has been resolved.