Omnivore outage history

Today around 14:52 UTC, we observed a spike in the number of locations that are flapping between 'online' and 'degraded'. We are currently working to diagnose the issue and restore agent connectivity back to normal levels. We will provide further updates as soon as we have more information to share.

Around 16:30 UTC, we applied a change and observed that the number of locations flapping between 'online' and 'degraded' dropped back to baseline levels. We will continue monitoring this issue to make sure our systems remain stable.

We continue to observe location status stability.

Around 9:02 UTC, we noticed a steady increase in the number of Locations entering a "degraded" state. This seems related to a small increase in average Agent Heartbeat response times to slightly above 2 seconds. This is causing some Locations to flap between "online" and "degraded". Customers may see a minor increase in API response times as a result. We are actively investigating the cause at this time.

We have identified a possible cause of the increased latency and are working on a fix.

At this time, we believe the impact for the duration of this incident was limited to incorrectly reporting the Location Status. We no longer believe that API calls experienced increased latency or error rates.

We released a fix around 17:08 UTC and verified that Location Status reporting has returned to normal behavior.

We are experiencing issues polling tickets for all Brink locations. This started at approximately 17:15 UTC and is ongoing. During this time, we are seeing a lower than expected volume of tickets for all of our Brink locations. Brink is aware of the issue and is working to resolve. We will post updates as they become available.

Brink has addressed the issue and we are seeing an influx of new tickets as we move to recover tickets missed during the window. We will continue to monitor and post another update once levels get back to normal.

We have observed ticket volume return to baseline levels.

We are seeing issues occurring with our 3rd-party cloud connectivity provider. This affects most Omnivore systems to varying degrees. We will update further as we have additional details.

Connectivity to all services have recovered to normal levels. We are continuing to monitor.

This incident has been resolved

Today at 20:52 UTC, we identified an issue with a change to our Request Indexer service that was released yesterday at 20:04 UTC. This change was rolled back at 21:36 UTC. The issue caused all API Requests made between the initial release and the rollback to not display in API Activity. The issue is currently remediated and we are evaluating options to repopulate the missing data.

Around 07:11 UTC this morning, Amazon Web Services began experiencing an outage (https://health.aws.amazon.com/health/status). The Omnivore team was alerted at 07:24 UTC and has been monitoring ever since. So far, we haven't seen any clear signal of disruption in Omnivore services other than the following: * ~2.4k agents have entered a degraded state. They are not going offline, so API calls to these locations are succeeding, however customers may be experiencing slower than normal response times. * The Brink API experienced elevated error rates from 15:21-15:41 UTC. Omnivore customers would have also seen an elevated error rate for Brink locations and possibly delayed webhooks. We will continue to monitor for new impacts and will send out updates as needed.

Around 17:07 UTC the number of degraded agents returned to baseline levels. We have not observed any other spikes in the error level for Brink API. We are currently not seeing any other areas of concern, but since AWS's incident is still open, we will continue monitoring all of our services for new issues.

Around 19:27 UTC, we observed a spike in the error rate for our calls to the Toast API, approaching a 100% failure rate. At this time there are no actions we can take and we will continue to monitor the situation.

We are still seeing a near 100% error rate for the Toast integration. We will continue to monitor for improvement.

We are still seeing a near 100% error rate for the Toast integration. For further updates, check the Toast status page (https://status.toasttab.com).

At 00:03 UTC on 10/21 error levels for the Toast integration returned to baseline. As this was the last remaining impacted Omnivore service, we are marking this incident as resolved.

Around 16:47 UTC, the edge nodes serving the Omnivore API went offline. We are currently investigating and hope to provide a better update once we have more information.

We have identified the trigger to be a bad patch that was released. This patch caused all of our edge servers to become unhealthy. We are currently working to rollback the edge servers to a known good state.

We are continuing to work on a fix for this issue.

We have partially restored some traffic to Omnivore API. We are still working on full resolution and stabilization.

We are seeing traffic approaching baseline levels. Webhooks have begun firing again. We are still working to improve stability.

A fix has been implemented, with all edge nodes now back in rotation and reporting healthy. We will continue to monitor for any further impacts.

Omnivore API and webhooks have been restored to full functionality.

We are seeing issues occurring with our 3rd-party cloud connectivity provider. This affects most Omnivore systems to varying degrees. We will update further as we have additional details.

We are continuing to investigate this issue.

The issues have subsided and we are seeing error rates return to baseline.

Beginning around 19:20 UTC, we began seeing an increased number of timeouts when calling the NCR CloudConnect API impacting ticket and clock entry reads. All API calls to fetch ticket and clock entry data are failing and webhooks may be delayed until service is restored. We are reaching out to our NCR contacts. We will continue to monitor for the issue to resolve. There are no further technical actions we can take to resolve the issue at this time.

At this time, we have no further updates from NCR regarding an estimated resolution. We will continue to monitor the situation and update as soon as the situation changes.

NCR posted an update to their statuspage at 22:13 UTC indicating they have found a possible solution and are working on stabilizing services. Omnivore calls to the NCR CloudConnect API began seeing a return to more normal error rates at 22:10 UTC and have been maintaining those rates since. We will continue to monitor error rates until NCR has confirmed they have resolved the issue.

NCR has marked their ordering components as operational and Omnivore has continued to see normal error rates since recovery.

We have identified an issue where writes to the Omnivore API are failing 100% of the time. We are implementing actions to recover. During this time, API reads will still function normally. We will provide updates as they are available.

We have implemented a fix and are observing successful API writes. We will continue to monitor.

API has resumed normal functionality

Starting at 18:25 UTC, we began seeing an increased number of errors when calling the Brink API, impacting ticket reads. API calls to ticket reads will likely fail at an increased rate and webhooks may be delayed until service returns to normal operation. So far, this seems isolated to a single Brink host. We have reached out to our Brink contacts and will continue to monitor until the situation is resolved. There are no other technical actions we can take to resolve the issue at this time.

We have observed Brink API calls return to their normal success levels for the past hour. Functionality of dependent systems have returned to normal.

At approximately 20:30 UTC, we identified an issue causing many locations to enter either a degraded or offline state. We have identified the issue and are working to resolve it.

A fix has been implemented and location status have returned to normal. We will continue to monitor at this time.

Affected locations have returned to online status and are operating normally

## Overview On May 29, 2024, Olo’s Omnivore Platform experienced agent degradation between 20:30 UTC and 21:55 UTC. Some API calls were failing during this time, and some agents went offline at 21:30 UTC. ## What Happened On May 29, 2024, during a routine instance resizing operation for our Connect service cluster, our configuration management system misidentified the IP addresses for the newly deployed instances, causing them to get bootstrapped incorrectly. This resulted in elevated error rates for Omnivore API calls beginning at 20:30 UTC, with 32% of connected Omnivore agents becoming degraded. At 21:30 UTC, some more API calls began to fail, causing 6% of connected agents to go fully offline. We initiated an accelerated rollback of the change, which fully restored service to a healthy state by 21:55 UTC. ## Next Steps * Improve the provisioning process to better detect and alert on this kind of misconfiguration earlier, and before the new instances are put into rotation. * Create additional alerting around agent errors to improve investigation speed.

Beginning around 00:02 UTC, we began seeing an increased number of timeouts when calling the NCR CloudConnect API impacting ticket and clock entry reads. API calls to fetch ticket and clock entry data will likely fail at an increased rate and webhooks may be delayed until service is restored. We are reaching out to our NCR contacts. We will continue to monitor for the issue to resolve. There are no further technical actions we can take to resolve the issue at this time.

Calls to the NCR CloudConnect API have returned to base level timings and error rates. We will continue to monitor the success of calls to the NCR CloudConnect API.

Beginning around 2:47 UTC, we began seeing an increased number of timeouts when calling the NCR CloudConnect API impacting ticket and clock entry reads. API calls to fetch ticket and clock entry data will likely fail at an increased rate and webhooks may be delayed until service is restored. We are reaching out to our NCR contacts. We will continue to monitor for the issue to resolve. There are no further technical actions we can take to resolve the issue at this time.

At 8:32 UTC, we saw calls to the NCR CloudConnect API return to base level timings and error rates. We will continue to monitor the success of calls to the NCR CloudConnect API.

Beginning around 00:00 UTC, we began seeing an increased number of timeouts when calling the NCR CloudConnect API impacting ticket and clock entry reads. API calls to fetch ticket and clock entry data will likely fail at an increased rate and webhooks may be delayed until service is restored. We are reaching out to our NCR contacts. We will continue to monitor for the issue to resolve. There are no further technical actions we can take to resolve the issue at this time.

Beginning around 02:26 UTC, calls to the NCR CloudConnect API began to succeed.

Beginning around 17:03 UTC, we began seeing an increased number of timeouts when calling the NCR CloudConnect API impacting ticket and clock entry reads. API calls to fetch ticket and clock entry data will likely fail at an increased rate and webhooks may be delayed until service is restored. We are reaching out to our NCR contacts. We will continue to monitor for the issue to resolve. There are no further technical actions we can take to resolve the issue at this time.

The number of timeouts has returned to normal levels.

At around 19:18 Omnivore engineers cleaned up an MMS ingress that was believed to be unused as part of a larger release. During the release, we noticed MMS order counts dropped and immediately began to rollback. We were completely rolled back by 19:37 with ordering traffic returning back to normal at that time. Upon investigation, we found that a manual DNS entry was in place that referenced the removed ingress. Because this entry was not committed to our our infrastructure repository, we mistakenly believed the ingress to be unused. We will audit for any other manual DNS entries in our environment before continuing with this release.

Beginning around 07:40, we began seeing an increased number of errors when calling the Brink API impacting ticket reads and clock entries. API calls to ticket reads and clock entries will likely fail at an increased rate and webhooks may be delayed until service is restored. We are reaching out to our Brink contacts. We will continue to monitor for the issue to resolve. There are no further technical actions we can take to resolve the issue at this time.

Around 9:15 UTC Brink API calls began to succeed.

Beginning around 8:21 UTC, we began seeing an increased number of timeouts when calling the NCR CloudConnect API impacting ticket and clock entry reads. API calls to fetch ticket and clock entry data will likely fail at an increased rate and webhooks may be delayed until service is restored. We are reaching out to our NCR contacts. We will continue to monitor for the issue to resolve. There are no further technical actions we can take to resolve the issue at this time.

Error rates returned to normal levels around 20:58 UTC. We will continue to monitor for any further increases.

Beginning around 16:50 UTC, we began seeing an increased number of timeouts when calling the Brink API impacting our Brink Locations. API calls to fetch Tickets will likely fail at an increased rate and webhooks may be delayed until service is restored. We are reaching out to our Brink contacts. We will continue to monitor for the issue to resolve. There are no further technical actions we can take to resolve the issue at this time.

After further investigation, we have found that the timeouts are only happening when making calls to https://api22.brinkpos.net. All other Brink hosts seem to be operational.

Error rates returned to normal levels around 17:20 UTC. We will continue to monitor for any further increases.