Wyng outage history

Some clients are experiencing delays in processing integration data flows outbound from Wyng to external systems. Users may notice more integration transactions in a Pending state, and longer elapsed times between data collection by an experience and the delivery of that data to their external system or API. Data collection and storage are not impacted, and are operating normally. [UPDATE 08:30 AM EDT] We have taken steps to address the issue and processing delay for impacted clients is recovering now.

Delays have been resolved for all impacted clients.

An issue with event processing streams caused delays in integration processing affected a subset of Wyng clients. Initial behavior was intermittent, with delays occurring sporadically on July 1, that progressed to a backlog of up to 9 hours of transactions for affected clients starting at about 4:00 AM EDT on July 2. Wyng ops staff took initial action on the issue at 8:30 AM EDT on July 2. By 12:00 EDT the backlog reduced to 2 hours, by 4:20 PM EDT the backlog was 1 hour, and the issue was resolved for all affected clients at 6:38 PM EDT on July 2. The root cause was interdependence between a set of clients whose transactions were being processed in a shared stream. Significant delays in downstream processing by one client caused a backlog across all clients sharing the stream. To avoid recurrence of this failure mode, Wyng is prioritizing a infrastructure update to move from shared to dedicated streams for transaction event processing.

Instagram Content Streams are not collecting new content or experiencing significant delays in collecting new content, due to an outage affecting Meta and Instagram APIs. We are monitoring the situation. All other Wyng content features including UGC moderation, display, and Content API are operating normally.

We are continuing to monitor and will update on a change in status.

The issue affecting Instagram Content Streams has been resolved. New Instagram content is now being collected normally following the restoration of Meta and Instagram API services. We will continue to monitor the services, but no further impact is currently expected.

Some Wyng users are unable to log into the Platform. Additionally some users who are already logged in may encounter delays using Internal Wyng dashboard functions, including editing and publishing experiences. We are investigating. Live experiences are operating normally and not affected.

The root cause has been identified and issues are recovering. We are monitoring.

The incident has been resolved.

Some Wyng users are unable to log into the Platform. Additionally some users who are already logged in may encounter delays or intermittent issues using Internal Wyng dashboard functions, including editing and publishing experiences. We are investigating. Live experiences are operating normally and not affected.

Login is being restored for affected users, and degraded performance of some internal Wyng dashboard functions is also recovering, We are continuing to monitor.

Login and all Wyng dashboard functions have recovered are operating normally. We are continuing to monitor.

We are seeing delays in processing integration data flows outbound from Wyng to external systems. Users may notice more integration transactions in a Pending state, and longer elapsed times between data collection by an experience and the delivery of that data to their external system or API. Data collection and storage are not impacted, and are operating normally. We are investigating.

The backlogs in processing integration data flows have been cleared, and processing times have returned to normal. We are continuing to actively monitor.

This incident was triggered inaccurately, due to outdated events in a log stream, and can be ignored. We are reviewing and taking action to correct similar cases in the future.

We are seeing delays in processing integration data flows outbound from Wyng to external systems. Users may notice more integration transactions in a Pending state, and longer elapsed times between data collection by an experience and the delivery of that data to their external system or API. Data collection and storage are not impacted, and are operating normally. We are investigating.

We have identified the root cause of the backlog and are continuing to monitor it. Users will notice more integration transactions in Pending state, and delays of up to 10 hours between data collection and delivery of data to their external system or API. No data is at risk, and excluding the delay, all events are being processed normally. We will post again on a change in status.

The backlogs in processing integration data flows have been cleared, and processing times have returned to normal. We are continuing to actively monitor.

We are seeing delays in processing integration data flows outbound from Wyng to external systems. Users may notice more integration transactions in a Pending state, and longer elapsed times between data collection by an experience and the delivery of that data to their external system or API. Data collection and storage are not impacted, and are operating normally. We are investigating.

We have identified the root cause of the backlog and are continuing to monitor it. Users may notice more integration transactions in Pending state, and longer elapsed times between data collection and delivery of data to their external system or API. Aside from these impacts, all systems are operating normally. We will post again on a change in status.

The backlogs in processing integration data flows have been cleared, and processing times have returned to normal. We are continuing to actively monitor.

We are seeing delays in processing integration data flows outbound from Wyng to external systems. Users may notice more integration transactions in a Pending state, and longer elapsed times between data collection by an experience and the delivery of that data to their external system or API. Data collection and storage are not impacted, and are operating normally. We are investigating.

The backlogs in processing integration data flows have been cleared, and processing times have returned to normal for all customers. The root cause of this incident and similar incidents reported early today is unusually high, temporary spikes in volume of incoming data, related to the activity of a subset of Wyng customers. While the pattern continues, some customers may experience intermittent delays in processing integration data flows. We will keep this incident report open while we monitor traffic patterns.

We opened a new incident report to track this.

We are seeing delays in processing integration data flows outbound from Wyng to external systems. Users may notice more integration transactions in a Pending state, and longer elapsed times between data collection by an experience and the delivery of that data to their external system or API. Data collection and storage are not impacted, and are operating normally. We are investigating.

The backlogs in processing integration data flows have been cleared, and processing times have returned to normal. We are continuing to actively monitor.

We are seeing delays in processing integration data flows outbound from Wyng to external systems. Users may notice more integration transactions in a Pending state, and longer elapsed times between data collection by an experience and the delivery of that data to their external system or API. Data collection and storage are not impacted, and are operating normally. We are investigating.

The backlogs in processing integration data flows have been cleared, and processing times have returned to normal. We are continuing to actively monitor.

We are seeing delays in processing integration data flows outbound from Wyng to external systems. Users may notice more integration transactions in a Pending state, and longer elapsed times between data collection by an experience and the delivery of that data to their external system or API. Data collection and storage are not impacted, and are operating normally. We are investigating.

The backlogs in processing integration data flows have been cleared, and processing times have returned to normal. We are continuing to actively monitor.

We are seeing delays in processing integration data flows outbound from Wyng to external systems. Users may notice more integration transactions in a Pending state, and longer elapsed times between data collection by an experience and the delivery of that data to their external system or API. Data collection and storage are not impacted, and are operating normally. We are investigating.

We have identified the cause of the delay in integration data flows is an unusually high spike in transaction volume, and will resolve on its own as the backlog of queued transactions clears. Data collection and storage are operating normally. We will update again once the issue is resolved.

The backlogs in processing integration data flows have been cleared, and processing times have returned to normal. We are continuing to actively monitor.

There is a disruption affecting some Wyng customers and services. The root cause is an Amazon Web Services outage. Currently the following Wyng components and features are delayed or not working: * Experience Metrics * Weekly performance reports sent by email * Content Streams * Integrations and Webhooks * Identity Resolution by Email and SMS * Experiences with Custom Domains AWS indicates that impacted services are in process of being restored. AWS Service health can be found here: https://health.aws.amazon.com/health/status

We have updated the status of some components. We will post another update in 60 minutes.

Amazon Web Services continues to work on resolving the outage, and we continue to track updates and monitor for recovery. Here is the latest update from AWS, posted at 11:04 AM U.S. Eastern time: "[08:04 AM PDT] We continue to investigate the root cause for the network connectivity issues that are impacting AWS services such as DynamoDB, SQS, and Amazon Connect in the US-EAST-1 Region. We have identified that the issue originated from within the EC2 internal network. We continue to investigate and identify mitigations.".

Amazon Web Services continues to work on full resolution, but at this time, all Wyng services are operational or recovering. Integrations and web hooks are operating normally, and all queued transactions have been processed. Weekly reports have been delivered. We continue to monitor. Here is the latest update from AWS, posted at 12:13 PM U.S. Eastern time: "[09:13 AM PDT] We have taken additional mitigation steps to aid the recovery of the underlying internal subsystem responsible for monitoring the health of our network load balancers and are now seeing connectivity and API recovery for AWS services. We have also identified and are applying next steps to mitigate throttling of new EC2 instance launches. We will provide an update by 10:00 AM PDT."

Amazon Web Services continues to work to resolve the outage, and we are continuing to monitor. We have identified a new, related issue, which is that users editing experiences cannot view or edit sections with formatted text content. This is due to the AWS outage impacting a third-party vendor whose formatted text editor component we use. While this issue is outstanding, we recommend users hold off on editing and publishing changes to experiences that use formatted text content. Formatted text will continue to display normally in any experiences that are already live. Here is the latest update from AWS, posted at 3:15 PM U.S. Eastern time: "[12:15 PM PDT] We continue to observe recovery across all AWS services, and instance launches are succeeding across multiple Availability Zones in the US-EAST-1 Regions. For Lambda, customers may face intermittent function errors for functions making network requests to other services or systems as we work to address residual network connectivity issues. To recover Lambda’s invocation errors, we slowed down the rate of SQS polling via Lambda Event Source Mappings. We are now increasing the rate of SQS polling as we experience more successful invocations and reduced function errors. We will provide another update by 1:00 PM PDT."

At this time all Wyng applications and services are operating normally. Amazon Web Services continues to work on full resolution of the root cause incident, which remains open as of the last update, at 5:48 PM U.S. Eastern Time. We are continuing to monitor.

This incident has been resolved.

We are seeing delays in processing integration data flows outbound from Wyng to external systems. Users may notice more integration transactions in a Pending state, and longer elapsed times between data collection by an experience and the delivery of that data to their external system or API. Data collection and storage are not impacted, and are operating normally. We are investigating.

The backlogs in processing integration data flows have been cleared, and processing times have returned to normal. We are continuing to actively monitor.

We are seeing delays in processing integration data flows outbound from Wyng to external systems. Users may notice more integration transactions in a Pending state, and longer elapsed times between data collection by an experience and the delivery of that data to their external system or API. Data collection and storage are not impacted, and are operating normally. We are investigating.

The delay is affecting a subset of integration transactions processed today after 10:00 AM EDT. We are continuing to monitor.

Some integration transactions continue to be delayed as our systems catch up. We are continuing to monitor and will provide a further update once the issue is fully resolved.

The incident has been resolved and all pending transactions have been processed.

We are seeing delays in processing integration data flows outbound from Wyng to external systems. Users may notice more integration transactions in a Pending state, and longer elapsed times between data collection by an experience and the delivery of that data to their external system or API. Data collection and storage are not impacted, and are operating normally. We are investigating.

The backlogs in processing integration data flows have been cleared, and processing times have returned to normal. We are continuing to actively monitor.

Some users reported degraded performance and intermittent failures in content management features. Wyng support became aware of the issue at 07:53 GMT on June 10, and the issue was resolved by manual intervention by our production ops team at 10:08 GMT on June 10.

We are seeing delays in processing integration data flows outbound from Wyng to external systems. Users may notice more integration transactions in a Pending state, and longer elapsed times between data collection by an experience and the delivery of that data to their external system or API. Data collection and storage are not impacted, and are operating normally. We are investigating.

We have identified the root cause, which is a temporary spike in integration data stream volume. While the spike persists, users will continue to see delays of up to 60 minutes between the time that is collected in Wyng, and the delivery of that data to their external system or API. Data collection and storage are not impacted, and operating normally. We will provide an update when latency returns to normal.

Delays are resolved. We are continuing to monitor.

The backlogs in processing integration data flows have been cleared, and processing times have returned to normal. We are continuing to actively monitor.

Experience metrics are delayed for some users. Historical metrics are still available. Data is being collected and will be updated once the issue is resolved. We have isolated the problem and are working on a fix.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Some customers are reporting an issue accessing Metrics sub-tabs. The issue is limited to a subset of customers, and does not affect accuracy of metrics or related event processing. Summary metrics continue to work normally. We are currently investigating the issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.