Skai 장애 기록

Our engineering team identified a delay in our internal data synchronization process caused by a third party. The outcome of that data delay is a disruption to some of our features as will be listed below. We have already reached out to the third party in charge and are waiting on answers.

Our engineering team has continued investigating the delay affecting our internal data synchronization processes. We’ve now identified the issue as originating within our own systems rather than with an external provider, as initially believed. The issue has been isolated and remediation is currently underway. We currently expect full recovery within approximately 2 hours. Once recovery is complete, affected processes and features will resume normal operation, and we will also handle any impacted portfolios immediately. We apologize for the disruption and appreciate your patience while we complete recovery efforts

This incident has been resolved.

AWS is currently experiencing a major outage affecting the US-East-1 region, which began at approximately 8:30 PM EST. AWS engineers are actively working to restore the affected services. As a result, our platform is experiencing intermittent issues across a small number of servers. Our engineering team is actively working to restore full service as quickly as possible. AWS Status Page: https://health.aws.amazon.com/health/status

AWS experienced a power issue in one of their US-East data centres, which caused some servers and storage systems to shut down unexpectedly. During recovery, cooling systems in that area were also affected, so AWS has had to restore services gradually and carefully to avoid further disruption. Due to this, we are continuing to see a small number of servers remain unavailable. Our engineers are making every effort to restore full service as quickly as possible once AWS has completed recovery activities. Amazon AWS Status Page: https://health.aws.amazon.com/health/status

We wanted to provide an update regarding the earlier AWS infrastructure issue affecting a small number of servers hosted in the US-East region. While AWS is still continuing recovery activities within their environment, we have successfully mitigated the impact on our side for the limited number of affected servers. Once services were restored, our automated processes resumed normally and began recovering any missing daily performance data. Our engineering team has confirmed there was no impact to Optimization. We are continuing to review and correct a small number of remaining data inconsistencies on the affected servers. Where applicable, your Client Success team will proactively reach out with any relevant updates regarding your Skai platform. We will continue to monitor the situation closely until AWS confirms full resolution.

We are aware that Amazon performance data for yesterday and today is missing for some accounts in the Skai Platform. Our engineering team is currently investigating the issue. We will provide further updates as more information becomes available.

We have identified the internal issue causing Amazon Stream performance data to be missing for yesterday and today. Scheduled Reports, Automation Actions, and Optimization budgets may be affected. Our engineers are actively working to restore the missing data for the affected clients, and once resolved, budgets will be automatically adjusted back to expected values.

This issue is resolved. Amazon performance data is now updated in Skai and the campaign budgets will get updated accordingly.

Criteo is currently experiencing an API outage. We have notified Criteo, and their team is actively investigating the issue. For further updates, please refer to the Criteo Status Page: https://status.criteo.com/incidents/c7j0nz3rp682

This incident has been resolved.

We are currently experiencing delays in performance data from the Walmart API. At this time, data for 28 February is delayed within the Skai platform. The issue has been escalated to Walmart’s engineering team, and they are investigating. We will provide an update as soon as the issue has been resolved.

This incident has been resolved.

Please note that we are experiencing a delay with processing Amazon DSP data. Our engineers are working on fixing the issue and we will update as soon as we can.

This incident has been resolved.

Ever since this morning (Sunday, December 28th), Microsoft Ads are experiencing delays in their reporting API. The Skai team reached out to Microsoft on that, yet Microsoft haven't acknowledged that issue yet.

MSN are still investigating this issue on their end and we are waiting for their response. In the meantime, we started seeing impact on our optimization portfolios affecting both Budget Navigator and Portfolios optimizations. Please consider moving to manual bidding until this is resolved. Once MSN data is available, we will resume optimization and adjust budgets and bids accordingly.

Looks like Microsoft started recovering and we see reports coming in to our platform. There may still be delays in features that rely on yesterday's data until all of yesterday's data will be fetched and processed. As for optimization, no need to move to manual bidding, Skai platform will recover automatically.

The issue has been resolved, all features are back to normal

Starting this morning at 2 am EST, Microsoft Ads are experiencing delays in their reporting and API. Their team is investigating the issue, and we are monitoring the progress. https://status.ads.microsoft.com/

The issue was resoled by Microsoft and the reports are coming in. We will continue to monitor.

This incident has been resolved.

AWS experienced a major outage at around 3 AM EST, centered on the US-East-1 region. The disruption impacted a wide range of major apps, websites, and services globally. The outage lasted roughly three hours, with most services now reported as fully mitigated. AWS Status page: https://health.aws.amazon.com/health/status Our platform is fully operational for all clients. Performance data has already begun to recover automatically, and any processes waiting for daily data will resume as soon as the data becomes available. Our teams continue to monitor the situation closely to ensure there is no further impact.

Amazon continues to work on mitigating the issue on their end. You can monitor their official status page for updates: https://health.aws.amazon.com/health/status We are actively monitoring the situation while awaiting a full resolution from Amazon. We are still experiencing delays in Amazon Performance data being received due to the ongoing incident. Data is continuing to process, albeit more slowly than usual. Some users may notice slower performance when loading analysis grids; this can often be resolved by refreshing the page. A few users have encountered intermittent “Internal 500” errors. If this occurs, please try again after a short interval. We hope Amazon will resolve the underlying issue soon to restore normal service levels.

We are continuing to monitor for any further issues.

Amazon AWS has confirmed that the issue has been resolved on their side. For reference, see the AWS Status page: https://health.aws.amazon.com/health/status A backlog of Amazon performance reports remains on our side. We are actively monitoring and processing these reports to ensure normal operations are fully restored.

Amazon has confirmed that the earlier issue has been resolved. However, we are still seeing delays in Amazon Performance data being received. Our engineering team is actively working with Amazon to investigate and expedite a full resolution. We appreciate your patience and understanding while this is being addressed.

We can confirm that Amazon Performance data has now fully recovered, and all services are fully operational for all clients. Our team has verified that there is no further impact across processes, including: - Scheduled Reporting - Automated Actions - Optimization Thank you for your patience while this issue was being resolved.

This incident has been resolved.

We are aware of an issue in the Skai Platform where Performance Metrics for October are currently missing for Amazon DSP. Amazon is currently investigating the issue on their side, and we will provide updates as soon as more information becomes available.

This incident has been resolved.

Ever since yesterday (Sunday, August 3rd) afternoon, Microsoft Ads are experiencing delays in their reporting and API. They have acknowledged that and even posted that in their status page: https://status.ads.microsoft.com/

Daily reports have been processed successfully. We will continue to monitor.

This incident has been resolved.

Ever since yesterday, we are experiencing API issues with Criteo. These issues affect performance data fetching, as well as sync with Criteo platform. Criteo team was notified about these issues

Criteo informed us that they have escalated the issue on their end, we are still waiting on their investigation.

Criteo informed us that the issue was fixed, and we do see an improvement in our API calls to Criteo, we are still monitoring the situation

Our API calls go through to Criteo, and we do see a decrease in our requests queue. However, due to Criteo's API call limit, the decrease is slow. We hope to get it all done until the beginning of the business day tomorrow (US). Operations that rely on performance data availability may still have degraded performance, as performance data us still being fetched, limited by Criteo's API call limit. Uploads and Downloads work with no issue.

The issue was resolved, all performance data for the impacted dates is currently in place and the system is fully operational

Please note that Meta performance data may be missing from the grids for the last 7 days. The issue was escalated to Meta's team with the highest priority. Note that Automated Actions that rely on performance data metrics may not run correctly. Please consider disabling Automated Action that use filters on performance data until the issue is resolved.

The issue has been identified and a fix is being implemented.

Meta has resolved the issue. We have completed data backfill in Skai to ensure all data is now accurate. If you have disabled any Automated Actions that used filters on performance data, please go ahead and re-enable them.

We are aware that some clients are seeing inflated Amazon performance metrics for 23rd June. Our engineering team is currently investigating the issue.

Our engineers have identified the issue and are working to correct the data for 23rd June. Intra day data will be fixed retroactively.

The issue is Resolved. Data for 23rd June was corrected on all affected servers.

We identified an issue with retrieving performance data from Microsoft Ads platform. Our developers have already sent an inquiry to Microsoft support. Note that this issue is affecting only a few of our customers and not all.

We are continuing to work on a fix for this issue.

A Microsoft Advertising API issue caused delays in receiving performance data reports. This issue was resolved and all Microsoft Advertising daily reports were processed.

We're currently experiencing an issue with scheduled reports. Some fusion and structure scheduled reports are currently failing to run across many Skai servers.

The data in Skai is not affected by this issue and is up to date.
 All analysis view reports are unaffected. Our engineers are actively investigating the issue and we will continue to update you on this issue.

The issue has been identified and our engineers have released a fix which is currently being deployed to all affected servers.

The issue is now fully resolved. All affected failed reports were successfully re-run by our team.

A Microsoft Advertising API issue is causing delays in receiving performance data reports. We have escalated the issue with Microsoft's team.

This incident has been resolved.

Sam's Club performance data is missing starting April 23rd. We have informed Sam's Club engineers of the issue. Once API access is restored, missing days will automatically be backfilled.

Sam's Club daily performance data is processed successfully. Data is still missing for April 23rd. Sam's Club's team is looking into the issue

Kenshoo Social is currently unavailable. We are working to resolve the issue and will provide regular status updates until resolved.

We are currently experiencing a partial outage of the Performance Grids. This may cause slow loading, missing data or timeout errors. We are working to resolve the issue, and will provide regular status updates until resolved.