Stamped 장애 기록

From 6:45am PST, our identity service had a cache issue which resulted in an hour of instability across services. Login was inaccessible, some widgets may not have been loading, and some API endpoints may have been intermittent.

We have identified the issue and implemented a fix that we are currently monitoring.

Confirmed resolved at 9:15am PST. Everything has been stable and within nominal range since then.

With changes in Twitter's API policy and use, we will be working with Twitter on re-enabling our Social Push and Smart Banner features. What this means for your usage of these features during this period: - Smart Banners: Direct sharing on Twitter will not function however you can continue to use Manual Sharing - Social Push to Twitter will be unavailable

This incident has been resolved.

We are currently undergoing scheduled maintenance. Access to the Stamped dashboard will be unavailable during this time.

Stamped dashboard access is restored however new sign-ups will not be able to connect to your store during this maintenance period.

Connecting your store to all platforms for new sign-ups is restored through the Stamped dashboard.

Database Load Spike A spike in our database operational load was noticed over the past several hours which is affecting our platform's performance. We are currently investigating the cause.

We have identified a widget API call behaviour that was causing spikes in our database operational load. This issue has now been resolved and our database operations are operating as normal now.

We have been experiencing ongoing performance degradation since last evening in addition to delays in the reviews messaging queue that we are actively investigating the cause. As a result of the performance degradation, you may experience an issue while accessing Stamped services during this time. This includes our Dashboard and the Stamped API. With the reviews messaging queue, you may see delays in order status and the sending out of review requests.

We have made some patches that improved overall performance and increased the processing of the review messages backlog.

Intermittent performance degradation was noticed for our servers and we are currently investigating this issue.

We discovered Shopify was having instability issues on their API that was impacting third party services which has now been resolved.

We have been experiencing ongoing performance degradation since this morning. From our investigation, we have isolated the cause of the issue to be our server cluster on AWS and are continuing to investigate in greater detail. As a result of the performance degradation, you may experience an issue while accessing Stamped services during this time. This includes our Dashboard and the Stamped API. Our team is working tirelessly to resolve this and will be scheduling a maintenance period overnight during North America hours to implement necessary updates to our AWS server. Please note there will be no impairment to your data during the maintenance period but access to some of our services may be restricted during that time.

Maintenance has been completed, and all services have resumed and are showing performance within expected ranges.

This incident has been resolved.

A spike in our database operational load that hit our max load between 8:45am to 10:20am PST was noticed that affected our platform's performance during that period. The operational load is back to normal currently but we are continually investigating the cause.

Minor backlog identified in the sending of review requests, reaching 1-2 day delay for certain larger customers. We expect a resolution in the next 2-3 days.

The current review requests sending backlog has been resolved.

An incident has occurred with the Stamped AWS Autoscaling cluster. This has resulted in degraded performance impacting multiple services. It has been identified, a fix has been implemented and is expected to be resolved shortly. We will be continuing to monitor and provide updates on this incident as it occurs.

The fix has been deployed, and operations should be resuming normal behavior, but we are continuing to monitor.

This incident has been resolved.

Some users are experiencing unresponsive or slow load times on review display widgets. We are aware of the issue and are currently looking at potential causes. We apologize for any inconvenience this has caused.

The issue has been identified and a fix is being implemented.

This issue has been resolved and all effected review widgets should be delivering properly.

At 23:00(GMT+8) one of our servers became non-responsive. Due to this issue, and the volume of reviews, our software created a data dump into an XML file. This prevented users from logging into to their dashboards. Resolution: After analysis of the issue, it was determined the quickest solution was to reboot our servers in sequence. We instituted additional monitoring, and added a routine to prevent a similar issue from reoccurring. We are actively monitoring the fix and no additional issues have been observed. The issue has been resolved at 00:07(GMT+8).