52 Bonusly incidents · aprile 2022 — official updates, affected components, duration and resolution details.
Stiamo cercando un problema che riguarda alcune redenzioni di ricompensa. Se hai recentemente riscattato punti per una ricompensa, può mostrare come “a pagamento” nel tuo account. Il nostro team sta indagando attivamente, e pubblicheremo un aggiornamento qui non appena ne sapremo di più.
Tradotto automaticamente dall'aggiornamento ufficiale dell'incidente.
Abbiamo identificato un problema con il nostro sistema di ricompense che impedisce la maggior parte delle redizioni che passano. Stiamo lavorando alla soluzione.
Ok, abbiamo implementato la soluzione e ora stiamo monitorando per assicurarsi che tutte le ricompense passino senza intoppi.
Stiamo lavorando ora per far passare tutte le redenzioni di ricompensa fallite. Ci aspettiamo che ci vorranno due ore.
Ok, tutte le redenzioni della ricompensa fallite sono state ritrattate. Siamo a posto, e il sistema funziona nominalmente.
Tradotto automaticamente dall'aggiornamento ufficiale dell'incidente.
We are currently experiencing intermittent issues with rewards redemption. Some users may encounter timeouts or delays when attempting to redeem rewards. Our engineering team is actively investigating the root cause. We will provide updates as we learn more.
We identified a slow query in our rewards infrastructure and implemented a query improvement to resolve.
This incident has been resolved.
We are currently investigating reports of slowness impacting the activity feeds on the Bonusly homepage. Please monitor this thread for updates.
Performance has stabilized - Our teams will continue to monitor.
Monitoring the situation confirms performance is nominal. Thanks for your patience!
We're seeing an increased error rate with Rewards fulfillment for gift cards. Investigating
One of our fulfillment vendors is experiencing a high error rate. We're working through the issue with them to help resolve it.
Almost all of the failed reward redemptions from a couple hours ago have now been retried and succeeded. All will be pushed through in the next few minutes. We'll monitor things after that.
We've successfully retried all failed redemptions from this morning. New rewards are being fulfilled as expected. Thanks for riding through this with us!
We're seeing our web app take longer to load than normal. We've noticed cyclical spikes on our DB cluster CPU and are investigating root cause.
Ok, we've identified the issue as a cluster capacity issue, which is resolvable with scaling. Stay tuned.
Telemetry across our system indicates nominal performance. We'll monitor the situation to ensure that continues.
Performance continues to be nominal after several hours of monitoring, and we're marking this incident as resolved.
Bonusly’s support messaging system, Intercom, is currently experiencing a widespread issue. Some users may be unable to access the Bonusly Support menu from their dashboard or communicate with the Bonusly Support team. We are actively monitoring the situation and will share updates as they become available.
Our partners have resolved the issue.
We've identified an issue where people who login with a username and password are receiving a Captcha verification failure. Note, login for people with SSO is unaffected.
We think we identified the issue and are deploying a hotfix. Stay tuned.
Ok, we've confirmed our fix is working and are rolling it out to our servers now. ETA is 30 minutes.
The fix is still rolling out. In the interim, root cause was an upgrade to our net-http library. That upgrade removed a default header, "application/x-www-form-urlencoded," from http requests our code issues to Cloudflare. The Cloudflare Turnstile API, which we use in conjunction with Captcha, expects this header to be present, and started rejecting our requests as a result.
Our fix is now deployed and we're seeing normal volume of successful username/password logins.
Bonusly is currently experiencing minor performance issues that may result in longer-than-normal load times. Our teams have identified the cause and are implementing a fix.
A fix has been implemented and performance is returning to normal. We will continue to monitor and provide another update soon.
This is now resolved
We are currently investigating an issue some users are having when attempting to access the Rewards Catalog. We will provide updates as soon as they become available.
This issue has been resolved, Rewards are now accessible.
Bonusly is currently experiencing intermittent availability issues due to the CloudFare outage causing widespread issues today. We are actively monitoring the situation and will provide updates here.
The global Cloudflare outage has been resolved, and users are now able to access Bonusly normally. We will continue to monitor the situation closely.
Service continues to operate as normal. This issue with Cloudfare is now resolved.
We experienced a temporary performance issue due to an unexpected spike in background job processing. Our engineering team quickly scaled system resources and restored normal performance within minutes. All services are now fully operational and we’re implementing safeguards to prevent a recurrence.
We’re closely monitoring the current Amazon AWS service disruptions. While several partners are affected, the Bonusly core platform remains operational with only a minor impact on performance. We’ll continue to monitor this situation and will post updates as they become available.
All services using AWS have been restored. The issue is now resolved.
We are investigating an issue some users have reported where the login screen does not respond after entering their password. Users who are already signed in can navigate without issue. We will provide updates as soon as they become available.
This issue is now resolved.
This incident has been resolved.
Our hosting provider, Heroku, has returned to normal service. The Bonusly Platform is back online and confirmed to be operating normally. We are currently monitoring to ensure stable service remains.
Service Remains stable. This incident is now closed.
Our third-party vendor, Heroku, is experiencing service issues that are affecting the Bonusly Platform. We have escalated this issue to this team and await service restoration updates. We will post updates here when available.
Our third-party vendor, Heroku, is experiencing global service issues that are affecting thousands of platforms, including the Bonusly Platform. Heroku continues to investigate and Bonusly will continue monitoring this situation to provide updates as they become available.
Bonusly continues to be affected by the Global issues our hosting provider, Heroku, is experiencing. We are actively monitoring the situation and will continue to share updates as they become available.
Our hosting provider, Heroku, has returned to normal service. The Bonusly Platform is back online and confirmed to be operating normally.
We are currently investigating an error users are receiving when selecting on categories within the Rewards page. This is a top priority for us. Please subscribe to this page for future updates. Bonusly Support
This incident has been resolved.
Our teams are actively working to resolve a performance issue with the Participation and Recognition Analytics pages. Please monitor this page for updates as they become available.
This issue is now resolved.
We are currently investigating this issue.
This incident has been resolved.
Engineering has been notified and is currently investigating the issue.
The issue has been identified and a fix is currently being implemented.
A fix has been implemented and the team is monitoring results
The fix has been live and monitored and we are no longer seeing errors.
