Keeping outage history

We are seeing occasional slowdowns of the app, possibly related to upstream issues with our GCP hosted load balancer. We are investigating.

We've identified the issue and are working with Google Cloud Platform to resolve. Some users may experience slowdowns or intermittent errors when accessing the app.

We are continuing to monitor has a fix is rolled out across our load balancer. Users may continue to experience an occasional long page load.

We are continuing to monitor as a fix is rolled out across our global load balancer.

This incident has been resolved.

We are currently investigating this issue.

We are continuing to investigate this issue.

A system wide Google Cloud is affecting Keeping - we're monitoring as the system recovers.

Google Cloud Platform experienced a major outage starting at 10:44 EDT. Keeping is hosted on Google Cloud Platform, so our services were affected. Remediation is in affect and service is being restored. https://status.cloud.google.com/incidents/J5ia5t9p3g9Q5Wi7r8Ev

GCP is still resolving the root cause but all Keeping systems are fully operational and we are continuing to monitor.

This incident has been resolved.

We are currently investigating an issue with ticket delivery.

This incident has been resolved.

We are currently investigating this issue.

Ticket delivery has returned to normal.

We are investigating issues with Cloudflare, our global DNS provider.

Cloudflare appears to have resolved the issue, we are continuing to monitor.

We are continuing to monitor for any further issues.

Cloudflare is still resolving the issue and service remains intermittent. We are investigating workarounds. https://www.cloudflarestatus.com/

We are continuing to monitor and investigate workarounds.

Cloudflare is continuing to remediate the issue and we are investigating workarounds. Service remains intermittent based on your geographic location.

We've implemented a workaround which should restore service for most users. Full recovery is ongoing as Cloudflare continues to remediate on their end.

This issue appears resolved - please restart your Chrome Browser if you are still experiencing issues, or reach out to [email protected].

We've identified an issue that caused a slowdown in website performance and ticket syncing. The issue has been resolved and we are monitoring the system as it recovers.

We're still seeing slowdowns in app performance and are continuing to investigate

We're still seeing slowdowns in app performance and are continuing to investigate.

A fix has been implemented and we are monitoring the results. Thanks for your patience here!

This incident has been resolved - please contact [email protected] if you are still experiencing issues.

We've identified an issue that prevents the "lightning bolt" (Knowledge Base) button from appearing when replying to a Keeping ticket. This is affecting a small subset of users. For now, the workaround is to "pop out" the reply via the button in the upper right corner when composing a reply. This will restore the buttons until we resolve the issue.

The latest version of the Keeping Chrome Extension resolves this issue. Please update your Chrome Extension if you are experiencing this issue.

We are currently investigating issues with "Sign in with Google". Our initial investigation would indicate that there is a global outage for Google's OAuth platform.

We can confirm there there is a global outage of Google's OAuth service ("Sign in with Google"). We are monitoring, there is no current workaround.

Google appears to have resolved the issue. We are monitoring.

This incident has been resolved. We will continue to monitor for any regressions.

We are currently investigating sending errors related to our email sending provider, Mailgun. https://status.mailgun.com/

Mailgun has identified the issue and is working on a fix. In the meantime, we recommend responding to Keeping tickets in the Keeping app at https://app.keeping.com/inbox.

Mailgun has identified the issue and has implemented a fix. We are monitoring the results.

This incident has been resolved.

Google has made a change to how it filters the Gmail Inbox, preventing Keeping tickets from being displayed in some cases. We've pushed an update to the Chrome Webstore which resolves the issue - it is currently being reviewed. If you are affected by this issue, you may use the Keeping App to access your tickets, or contact [email protected] if you would like instructions to manually update your Chrome Extension. We expect the new Extension to be approved by the Webstore team in the next 48 hours.

The Keeping Chrome extension has been updated in the Chrome Webstore. The update should be pushed to you automatically, but you can uninstall and reinstall the Keeping Chrome extension to speed up the process. Please feel free to reach out to [email protected] if you are still experiencing issues after updating your Extension.

This incident has been resolved.

We are currently investigating performance issues with the Keeping Sidebar.

A fix has been implemented and we are now monitoring.

This incident has been resolved.

We are investigating issues with signing into the main Keeping application.

Due to a widespread Google Cloud outage, parts of Keeping are temporarily down. We're monitoring closely and will be back to normal as soon as it's resolved. Thank you for your patience!

Google has identified the root cause and is rolling out a fix. We are still experiencing issues with email syncing and login - we'll post another update as the situation develops.

Google has fixed the issue, and services are beginning to return to normal. We are continuing to monitor.

This incident has been resolved.

We are investigating an issue with our email backbone provider, MailGun. Incoming email is delayed.

MailGun is not accepting email to incoming.keeping.com. We are working with MailGun to resolve the issue.

The issue has been identified - we are waiting for MailGun to resolve the issue. Only mailboxes connected via forwarding are affected.

MailGun has resolved the issues and emails should be flowing again. We are monitoring the fix.

Starting at approximately 5:05 AM Eastern Daylight Time, our primary email backbone provider, MailGun, experienced an issue with the domain where we receive inbound emails on our customer's behalf. They resolved the issue at approximately 10:05 Eastern Daylight Time.

We are currently experiencing delays with email syncing - the team is investigating.

The issue has been identified, and mitigation is in progress.

The issue has been resolved - the team will continue to monitor.

This incident has been resolved.

Google is rolling out a new version of Gmail that may disrupt the opening of the right sidebar when viewing a Keeping ticket. A fix for this issue is included in version 6.6.3 of the Keeping Chrome extension, which has been submitted to the Chrome Web Store. The update will be automatically pushed to users once approved. For users affected by this issue, please access your tickets via https://app.keeping.com/inbox until the issue is resolved.

Version 6.6.3 of the Keeping Chrome Extension is now available in the Chrome Web Store and resolves this issue. If your extension hasn’t updated automatically, please follow these instructions to update it manually: https://docs.keeping.com/help/setup/manually-updating-the-extension

We are seeing elevated errors from the Gmail API when syncing a connected mailbox. We are investigating.

The API Keeping uses to connect to Gmail is still returning elevated errors. We are continuing to investigate the issue.

We've identified the issue and are working to mitigate.

We are continuing to work on a fix for this issue.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

This incident has been resolved.

The Keeping Chrome Extension and web application are experiencing connectivity issues due to an outage with our primary hosting provider. We've remediated the issue and are monitoring.

This incident has been resolved.

### Postmortem: Service Outage on July 16, 2024 **Incident Summary:** On 9:05 am EDT July 16, 2024, Keeping experienced intermittent connection issues that lasted approximately 2 hours. The outage affected both the Keeping Chrome Extension and the main application website at https://app.keeping.com/ **Timeline:** * **9:05 am EDT:** Initial reports of the website being inaccessible were received. * **9:07 am EDT** : Incident response team was alerted and began investigation. * **9:25 am EDT** : Root cause identified as a Denial of Service attack on Keeping’s hosting provider, Gigalixir. * **9:30 am EDT**: Mitigation steps initiated. * **10:25 am EDT**: Website functionality partially restored. * **11:00 am EDT**: Full service restored and confirmed stable. **Root Cause:** The outage was caused by [a SYN flood attack on Gigalixir’s load balancers](https://gigalixir.com/blog/gcp-us-central-service-outage/). Gigalixir is Keeping’s primary web hosting provider. **Impact:** During the outage, access to Keeping’s main web application was severely limited \(or impossible\), and syncing between agent accounts and shared mailboxes was paused. No data was lost.

Due to an outage with our primary hosting provider, the Keeping Sidebar and app experienced connectivity issues from 9:57 am EDT to 10:14 am EDT. This issue is now resoloved.