18 Keeping incidents · 2024年7月 — official updates, affected components, duration and resolution details.
我们发现应用程序偶尔会减速, 可能与上游问题有关, 我们正在调查.
我们已经查明了这个问题,并与谷歌云平台合作解决. 一些用户在访问该应用程序时可能会遇到减速或间歇性错误.
我们正在继续监测 我们的负载平衡器都有一个固定装置 用户可能偶尔会遇到长长的页面负荷.
我们正在继续监测,因为我们的全球负载平衡器正在推出一个解决方案.
这一事件已经得到解决.
自动翻译自官方事件更新。
我们目前正在调查这一问题.
我们正在继续调查这一问题.
一个全系统Google Cloud正在影响Keeping——随着系统恢复,我们正在监测.
Google Cloud平台从10:44 EDT开始经历了重大停电. 保存在谷歌云平台上,因此我们的服务受到影响. 补救正在产生影响,服务正在恢复。 https://status.cloud.google.com/incidents/J5ia5t9p3g9Q5Wi7r8Ev 互联网档案馆的存檔,存档日期2013-12-21
全球合作方案仍在解决根源问题,但所有维持系统都已充分运作,我们继续进行监测.
这一事件已经得到解决.
自动翻译自官方事件更新。
We are currently investigating an issue with ticket delivery.
This incident has been resolved.
We are currently investigating this issue.
Ticket delivery has returned to normal.
We are investigating issues with Cloudflare, our global DNS provider.
Cloudflare appears to have resolved the issue, we are continuing to monitor.
We are continuing to monitor for any further issues.
Cloudflare is still resolving the issue and service remains intermittent. We are investigating workarounds. https://www.cloudflarestatus.com/
We are continuing to monitor and investigate workarounds.
Cloudflare is continuing to remediate the issue and we are investigating workarounds. Service remains intermittent based on your geographic location.
We've implemented a workaround which should restore service for most users. Full recovery is ongoing as Cloudflare continues to remediate on their end.
This issue appears resolved - please restart your Chrome Browser if you are still experiencing issues, or reach out to [email protected].
We've identified an issue that caused a slowdown in website performance and ticket syncing. The issue has been resolved and we are monitoring the system as it recovers.
We're still seeing slowdowns in app performance and are continuing to investigate
We're still seeing slowdowns in app performance and are continuing to investigate.
A fix has been implemented and we are monitoring the results. Thanks for your patience here!
This incident has been resolved - please contact [email protected] if you are still experiencing issues.
We've identified an issue that prevents the "lightning bolt" (Knowledge Base) button from appearing when replying to a Keeping ticket. This is affecting a small subset of users. For now, the workaround is to "pop out" the reply via the button in the upper right corner when composing a reply. This will restore the buttons until we resolve the issue.
The latest version of the Keeping Chrome Extension resolves this issue. Please update your Chrome Extension if you are experiencing this issue.
We are currently investigating issues with "Sign in with Google". Our initial investigation would indicate that there is a global outage for Google's OAuth platform.
We can confirm there there is a global outage of Google's OAuth service ("Sign in with Google"). We are monitoring, there is no current workaround.
Google appears to have resolved the issue. We are monitoring.
This incident has been resolved. We will continue to monitor for any regressions.
We are currently investigating sending errors related to our email sending provider, Mailgun. https://status.mailgun.com/
Mailgun has identified the issue and is working on a fix. In the meantime, we recommend responding to Keeping tickets in the Keeping app at https://app.keeping.com/inbox.
Mailgun has identified the issue and has implemented a fix. We are monitoring the results.
This incident has been resolved.
Google has made a change to how it filters the Gmail Inbox, preventing Keeping tickets from being displayed in some cases. We've pushed an update to the Chrome Webstore which resolves the issue - it is currently being reviewed. If you are affected by this issue, you may use the Keeping App to access your tickets, or contact [email protected] if you would like instructions to manually update your Chrome Extension. We expect the new Extension to be approved by the Webstore team in the next 48 hours.
The Keeping Chrome extension has been updated in the Chrome Webstore. The update should be pushed to you automatically, but you can uninstall and reinstall the Keeping Chrome extension to speed up the process. Please feel free to reach out to [email protected] if you are still experiencing issues after updating your Extension.
This incident has been resolved.
We are currently investigating performance issues with the Keeping Sidebar.
A fix has been implemented and we are now monitoring.
This incident has been resolved.
We are investigating issues with signing into the main Keeping application.
Due to a widespread Google Cloud outage, parts of Keeping are temporarily down. We're monitoring closely and will be back to normal as soon as it's resolved. Thank you for your patience!
Google has identified the root cause and is rolling out a fix. We are still experiencing issues with email syncing and login - we'll post another update as the situation develops.
Google has fixed the issue, and services are beginning to return to normal. We are continuing to monitor.
This incident has been resolved.
We are investigating an issue with our email backbone provider, MailGun. Incoming email is delayed.
MailGun is not accepting email to incoming.keeping.com. We are working with MailGun to resolve the issue.
The issue has been identified - we are waiting for MailGun to resolve the issue. Only mailboxes connected via forwarding are affected.
MailGun has resolved the issues and emails should be flowing again. We are monitoring the fix.
Starting at approximately 5:05 AM Eastern Daylight Time, our primary email backbone provider, MailGun, experienced an issue with the domain where we receive inbound emails on our customer's behalf. They resolved the issue at approximately 10:05 Eastern Daylight Time.
We are currently experiencing delays with email syncing - the team is investigating.
The issue has been identified, and mitigation is in progress.
The issue has been resolved - the team will continue to monitor.
This incident has been resolved.
Google is rolling out a new version of Gmail that may disrupt the opening of the right sidebar when viewing a Keeping ticket. A fix for this issue is included in version 6.6.3 of the Keeping Chrome extension, which has been submitted to the Chrome Web Store. The update will be automatically pushed to users once approved. For users affected by this issue, please access your tickets via https://app.keeping.com/inbox until the issue is resolved.
Version 6.6.3 of the Keeping Chrome Extension is now available in the Chrome Web Store and resolves this issue. If your extension hasn’t updated automatically, please follow these instructions to update it manually: https://docs.keeping.com/help/setup/manually-updating-the-extension
We are seeing elevated errors from the Gmail API when syncing a connected mailbox. We are investigating.
The API Keeping uses to connect to Gmail is still returning elevated errors. We are continuing to investigate the issue.
We've identified the issue and are working to mitigate.
We are continuing to work on a fix for this issue.
A fix has been implemented and we are monitoring the results.
We are continuing to monitor for any further issues.
This incident has been resolved.
The Keeping Chrome Extension and web application are experiencing connectivity issues due to an outage with our primary hosting provider. We've remediated the issue and are monitoring.
This incident has been resolved.
### Postmortem: Service Outage on July 16, 2024 **Incident Summary:** On 9:05 am EDT July 16, 2024, Keeping experienced intermittent connection issues that lasted approximately 2 hours. The outage affected both the Keeping Chrome Extension and the main application website at https://app.keeping.com/ **Timeline:** * **9:05 am EDT:** Initial reports of the website being inaccessible were received. * **9:07 am EDT** : Incident response team was alerted and began investigation. * **9:25 am EDT** : Root cause identified as a Denial of Service attack on Keeping’s hosting provider, Gigalixir. * **9:30 am EDT**: Mitigation steps initiated. * **10:25 am EDT**: Website functionality partially restored. * **11:00 am EDT**: Full service restored and confirmed stable. **Root Cause:** The outage was caused by [a SYN flood attack on Gigalixir’s load balancers](https://gigalixir.com/blog/gcp-us-central-service-outage/). Gigalixir is Keeping’s primary web hosting provider. **Impact:** During the outage, access to Keeping’s main web application was severely limited \(or impossible\), and syncing between agent accounts and shared mailboxes was paused. No data was lost.
Due to an outage with our primary hosting provider, the Keeping Sidebar and app experienced connectivity issues from 9:57 am EDT to 10:14 am EDT. This issue is now resoloved.
