Keypup 장애 기록

Google Cloud Platform is currently having issues with services related to background jobs, on which Keypup relies. This results in an elevated error rate in the data processing jobs, with a high number of jobs failing or being retried. We are monitoring the situation and will reprocess application updates since Thursday once the incident is over on the GCP side.

Data updates have been paused while we are processing a large backlog of jobs. Job processing is slowed down due to continued issues on Google Cloud Platform.

We have re-enabled data updates from third-party applications and are continuing to clear the backlog of jobs. On the reporting side, the loading of insights may feel slower due to the following: - The auto-refresh mechanism of dashboard insights is currently disabled. Insights will only be refreshed upon reloading the page (or visiting another dashboard, then coming back) - The ahead-of-time caching of dashboard insights is currently disabled. Insights are live-loaded if data has changed since the last load.

We continue the process the pending backlog and will provide another update in the next few hours.

The processing of the remaining backlog is almost over. Once processed, the team will trigger a job to reprocess the application updates for the last few days to ensure no updates were missed.

The processing backlog has been cleared, and our team is now reprocessing the last few days of updates to ensure no data updates were missed.

The application updates have been reprocessed, and the platform has resumed normal operations.

The team identified a rogue process on one of our databases tampering with index performance and causing delays in our processing pipeline. The team is currently taking actions to fix this and restore normal service. This action will involve a component restart, which means some platform components will be briefly unavailable.

The team has performed maintenance activities on the database to restore performance. The system is currently catching up on updates from the last two hours.

This incident has been resolved.

Intermittent traffic issues are preventing users from accessing the platform. This is due to a global outage on the Clouflare side. Cloudflare incident status page: https://www.cloudflarestatus.com/

Cloudflare has implemented a fix and is currently monitoring traffic through their network.

This incident has been resolved.

The Jira API is currently in the middle of an incident, returning an empty list of issues on the /search API endpoint. Other API endpoints are fine and returning data. This problem occurs for most customers but not all. We believe this issue started occurring on September 4th around 4 pm UTC, as Atlassian deployed a fix for another incident occuring the same day. Jira-related reports are still available on Keypup. However, we are currently not receiving updates for Jira issues. Jira-related reports will show information as of September 4th. We have reached out to the Atlassian team, provided a full incident report, and are waiting for next steps.

The issue retrieval flow has resumed normal operations. We are currently monitoring the logs to ensure proper activity. A resync of Jira projects will take place during the weekend to ensure the data are up to date.

This incident has been resolved.

We're currently investigating an elevated load on our databases related to third-party data processing.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently observing an elevated number of jobs in our queues related to updates on cross entities (e.g. PR attributes on a commit record). Our team is currently investigating this performance issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

The Google Cloud Tasks is frequently returning permission errors without apparent reasons. This is degrading the efficient of our background jobs and impacting the data connectivity of apps and projects. Our team is currently investigating.

Google Cloud seems to have resolved the issue. The errors have stopped for the last 10 minutes. We continue to monitor the logs.

This incident has been resolved.

We're observing degraded performance on our reporting engine and data processing platform. Our team is currently looking into it.

We are continuing to investigate this issue to find the bottleneck. We already cleared a few blocking processes on our reporting database and are continuing to dig down to find the root cause. Data updates from third-party apps are currently on hold.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating a slowdown in our data processing pipeline and our team is looking into it. Updates from third-party applications will be delayed in the meantime.

The issue is related to a set of recently introduced database indexes, which got corrupted during creation. Those indexes are currently being rebuilt.

The indexes have been rebuilt. We are now starting to process the backlog of updates.

The backlog has been processed and updates from third-party apps now flow as expected. We continue to monitor the activity.

This incident has been resolved.

We are currently investigating a high volume of processing activity on our data management platform. Updates from third-party apps are currently paused while we investigate this issue.

We have identified the issue and are currently working on a patch.

We are progressively resuming data updates across all our app integrations.

Our data processing platform has resumed normal activity.

We are currently experiencing delays in processing data from third-party applications. The import of data has currently been paused while investigations are in progress.

We are progressively reenabling the import of data from third-party applications.

The import of data from third-party applications has been fully re-enabled.

This incident has been resolved.

We are currently investigating a Jira issue where tickets from other Jira projects (not selected) may appear under selected projects on Keypup.

We are continuing the investigations. The issue is not systematic. It is currently unclear if the issue comes from the Keypup side or from the Jira side (= incorrect filtering based on project ID when retrieving issues on the API via JQL)

We have identified the source of the issue. Some logic in our webhook processor was leading to the retrieval of arbitrary issues by ID under each Jira instance (across projects for each instance). A fix is going to be developed and deployed to correct this issue. Once deployed we will start cleaning up the datasets to remove any unwanted issue in each Jira project on the Keypup side.

A fix has been developed and is currently being tested by our team.

A fix has been deployed to production. We have now started cleaning up the Jira projects on the Keypup side to remove unwanted issues.

All Jira projects have been cleaned up from unwanted issues. We are now resyncing all Jira projects from their respective Jira instance.

All Jira projects have been resynced. We are now doing some additional monitoring before closing the incident.

This incident has been resolved.

We're currently seeing a high number of queries being canceled and returning no data. Our team is investigating the issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We've noticed that insights and dashboards take a significant amount of time to load. Our team is currently investigating.

The team is still investigating the issue. We have identified some areas that may be creating excessive load on our data processing pipeline.

The investigation is still in progress. The team is refining its investigations around specific areas of our data processing pipeline.

A first fix has been applied to reduce the processing time of records. A second fix is currently in testing.

The second fix has been released. The overall performance of our data processing pipeline is now back to normal. We are currently processing the backlog of updates for the day.

Backlog of data updates has been processed. However we are still investigating slow queries for insights.

We've proceeded with an upgrade of our reporting datastore. We will be monitoring the performance of reporting queries over the next 12 hours.

Our team has identified an area of improvement in our reporting service. We will test a fix today and deploy based on performance results.

A fix for the slow queries has been developed by our data team and is currently being tested in our staging environment.

The fix for our slow queries has been released to production and our data team is currently monitoring the results.

Incident resolved.

We are receiving an elevated rate of errors related to our cache instances. We are currently investigating.

We have identified the issue and are currently deploying a fix.

The configuration of one of our cache nodes has been updated. Service has been restored and we are now monitoring the services internally.

We are continuing to monitor for any further issues.

This incident has been resolved.