Codetree 장애 기록

Web service became unstable and no longer accessible

The issue has been identified and currently being fixed

AWS Fargate changed the way containers share volumes within a task. This change in behavior caused a deployment failure along with failures on rollback. The solution required redesigning the container infrastructure to not require shared volumes

Search and Indexing services are degraded and we are working to fix this issue. When data is saved / updated, you may receive an error, but in most cases, all data is persisted (but not indexed).

A fix has been promoted and looking to see if performance improves

resolved

We are currently investigating slow performance with Codetree site.

This incident has been resolved.

The issue with burndown activities data has been identified and is being resolved

this incident is resolved and ongoing monitoring is in place

We are currently investigating a degradation of performance and partial outage today.

This incident has been resolved; We'll continue to monitor to ensure all clients are back to normal

Github authentication and API services are degraded due to Github itself currently experiencing issues

continuing to monitor Github for a resolution to their performance issues: https://www.githubstatus.com/incidents/4sh7f76nh5gq

Github resolved their outage as of 5:01 UTC

Some community members are having difficulty opening some issues due to a rendering error

We have identified the problem with a recent release and are working on a fix.. We hope to have something within the next hour or so

A fix has been implemented and we are monitoring results

This incident has been resolved.

We are currently experiencing issues with our caching and client side notifications, overall degrading the experience. We have identified a couple of the issues and working to get fixes in place as soon as possible

We are continuing to work on a fix for this issue.

This incident has been resolved.

We are currently experiencing some issues with search; The search indices are being rebuilt

The issue has been identified and the index is being rebuilt

The index issues have been resolved

GitHub API and other services are currently degraded and unreliable for usage

GitHub seems to be running smoothly again

There seems to be an issue with a recent change to the issues view for a handful of accounts. If you run into this issue, you should still be able to access your project view the board view. https://codetree.com/projects/:project_id/board

We are continuing to work on a fix for this issue.

fix has been pushed and issue has been resolved

A caching server failed leading a small portion of users to not be able to access the site. We are restoring service now

Caching services restored and full connectivity is available again

This incident has been resolved.

We are currently investigating this issue

Search Indexing server failed due to an AWS us-east event today. New Search Indexing instance is online and indexes are currently being rebuilt

Search Indexes have all been refreshed and operations are back to normal

We are currently investigating intermittent 500 gateway issues

This incident has been resolved.

We are currently experiencing intermittent access to website. investigation ongoing

We are continuing to investigate this issue.

resolved connectivity failure

connectivity has been restored; access will continued to be monitored for the next couple of hours

We are experiencing some delays syncing issues from GitHub. We are investigating the cause of the issue.

We're back up and issues are syncing from GitHub normally again. You may see a slight delay as we process the backlog.

GitHub is experiencing an API outage that is causing occasional errors with Codetree. Until GitHub resolves their outage Codetree service won't be 100%.

GitHub has "addressed the source of the exceptions and are continuing to monitor as the site recovers." At Codetree we're doing the same.

GitHub is experiencing API slowness which is affecting Codetree performance.

GitHub API slowdown has been resolved.

We are experiencing an outage right now. We're investigating and will update as soon as we know more.

The issue has been caused by an outage with the company hosting our Redis instance. We've contacted support and will update when we know more.

The outage has been resolved and we're back up.

We've identified the cause and are working towards a resolution.

The incident has been resolved and all systems are back online.