Chef outage history

We are currently investigating an issue affecting Chef 360 services. Customers may experience difficulty accessing or using Chef 360 functionality. Our engineering and operations teams are actively working to restore service. We will provide additional updates as more information becomes available.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

The issue affecting Chef 360 services has been resolved, and normal service has been restored. We apologise for any inconvenience caused and appreciate your patience while our teams worked to resolve the issue.

We are currently experiencing an issue impacting user sign‑in to the Supermarket. Our team has identified the problem and are actively working to restore normal service as quickly as possible. At this time, some users may be unable to log in; however, existing login sessions will continue to function. We understand the inconvenience this may cause and appreciate your patience while we work on resolving it. We will continue to provide updates as more information becomes available.

The issue impacting user sign‑in to the Supermarket has now been resolved, and normal service has been fully restored. Users should now be able to log in without any issues. We sincerely apologise for the inconvenience caused and appreciate your patience while we worked to resolve this issue.

We are currently facing some technical issue with supermarket and we are investigating on it.

Issue with supermarket is resolved. Currently monitoring the systems.

We are continuing to monitor for any further issues.

This incident has been resolved.

We are currently facing some technical issue with supermarket. We are on it and we will keep this page updated

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

The issue has re-surfaced and we are looking into it

The issue with supermarket is fixed and we are monitoring the results.

The issue with supermarket is resolved

There are currently reports coming in from users about degraded performance with our CDN in some regions. Users may encounter timeouts when downloading binaries using the Chef download API endpoints. Current Actions: We are actively working with Fastly to identify the root cause and resolve this issue. We will provide updates as more information becomes available.

The status has been resolved.

After our maintenance window today, we experienced issues with our download endpoint due to an underlying infrastructure problem. The team has resolved the issue. We apologize for any inconvenience caused.

We have an isolated incident with our habitat database and we are investigating the issue. We will update when we are back online. Please check our Slack channel for updates. https://chefcommunity.slack.com/archives/CAW6DQBV2

Issue is resolved, all Habitat service is functional now.

There was an incident with Hosted Chef today morning at 6:00AM UTC, the Hosted Chef application (manage.chef.io) was down for 15 mins. We identified the root cause and fixed the issue immediately. The systems are now healthy and up and running.

We have identified that sign in for supermarket is not working. We are currently investigating the issue and we will keep you posted

We have identified and fixed the issue, sign in is working as expected now. We will continue to monitor the system.

We are seeing increased Error rates in Hosted Chef , this seems to be because of an on going AWS network issue in us-east-1 region. Our Engineers are working on this and we will keep you posted.

We have identified our internal Search Service to be the problem and this is because of on-going AWS us-east-1 incident. Our Engineers are working to restore the service with normal performance.

The issue is fixed and both Hosted Chef Web console and Hosted Chef API are performing normally now. We will continue to monitor for a while.

This Incident is resolved and services are back to normal. Thanks for your patience.

We are experiencing network performance degradation due to a vendor.

We have implemented a workaround which should alleviate timeouts and error rates. We are continuing to monitor the upstream vendor outage.

Error rates have improved for package distribution infrastructure. We will continue to monitor the upstream outage.

Vendor has resolved their incident.

We are facing issues with our Package Distribution Infrastructure. We are Working on identifying the issue and a resolution.

This incident has been resolved.

We are facing increased 5xx Error rate on Hosted Chef. We have identified the problem and are working to resolve the issue. We will keep you updated.

We have not been able to fix the issue yet and are continuing our work to resolve the issue. Thanks for being patient.

We have fixed the issue and Error rate is close to 0. We will continue to monitor for a while. Thank you for your patience.

We are currently investigating increased error rate in Hosted Chef. We will keep you posted.

The issue has been resolved. We were facing error with our indexing cluster and that lead to increased 5xx for the API calls.

We are investigating an issue with Hosted Chef - Manage UI functionality. Please note this does not affect the Hosted Chef API and Knife functionality is still available.

We have deployed a previous release of Manage to Hosted Chef to fix the UI bug until we can fix it in the latest version of manage.

After upgrading PostgreSQL we are seeing database connections and CPU spikes. We're resizing the database to get more system resources and will provide additional updates as we have them.

We're investigating an unexpected elevation in fetches by the authz service from the database.

We're isolating the issue with authz service's database queries that is taking an abnormally long time to complete.

We've implemented a short term workaround to restore service. We're monitoring the service.

Traffic patterns and service have normalized to regular levels observed prior to the maintenance window. We will conduct an incident analysis and write up a blog post for this next week. Thank you for your patience and I'm sorry that this impacted your workflows.

This incident has been resolved.

We are investigating increased error rates and timeouts with supermarket.chef.io

We're investigating what is causing the load balancer to remove hosts sporadically.

The load balancers removing hosts sporadically was a symptom. We're continuing to investigate what appears to be regularly occuring spikes in cache gets at the :00 and :30 minute marks each hour.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are investigating increased error rates for Hosted Chef services

Backend services have stabilized, engineers are monitoring

This incident has been resolved.

We are investigating problems in the Hosted Chef backend. Unfortunately, we expect chef-client communications to be severely impacted at this time. We will post further info regularly, as we find it.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

https://omnitruck.chef.io/install.sh serves up bootstrap scripts for use by unattended installs like autoscaling and `knife bootstrap ...` initiated bootstraps. We have become aware of an omnitruck.chef.io issue affecting downloads of the bootstrap install script from opscode.com/chef/install.sh as well. Please upgrade if running older chef-client/knife code that would point at opscode.com We expect the issue to have been cleared at 2019-10-11 8am Pacific US