Historia awarii Skylight

There is a data processing lag starting 15:08 UTC. The problem was initially caused by a hardware failure, which has since been addressed. We are currently investigating and addressing the secondary errors stemming from the initial hardware issue. Currently agent data are still processed albeit in a slower rate.

We are making progress processing the agent report backlog. The majority of the data has been backfilled, there are some short/intermittent data gaps that are expected to be filled in over the next few hours.

Thank you for your patience. Data processing has fully caught up. If you are experiencing issues still, please reach out to support.

We are currently investigating this issue.

This incident has been resolved.

This appears to be causing slow and sometimes failing requests when requesting data from the frontend

We have resolved the performance bottleneck on the data backend. We do not believe agent reporting were affected and no data was lost. If you are still experiencing issue, please reach out to support.

Due to a limited data corruption in the production database, a small portion of shared users may be unable to access the Skylight apps they were invited to. App owners are unaffected and can manually re-invite any affected users in the meantime. Agent authentication and data collection also remain unaffected. We are looking into the possibility restoring the affected access from database backups. We are very sorry for any inconvenience and appreciate your understanding.

Shared user access has been restored. Once again, we apologize for the inconvenience. If you are still having access issues, please reach out to support.

One of our processors is encountering errors while processing traces, we have taken the worker offline to work on a fix. This is affecting a portion of our customers – if your app is affected you may see missing data on the Skylight dashboard starting from around 20:50 UTC. We expect the missing data to be filled in once we resume processing.

We have deployed a fix and processing has resumed for the affected customers. It may take some time for the processing to catch up, data will start to appear in the order they were received.

Processing is fully caught up.

Problems with an upstream provider (Heroku) have taken down our website as well as agent authentication. It does not appear to affect all customers uniformly. We are still able to resolve and connect to the site from some locations as of now. The Skylight collector is still operational and processing data from agents that were already authenticated. We are monitoring the situation. Upstream incident: https://status.heroku.com/incidents/2454

Heroku appears to have resolved the issue, and we have verified that agent authentication has returned to normal. We don't expect significant data loss. That said, if you're not seeing data on your dashboard, we recommend that you restart your app. This should update your authentication token and get things back to normal. If you still don't see data after an app restart, please reach out to us using in-app support.

Problems with an upstream provider (Heroku) have taken down our website as well as agent authentication. The Skylight collector is still operational and processing data from agents that were already authenticated. We are monitoring the situation. Upstream incident: https://status.heroku.com/incidents/2453

Heroku appears to have resolved the issue, and we have verified that agent authentication has returned to normal. We don't expect significant data loss. That said, if you're not seeing data on your dashboard, we recommend that you restart your app. This should update your authentication token and get things back to normal. If you still don't see data after an app restart, please reach out to us using in-app support.

This incident has been resolved.

We are investigating a "clog" in the data processing pipeline that is causing one of the worker servers to be "stuck". This only affect a portion of the customers. If you are affected, you will see missing data from the Skylight dashboard for the last few hours (which is slowly populating). At this stage, we believe the data is safely received in the queue for processing, and once we resolve the issue that causes the "clog" we will be able to backfill the missing data.

We have "unclogged" the worker and are "catching up" on the backlog.

We have completed processing of the backlog.

Our automation have failed to renew/replace the SSL certificates before they expired. We are currently replacing the certificates manually. In the meantime, both the Skylight dashboard and the data collecting servers (used by the agents to submit traces) are inaccessible. We are sorry for the inconvenience.

We have deployed new certificates. All issues should be resolved now, although it may take a while for the agents to notice they are able to reattempt the authentication process. This process can be sped up (or in some case necessary if the agent had given up) by restarting the Rails app, which resets the agent. Please reach out to support if you need additional assistance. As part of the response to the Heroku security incident earlier this year (https://status.heroku.com/incidents/2413), a token that was used to upload certificates was invalidated without being replaced. Since the renewal script runs rarely the problem was not noticed until today. We had previously set up secondary monitoring to alert us when a certificate is nearing its expiration date (i.e. the automation did not work as expected), but it appears that secondary monitoring had also failed for a different reason. We are very sorry for the inconvenience.

We believe all the associated issues with the certificate expiration has been resolved. If you need further assistance, please reach out to support. Sorry again for the inconvenience.

A subset of our customers are experiencing a delay in data processing. We've identified the cause and are working on a fix.

We have deployed a fix and are monitoring while the lag resolves.

We've successfully reduced the lag.

The Skylight dashboard is inaccessible currently due to a configuration issue. This outage also impacted agent authentication – new authentications from agents will not succeed at the moment. Agents that are already authenticated can continue to report data until the authentication session expires. The data processing pipeline is technically unaffected by this outage as it is hosted on a different provider. However, given that agents are failing to authenticate (and therefore failing to submit traces), we expect this to cause lapses in Skylight data during the outage period.

We have manually uploaded a new cert, have completed the migration and are monitoring the situation. The Skylight dashboard (skylight.io) should be immediately accessible. If you are still unable to access the site, please email [email protected] for assistance. Your Skylight agents should resume reporting data once it retries the previously failed authentication request. If this does not occur, you can try restarting your app, which would force the agent to restart and authenticate again. If that still doesn't work, please email [email protected] for further help. We are very sorry for the trouble.

Our metrics indicate that the agent report rate has recovered to the level before the incident. We believe most customer agents have resumed normal reporting and the issue has been resolved. If you continue to encounter issues, please email [email protected] for assistance. Unfortunately, if your agent was "locked out" from an expired authentication and was unable to report data during the outage, that unreported data will not be available to view on the dashboard. We are truly sorry about this.

The Skylight dashboard is inaccessible currently due to a potential configuration issue. This outage also impacted agent authentication – new authentications from agents will not succeed at the moment. Agents that are already authenticated can continue to report data until the authentication session expires.

The dashboard and agent authentication endpoints are affected by a service degradation on our hosting provider Heroku. This outage impacts their "SSL Endpoint" add-on and is expected to last for 8 hours. We began to process to migrate away from the add-on but this is normally expected to take up to 24 hours. We are investigating if there is any way we can speed up the process or re-route the affected endpoints. The data processing pipeline is technically unaffected by this outage as it is hosted on a different provider. However, given that agents are failing to authenticate (and therefore failing to submit traces), we expect this to cause lapses in Skylight data during the outage period. We are very sorry for the inconveniences.

We have completed the migration and monitoring the situation. Due to the nature of the incident and it requiring updating our DNS records, it may take some time to fully resolve. The Skylight dashboard (skylight.io) should be immediately accessible assuming your operating system has refreshed the DNS record, which should happen within minutes as we have a TTL of 300 seconds. If you are still unable to access the site, please email [email protected] for assistance. Your Skylight agents should resume reporting data once it retries the previously failed authentication request. If this does not occur, you can try restarting your app, which would force the agent to restart and authenticate again. If that still doesn't work, please email [email protected] for further help. Once again we are very sorry for the trouble.

Our metrics indicates an agent report rate have recovered to the level before the incident. We believe most customer agents have resumed normal reporting and the issue has been resolved. If you continue to encounter issues, please email [email protected] for assistance. Unfortunately, if your agent was "locked out" from an expired authentication and was unable to report data during the outage, those unreported data will not be available for view on the dashboard. We are truly sorry about this.

We are currently investigating this issue.

We believe that the issue has been resolved and that data will process going forwards.

Due to an AWS outage we were unable to process data between between approximately 07:15 and 08:15 UTC. We are fully recovered and anticipate no further issues. We apologize for any trouble.

A subset of our customers are experiencing a delay in data processing. We're investigating the cause.

We believe the issue is resolved. We are continuing to monitor while we catch up with processing.

This incident has been resolved.

We're experiencing a backlog in data processing for some apps. We believe this issue is related to a bad data entry in our queue. We're working to remove this entry and continue processing the remaining backlog.

We have removed the problematic data form the queue and started catching up on the backlog.

We are fully caught up.

We're experiencing a backlog in some of our data processing. We currently anticipate a full recovery once we've identified the underlying issue.

We believe this issue is related to a bad data entry in our queue. We're working to remove this entry and continue processing the remaining backlog.

We have resolved the issue and data processing has resumed. We will monitor the progress as it catches up.

We have caught up and are processing as normal.

This incident has been resolved.

We are currently investigating a delay in processing data for some customers. No data loss is anticipated.

We believe we have identified the underlying cause and have put a temporary fix in place to resume processing data.

Data is processing again. We will continue to keep an eye on the situation.

We've caught up with processing.

We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

We are back to processing data at a satisfactory rate. The majority of our customers will see their data in Skylight with less than 10 minutes of processing delay. We expect the delay to reduce further and will continue to monitor the situation and deploy additional capacity as needed.

This incident has been resolved.

We are currently investigating a delay in data processing.

We have identified and correct the root cause of the issue. Unfortunately at this time we do not believe we will be able to process data that was reported during the outage.

We have resolved the issue and are processing data again.

We are experiencing a major network outage on AWS affecting most of our EC2 instances. Due to the scale of this outage, we are expecting connectivity issues from agent and potential loss of report data submitted during this time.

The network issue has been resolved. We have determined that the data sent during the outage period is not recoverable, and have proceeded to unblock processing new data instead.

We have resumed processing data on all workers.

This incident has been resolved.