52 Logrocket incidents · dicembre 2020 — official updates, affected components, duration and resolution details.
Tra le 20:50 e le 21:10 America / New York il 21 luglio, ci sono stati blip di disponibilità durante un aggiornamento del nostro strato proxy nginx. I client SDK si saranno ritirati durante questo evento e ci siamo recuperati da un backlog entro le 21:30.
Tradotto automaticamente dall'aggiornamento ufficiale dell'incidente.
We are investigating elevated queue backlogs for our analytics ingestion. Analytics and alerts may be delayed.
Latency is returning to normal. We are waiting on our remaining queues to drain.
This incident has been resolved.
We're investigating slow and failing dashboard queries.
We've identified the problem and are rolling out a fix.
This incident has been resolved.
An outage from our upstream CDN provider is causing instability for our dashboard. Data collection is not impacted.
Error rates are improving, we're continuing to monitor stability of the system.
This incident has been resolved.
We are investigating instability in our ingestion endpoints.
This incident is fully resolved.
We are investigating performance issues with our API server causing dashboard instability. Data collection is not impacted.
We have identified the cause of the issue and are deploying a mitigation.
The performance issue with our API server has been resolved.
We are currently investigating this issue.
We are continuing to investigate the issue and are working with our vendor to restore service.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are investigating a wide-spread issue with our cloud provider causing ingestion issues and limited access to the dashboard.
GCP is suffering a service outage. Google has opened an incident tracking this issue at https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW#2c2sBHWU84yPDJ8y1ar4
The upstream issue has been resolved. We are monitoring and beginning to work through the backlog of session data.
We are continuing to monitor for any further issues.
This incident has been resolved.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are investigating issues causing ingestion delays. We're working with a vendor to identify the root cause of the problem
We are continuing to work with a vendor on identifying the root cause of our ingestion delays.
We've applied a fix and are monitoring the results
This incident has been resolved.
We are investigating issues causing ingestion delays and performance issues loading the dashboard and metrics.
Dashboard performance should be improved. We are continuing to work with a vendor on identifying the root cause of our ingestion delays.
We have identified a potential cause and observing significant improvements to ingestion delays. We are continuing to monitor.
This incident has been resolved.
The Streaming Data Export service is delayed. We've identified the issue and are working to remediate.
We are continuing to investigate this issue.
We've resolved the issue for most Streaming Data Export destinations. A few customers have been temporarily disabled until another fix goes out in the morning, at which time they will be caught up and we will resolve the incident.
We are continuing to work on a fix for this issue.
This incident has been resolved.
We are seeing significant infrastructure instability from our hosting provider.
The system has stabilized and our processing backlog is recovering.
We are investigating instability in our ingestion endpoints.
Ingestion has stabilized, we are continuing to monitor the recovery.
We are continuing to work down a backlog of ingestion from the initial outage, but have otherwise recovered from the outage.
The backlog has fully recovered.
We've identified an issue with our analytics database resulting in delayed ingestion and degraded alerting performance. The problem has been identified and we're working on a solution.
A fix has been implemented and ingestion is recovering
This incident has been resolved
We're seeing elevated error rates loading the UI.
This incident has been resolved.
Our Streaming Data Export system is not currently exporting data, we are working to resolve this. When this is reenabled it will catch up on data that would have been exported earlier.
The most recent export window ran successfully but we're continuing to monitor it for issues.
We've monitored several exports and believe this is resolved.
This incident has been resolved.
We are investigating an issue with slow and/or unresponsive application load times.
We've identified the source of the instability and are working through recovery.
This incident has been resolved.
We are investigating an issue with one of our search databases. Ingestion into this system is currently delayed, and loading data on the dashboard may be slow or timeout. Session Data collection is not impacted.
We are continuing to investigate this issue.
Our vendor has identified a root cause and we are working to restore the search database to full performance.
System performance has been restored. We are continuing to monitor as we work through the remaining backlog of events to process.
We are continuing to monitor for any further issues.
This incident has been resolved.
We are investigating an issue with our analytics data stores. Dashboard performance and data freshness may be degraded.
We've stabilized our infrastructure and are beginning to burn down our queues.
This incident has been resolved.