30 Gitguardian incidents · marzo de 2021 — official updates, affected components, duration and resolution details.
Status: Monitoring Service has returned to normal levels and we're no longer seeing degradation. Our engineering team has applied mitigations and continues to investigate the underlying cause to prevent recurrence. We're closely monitoring the situation and will update this page as needed. Affected components Secret Detection - Scan Check Run (Operational)
Status: Resolved GitHub has restored their API, and our public ingestion pipeline has fully recovered. Real-time event ingestion and commit retrieval are operating normally again. This incident was caused by an upstream GitHub API outage and is now resolved. Thank you for your patience. Affected components Data Collectors (Operational)
Traducido automáticamente desde la actualización oficial del incidente.
Status: Resolved The fix has successfully been deployed and the situation should be back to normal. Sorry for the inconvenience. Affected components Secret Detection - Scan Check Run (Operational)
Traducido automáticamente desde la actualización oficial del incidente.
Status: Resolved What happened:From June 1 12:00am UTC to June 1 08:45am UTC, one of the component serving the public monitoring experienced some delays and a growing backlog of unprocessed messages. Impact:No data was lost. The only impact for customers could have been a delay between the detection of a secret and the moment the corresponding incident was created in the customer's dashboard. The problem is now fixed, the faulty component was able to re-start and to process the backlog of messages.
Traducido automáticamente desde la actualización oficial del incidente.
Estado: Resuelvado Hemos implementado una solución hoy que abordaba los problemas de la carrera de cheques. Se observó un breve aumento de la carga de la base de datos después del despliegue sin ningún impacto en los resultados de la prueba. Este incidente ya está resuelto. Componentes afectados Secret Detection - Scan Check Run (Operational)
Traducido automáticamente desde la actualización oficial del incidente.
Due to an availability issue of github.com Actions APIs (https://www.githubstatus.com/incidents/z3c6q056q332), some check runs have timed out on our side, resulting in a neutral status.
We are currently experiencing an incident on GPUB that started around yesterday, 10PM Monday 13th of March. An unexpected activity led to an overload of the queuing system of GPUB. We are deploying a fix to resume processing and catch up with the lag. In terms of data, some events will need to be reprocessed afterward, and we need to assess if some events cannot be processed again.
This incident has been resolved.
The fix has been deployed, the processing has resumed at full speed and the awaiting queue will be processing within the next hour.
Following a upgrade operation of a network component, GIM endpoints have been unavailable most of the time during this incident between 09:48 and 10:24 UTC. Fix has been deployed, situation is still under monitoring.
During deployment of GIM new version, The app was malfunctioning for a few minutes. The events that failed to evaluate during the incidents have been processed again.
We have identified a Network issue on Internal Repositories Monitoring that impacted Hook, Dashboard and API.
Following a DNS misconfiguration, GitGuardian Internal Monitoring platform was unavailable for about 20 minutes. Impact: No realtime scan (webhook): - GitHub events from recent GHE instances and Github.com were redelivered after the incident - Other events (BitBucket, GitLab) have been lost but we fully rescanned repositories when possible No API for secrets detection: CI and pre-commit hook failed. Dashboard was unavailable. Impacted entries: - api.gitguardian.com - dashboard.gitguardian.com - hook.gitguardian.com
This incident has been resolved.
Since October 17th, 2022, 12:28am (CEST) we are sending notifications by email for certain incidents already present in your workspace. We are currently investigating this incident.
On October 17th 2022, at around 5pm GMT, The queuing system in charge of handling Github Check Runs and Real Time Monitoring accumulated tasks that were slow to process and impacted the whole system. The size of the task queue started increasing for several hours straight, delaying processing for all customers. until it started decreasing again and went back to normal the day after at around 1am GMT, where processing time was back to normal. This event occurred again on October 18th 2022, at around 4pm GMT, but has been detected earlier and have been solved at 5pm GMT. Actions were taken to avoid this issue to happen again: It includes a better monitoring of those queues to improve time to detect an issue. The montoring of processing latencies on a end-user point of view will be improved too. More long terms actions are also planned, including implementing mechanisms to avoid accumulating latency in case of slow tasks accumulations.
We are continuing to work on a fix for this issue.
This incident has been resolved.
Desde el viernes 9 de septiembre de 6:30pm UTC hasta el lunes 12 de septiembre de 7:30 UTC, los servicios de recopiladores de datos fueron parcialmente reducidos debido a un error interno. Los eventos de Github han sido recogidos correctamente en la plataforma GitGuardian pero el procesamiento de esos eventos fue bajado. No hay pérdida de datos, pero no se han enviado alertas durante este período. El procesamiento se ha reanudado correctamente el lunes 12 de septiembre de 7:30 UTC, se está analizando el backdata del fin de semana y se debe capturar alrededor de las 12pm. Actualización Lunes 12 de Septiembre 10:25am UTC: El procesamiento de las colas es más lento de lo esperado, el ETA para ponerse al día con el lag se pospone al menos hasta la mañana del martes (UTC), se comunicará el ETA más preciso cuando esté disponible. Actualizar Lunes 12 de Septiembre 2:05pm UTC: Problemas de lentitud se han resuelto, el procesamiento de colas se ha reanudado a la velocidad normal, ETA para ponerse al día con el lag es Lunes 12 de Septiembre 5:00pm UTC
Traducido automáticamente desde la actualización oficial del incidente.
This incident has been resolved.
A fix has been implemented and the systems are getting back to normal progressively.
Start of the issue.
We have an issue with our database. We are still evaluating the impact.
The incident has been resolved at 9h10 CEST.
We have detected a spike in errors on some endpoints of GitGuardian Internal Monitoring API. Our teams are investigating.
The incident has been resolved.
An incident was detected during a scheduled maintenance. Our teams are investigating.
This incident has been resolved.
A fix has been implemented. We are monitoring the situation.
The issue has been identified and a fix is being implemented.
Commits and patches are now indexed in real-time again.
Search was unavailable from Dec. 29 from 11:30 AM CET to Dec 30. 11:30 AM CET. Latest commits and patches are now being indexed again.
Our services were unavailable from 16h09 to 17h43 CET. The situation is now resolved.
We are continuing to investigate this issue.
Our cloud provider is currently having issues. We are investigating the situation.