Gitguardian 장애 기록

This incident has been resolved by Auth0 at 23:53 UTC yesterday. More infos here https://status.auth0.com/incidents/zvjzyc7912g5?u=v0zzz6jxvbv7

More info here https://status.auth0.com/incidents/zvjzyc7912g5?u=v0zzz6jxvbv7

Due to an Auth0 ongoing incident, users can't log in to our public monitoring app.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

We are continuing to work on a fix for this issue.

This incident has been resolved.

Due to a deployment incident, Public Monitoring's Data collector was not operational this morning between 8:00 AM UTC and 8:12 AM UTC.

Maintenance mode was removed and dashboard & workers were restored at 14:56 UTC. No incoming events were lost. A bug in the maintenance script maintained a lock on the release process. We had to manually remove the lock. A bug fix is in progress and will be deployed on the 20th.

Issue has been identified. A fix is in progress

During an application release the application frontend and workers were stuck in Maintenance mode for longer than expected.

OVH has resolved the routing issue. No data was lost, secret detection worked as usual. Only the "Explore" feature was unavailable.

The Explore feature is down on all Enterprise environments. OVH is working on recovering network connectivity.

We have an ongoing incident that may affect our Public Monitoring application. It is due to an OVH datacenter maintenance: https://twitter.com/ovh_status/status/1448185498812485633 We are investigating the impact.

This incident has been resolved. All the GitHub commits, except those that may have been deleted by their authors, have been recovered. Incident's impact has been lowered to "Major".

We are currently trying to recover missed GitHub events and backfill the Public Monitoring.

GitHub Public Monitoring is fully operational again. There was a lack of commit coverage between 7:28 and 7:37 UTC. We are analysing the issue and working on a fix.

The issue has been identified and a fix provided at 7:39 UTC

We received an alarm on GitHub's commit coverage at 7:28 UTC. We are currently investigating

Dear customers, At 14:30 UTC, GitGuardian Internal Monitoring's API suffered from increased latency. The API response time was fully back to normal at 15:00 UTC and the service remained available all the time. We apologise for this disturbance which was due to a database maintenance.

Our cloud provider is currently having issues. We are investigating the situation.

Our services were unavailable from 16h09 to 17h43 CET. The situation is now resolved.

We are continuing to investigate this issue.

GitHub API is back online, data collections is slowing recovering.

GitHub API is fully functional. Public GitHub data collection is up again.

Data collection is recovering

GitHub has an issue with their API and are investigating: https://www.githubstatus.com/incidents/r5qrpp2f5fc0?utm_ts=1638045817

The release process is finished.

We are working on a fix.

The GitGuardian Internal Monitoring release in progress is taking longer than expected.

This incident has been resolved.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

An incident was detected during a scheduled maintenance. Our teams are investigating.

This incident has been resolved.

A fix has been implemented. We are monitoring the situation.

The issue has been identified and a fix is being implemented.

Commits and patches are now indexed in real-time again.

Search was unavailable from Dec. 29 from 11:30 AM CET to Dec 30. 11:30 AM CET. Latest commits and patches are now being indexed again.

The incident has been resolved.

We have detected a spike in errors on some endpoints of GitGuardian Internal Monitoring API. Our teams are investigating.

This incident has been resolved.

On October 17th 2022, at around 5pm GMT, The queuing system in charge of handling Github Check Runs and Real Time Monitoring accumulated tasks that were slow to process and impacted the whole system. The size of the task queue started increasing for several hours straight, delaying processing for all customers. until it started decreasing again and went back to normal the day after at around 1am GMT, where processing time was back to normal. This event occurred again on October 18th 2022, at around 4pm GMT, but has been detected earlier and have been solved at 5pm GMT. Actions were taken to avoid this issue to happen again: It includes a better monitoring of those queues to improve time to detect an issue. The montoring of processing latencies on a end-user point of view will be improved too. More long terms actions are also planned, including implementing mechanisms to avoid accumulating latency in case of slow tasks accumulations.

We are continuing to work on a fix for this issue.

From Friday 9th Sept 6:30pm UTC to Monday 12th Sept 7:30 UTC, data collectors services were partially down due to an internal error. Github events have been correctly gathered into GitGuardian platform but processing of those events was down. There is no data loss but no alerts have been sent during this period of time. The processing has resumed correctly on Monday 12th Sept 7:30 UTC, the backdata from the week-end is being analyzed and should be caught up around 12pm. Update Monday 12th Sept. 10:25am UTC: Queues processing is slower than expected, ETA for catch up with the lag is postponed to at least to Tuesday morning (UTC), more precise ETA will be communicated when available. Update Monday 12th Sept. 2:05pm UTC: Slowness issues have been solved, queues processing has resumed at normal speed, ETA for catch up with the lag is Monday 12th Sept. 5:00pm UTC

Start of the issue.

A fix has been implemented and the systems are getting back to normal progressively.

This incident has been resolved.

We have an issue with our database. We are still evaluating the impact.

The incident has been resolved at 9h10 CEST.

Following a DNS misconfiguration, GitGuardian Internal Monitoring platform was unavailable for about 20 minutes. Impact: No realtime scan (webhook): - GitHub events from recent GHE instances and Github.com were redelivered after the incident - Other events (BitBucket, GitLab) have been lost but we fully rescanned repositories when possible No API for secrets detection: CI and pre-commit hook failed. Dashboard was unavailable. Impacted entries: - api.gitguardian.com - dashboard.gitguardian.com - hook.gitguardian.com

Since October 17th, 2022, 12:28am (CEST) we are sending notifications by email for certain incidents already present in your workspace. We are currently investigating this incident.

This incident has been resolved.