Ιστορικό διακοπών του Netdata

1. 23 Οκτωβρίου 2026 στις 12:34 μ.μ. UTCinvestigating

   Μια ευπάθεια στο Netdata Agent αποκαλύφθηκε δημοσίως. Ένας επιτιθέμενος με πρόσβαση στην τοπική θύρα ιστού (19999) ή τον χώρο Νέκτατα Cloud που ισχυρίζεται ο Πράκτορας, μπορεί να εκτελέσει αυθαίρετες εντολές ως χρήστης `netdata'. Οι πλήρεις λεπτομέρειες της ευπάθειας είναι στην αποποίηση εδώ: https://codeberg.org/x509dot/vulns/src/branch/main/netdata-rce.md. Έχουμε διορθώσει το θέμα και έχουμε δημιουργήσει ένα νέο nightly/edge build του Agent (https://github.com/netdata/netdata- nightlies/releases/tag/v2.12.0-267-nightly) και ένα νέο patch release v2.12.2 για το σταθερό κανάλι μας θα πρέπει να δημοσιευθεί σύντομα. Θα ενημερώσουμε αυτό το περιστατικό όταν η κατασκευή έχει δημοσιευθεί. Μετριασμός: ο περιορισμός της πρόσβασης στην διαδικτυακή θύρα του Πράκτορα μειώνει τη δυνατότητα ενός επιτιθέμενου να εκμεταλλευτεί αυτή την ευπάθεια. Οδηγίες για την εξασφάλιση των Agents μπορείτε να βρείτε εδώ: https://learn.netdata.cloud/docs/netdata-agent/configuration/securing-agents, με serveral προσεγγίσεις συμπεριλαμβανομένης της σύνδεσης της θύρας με το localhost, που απαιτούν κουπόνια κομιστή, ή απενεργοποίηση της θύρας ιστού εντελώς. Οι Parent Agents, ωστόσο, απαιτούν από τη θύρα τους να παραμένει διαθέσιμη στους Child Agents για streaming. Οι λίστες πρόσβασης μπορούν να χρησιμοποιηθούν για τον περιορισμό της πρόσβασης: https://learn.netdata.cloud/docs/netdata-agent/configuration/securing-agents/web-server-reference#access-lists Στοιχεία συμβιβασμού μπορούν να βρεθούν με αυτό το σενάριο: https://gist.github.com/ktsaou/b3fcf41c8ce0a01effb6df9d82a14858

2. 23 Οκτωβρίου 2026 στις 12:35 μ.μ. UTCidentified

   Το θέμα έχει εντοπιστεί και υλοποιείται μια λύση.

3. 23 Οκτωβρίου 2026 στις 1:34 μ.μ. UTCmonitoring

   Η έκδοση 2.12.2 patch release έχει δημοσιευθεί στο σταθερό κανάλι απελευθέρωσης με ένα fix για το τεύχος.

Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

1. 30 Μαΐου 2026 στις 11:21 π.μ. UTCresolved

   Following a scheduled TLS certificate renewal, a subset of Netdata agents failed to reconnect to Netdata Cloud. The issue stems from outdated CA (Certificate Authority) trust stores on certain legacy Linux distributions, like: • CentOS 7 / RHEL 7 • Amazon Linux 2 • Debian 9 (Jessie) • Ubuntu 16.04 LTS • Oracle Linux 7 A quick way to test for this: curl -v https://app.netdata.cloud If the output shows an error indicating the certificate has expired, the system needs an updated CA bundle.

1. 8 Μαΐου 2026 στις 1:13 π.μ. UTCmonitoring

   We are currently tracking an issue with our upstream cloud provider that is affecting a subset of our Kubernetes nodes. As a result, some customers may receive false-positive alerts regarding node reachability. Please be assured that your underlying services and workloads remain operational, and these alerts can be safely ignored at this time. We are closely monitoring the situation with our provider.

2. 8 Μαΐου 2026 στις 6:00 π.μ. UTCresolved

   This incident has been resolved.

1. 8 Μαΐου 2026 στις 12:11 π.μ. UTCmonitoring

   We are currently tracking an issue with our upstream cloud provider that is affecting a subset of our Kubernetes nodes. As a result, some customers may receive false-positive alerts regarding node reachability. Please be assured that your underlying services and workloads remain operational, and these alerts can be safely ignored at this time. We are closely monitoring the situation with our provider.

2. 8 Μαΐου 2026 στις 12:12 π.μ. UTCmonitoring

   We are continuing to monitor for any further issues.

3. 8 Μαΐου 2026 στις 12:32 π.μ. UTCresolved

   This incident has been resolved.

1. 15 Απριλίου 2026 στις 6:38 π.μ. UTCresolved

   At ~02:20 UTC, a Kubernetes node running part of our ingress controller failed and was auto-replaced. Some clients experienced brief agent disconnections and false-positive node-unavailability alerts during the failover. Service has fully recovered after few minutes. We apologize for the inconvenience.

1. 9 Μαρτίου 2026 στις 1:49 μ.μ. UTCresolved

   Yesterday, a brief infrastructure incident occurred due to certain processes on our load balancer failing (exactly at 16:34 UTC). This caused more than half of our globally connected nodes to reconnect, which in turn triggered false positive alerts notifying clients of node downtime. We apologize for the confusion and are taking steps to ensure this does not happen again.

1. 22 Ιανουαρίου 2026 στις 12:47 μ.μ. UTCidentified

   We have identified an issue with the latest version of the repository configuration packages for our RHEL 8.x repositories that causes systems which update to that version of the repository configuration package to become unable to access our repositories. This affects all systems that use our repository configuration packages to access our RHEL 8.x repositories, including but not limited to RHEL 8.x, Alma Linux 8.x, Rocky Linux 8.x, and Oracle Linux 8.x. Affected systems will report a failure to download repository metadata for Netdata repositories. A fix is currently being tested and is expected to be deployed soon.

2. 22 Ιανουαρίου 2026 στις 12:48 μ.μ. UTCidentified

   We are continuing to work on a fix for this issue.

3. 22 Ιανουαρίου 2026 στις 2:06 μ.μ. UTCmonitoring

   The primary fix has been deployed and appears to be working correctly. We are currently working on a followup fix to allow affected systems to update properly without requiring user intervention. Until that additional fix is deployed, affected systems can be manually updated to fully resolve the issue locally by fetching the latest version of the repository configuration package manually and installing it on the system: - For most RHEL 8 equivalent systems using our stable repositories: https://repository.netdata.cloud/repos/repoconfig/el/8/x86_64/netdata-repo-5-4.noarch.rpm - For most RHEL 8 equivalent systems using our nightly repositories: https://repository.netdata.cloud/repos/repoconfig/el/8/x86_64/netdata-repo-edge-5-4.noarch.rpm - For Oracle Linux 8 systems using our stable repositories: https://repository.netdata.cloud/repos/repoconfig/ol/8/x86_64/netdata-repo-5-4.noarch.rpm - For Oracle Linux 8 systems using our nightly repositories: https://repository.netdata.cloud/repos/repoconfig/ol/8/x86_64/netdata-repo-edge-5-4.noarch.rpm

4. 22 Ιανουαρίου 2026 στις 3:20 μ.μ. UTCresolved

   In addition to the fix for the repository configuration packages, a fix has been deployed to the repositories themselves that will enable affected systems to update successfully to the latest version of the repository configuration packages, and as such no further issues are expected.

1. 18 Νοεμβρίου 2025 στις 12:11 μ.μ. UTCinvestigating

   As Cloudflare is experiencing network issues, Netdata Cloud, as well as our package repositories are currently unavailable.

2. 18 Νοεμβρίου 2025 στις 12:23 μ.μ. UTCmonitoring

   Connectivity is recovering, we'll keep this open until we see traffic at previous levels. See https://www.cloudflarestatus.com/incidents/8gmgl950y3h7 for details on the Cloudflare outage itself.

3. 18 Νοεμβρίου 2025 στις 7:22 μ.μ. UTCresolved

   This incident has been resolved.

1. 29 Σεπτεμβρίου 2025 στις 11:36 π.μ. UTCidentified

   We've identified an issue with the latest Agent release on Windows that causes the go.d plugin to not be started. The impact is that for Windows nodes, none of the metrics from the go.d plugin are being collected and corresponding alerts don't fire. We will be releasing a patch release to address this as soon as possible today. As a workaround the plugin binary in Program Files can be renamed to add the missing .exe extension.

2. 30 Σεπτεμβρίου 2025 στις 5:04 π.μ. UTCmonitoring

   We have released Netdata Agent v2.7.1 and recommend all Windows users to upgrade. Release notes: https://github.com/netdata/netdata/releases/tag/v2.7.1

3. 30 Σεπτεμβρίου 2025 στις 6:29 μ.μ. UTCresolved

   This incident has been resolved.

1. 21 Αυγούστου 2025 στις 7:35 μ.μ. UTCinvestigating

   We are currently investigating issues with availability of our nightly DEB and RPM packages which are preventing updates and installs from working correctly.

2. 21 Αυγούστου 2025 στις 7:39 μ.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

3. 21 Αυγούστου 2025 στις 7:42 μ.μ. UTCmonitoring

   A fix has been implemented and we expect the repositories to be functioning normally within the next 30 minutes.

4. 21 Αυγούστου 2025 στις 7:58 μ.μ. UTCresolved

   All repositories are functioning normally again.

1. 14 Αυγούστου 2025 στις 3:03 μ.μ. UTCmonitoring

   Due to issues in our package upload process, the v2.6.2 stable release and the nightly builds for 2025-08-13 and 2025-08-14 were not fully uploaded to our official DEB/RPM repositories. We have already identified the root cause and implemented a fix, and are currently monitoring the processing of the package upload backlog to ensure that all of the packages are properly available.

2. 14 Αυγούστου 2025 στις 3:12 μ.μ. UTCresolved

   We have confirmed that the issue that caused the incomplete package uploads has been fully fixed, and all packages that should have been uploaded since the issue first occurred have been fully uploaded and are available in the repositories.

1. 11 Αυγούστου 2025 στις 12:27 μ.μ. UTCinvestigating

   We are investigating an issue with increased data request latencies that can result in slow or completely empty charts for some customers. The affected Netdata Agent versions are v2.6.0 and later, for all stable and nightly builds.

2. 11 Αυγούστου 2025 στις 2:24 μ.μ. UTCinvestigating

   We will be adding additional debugging code in both the Agent (next nightly) and Cloud Backend to track what is causing these spurious latencies.

3. 12 Αυγούστου 2025 στις 9:28 π.μ. UTCidentified

   We have identified the cause of this issue and have been creating a fix which will appear in the upcoming nightly build. When we are confident that there are no negative effects on that build, we will do a new patch release on the stable channel the next day. What happened is that we were not correctly clearing MQTT messages that had been acknowledged by the broker, causing newer messages to not be correctly recognized as acknowledged in certain conditions.

4. 13 Αυγούστου 2025 στις 3:30 μ.μ. UTCmonitoring

   We've implement a change for this issue which appeared in our latest nightly, and resolved the problem. We are now issuing a new stable release v2.6.2 to include that fix, and will resolve this issue when all packages are available.

5. 20 Αυγούστου 2025 στις 1:09 μ.μ. UTCresolved

   This incident has been resolved. We recommend that Agents are upgraded to v2.6.2 on the stable channel, or the latest nightly build at the earliest convenience.

1. 25 Ιουλίου 2025 στις 12:27 μ.μ. UTCmonitoring

   We identified monitoring issues affecting customers using Netdata Agent version 2.6.0 on Windows servers. These issues were caused by upstream changes in Windows libraries that impacted agent functionality. Resolution Fixed Version Available: Netdata Agent v2.6.1 Release Date: July 25, 2025 (16:00 UTC) Preview Available: Nightly release v2.6.0-2 Required Action Immediate upgrade recommended - All users running Netdata Agent v2.6.0 on Windows servers should upgrade to v2.6.1 as soon as possible to restore full monitoring capabilities. We appreciate your patience as we continue working on additional improvements related to the upstream Windows library changes.

2. 28 Ιουλίου 2025 στις 10:44 π.μ. UTCresolved

   This incident has been resolved.

1. 4 Ιουνίου 2025 στις 12:59 μ.μ. UTCidentified

   We had a major problem with our MQTT broker. It is currently up and running and agents are reconnecting to the cloud.

2. 4 Ιουνίου 2025 στις 1:42 μ.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 4 Ιουνίου 2025 στις 1:49 μ.μ. UTCresolved

   This incident has been resolved.

1. 1 Απριλίου 2025 στις 9:00 π.μ. UTCinvestigating

   We have found that a recent change in the nightly static builds of Netdata Agent causes meta data on the Agent to be overwritten. Specifically the sqlite3 database that keeps meta data on which timeseries stored in dbengine correspond with which metrics, and the information on the Agent's "machine GUID" are overwritten with the same copy in the build package. Not affected are: - All stable releases - Native packages (.deb and .rpm) Affected are all nightly static builds with the following version numbers: - 2.3.0-50-nightly - 2.3.0-60-nightly - 2.3.0-72-nightly - 2.3.0-78-nightly - 2.3.0-87-nightly The initial impact is that all affected Agent installs, even though they still have the timeseries data stored on disk, have lost all meta data associated with it, so these timeseries become inaccessable. This is unrecoverable. Additionally, the main form of identification is overwritten, too. We are assessing what the impact is for users of Netdata Cloud, and will update this incident with more information when the investigation is completed. The bug itself has been fixed and merged. We will issue a new nightly build shortly.

2. 2 Απριλίου 2025 στις 9:25 π.μ. UTCresolved

   Affected Agents can cause the creation of multiple duplicate nodes in Netdata Cloud. All but the last one will appear as offline, and the last one will be as if it was created from scratch with no data. Unfortunately, the previously stored metrics for the affected nodes can not be recovered. The duplicate offline nodes can be safely deleted from Space Settings -> Nodes. Note that you may have to add the newest copy of these nodes to the appropriate rooms. The fixed nightly static build is v2.3.0-102.

1. 25 Φεβρουαρίου 2025 στις 9:41 π.μ. UTCinvestigating

   We are investigating an issue with today's nightly (v2.2.0-245), causing alerting to not work ("health") and external plugins, including go.d, to not connect properly. This may be resolved by restarting the Agent. Stable versions of the Agent are not affected.

2. 25 Φεβρουαρίου 2025 στις 11:04 π.μ. UTCidentified

   We have identified the issue, committed a fix, and initiated new nightly builds for all platforms. This will take several hours. In the mean time, please restart Netdata to work around the issue.

3. 25 Φεβρουαρίου 2025 στις 1:14 μ.μ. UTCmonitoring

   The builds are completed, so we are watching out for any remaining related issues.

4. 25 Φεβρουαρίου 2025 στις 1:29 μ.μ. UTCmonitoring

   We are continuing to monitor for any further issues.

5. 25 Φεβρουαρίου 2025 στις 3:25 μ.μ. UTCresolved

   This incident has been resolved.

1. 15 Νοεμβρίου 2024 στις 8:23 π.μ. UTCresolved

   Our alarm processing infrastructure was running behind which is causing inaccuracies alarms for some nodes. No data has been lost and the systems should be already up to date.

1. 8 Νοεμβρίου 2024 στις 8:07 π.μ. UTCinvestigating

   Due to the release of Netdata Agent 2.0 we have quite a big backlog for alarms. We are investigating this issue.

2. 8 Νοεμβρίου 2024 στις 9:49 π.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

3. 8 Νοεμβρίου 2024 στις 11:01 π.μ. UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 8 Νοεμβρίου 2024 στις 1:36 μ.μ. UTCresolved

   This incident has been resolved.

1. 18 Ιουλίου 2024 στις 4:45 π.μ. UTCinvestigating

   We were alerted to a delay in alarms for some users and are investigating the matter.

2. 18 Ιουλίου 2024 στις 5:37 π.μ. UTCidentified

   The issue has been identified and a fix is being implemented.

3. 18 Ιουλίου 2024 στις 8:17 π.μ. UTCidentified

   Currently, we are waiting for the fix to take effect, and some users might experience delays in all cloud operations.

4. 18 Ιουλίου 2024 στις 11:41 π.μ. UTCresolved

   This incident has been resolved.

1. 1 Μαΐου 2024 στις 3:44 μ.μ. UTCidentified

   We've identified an issue with static and local builds of the Netdata Agent, that causes its main configuration in `/etc/netdata/netdata.conf` or `/opt/netdata/etc/netdata/netdata.conf` to be overwritten with the default. The `netdata-updater.conf` file is similarly affected. Depending on your configuration settings that have been changed with respect to the defaults, this may result in data loss. We will update this incident with more detailed information on the impact as soon as possible. Docker image or native package builds, as well as stable builds, are not affected. We have created a fix (https://github.com/netdata/netdata/pull/17572) and have triggered a new nightly build. As soon as those are available, we will also update this incident.

2. 1 Μαΐου 2024 στις 3:59 μ.μ. UTCidentified

   The affected build number is v1.45.0-315-nightly, and local builds starting with commit https://github.com/netdata/netdata/commit/5973417027606bacf044b3ead40a882931ce773f (April 30, 11:45 UTC) up until commit https://github.com/netdata/netdata/commit/0f2a261839d5ffc42f17383b4292673aa93d6a1f (May 1, 15:13 UTC).

3. 1 Μαΐου 2024 στις 4:28 μ.μ. UTCidentified

   Update regarding potential data loss. This will happen if the configuration had been changed to increase metric retention (with respect to the defaults). Unfortunately, any stored data beyond the default metric retention will be lost on running installs of the affected builds. The only way to prevent this is by not using (of having used) version v1.45.0-315-nightly. We have made sure that the corresponding artifacts are no longer accessible by the installer.

4. 1 Μαΐου 2024 στις 5:22 μ.μ. UTCresolved

   The build artifacts for the new nightly release (1.45.0-326) are now available, and consider the incident resolved. Should you experience any issues, please let us know!

5. 2 Μαΐου 2024 στις 10:02 π.μ. UTCpostmortem

   Prior to [netdata/netdata#17475](https://github.com/netdata/netdata/pull/17475), the `netdata.conf` and `netdata-updater.conf` files where handled by the installer code outside of the build system. With the shift to using the build system to produce packages, handling for them needed to be moved into the build system. However, insufficient testing was performed to confirm that this would not break other installation types, and the change was not properly made conditional on packages being built. As a result, the static and local builds with version `v1.45.0-315-nightly` will overwrite these configuration files with the default templates for those files. This causes all local changes to those files to be lost. In particular, if the Agent configuration had been changed for longer retention, the overwritten configuration will have undone those settings, causing any metrics data **beyond the _default_ retention to be lost** on the first run of this version. We have pulled the affected build artifacts to prevent our installer from using them. While [the fix](https://github.com/netdata/netdata/pull/17572) ensures the issue won't occur in future versions, starting with version `v1.45.0-326-nightly`, it is important to note that affected installations **will not automatically recover** their previous configurations. If you were using a non-default `netdata.conf` and/or `netdata-updater.conf` and experienced this bug, you will need to **manually reconfigure** your Netdata install. As we aim to carefully develop Netdata for many platforms and hardware architectures, we release nightly builds of the Netdata Agent to catch any issues our changes may have caused, beyond our own internal testing. Unfortunately, we make mistakes that we did not catch in our testing, with data loss as an extreme possible outcome. Therefore we strongly recommend using our **stable releases for production systems**. You can review the [difference between nightly and stable builds](https://learn.netdata.cloud/docs/netdata-agent/installation#nightly-vs-stable-releases), and our recommended [best practices](https://www.netdata.cloud/blog/netdata-best-practices/). If you have been affected by this issue and/or have any questions, please let us know.