Seravo 장애 기록

1. 2026년 8월 11일 오전 11:11 UTCinvestigating

   We are currently investigating this issue.

2. 2026년 8월 11일 오전 11:26 UTCresolved

   This incident has been resolved.

1. 2026년 7월 21일 오전 10:12 UTCinvestigating

   우리는 현재이 문제를 조사하고 있습니다.

2. 2026년 7월 21일 오전 10:32 UTCidentified

   문제는 확인되었습니다. 우리는 현재 우리의 서버 주인 공급자를 가진 해결책에 일하고 있습니다.

3. 2026년 7월 21일 오전 11:19 UTCmonitoring

   수정이 시행되어 현재 결과를 모니터링하고 있습니다.

4. 2026년 7월 21일 오후 12:24 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 6월 27일 오후 6:42 UTCidentified

   We identified a networking issue in cluster de-dauner. Our team is working on it to fix it. We apologise for any inconvenience.

2. 2026년 6월 27일 오후 7:14 UTCidentified

   We are continuing to work on a fix for this issue.

3. 2026년 6월 27일 오후 7:36 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2026년 6월 27일 오후 8:03 UTCresolved

   This incident has been resolved.

1. 2026년 6월 26일 오후 8:53 UTCinvestigating

   We are experiencing availability issues. Our team is actively investigating and resolving the situation. We apologise for any inconvenience. Thank you for your patience.

2. 2026년 6월 26일 오후 9:51 UTCresolved

   This incident has been resolved.

1. 2026년 6월 25일 오후 6:19 UTCidentified

   We identified a networking issue in cluster. Our team is working on it to fix it. We apologise for any inconvenience.

2. 2026년 6월 25일 오후 7:07 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2026년 6월 25일 오후 7:20 UTCresolved

   This incident has been resolved.

1. 2026년 6월 17일 오전 7:06 UTCinvestigating

   We have noticed database connectivity problems in cluster fi-tolonen. We are investigating the issue.

2. 2026년 6월 17일 오전 7:09 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2026년 6월 17일 오전 7:36 UTCresolved

   This incident has been resolved.

1. 2026년 6월 15일 오후 1:36 UTCmonitoring

   Customers that have opted to use external DNS ( dns{1..4}triuvare-webhosting.fi) for their site might encounter availability issues with their sites. Incident does not affect customers using only Seravo-provided hosting services.

2. 2026년 6월 15일 오후 7:25 UTCresolved

   This incident has been resolved.

1. 2026년 6월 10일 오전 4:30 UTCidentified

   We are experiencing availability issues in cluster fi-haarla. Some websites may experience interruptions. Our team is actively investigating and resolving the situation. We apologise for any inconvenience. Thank you for your patience.

2. 2026년 6월 10일 오전 4:36 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2026년 6월 10일 오전 4:41 UTCresolved

   This incident has been resolved.

1. 2026년 6월 9일 오전 3:25 UTCidentified

   We identified a networking issue in cluster. Our team is working on it to fix it. We apologise for any inconvenience.

2. 2026년 6월 9일 오전 4:23 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2026년 6월 9일 오전 4:23 UTCmonitoring

   We are continuing to monitor for any further issues.

4. 2026년 6월 9일 오전 4:24 UTCresolved

   This incident has been resolved.

5. 2026년 6월 9일 오전 10:18 UTCpostmortem

   # Häiriö Suomessa sijaitsevissa palvelinklustereissa 9.6.2026 \(English follows\) Ubuntu-käyttöjärjestelmän kehittäjä Canonical julkaisi maanantaina 8.6.2026 korjauksen www-palvelinohjelmisto Nginx:n HTTP/2 Bomb -haavoittuvuuteen \([https://ubuntu.com/security/notices/USN-8398-1](https://ubuntu.com/security/notices/USN-8398-1)\). Nginx on käytössä myös Seravolla, ja ohjelmisto päivitettiin automaattisesti aamulla 9.6.2026. Päivityksessä oli mukana regressio, joka rikkoi Seravolla käytössä olevien Nginx-moduulien yhteensopivuuden, minkä seurauksena Nginx ei toiminut normaalisti eikä pystynyt vastaamaan HTTP-pyyntöihin. Ongelma on maailmanlaajuinen, eikä Canonical ole toistaiseksi julkaissut päivityksiä ko. moduuleihin. Seravolla ongelma vaikutti suurimpaan osaan asiakkaidemme sivustoja Suomessa sijaitsevissa palvelinklustereissa, ja sen seurauksena sivustot eivät toimineet. Ongelma ei koskenut sivustoja palvelinklustereissamme, jotka sijaitsevat muissa maissa \(Ruotsi, Saksa, Yhdysvallat\). Ongelma saatiin korjattua palauttamalla väliaikaisesti käyttöön Nginx:n aiempi versio. Seuraamme saatavilla olevien korjausten tilannetta ja teemme tarvittavat päivitykset mahdollisimman pian. Ilmoitamme reaaliaikaisesti kaikista palveluitamme koskevista häiriöistä erillisellä statussivulla osoitteessa status.seravo.com. Suosittelemme häiriöilmoitusten tilaamista esim. sähköpostiin. ## Aikajana \(UTC\+3\) * 06:00 Päivitykset alkoivat * 06:13 Ensimmäinen sivustohälytys * 06:15 Seravon päivystäjä tarkistaa tilanteen * 06:21 Ongelma rajataan Nginx:ään * 06:25 Häiriöilmoitus julkaistaan \(status.seravo.com\) * 06:30 Ongelma rajataan ohjelmistopäivityksiin * 06:44 Päivystäjä eskaloi ongelman tiimin sisällä * 06:53 Käyttöön palautetaan aiempi Nginx-versio * 07:12 Kaikkien palvelimien Nginx palautettu, sivustot toiminnassa * 07:18 Ongelman leviäminen laajemmalle estetty # Incident on Finnish server clusters on June 9, 2026 On Monday, June 8, 2026, Canonical \(developer of the Ubuntu operating system\) released a security patch for the Nginx web server software to address the HTTP/2 Bomb vulnerability \([https://ubuntu.com/security/notices/USN-8398-1](https://ubuntu.com/security/notices/USN-8398-1)\). Nginx is used by Seravo, and the software was automatically updated on the morning of June 9, 2026. The patch contained a regression that broke compatibility with the Nginx modules used by Seravo. As a result, Nginx failed to handle incoming HTTP requests. The issue affects systems globally, and Canonical has not yet released updates for the affected modules. At Seravo’s hosting, the issue affected the majority of our customers’ sites hosted on server clusters located in Finland, causing downtime. Sites hosted on our server clusters in other countries \(Sweden, Germany, and the United States\) were not affected. The issue was resolved by temporarily rolling back to the previous version of Nginx. We are closely monitoring the availability of official fixes and will apply the necessary updates as soon as possible. Seravo provides real-time updates on all disruptions at status.seravo.com. We highly recommend subscribing to incident notifications via email, for example. ## Timeline \(UTC\+3\) * 06:00 Updates begin * 06:13 First site alert triggered * 06:15 Seravo’s on-call expert investigates the situation * 06:21 The issue is isolated to Nginx * 06:25 Incident notification published \(status.seravo.com\) * 06:30 The issue is limited to software updates * 06:44 On-call expert escalates the issue within the team * 06:53 Nginx is rolled back to previous version * 07:12 Nginx rollback on all servers successful, sites back in operation * 07:18 Issue successfully mitigated, further propagation prevented

1. 2026년 5월 5일 오후 9:39 UTCinvestigating

   Due to a fault, there is currently limited availability to the recursive DNS resolution for "de" domains. This might affect some of our customers. More info here: https://status.hetzner.com/incident/c5e4ec2e-82b5-4f1a-a7ce-37127a6f2aa9

2. 2026년 5월 6일 오전 4:01 UTCresolved

   This incident has been resolved.

1. 2026년 4월 22일 오전 11:08 UTCidentified

   We identified a networking issue in cluster fi-sestak. Our team is working on it to fix it. We apologise for any inconvenience.

2. 2026년 4월 22일 오전 11:40 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2026년 4월 22일 오전 11:52 UTCresolved

   This incident has been resolved.

1. 2026년 3월 31일 오전 12:13 UTCidentified

   We identified a networking issue in cluster. Our team is working on it to fix it. We apologise for any inconvenience.

2. 2026년 3월 31일 오전 12:25 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2026년 3월 31일 오전 12:42 UTCresolved

   This incident has been resolved.

1. 2026년 3월 25일 오전 11:22 UTCidentified

   We identified a networking issue in cluster us-davis. Our team is working on it to fix it. We apologise for any inconvenience.

2. 2026년 3월 25일 오전 11:25 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2026년 3월 25일 오전 11:40 UTCresolved

   This incident has been resolved.

1. 2026년 3월 24일 오전 6:14 UTCidentified

   We identified a networking issue in cluster fi-sestak. Our team is working on it to fix it. We apologise for any inconvenience.

2. 2026년 3월 24일 오전 6:21 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2026년 3월 24일 오전 7:22 UTCresolved

   This incident has been resolved.

1. 2026년 3월 12일 오후 3:45 UTCinvestigating

   We are currently investigating this issue.

2. 2026년 3월 12일 오후 4:18 UTCresolved

   This incident has been resolved.

1. 2026년 3월 12일 오전 3:10 UTCinvestigating

   We are currently investigating this issue.

2. 2026년 3월 12일 오전 4:33 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2026년 3월 12일 오전 6:21 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2026년 3월 12일 오전 7:47 UTCresolved

   This incident has been resolved.

5. 2026년 3월 13일 오전 9:37 UTCpostmortem

   # Notice of a Server Outage on March 12, 2026 On Thursday March 12, 2026, a failure related to the server resource usage caused an outage in the cluster fi-perko. The outage was caused by a newly implemented security feature which led to increased resource usage. Monitoring failed to notice and pinpoint the increased resource usage in time, which led to the outage. The outage was aggravated by an unrelated networking change which led to delay in recovery. This caused an outage to all sites in the fi-perko cluster. The problem was noticed immediately by our on-call monitoring processes, and measures were immediately implemented to begin recovery. The disruption first began at 04:00 \(UTC\+2\), escalated at 05:00 and ended at 08:19 \(UTC\+2\). We apologise for any inconvenience caused by the disruption. ## Timeline \(all timestamps UTC\+2 \(EET\)\) * 12.3.2026 03:47 Partial outage * 12.3.2026 03:52 On-call performs initial recovery measures * 12.3.2026 04:00 Recovered from partial outage * 12.3.2026 05:00 Incident escalated again * 12.3.2026 05:10 On-call attempts recovery measures * 12.3.2026 05:15 Full outage * 12.3.2026 05:27 On-call performs recovery measures * 12.3.2026 06:17 Recovered from the outage - begin site recovery * 12.3.2026 08:19 All affected sites were back online ## Follow-Up Action As a result of the incident, we at Seravo have identified the need for the following measures: * Testing new features with extended data-sets to be able to catch issues caused by the sheer volume of processed data * Improvements to system resource monitoring

1. 2026년 3월 12일 오전 1:56 UTCinvestigating

   We are currently investigating this issue.

2. 2026년 3월 12일 오전 2:35 UTCresolved

   This incident has been resolved.

1. 2026년 3월 11일 오전 5:19 UTCinvestigating

   We are currently investigating this issue.

2. 2026년 3월 11일 오전 5:47 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2026년 3월 11일 오전 6:52 UTCresolved

   This incident has been resolved.

1. 2026년 3월 5일 오후 3:08 UTCidentified

   We identified an upstream service provider networking issue causing slowness on some sites. We apologise for any inconvenience.

2. 2026년 3월 5일 오후 4:01 UTCidentified

   We are continuing to work on a fix for this issue.

3. 2026년 3월 5일 오후 4:24 UTCidentified

   The problem is related to Meta API services that slow down the sites. The sites will remain to work normally after the issue is fixed by the service provider.

4. 2026년 3월 5일 오후 5:36 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

5. 2026년 3월 5일 오후 6:59 UTCresolved

   This incident has been resolved.

1. 2026년 2월 25일 오후 9:14 UTCinvestigating

   Customers that have opted to use external DNS for their site might encounter availability issues with their sites. Incident does not affect customers using only Seravo-provided hosting services.

2. 2026년 2월 25일 오후 9:19 UTCresolved

   This incident has been resolved.