Ιστορικό διακοπών του Flying Sphinx

1. 7 Σεπτεμβρίου 2026 στις 9:23 μ.μ. UTCmonitoring

   Ο κεντρικός διακομιστής Robigus συνετρίβη απρόσμενα, προσκρούοντας στην αναζήτηση και την ευρετηρίαση για ορισμένους πελάτες. Αποκαταστάθηκε και οι επιχειρήσεις επανήλθαν στο φυσιολογικό. Αυτή τη στιγμή παρακολουθώντας τα πράγματα για να διασφαλιστεί η σταθερότητα έχει επιστρέψει.

2. 7 Σεπτεμβρίου 2026 στις 9:48 μ.μ. UTCresolved

   Όλα έχουν επιστρέψει στο φυσιολογικό, αυτό το περιστατικό θεωρείται ότι έχει επιλυθεί. Συγγνώμη για τη διακοπή!

Αυτόματη μετάφραση από την επίσημη ενημέρωση του συμβάντος.

1. 13 Σεπτεμβρίου 2024 στις 4:20 μ.μ. UTCinvestigating

   The Mors server has unexpectedly crashed, working on bringing up a replacement.

2. 13 Σεπτεμβρίου 2024 στις 4:26 μ.μ. UTCidentified

   A server is now booted, working on restoring Sphinx daemons.

3. 13 Σεπτεμβρίου 2024 στις 4:39 μ.μ. UTCmonitoring

   Sphinx daemons and all other services are now running as expected. Please get in touch if problems are persisting.

4. 13 Σεπτεμβρίου 2024 στις 8:52 μ.μ. UTCresolved

   No further issues have cropped up. This incident is resolved.

1. 7 Νοεμβρίου 2022 στις 12:33 π.μ. UTCidentified

   We're seeing a failure from an essential third-party API which is stopping our own requests from working - actions that involve reindexing or restarting Sphinx daemons aren't being completed. Search requests are not impacted by this change, though. The issue has been raised with the third-party vendor, and we're hoping for a swift resolution.

2. 7 Νοεμβρίου 2022 στις 2:02 π.μ. UTCmonitoring

   The third-party has implemented a fix, errors on our side have stopped but will continue to monitor the situation.

3. 7 Νοεμβρίου 2022 στις 2:30 π.μ. UTCresolved

   All functionality has returned to normal. Thanks for your patience, and if you're finding any problems are persisting, please do get in touch.

1. 14 Ιουλίου 2021 στις 1:46 π.μ. UTCidentified

   The primary server for Robigus has stopped responding unexpectedly. Impacted customers have been switched over to the failover server, and should have search queries working again. A new failover will be created shortly.

2. 14 Ιουλίου 2021 στις 2:57 π.μ. UTCmonitoring

   New failover is now running. If any customers are seeing issues persist, please do get in touch.

3. 14 Ιουλίου 2021 στις 3:12 π.μ. UTCresolved

   All systems are back to normal now.

1. 4 Μαΐου 2021 στις 7:01 π.μ. UTCresolved

   Both Averna and Strenua servers had some instability in their services, but that has been resolved and everything is back to normal. If any customers continue to be affected, please do get in touch.

1. 3 Μαρτίου 2021 στις 3:39 π.μ. UTCmonitoring

   The robigus server had stopped responding, but fixes have been applied, things should be working again, but monitoring the situation to confirm.

2. 3 Μαρτίου 2021 στις 3:48 π.μ. UTCresolved

   The incident has been resolved, all services are functioning as expected. If you're seeing any lingering issues, please do get in touch!

1. 4 Σεπτεμβρίου 2020 στις 9:47 μ.μ. UTCmonitoring

   Robigus was temporarily overloaded, but is now responding to search requests and API calls. Further investigation is underway to determine the initial cause of the problem.

2. 4 Σεπτεμβρίου 2020 στις 10:39 μ.μ. UTCmonitoring

   The primary Robigus server has gotten worse, so the failover has been promoted as the new primary, with all affected customers migrated. If you're still seeing issues, please get in touch. A new failover server is now being prepared.

3. 4 Σεπτεμβρίου 2020 στις 10:56 μ.μ. UTCresolved

   New failover is standing by, everything should be functioning properly. Please get in touch if that's not the case.

1. 25 Αυγούστου 2020 στις 10:38 μ.μ. UTCidentified

   The primary server for Robigus-hosted customers has stopped responding unexpectedly, so the failover has now been promoted (and is actively serving search queries). A new failover will be set up.

2. 25 Αυγούστου 2020 στις 11:16 μ.μ. UTCmonitoring

   The new failover is in place, the new primary is running as expected.

3. 25 Αυγούστου 2020 στις 11:31 μ.μ. UTCresolved

   Everything's back to normal - but if any customers are still coming across issues, please get in touch.

1. 2 Ιουλίου 2020 στις 10:00 μ.μ. UTCidentified

   The primary that was switched over to yesterday is having some teething issues, so I've shifted affected customers over to the new failover instead while things are fixed up. Everything should already be working again, and I'll get the underlying problem fixed.

2. 2 Ιουλίου 2020 στις 10:18 μ.μ. UTCmonitoring

   The old primary is now in a happy state (and thus can be considered the new failover). The new primary is serving queries and processing API requests without any issues. Keeping an eye on things.

3. 2 Ιουλίου 2020 στις 10:31 μ.μ. UTCresolved

   No further issues have cropped up, everything's back to working smoothly, so this issue is considered resolved. If anyone's still seeing problems on their side, please do get in touch.

1. 1 Ιουλίου 2020 στις 7:02 μ.μ. UTCidentified

   The primary server for Robigus has failed unexpectedly. Impacted customers have now been switched to the failover server, and search requests should be being served again. A new failover is now being prepared.

2. 1 Ιουλίου 2020 στις 7:29 μ.μ. UTCmonitoring

   Robigus replacement failover is set up, and search queries and API calls should be operating normally for all customers.

3. 1 Ιουλίου 2020 στις 7:37 μ.μ. UTCresolved

   Everything's continuing to work as expected, this issue is considered resolved. Please get in touch if you're finding any problems persist.

1. 14 Απριλίου 2020 στις 9:14 π.μ. UTCidentified

   Our DNS provider is having issues from their European node that impacts Lime (the server for our eu-west customers). Searching should remain unaffected, but API requests for indexing and other daemon actions are currently failing. We've been in touch with the DNS provider, and expect there'll be a resolution soon.

2. 14 Απριλίου 2020 στις 9:15 π.μ. UTCidentified

   We are continuing to work on a fix for this issue.

3. 14 Απριλίου 2020 στις 10:42 π.μ. UTCmonitoring

   Our DNS provider has fixed the issue and API requests are now working again.

4. 14 Απριλίου 2020 στις 11:14 π.μ. UTCresolved

   There have been no further disruptions, this incident is considered resolved.

1. 4 Νοεμβρίου 2019 στις 1:29 μ.μ. UTCinvestigating

   The current More instance is not responding, so a replacement server is being fired up to take its place.

2. 4 Νοεμβρίου 2019 στις 1:39 μ.μ. UTCmonitoring

   The new Mors instance is in place and responding to API calls and search queries.

3. 4 Νοεμβρίου 2019 στις 1:53 μ.μ. UTCresolved

   This incident has been resolved, and all systems are operating as expected. If you're finding any problems persisting, do get in touch.

1. 17 Σεπτεμβρίου 2019 στις 10:30 μ.μ. UTCidentified

   We've switched over to the secondary server, so customers should have queries and API calls working again. The cause of the initial issue is not yet clear.

2. 18 Σεπτεμβρίου 2019 στις 1:34 π.μ. UTCresolved

   Robigus has a new secondary server (with the old secondary working smoothly as the new primary). Everything should be responding as expected now - do get in touch if any issues are still cropping up.

1. 6 Ιουνίου 2019 στις 12:31 π.μ. UTCmonitoring

   Lime unexpectedly restarted (possibly due to a broader issue in eu-west-1), but came back up within a few minutes with all Sphinx daemons. Everything is back to normal.

2. 6 Ιουνίου 2019 στις 12:34 π.μ. UTCresolved

   Lime has returned to being fully operational. If any customers are seeing issues persist, please do get in touch.

1. 24 Ιανουαρίου 2019 στις 10:05 μ.μ. UTCidentified

   Averna has crashed, currently bringing up a replacement server.

2. 24 Ιανουαρίου 2019 στις 10:12 μ.μ. UTCmonitoring

   Replacement is up and running, with indexing and search requests responding accordingly.

3. 24 Ιανουαρίου 2019 στις 10:27 μ.μ. UTCresolved

   This issue is now resolved: Averna's returned to operating smoothly. If you're still seeing issues, please get into touch.

1. 21 Νοεμβρίου 2018 στις 5:02 μ.μ. UTCinvestigating

   Working on getting it restored.

2. 21 Νοεμβρίου 2018 στις 5:11 μ.μ. UTCmonitoring

   New instance for averna is now up and running.

3. 21 Νοεμβρίου 2018 στις 5:19 μ.μ. UTCresolved

   Everything's operating as expected again. Please get in touch if you're finding that's not the case for you!

1. 3 Οκτωβρίου 2018 στις 8:49 π.μ. UTCresolved

   Earlier today, Robigus stopped responding to search requests (connections to Sphinx daemons). The part of the Flying Sphinx infrastructure that failed was the Sphinx proxy on your original server - it stopped responding to any TCP requests (though the logs had no suggestion as to why). Clearly, this is a critical part of everything - if the proxy’s down, you can’t connect to your Sphinx daemon at all (and that’s essential in both searching and regenerating). I usually get downtime alerts (with a dedicated phone + SMS messages which should wake me up if it’s the middle of the night), but this wasn’t triggered by the proxy failing. So, I’ve made the following changes: * If the proxy does not respond to health checks, it’s considered a major server failure, and thus I will get SMS alerts. * Also, if it’s not responding to health checks, Monit will restart the proxy process, so resolution should be sorted out within a minute. This is now all in place, but I’ll continue to think through better ways of handling such situations. I’m very sorry for the downtime!

1. 21 Σεπτεμβρίου 2018 στις 7:52 π.μ. UTCidentified

   Greatly delayed alert, but Carmenta has unexpectedly restarted itself and crashed halfway through the process. Working on getting a replacement sorted now, expected resolution in 40 minutes.

2. 21 Σεπτεμβρίου 2018 στις 8:24 π.μ. UTCmonitoring

   A new server for Carmenta is running, affected customers should find things are working now. Continuing to monitor to confirm that the issue is resolved.

3. 21 Σεπτεμβρίου 2018 στις 8:45 π.μ. UTCresolved

   Everything's now back to normal. Will make sure such fixes don't get delayed so badly again! If anyone's still seeing problems persist, please get in touch.

1. 20 Απριλίου 2018 στις 4:41 π.μ. UTCidentified

   As part of some DNS changes, SSL requests for flying-sphinx.com were briefly returning an invalid certificate (lasting about 10 minutes). This has been resolved, but if you're seeing the problem persist, please do get in touch.

2. 20 Απριλίου 2018 στις 4:42 π.μ. UTCmonitoring

   Marking this issue as resolved given the certificate is now correct again. Will continue to monitor for any affected services.

3. 20 Απριλίου 2018 στις 6:21 π.μ. UTCresolved

   There was one more glitch from the certificate provider, but things have been smooth since, so marking this as resolved.

1. 29 Δεκεμβρίου 2017 στις 12:26 μ.μ. UTCidentified

   While general server maintenance was happening, a bug was introduced for API actions. This has been fixed, and behaviour has returned to normal. Search daemons/requests were unaffected.

2. 29 Δεκεμβρίου 2017 στις 12:34 μ.μ. UTCmonitoring

   Maintenance has finished, behaviour remains as expected.

3. 29 Δεκεμβρίου 2017 στις 12:45 μ.μ. UTCresolved

   This issue is now resolved. If you're seeing any persistent problems, please do get in touch.