Sirv outage history

Uploads to some Sirv accounts are failing, due to a server problem. This may also prevent images from displaying occasionally. We are working to resolve the issue as soon as possible.

The cause of the issue was identified and it has now been fully resolved. During the period, about 25% of Sirv accounts may have experienced a proportion of file uploads failing. Retrying the uploads a short while later may have been successful because this was a only affecting a small number of requests. A review is now taking place and further information will be provided here later.

A small number of uploads and requests are returning errors due to multiple network switch failures in Sirv's primary datacentre. CDN requests are working as normal. We are closely monitoring the situation and will provide updates as more information becomes available.

All services are back to normal and requests are completing.

This issue was caused by a core router fault affecting Sirv's primary data centre. This caused brief, intermittent outages on file storage and uploads, across all transfer types: API, S3 and FTP. Sirv automatically switched to its failover service, so file processing and delivery continued as normal. CDN requests also continued as normal. All services were restored once the datacentre had resolved the core router outage.

Response time for some requests in Germany are slower than normal and a small number are timing out. The issue is being investigated.

The issue has been resolved. One of the servers serving Germany was overloaded and responding slower than usual. The server is now healthy again. We are reviewing what actions to take to reduce the chance of this happening again.

Some requests from California are slow and a small number (under 2%) are failing. This is being investigated and likely related to new CDN servers that were deployed today in Los Angeles.

The issue has been resolved. The issue was caused by packet loss affecting our new Los Angeles datacenter. The datacenter traced the issue to their upstream provider, Arelion, which was experiencing network instability. The issue was resolved when Arelion was temporarily disabled as an upstream provider and traffic was rerouted through other providers. We require that our datacenters identify and reroute traffic automatically and quickly in such scenarios. However, the resolution of this issue was not fast enough and given that it occurred soon after adopting this new datacenter, we have decided to cease using it. All traffic is now being routed from our original Los Angeles datacenter (as of 7 April 2025).

Some file uploads are not completing due to a server cluster issue. The cause is being investigated. The issue is affecting a small number of accounts. If your file upload fails, please wait until this issue has been resolved.

This issue remains in progress. The cause of the issue has been identified and we are working to resolve it as soon as possible.

The issue has been resolved. The issue was caused by two servers in one cluster failing simultaneously. This impacted the uploading of new files to about 20% of Sirv accounts. Uploads are designed to continue as normal when one server is down but when two servers are down, some uploads can fail, which is what happened. They failed due to an exporter service that stopped running on one server, then the other server. Errors prevented the services from restarting. Then a second issue caused the resolution to take much longer than expected because the servers couldn't be rebooted. This was due to an outdated BIOS. Once the BIOS had been updated, the servers were rebooted and the cause of the underlying issue was resolved. To prevent this from happening again, we have implemented a new BIOS management process with our datacenter. We are also shortening our hardware refresh cycle, for more frequent hardware upgrades. To reduce the chance of the exporter service failure from recurring, we have disabled a process and are monitoring the server metrics.

For a 3 minute period between 09:29 and 09:32 UTC, some requests were not returned. This issue affected only certain requests and only certain accounts. Most accounts were unaffected.

The issue was resolved within 3 minutes. The cause was the rebooting of multiple servers at the same time, whilst only 1 server should have been rebooted. This was an oversight in our standard operating procedures. We are investigating ways to automatically protect and prevent this from recurring.

A server fault is causing some file uploads to fail for a limited number of accounts. The issue is being resolved. If you experience failed uploads, please pause and try again later.

The issue has been resolved. An unusually large number of servers failed at the same time. Sirv is designed to continue normal operation during a server outage and all systems did continue as normal, but some file uploads failed. We are considering what possible changes could be made to assist in such situations in future.

An elevated number of 504 timeouts are being returned for API file upload requests.

There continues to be heavy load on the API from upload requests. A contributing factor is the large number of AI image upscaling requests, which are taking longer than usual to process. The delay caused by the queue of requests is causing some API file upload requests to fail with a 504 response.

Changes have been made and the issue is almost resolved. The majority of upload requests are successful.

The API upload overload is almost fully resolved.

The issue has been fully resolved. The issue started when heavy image upscaling API requests overloaded the upload API, causing some API requests to return 504 errors. This was solved but a knock-on effect caused approximately 5% of API requests to fail with a 502 response. To prevent this issue from happening again, improvements have been made to reduce the load of image upscaling and better distribute load. Changes are also being made to logging and alerting, for faster identification of such scenarios. The greatest impact of this issue was on the Sirv WordPress plugin. The plugin was sending an API request on all admin pages, which blocked the page from loading until the response had been received - typically less than 300ms but up to 20 seconds if API requests timed out. Those API requests were not required, so a new WordPress plugin version v7.3.3 was released at 1805 UTC (14 November) to remove such requests and prevent this issue from recurring: https://wordpress.org/plugins/sirv/ ------ If you use the Sirv API, we recommend that your scripts are written to retry requests that receive a 502 response. Recommendations for handling 502 and other error responses are provided here: https://sirv.com/help/articles/sirv-rest-api/#error-handling-guide

Some files are not being returned for a small number of accounts. This issue is being addressed urgently.

The issue has been resolved. At 1052 UTC, about 11% of requests to a small number of accounts (12 accounts, out of 50,000+) were returning 404 responses (file not found) when they should have returned 200 responses (OK). When the Sirv team identified the issue, they quickly worked to solve it and by 1206 UTC all such requests were returning the normal 200 response. The issue occurred due to a misconfiguration by Sirv engineers while they were configuring a new storage cluster shard. Some requests were going to the new shard before it was ready to receive them. Classified as severity level 3: Minor impact.

A high number of requests to Sirv's 25 CDN locations are returning an error. Our engineers are working quickly to identify and resolve the issue.

The issue has been resolved. It was caused by the failure of a software daemon simultaneously on multiple servers. This is a lightweight and highly efficient service manager that has never failed in the past. During the period, between 3% and 58% of HTTPS requests returned an error, depending on which of the 25 CDN locations received the request. We are now investigating why this happened and will be taking actions to prevent this from impacting service availability in the future. Classified as severity level 2: Meaningful impact.

An elevated level of 415 errors is occurring on the CDN. Our engineers are investigating the cause.

We are continuing to investigate the cause of this issue. The nature of it is complex. It represents itself as a 415 error, from files of zero bytes. There is also an elevated though lower rate of 500 errors. The majority of requests are successful 200 responses. Our engineers are working hard to understand and resolve this.

The 415 error issue was caused by some files becoming truncated, meaning they lost part of their contents. These files show in the control panel at my.sirv.com as having 0 bytes. Files uploaded today were more likely to become truncated, though it seems that truncation has been happening to files for a long time. This was not identified because they performed entirely as normal, due to caching of non-truncated versions. The truncated file issue only became noticeable today, after an infrastructure upgrade. Server caches were cleared and the truncated versions became apparent. The cause of truncation is believed to have been a software bug that has now been resolved. We are currently working to restore truncated files - a process that may take from hours to days. If you are able to reupload any files that are not currently loading, please do that now. The issue is estimated to be affecting about ~0.2% of files.

Approximately 80% of truncated files have been recovered. The number of 415 errors have fallen significantly, though they remain elevated. To fully resolve the problem, the remaining truncated files need to be replaced with the original file. We are working on ways to recover those remaining files, though they are hard to obtain and it may take hours or days. For the fastest possible resolution, it is recommended that you re-upload the files. Such files will show in your account as having 0 bytes. Start at the following URL, to see a list of all files uploaded in the last 30 days: https://my.sirv.com/#/search/?created=30d Find any zero byte files by sorting the results by Size, Ascending. Another update will be posted here once we have news on the remaining files.

We are continuing to recover the small number of remaining truncated files. This is estimated to require a further 12-24 hours. If you have any files that are not being served, you can solve this straight away by reuploading them. We are creating a tool to generate a list of zero byte files. If you'd like a list of any zero byte files in your account, please submit a request at https://sirv.com/help/support/#support and we'll prepare a list once the tool is ready. Regardless, we will continue work to recover all files. If you upload files using Sirv's Autofetch feature, you can simply delete the zero byte files and they'll be re-fetched from your remote location upon the next request.

More files have been restored. The rate of 415 errors continues to fall. You can see a list of 415 errors on this page of your account: https://my.sirv.com/#/analytics/errors/415?from=20231206&to=20231206 You can upload files to solve any 415 errors in your files. We can provide a list of zero byte files if you need - please ask the Sirv support team via this form: https://sirv.com/help/support/#support We are continuing to restore as many files as possible.

The issue that caused the elevated rate of 415 errors has been resolved. The truncated files were caused by a software issue that wrote some files with zero bytes. As it was software, this caused the truncated files to be replicated across all versions of the master file. Sirv triplicates every file to protect against hardware issues (server or hard drive failures), however, the updated files passed all checks and faithfully triplicated the files. It existed without detection because cached versions of the files were used for processing and delivery. All systems worked perfectly, until yesterday's infrastructure upgrade when cached files were cleared. The root cause was solved by identifying and correcting the software issue that caused truncation. This was completed by 1500 UTC on 5 December. Then the task was to restore those files that had been truncated. Across the platform, approximately 0.2% of files had been affected. Of those files, we were able to restore between 95% and 100%, depending on the account. Any account which had less than 100% restoration is being sent a list of zero-byte files that should be restored by uploading the files again. Please look out for our email with a list of files. Please message us via the following URL if you need any help: https://sirv.com/help/support/#support A thorough investigation of the issue is underway, which will be reported in detail by email. Within the report will be a list of actions being taken to prevent any possible repeat of this issue, along with improvements and new features that are already being designed. We apologise to all customers who were impacted by this issue. It is the first time master data has been lost. We will learn every possible lesson from it and make Sirv even stronger and reliable than before.

More than 99% of the remaining truncated files have also now been restored. Any impacted user has been provided with an exact list of files that were impacted. There is no residual impact remaining from this issue. Classified as severity level 1: Critical impact.

From 0419 UTC, an elevated level of 429 errors were returned from the Sydney CDN location.

The issue was resolved at 1031 UTC. The cause was a server misconfiguration which made the Sydney CDN location see all requests coming from the same IP address, instead of the true client IP address. This triggered DDOS protection, to rate-limit the excessive number of requests from the same IP. 429 responses were returned for many of the requests. The issue appeared to have been an extremely short issue when the Sydney datacentre network became unavailable. That issue lasted approximately 1 minute only. However, the actual incident commenced immediately after our team had restored the Sydney CDN location, which is when the server misconfiguration was applied. The issue was solved once we discovered the misconfiguration and corrected it. Multiple actions are being taken to prevent this issue from recurring and to reduce the impact of similar but different issues in future: - Additional monitoring and alerting - New issue submission form - Context sensitive resolution advice - Wider internal alerting Only the Sydney CDN location was affected. Classified as severity level 3: Minor impact.

At approximately 1830 UTC on 22 November, we began an upgrade to the primary file database to a new, more powerful cluster. While the upgrade took place, API search queries were failing.

At 1035 UTC on 23 November, the upgrade was complete and searches resumed normal operation. There was no downtime and all systems functioned as normal, except that API search requests had been returning error responses. The issue was not identified because other API requests were returning as normal, so total API error responses were only slightly elevated and no alerts were triggered. The issue was identified after the event, after a report by a user prompted an investigation. To automatically identify such issues in future, we are evaluating ways to monitor the search API specifically. We will also be updating the Sirv WordPress plugin to show a clearly-described message in case of a failed search. Classified as severity level 4: Minimal impact.

The file management tools at my.sirv.com are loading slowly due to a failed server. File listing and file searching is taking from 2 to 20 seconds, whereas it normally takes 1 to 4 seconds. The failed server is being replaced and we expect speed to return to normal within 72 hours. This will be followed by a major upgrade of the entire control panel server cluster over the next 3 months.

We are waiting for provisioning of a new server to add to the cluster, then UI navigation speed will become much faster.

The new server was deployed in the evening of 14 November and gradually filled its caches. The control panel has now returned to its original fast performance. Additional servers are now being upgraded, so the control panel will load faster still over the coming 2 weeks. Classified as severity level 4: Minimal impact.

A dramatic increase in requests to the Washington D.C. CDN location occurred at 14:10 UTC on 23 October, causing a small proportion of requests to be returned slowly - greater than 2 seconds. The issue subsided over the next 3 hours, though average response time remained elevated. The issue returned the following day at 14: 32 UTC on 24 October, more severely than before causing 17% of requests to load slowly - greater than 2 seconds and some as long as 30 seconds. The issue impacted two of Sirv's 25 CDN locations.

The issue was resolved by 06:48 UTC on 25 October. Investigation points towards two causes - the spike was caused by Sirv's DNS provider routing Los Angeles POP requests to the Washington D.C. POP instead. This would have been fine in isolation but the datacentre appears to have throttled the bandwidth of the receiving servers, causing the slow responses. The investigation is ongoing, has not reached definitive conclusions yet and requires collaboration with our DNS and datacentre providers. Actions are already being taken to prevent any possible repeat of this issue. A fully detailed report will be provided here later.

The detailed investigation into this issue found two contributing factors. The first was an update to AWS Route53, which appears to have been implemented by AWS on 17 October. This change was not made by Sirv to its Route53 rules but a change within the Route53 service itself. The Los Angeles POP had slightly different routing logic to other POPs and the AWS update caused routing to behave differently, with traffic becoming routed to the Washington DC fallback. This went unnoticed as he higher load was still within its capacity and requests were returned successfully, so this went unnoticed as Washington DC was able to handle the additional load. The second contributing factor was higher than normal traffic during peak US trading hours on 23 and 24 October. The two factors combined caused the Washington DC hard disks to return requests significantly slower than normal. Since the issue occurred, we have taken multiple actions to prevent it repeating; to mitigate against similar possible incidents; and to accelerate resolution time in the event of an incident. Actions include: - New routing logic has been applied. - Washington DC capacity has been increased. - Reserve capacity has been increased in all other CDN locations. - New tolerance introduced for a CDN location to be removed from routing. - Additional disk monitoring, to help early preparation of capacity upgrades. - HDDs are being replaced with SSDs. - Updated SOP for our support team to investigate and respond to similar issues. Classified as severity level 3: Minor impact.

The earlier issue of high upload usage returned, with a small number of API upload failing.

The issue resolved itself within 9 minutes once upload volume normalised. During the period, approximately 98% of uploads succeeded and 2% failed. REST API uploads are limited per hour, to the amount specified on the Usage page of your Sirv account https://my.sirv.com/#/account/usage. If you use the REST API for uploading images, best-practice is to check the response for each upload request and if you receive an error response (400, 401, 429 etc.) then pause uploads and retry later. Refer to the docs here: https://sirv.com/help/articles/sirv-rest-api/#error-handling-guide Classified as severity level 4: Minimal impact.

A small number of REST API upload requests are failing due to very high load.

This issue resolved itself once the load on the upload API started falling. During the period, approximately 97% of uploads succeeded and 3% failed. REST API uploads are limited per hour, to the amount specified on the Usage page of your Sirv account https://my.sirv.com/#/account/usage. If you use the REST API for uploading images, best-practice is to check the response for each upload request and if you receive an error response (400, 401, 429 etc.) then pause uploads and retry later. Refer to the docs here: https://sirv.com/help/articles/sirv-rest-api/#error-handling-guide Classified as severity level 4: Minimal impact.

One of the Los Angeles CDN servers became unhealthy and started returning 4xx errors at 0000 UTC. It was removed from routing after 90 seconds and within a further 60 seconds, all traffic was being routed to healthy servers.

The issue was automatically resolved within 3 minutes. Classified as severity level 4: Minimal impact.

Some requests to the London POP of Sirv CDN, are returning errors due to high load. This POP serves UK and Ireland. We are working on a resolution.

The issue has been resolved and is now being monitored.

Some requests received by one of the London CDN servers failed due to a combination of local DNS resolution and pulsating network timeouts. The issue began at 1215 UTC and was resolved at 1332 UTC. During this period, 25% of requests from UK and Ireland returned a 404 or 500 error. The issue would have been solved sooner but an unrelated SSL certificate validation failure a few hours earlier blocked alerts from being received by our monitoring system. Changes have already been deployed to prevent this issue from recurring. The SSL certificate issue has been resolved; a second level of independent monitoring has been configured; an additional health automation is being trialled that has the potential to resolve such issues automatically. Classified as severity level 3: Minor impact.

The issue was resolved at 0755 UTC. At 0625 UTC, some requests to the Indore CDN POP which serves India started to return a 404 response (file not found), instead of a 200 response (OK). The issue occurred 14 hours after an update to all CDN servers worldwide. All POPs behaved healthily, including the India POP but the Indore POP hadn’t updated successfully and eventually started to return errors. Once the issue had been identified, it was resolved by re-deploying the update to the India POP. During the incident, 94% of requests returned an error. Requests to all other countries were returned as normal. To prevent this issue from recurring, a new automated deployment rule is being created for a part of the deployment process where there had been only a human check. An additional alerting metric is also being created, to identify a situation in which a deployment has failed. We apologise to all customers who were impacted by this incident and we are evaluating other changes that may help prevent any possible repeat of the issue. If a CDN issue like this occurs, one immediate solution within your control is to disable the CDN temporarily in your Sirv settings here: https://my.sirv.com/#/account/settings/domains Classified as severity level 2: Meaningful impact.