Localize outage history

Localize is impacted by the Cloudflare Global Network Experiencing Issues: https://www.cloudflarestatus.com/

We are continuing to work on a fix for this issue.

We rerouted the majority of traffic around Cloudflare, bringing the system back to operational, but degraded. There are signs that Cloudflare is coming back to fully operational. When they are back to stable, we will re-route traffic back through Cloudflare. Degraded performance is related to the intermittent nature of all third-party services. We will continue to watch Cloudflare's status and adjust as necessary. An update will be sent around 11 AM Eastern.

We've updated most of our statuses to operational. We are continuing to monitor the situation with Cloudflare and will post updates as they become available.

Cloudflare, our upstream CDN provider, has resolved their delivery issues. We have began routing traffic through Cloudflare once again, and will continue to monitor the situation.

The upstream provider issue has been resolved and functionality is back to normal for all users.

Localize is experiencing intermittent problems with saving new phrases. This may result in some phrases not being "found" on your website. This is a high priority incident, and we are actively working to resolve it.

We are deploying changes that we believe will alleviate the issue. Our team will continue to actively monitor the situation.

The root cause has been identified and the incident has been resolved. We’re seeing phrases being saved as normal.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

This incident only affects customers who use the auto-approve option when initializing Localize on their website. While newly found phrases are still being automatically moved to the Published bin, machine translations are not immediately being generated. Machine translations can still be manually ordered, or they will be auto-generated within 48 hours, they just won't be generated immediately. Our development team is actively looking into the cause of this issue.

We are continuing to investigate the issue.

The problem was identified and a fix was deployed. Thank you for your patience, the Localize team.

Between 7:04 AM ET and 7:49 AM ET (11:04-11:49 UTC) on August 16, Localize's infrastructure monitoring system notified our team of increased error rates affecting a group of servers responsible for (1) processing phrases ingested by the Localize Javascript SDK and (2) delivering translation dictionary files requested by the Localize Javascript SDK. During this 45 minute time period, the Localize Javascript SDK experienced an interruption of phrase ingestion and translation delivery that affected 8% of customers. We have taken steps to identify and rectify the underlying issue. If you have any questions or concerns, please contact your account manager or email [email protected].

We are investigating a networking and connectivity problem causing increased latency and connectivity issues between a subset of our servers and other key infrastructure components. Currently this is affecting a small subset of servers which are responsible for performing background tasks affecting certain services noted below. Service status: Phrase Ingestion -- Degraded Performance. Our "overflow" system which is part of a redundancy system relating to phrase ingestion is experiencing an outage, however, the primary system responsible for phrase ingestion remains unaffected. The overflow system affected by this incident is not expected to affect Phrase Ingestion. In the unlikely event that this changes, the status of the Phrase Ingestion component will be updated. Translation ordering -- partial outage. Orders placed on the dashboard will not be immediately sent to the third-party translation provider. You may safely place new orders, and existing orders are unaffected. Newly placed orders will be received, but will experience a delay in the order's submission to the third-party translation provider pending resolution. Realtime translation deployment - partial outage. Real-time translation deployments are paused and queued to be performed when the Worker Service becomes fully operational. Changes to phrases and translations made during this incident will be deployed and will reflect on your live website when the incident is resolved. File Import / Export - degraded performance. File import and export may experience degraded performance.

The issue has been identified and a fix is being implemented.

The underlying issue has been resolved and the affected systems are being monitored to confirm resolution of the affected system components.

The underlying issue has been resolved and the affected systems are being monitored to confirm resolution of the affected system components.

This incident and the underlying network connectivity issue has been resolved after switching to an alternate DNS provider.

We are monitoring service availability in response to an issue reported by an underlying provider (https://status.npmjs.org/). Localize services are currently not affected and remain available at the present time. We will update this incident if errors are observed, and will resolve the incident when the underlying provider reports that the issue is resolved.

The monitoring period has concluded and the incident has been resolved. Service availability was not affected..

We experienced 36 minutes of downtime on our help center at help.localizejs.com. This issue is now resolved and the help center is active. This issue was due to a provider outage. You can read more about the provider outage on the readme.io statuspage: http://status.readme.io/ Feel free to email [email protected] with any questions.

11:29 AM PDT: We are investigating reports of elevated error rates within the Localize platform. 11:35 AM PDT: The reports have been confirmed, and the issue has been resolved. ____ 1:51 PM PDT: Between 7:00 AM PDT and 7:15 AM PDT, customers began experienced errors that interrupted the availability of the Localize platform. After the issue began, customers may have continued to experience issues until 11:35 AM PDT. The issue has been resolved and the service is operating normally.

We are currently experiencing an issue preventing users from enrolling or up/downgrading subscriptions from within the Localize app when logged in. We have identified the issue and will provide an update here once the subscription issue is resolved. In the meantime, please do not hesitate to email [email protected] with any subscription questions or needs. We are happy to help with any plan enrollments, upgrades, downgrades, or cancellations as to not cause any delays.

In-app subscription change and plan enrollment link fixes have been deployed. We will be monitoring the issue over the next 24 hours and please don't hesitate to reach out to [email protected] with any questions or feedback.

The in-app subscription issue has been resolved and functionality is back to normal for all users. We will be providing a formal postmortem in the next 24 hours. Please do not hesitate to reach out to [email protected] with any questions.

A subscription bug was introduced on Friday, March 23rd preventing any users on a free trial plan from enrolling in a paid subscription from within the Localize app. This issue also prevented any customers currently on a paid plan from upgrading or downgrading subscriptions from within the Localize app. This issue happened during a pricing and plan change deployment for future customers and occured due to an incomplete front end testing setup in place to ensure in-app plan enrollment actions worked as normal. The issue impacted users on a free trial trying to move to a paid subscription and current paid customers attempting to upgrade or downgrade their plan subscription. Moving forward, we will be taking steps to mitigate the chances of such issues from occurring again. Additional QA processes will be enacted across code deploys; especially ones that directly impact end user interactions and additional integration tests will be implemented that ensure specific high traffic paths are tested on a hourly and code change basis. Please do not hesitate to reach out to [email protected] with any questions.

We are currently experiencing issues with our email system causing intermittent failed delivery of account notification emails. The issue is being addressed and we will provide an update here as soon as the email system is back to normal. Please don't hesitate to reach out to [email protected] in the meantime with any questions.

The backend email system is now triggering account notification emails without delay. We will be monitoring the issue over the next 24 and please don't hesitate to reach out to [email protected] with any questions or feedback.

The backend email issue resulting in intermittent failed delivery of account notification emails has been resolved. All emails are now triggering without delay. Please do not hesitate to reach out to [email protected] with any questions.

We are currently investigating issues with archived (deprecated) phrases. Phrases get deprecated after 14 days of not being detected (visited) on your website. Typically, when your end-user visits the page the deprecated phrase is automatically reactivated and moved back into Published and the translation will render on your application. We have noticed a minor bug where deprecated phrases are not being moved back to active once the page is visited. Please, contact [email protected] if you have any questions or need any assistance. When the deprecation issue is resolved, we will provide an update here. We appreciate your patience.

We are currently monitoring the situation closely. We will provide an update as soon as the issue is completely resolved. Thank you for your patience and understanding. Please, contact [email protected] if you have any questions or need any assistance.

We have mitigated the problem and are closely monitoring the situation.

We continue to monitor the status of our site and all related services. If you are experiencing problems, please email [email protected] for assistance.

We’ve identified and addressed the cause of the issues related to deprecated phrases. The incident has now been fully resolved. Issue: Deprecated phrases were not being reactivated when the application is reloaded. Status: Resolved. The Deprecated phases functionality is now working as intended. Localize archives (deprecates) content that has not been detected on your website after 14 days, to optimize the load speed of our library. If that deprecated content is detected again on your application, it will be restored automatically. The default setting for this feature is enabled. If you turned this feature off and would like it re-enabled, you can turn deprecation back on in your project settings under "archive unused phrases" in your project settings.

We're currently experiencing delays detecting source phrases, this includes reactivating deprecated phrases. Our team is currently investigating and are working on getting this resolved as quickly as possible. Thank you for your patience.

We're seeing improvement with phrases being detected. However, the team will continue to investigate any delays and provide updates as soon as it's resolved.

Thanks for your patience. We're still experiencing delays detecting source phrases. Our team is working on resolving this as soon as possible.

Thanks for your patience. We're no longer seeing delays detecting source phrases but are closely monitoring these functions. Please contact [email protected] if you have any questions.

Our escalation team is currently working with engineering to resolve the issues regarding delays in detecting source phrases. Please contact [email protected] if you are being affected by this delay. A member of the escalation team will contact you after receiving your request.

New phrase detection has been resolved, we are closely monitoring this system. Please contact [email protected] if you have any questions.

This incident has been resolved.

Our team is performing unplanned maintenance to proactively avert future system latency. Updates to your account and availability of core services remain operational. There is delay in certain services, including: - Delayed file import / exports - Delayed deployment of updated translations - Delayed email notifications - Other background system operations that may delay reporting These systems will not fail, but will resume operation and complete operations in a delayed timeframe. The operations are queued and will run at the end of the maintenance window. We expect these delayed systems to be back to normal operation by 6:30pm PT. Please contact [email protected] if there is data that requires immediate update or download. Please email [email protected] if you see degraded performance outside of what is listed above, or in cases that impact your systems in a manner requiring immediate resolution prior to the expected resolution time. We will be performing additional database maintenance gradually over the next week and will notify users at least 24 hours prior to such maintenance, excluding critical unplanned incidents, which we do not anticipate. Our team of globally located database administrators continue to monitor all systems 24/7 across all timezones. Should you have any questions, please don't hesitate to contact support.

Delayed background systems have returned to normal operation as of 1:22:00 UTC on Sunday, October 1, 2017. We will announce a 3 day maintenance window within the next 14 days, during which our infrastructure engineers will be traveling to Localize Headquarters in California to run diagnostics and modify certain system to improve the speed and efficiency of core systems. This will not have an impact on Localize systems. Please email [email protected] if you have concerns or questions prior to our post detailing the scheduled system diagnostics. We appreciate your continued patience.

Some of our application servers are experiencing increased latency. Our engineering team is investigating. Translations are still rendering for your end-users and no data has been lost.

Our engineering team has been diligently working with our database provider to resolve the issue. Translations are still rendering for your end-users and no data has been lost.

The issues with latency has been resolved, but we're closely monitoring our systems in the event of any degraded service. Please report any issues to [email protected].

The issues with latency has been resolved, but we're closely monitoring our systems in the event of any degraded service. As part of our discovery process for this issue, we identified areas of improvement and are performing updates and maintenance. Localize will be performing ongoing maintenance this week, starting on October 2, 2017. During this maintenance window there may be sporadic service unavailability. We will provide updates as necessary.

Localizejs.com may not be available. Translations are still rendering for our customers.

Amazon S3 is reporting increased error rates. We're monitoring the status of Amazon S3. Localizejs.com is currently up and translations are rendering for our customers.

Amazon appears to have resolved their S3 issues, which means you can access your Localize projects without delay.

Update from AWS about their outage earlier today, September 14, 2017. Outage period:  Detailed log:  During this period Localize may have been unaccessible. This meant our customers were not able to access their Localize projects. However, existing translations continued to render on our customer's applications (unless they were also directly affected by the AWS S3 outage.) We apologize for any impact this may have caused for our customers.

We experienced an outage with our emails. This has been resolved and all emails have since gone out, which maybe result in multiple emails at once. We apologize for the delay and any confusion.

We are currently investigating this issue.

Localize and our application is back up. The outage was isolated to our website; the deliverability of translations to Localize customers translating their websites or web applications using our service remained available.

Some accounts are being automatically logged out after a certain period of time.

Resolved: Some accounts are being automatically logged out after a certain period of time.

We apologize for any issues you may experience with Localize. We are currently investigating issues involving slow performance, machine translations, emails, changes in phrase states. Services remain degraded due to a provider issue with AWS.

Services are starting to recover as a result of Amazon Web Services stabilizing: https://status.aws.amazon.com. Emails are being sent out as expected. We expect other services to recover quickly. We apologize for the interruption to service and will continue to monitor our systems to ensure services remain stable.

Following the stabilization of AWS service, all Localize services have been fully restored and are stable. Thanks for your patience.