16 Liquidplanner incidents · November 2017 — official updates, affected components, duration and resolution details.
A fix has been implemented and we are monitoring the results.
Our team is investigating reports of a service issue. We'll update this page as soon as we have more information to share. Thanks for your patience.
This incident has been resolved.
Our website and support hub may not be accessible for some users. We are currently investigating the issue. Access your workspace here: Classic LiquidPlanner: https://app.liquidplanner.com/login. New LiquidPlanner: https://next.liquidplanner.com/login Until self help is up and running again, feel free to contact [email protected]. For help purchasing, please contact [email protected]. We are investigating and will post updates here when we have more information to share. This issue does not affect workspace performance or access; our apps are working as expected.
This incident has been resolved.
Our provider reports they have completed a series of successful changes and will monitor traffic closely over the weekend. We will do the same. Our next update will be on Monday, February 27.
The email processing provider for LiquidPlanner Classic is experiencing an issue affecting inbound and outbound email traffic for workspaces that may be causing delays and intermittent failures. To get around this, please log into your workspace in order to: View comments Comment on plan items View the activity stream on the Home tab for items you own or follow Create plan items All other aspects of LiquidPlanner are functioning normally and there is no impact to customer data. If you are trying to invite a new member to the workspace, please contact [email protected]. We'll update this page when the provider confirms the problem is resolved.
Our service provider reported their email services are fully operational again and this incident is resolved.
The issue our provider is experiencing remains active and they're working toward resolution.
Our email processing provider is currently experiencing an issue which affects inbound email traffic to your workspace. This means if you reply to a notification from your personal email client, it may not appear in the workspace as expected. Tasks created via email are not reaching the workspace in a timely manner. You may receive an error stating that there was an error processing your email or task. All other aspects of LiquidPlanner Classic are functioning normally. There is no impact to customer data. Please log in to your workspace in order to: View comments Comment on plan items View the activity stream on the Home tab for items you own or follow Create plan items We'll update you as we know more. Thank you for your patience.
Inbound email traffic to LiquidPlanner Classic workspaces is processing normally now. Customers are able to: View and reply to comment notifications from within their personal email client. View and add comments to recent change notifications from within their personal email client. Create tasks via email. It may take some time for systems to catch up. We will monitor this issue until delivery is complete.
This incident has been resolved.
We're starting to see email notifications going out again. We're monitoring the queue as the remaining notifications are sent.
We have nothing to report at this time. Notifications are still delayed.
Our notification queue is now empty, and notifications are functioning normally. This incident has been resolved.
Our email processing provider is currently experiencing an issue which affects both inbound and outbound notifications. This means you will not receive an email when someone comments at you, or when there is a change to an item you own or follow. If you reply to a notification from your personal email client, it will not appear in the workspace as expected. Tasks created via email are not reaching the workspace in a timely manner. All other aspects of LiquidPlanner are functioning normally. There is no impact to customer data. Please log in to your workspace in order to: View comments Comment on plan items View the activity stream on the Home tab for items you own or follow Create plan items If you are trying to invite a new member to the workspace, please contact [email protected]. We'll update you as we know more. At a minimum, we'll post something here every 30 minutes. Thank you for your patience.
AWS confirms that the incident is resolved and systems are operating normally.
Our service provider announced the issue has been resolved and as of 8:14am Pacific time, access has been restored. We'll continue monitoring and update this page as needed.
There is an issue with AWS servers affecting internet connectivity in Northern California and Oregon preventing some users from accessing LiquidPlanner and other software platforms. We'll monitor the issue with our provider and update here as we learn more. Thank you for your patience.
Our team is investigating reports of a service issue for LiquidPlanner Classic. We'll update this page as soon as we have more information to share. Thanks for your patience.
Our server provider, AWS, is investigating an issue affecting network traffic that may affect some actions in our platforms. -In our Classic version, you may see a slow down in notifications. -In the New LiquidPlanner live scheduling updates may be delayed and billing changes may not sync right away. If you have concerns about billing updates during this time, please contact [email protected] for assistance. We are monitoring the issue with our provider and will post an update here when it is resolved.
This incident has been resolved. Our provider confirms that service is operating normally.
Our email service provider continues to experience issues which prevent inbound notifications from reaching your workspace. All other aspects of LiquidPlanner are functioning normally, and there is no impact to your workspace data. Until this issue is resolved teams who rely on email to create or update plan items and send in comments will need to log in to the workspace to complete these actions. We apologize for the disruption to your normal workflow and thank you for your patience as we wait for it to be resolved.
Our email processing provider has identified the issue and is working to resolve it.
Our email processing provider is currently experiencing an issue which is affecting inbound notifications. This means if you reply to a notification from your personal email client, it will not appear in the workspace as expected. Tasks created via email are not reaching the workspace. All other aspects of LiquidPlanner are functioning normally. There is no impact to customer data. Please log in to your workspace in order to: View comments Comment on plan items View the activity stream on the Home tab for items you own or follow Create plan items We'll update you as we know more. At a minimum, we'll post something here every 30 minutes. Thank you for your patience.
Our systems show that inbound email has started flowing again. We expect that you will be able to successfully send in comments, update items, and add tasks via email from this point. We'll continue monitoring until our provider officially lets us know the problem is resolved, and then we'll close out the incident. Please contact [email protected] if you need further assistance. Thanks for your patience.
Notifications are processing normally now. Customers are able to: View and reply to comment notifications from within their personal email client. View and add comments to recent change notifications from within their personal email client. Create tasks via email. It may take some time for systems to catch up. We'll monitor until delivery is complete. Thanks for your patience. If you need additional help, please email [email protected].
We are continuing to monitor for any further issues.
Our email service provider resolved the issue and systems have returned to normal.
We've made temporary mitigations to reduce the database load, and the site's performance is getting closer to normal levels. We are still investigating the cause. Thank you for your patience while we are continuing to work on the resolution.
Some users may be experiencing trouble logging into LiquidPlanner and navigating around LiquidPlanner. Our team is currently investigating issues related to slow page loads and intermittent timeouts. We'll update you again in 30 minutes. If we have we have a better idea about the resolution time before then, we'll add it here immediately.
We determined that the database came under heavy load and while it was catching up on requests, the site performance degraded for some users. There was no impact to customer data during this period. The application is accessible and operating as expected at this time.
Our team investigated the incident and determined that some of the recurring automated tuning operations performed by the PostgreSQL database platform created a bottleneck in the LiquidPlanner operating environment. Requests backed up and some users experienced slow operations and errors. We've since added additional database I/O capacity and adjusted the automated tuning operations to prevent this issue in the future. Thanks for your continuing support and patience.
Our team is investigating isolated customer reports about slow performance. At this time systems are operational and we're closely monitoring the situation. We'll update this page again as soon as we have more information to share. Thanks for your patience.
Degraded performance in a back up system resulted in isolated slow-downs in service. That infrastructure has stabilized and this incident is resolved.
Systems are stable and there have been no further reports.
We are continuing to investigate this issue.
Some customers have reported issues with email integration affecting the ability to comment or create tasks via email. We are working with our provider to understand the timing for resolution. This is related to notifications only, all other aspects of LiquidPlanner are functioning normally. There's no impact to customer data. We'll update you as we know more. Thank you for your patience.
Our email processing provider confirms services are operating as expected. Email to your workspace has resumed. Items previously sent have been processed.
Our email processing provider experienced an issue which affected both inbound and outbound email processing. They have implemented a fix and emails are processing again. While the service was disrupted, you would not have received a notification if someone commented at you, or if there was a change to an item you own or follow. If you replied to a previous notification from your personal email client, it was delayed and did not appear in the workspace as expected. Tasks were not created via email. There is no need to resend emails, the system received them and will complete the send cycle shortly. If you invited a new member to the workspace, please contact [email protected] if that invitation is not received.
This incident has been resolved.
A fix has been implemented and we are monitoring the results.
Wir haben festgestellt, dass ein bestimmtes Anforderungsmuster mit schlechter Leistung alle verfügbaren Ressourcen eines Webservers verbraucht. Dies veranlasste unseren Gesundheitscheck, diesen Wirt aus dem Lastausgleichspool zu ziehen. Der erhöhte Traffic zu anderen Hosts führte dazu, dass sie ebenfalls überwältigt wurden, was zu einem Brown-Out der Anwendung führte. Unser Team handelte schnell und der Service wurde in zehn Minuten wiederhergestellt. Es gab keine Auswirkungen auf die Kundendaten in diesem Zeitraum. Die Anwendung ist zugänglich und funktioniert zu diesem Zeitpunkt wie erwartet.
Unser Team untersucht ein Serviceproblem. Einige Kunden berichten, dass sie "Site unter schwerer Last" sehen Wir werden diese Seite erneut aktualisieren, sobald wir weitere Informationen zu teilen haben. Danke für deine Geduld.
Normalbetrieb. Wir sind Überwachungssysteme.
Automatisch aus der offiziellen Störungsmeldung übersetzt.
To minimize further disruption, we moved our planned update outside of US business hours. Changes will release at 6pm Pacific Daylight time. We apologize for the performance issues today and thank you for your patience while we determine the best course of action to rectify the situation.
The LiquidPlanner application is accessible. Some users may find it slow to reschedule or load pages. We'll address these issues in a release going out at 6pm Pacific Daylight time today, April 23, 2018. Thank you for your continued patience.
6pm - 7pm Pacific Daylight time, LiquidPlanner is undergoing maintenance to improve performance. The application is available, but you may experience delays in scheduling and page loads while changes roll out across the system.
The LiquidPlanner application is working as expected following the update. We are continuing to monitor the systems closely.
After thorough investigation, we determined that we hit some unforeseen I/O limits with our database. The combination of a cache flush due to a restart, recalculation of some statistics, and peak load resulted in brownouts and occasional errors in our application. To remediate this type of resource exhaustion, we've purchased additional I/O capacity from AWS. As always, we'll continue to monitor and tune our database configuration in response to the large amounts of data that we process. There was no impact to customer data during this period. The application is accessible and operating as expected at this time.
We've identified the source of the performance issues, and will make changes later this afternoon (Pacific time) to correct the problem. There may be intermittent performance issues in the meantime (slow page loads and scheduling). We're currently analyzing site traffic history to determine the optimum time to make updates. We'll post again prior to updating to make you aware of the time we expect to make changes. The app will be accessible during the maintenance period.
We continue to see improved performance although some monitors still show slower response times than normal. All areas of the application are accessible. We will update this page until resolved.
Engineers have made changes to reduce load, and application performance is back to normal levels. We're still monitoring the situation, and will provide a more thorough update soon. We apologize for the inconvenience and appreciate your patience.
Some customers are seeing a message that the website is under heavy load. We're tracking down the cause.
We are experience slow load times. The app is accessible and performance is improving. Our team is monitoring continuously and working to resolve.
We are experiencing a service disruption. Our team was notified and they are investigating the issue. This page will be updated again as soon as we have more information about impact and scope.
At 2pm Pacific Daylight time our team will release an update to address performance issues. There may be a degradation in service while changes roll out until the system stabilizes.
Performance is back to normal levels and the site is responding as expected now. We're still investigating the cause. Thank you for your patience this morning while we sorted it out.
We determined that a particular request pattern with poor performance consumed all of a webserver's available resources. This caused our health check to pull that host out of the load-balancing pool. The increased traffic to other hosts caused them to become overwhelmed as well, resulting in a brown-out of the entire application. There was no impact to customer data during this period. The application is accessible and operating as expected at this time.
Some users may be experiencing trouble logging into LiquidPlanner. Our team is currently investigating issues related to slow page loads and intermittent timeouts. We'll update you again in 30 minutes. If we have we have a better idea about the resolution time before then, we'll add it here immediately.
We completed the roll out and consider the issue resolved, but we'll maintain monitoring status until our development team gives the all clear.
This incident has been resolved.
Some people are encountering a 500 error when accessing a member's profile from the people tab. Clicking the back button returns you to the workspace. This is affecting member invitations. New members may encounter 500 error after they accept the invitation. We are investigating and will post updates here when we have more information to share. All other aspects of the app are working as expected.
We've identified the issue and are deploying a solution. We'll update this page again at 8:45am Pacific time.
Some customers reported issues with email delivery (sending invitations, password reset, comments, recent change notifications, daily digest). Mail had been queued on our local servers following the release this weekend (7am Pacific, Saturday, November 4). Email notifications are flowing as expected now. We apologize for the inconvenience. You will receive all mail that had been generated, and you do not need to resend anything that was initiated while the mail was in queue.