Innovo 장애 기록

We're currently investigating an issue with the Google Maps display in the Customer Portal. It appears to be caused by a broken Google library. This is only affecting Route — all other products are working as expected. We'll keep you posted on our progress.

We've identified the issue and have a fix. We'll be pushing out the update shortly.

A fix has been implemented and we're monitoring the results. Route should be fully functional again. Thanks for your patience!

This incident has been resolved. We really appreciate your patience and ongoing support!

It appears we are currently experiencing an outage. We will investigate and let you know as soon as we know more.

We are continuing to investigate this issue.

We've pointed our DNS to our backup server so Route and Deliver Web should be coming back online. We are still investigating what caused the outage and will keep you posted.

The DNS change worked for a bit and then we went down again. We have been on the phone with our database provider and AWS all morning to try to get to the bottom of things. We will keep you posted as we learn more.

We have identified the issue to be with MongoDB (our database provider) and are working with their engineers on implementing auto scaling of our cluster to mitigate the issue. This seems to be working for the moment. We will continue to monitor throughout the day. Thank you all for your patience. All of your signatures, photos, comments, etc. were being transmitted to Eclipse during the outage just as normal. The outage was only on the portal side.

Unfortunately we are still experiencing issues and connections are intermittent. We will keep you updated.

We have finally identified root cause of the outage. In January we made a change to turn on Deliver alerts for everyone by default. Because of this change, this is now our third outage so far this year. This is more than we've experienced in past years. The two previous times we put some new indexes in place and thought the issue was resolved. Unfortunately that was not the case. Today the DB server pegged at 100% CPU and when that happens the application server starts dropping connections and then struggles to reconnect to get a connection to the database. Luckily the connections still went through, however slow so theoretically the portal should still show the stops as delivered, skipped, etc. And like I said previously all signatures, photos, and comments were successfully updated to Eclipse. To fix this today, we increased the RAM on our DB server based on the recommendations from Mongo. On Sunday we will also be creating additional indexes to mitigate any further slowness. In addition, we will be working with Mongo next week to implement further performance recommendations. Again, I am really sorry this happened today and we do really appreciate your patience. We will continue to keep you updated as we learn more.

We had a great day yesterday! During our peak time (10am-2pm EDT), we averaged about 350 transactions a second and 2,800+ reads per second. Average CPU usage remained steady at 20-40%. In addition to the increase of RAM on Friday, we rebuilt indexes over the weekend which has really seemed to help. We will continue to monitor numbers this week and if we find anything else that needs to be changed to keep up with the load, we will do so. Thank you for your continued patience - we really appreciate you!

We are just receiving word that Route and Deliver Web are currently not functioning. We are investigating now and will provide status updates.

We have identified the issue and pushed traffic for now to our backup server. This is a DNS change so it might take a bit for everyone to come back online. You may have to close and re-open your browser to see activity again. Just a reminder, the Deliver app was fully functional during this outage. Unfortunately this was the same issue as what happened last time. We will be making additional adjustments to fix. We will continue to monitor throughout the day and provide additional updates as needed. Thank you so much for your patience. We know how critical these services are and will work to ensure this does not happen again.

The system is down again. We are investigating.

We have escalated the issue with our provider and they have responded with some recommendations we can put in place immediately. I will keep updating the status as we know more.

We will continue to monitor traffic today and make sure most systems are up and running. Currently the system is back up. We will be working off-hours tonight with our provider to make more permanent changes.

Closing

We are currently investigating the issue and will provide status updates.

We have identified and fixed the issue and all services should be coming back online. This will be a rolling process as we had to make a DNS change. We have not yet found the root cause so we will continue to monitor the systems throughout the day and provide additional updates as needed. Unfortunately any breadcrumb data that was accumulated in the Deliver app during the time of the outage will not show in Deliver Web. And you may see manifests that were completed during the outage not reflect a completed status in Deliver Web. During the time of the outage, all communication with Eclipse was intact so signatures, photos, comments, etc. were updated as normal. We really appreciate your patience and will let you know as soon as we know more.

We still haven't found the actual root cause for the outage yesterday but we did find there was a spike in IOPS which is the standard unit of measurement for the maximum number of read/writes per second. This could have been caused by the portal change implemented this past weekend to turn on Deliver alerts by default for everyone or it could have been caused by a bad query. We are just not sure yet. There were also out of memory issues so we increased the heap size to account for that. We will monitor the IOPS today to see if that needs to be increased as well. Again, we really appreciate everyone's patience yesterday and apologize for the inconvenience. Fortunately operations were only affected for a short time.

This incident has been resolved. We created a couple indexes to address some slower queries over the weekend and believe that should address this issue moving forward. We appreciate everyone's patience!!

We are currently investigating reports of Route and Deliver being down. We'll keep you updated as we learn more.

Just a quick update - This seems like the same issue we experienced yesterday and we are currently working with AWS to address the issue. Drivers should still be able to download manifests, capture signatures, and upload photos however it may be a little slower than normal. We will update you as soon as we know more!

We are still working with AWS on the root cause however our fail over server is now online and seems to be holding strong so far. Deliver Web and Route are back online. We'll send out another update when we have completely resolved the issue. Thank you for your patience!!

We are continuing to monitor for any further issues.

Our servers experienced a bottleneck today at peak run time (9am-2pm EDT), similar to what happened yesterday. As soon as we noticed the bottleneck, we pushed all traffic to our failover server and recovered within five minutes. Unfortunately we are still not sure what is causing these bottlenecks so for now we will be increasing the compute power on the production server to handle the network spikes and adjusting the load balancing configuration to fine tune the instance. We will keep you updated with our progress and findings. Thank you for your continued patience!

The increase of compute power along with adjustments to our load balancer configuration has resolved the issue. Everything has been running smoothly for the past week.

We are currently investigation reports of Route and Deliver being down. We'll keep you updated as we learn more.

Just a quick update - we are currently working with AWS to address the issue. Looks like we went down at 9:25am Eastern Time today. Drivers should still be able to download manifests, capture signatures, and upload photos however it may be a little slower than normal. We will update you as soon as we know more!

We are still working with AWS on the root cause however our fail over server is now online and seems to be holding strong so far. Deliver Web and Route are back online. There was some data loss while manifests were completed during the outage however if your manifest is still on the device and not yet been completed, all bread crumbs from those routes will eventually be sent to our platform. We'll send out another update when we have completely resolved the issue.

We are continuing to monitor for any further issues.

Thank you all for your patience today! We will be deploying a fix tonight and pointing all traffic back to our primary server. As mentioned above, any manifest completed during the outage will not show breadcrumbs for the remainder of that route. Any manifest that was in progress and completed after the outage should show as completed with all breadcrumb data intact. The data impact only pertains to Deliver Web. All Eclipse processes were functioning as normal with signatures and photos successfully uploaded as stops were completed. If you have any questions about the data, please let us know. We are really sorry this happened and appreciate everyone's support!

We are getting reports that Deliver Web and Route are not displaying any data this morning. Users can still load manifests and deliver as normal through the Deliver app. We are investigating ASAP and will send out updates as we have them. Sorry for the inconvenience this morning. We understand the effect to your business operations.

We are in communication with AWS (Amazon) to see if they can help us identify the issue.

Just wanted to provide an update. We are still working with AWS on identifying the cause. We do not yet have an ETA on when this will be resolved. Please know we are doing everything we can to get the site back up and running ASAP. We really appreciate your continued support and patience!

We are working diligently to resolve the issue. We did identify the root cause of the failure. A scheduled report that runs nightly kept failing to run and locked up our storage nodes. Storage nodes connect to the database to perform operations, such as update, delete, insert, etc. Usually this is something that resolves on its own however due to a large spike in web traffic at the same time, our load balancer struggled to catch up. This ended up causing a bottleneck and the web traffic to eventually time out. This will resolve on its own when the web traffic subsides however we are still working with AWS to get it resolved ASAP. This is something that AWS should be able to help with but it is taking longer than we thought. We will keep you updated and let you know as soon as we know more. We will also try to identify the root cause of the report failure once our system is back online.

Good news! It looks like Route and Deliver Web are back online. We did have to update the DNS so it may take a bit for those changes to propagate. The data being displayed currently in Deliver Web Live does not look current. We are in the process of determining if any data was lost and if not, how to restore. We will send you another update when we have information on that. For all of you that need to start routing for tomorrow, you should be able to do that now.

Spoke too soon. We were up for a bit and then went down again. We are working through this as best we can and right now I have no update other than my previous update was incorrect. We will update you as soon as we know more. I so apologize for the issues today. Please know we are all doing the best we can to get you up and running again.

We are FINALLY back up! We have identified and fixed the issue. There should be no more disruptions to Route and Deliver Web operations moving forward. We are still investigating if there was any data lost and will send out an update as soon as we know more. You may notice the data listed in the Live view is old. This will clear out tonight. Thank you SO much for your patience and support. It's been a day.

As for the data... Signatures, photos, comments, etc. were all updated to Eclipse so that is good news! Unfortunately, any stop or manifest that was completed this morning AFTER the system went down will not display in their associated tables in Deliver Web. We do apologize for that, and for everything that happened today. We always learn something when these types of situations happen, and those learnings make us stronger as a team moving forward. Again, we are so sorry for the inconvenience this has caused you today. Thank you for your continued partnership and your patience.

We upgraded our AWS instance yesterday and it is causing response times to be slow or appear unresponsive this morning. We are in the process of pointing traffic back to the old server and will let you know when it's complete. This should not be affecting the Deliver app, only the portal. And most notably Deliver Web and Route. We are so sorry for the inconvenience this morning and should have you back up and running soon.

We have pointed traffic back to the old server and are standing up new instances to keep up with the traffic. I see systems slowly coming back online. I will send out another update as soon as things appear more stable.

For Route users, you may also notice that scheduled orders are not appearing on the Schedule page. This will also be resolved once the server is fully functional. All of your scheduled orders are still scheduled.

We are still seeing very slow response times. Our database is 99% utilized and that is what is causing the slowness to continue. We are working with our database provider and should have an update soon. Thank you for your continued patience! I know this is causing a lot of confusion and frustration and we are working to get it resolved as soon as possible.

It looks like data is showing up again in the Live page as well as in Route. We will continue to monitor today and will plan for another update to AWS in the coming weeks once we have identified the root cause of what happened today. Please keep us updated throughout the day if you see any more issues. Thank you so much for your patience as we worked through the issues. We appreciate all of your support!

This incident has been resolved for now. We will be investigating root cause and deploying the AWS update in the next few weeks.

The Google Maps Platform is experiencing elevated error rates and increased latency for the Directions and Route APIs. This is affecting how our maps display in Deliver, Deliver Web, and Route and are in the process of investigating other potential issues. You can check the status of the Google Maps Platform here - https://status.cloud.google.com/maps-platform/. We will send out another update when we know more.

The issue seems to be mostly resolved. According to Google's status page (https://status.cloud.google.com/maps-platform/) they are still down however all of our apps seem to be working as normal. We'll continue to monitor and provide additional status updates as needed.

As you may be aware, a critical vulnerability (CVE-2021-44228) was recently discovered in Log4j, a java library that is used by millions of software applications including Innovo. This vulnerability affects versions 2.0 to 2.14.1. We are currently using version 1 which is not affected. No updates to our service are needed at this time. Please see https://nvd.nist.gov/vuln/detail/CVE-2021-44228 for more information. If you have any questions, please reach out to us at support@goinnovo.com.

We continue to have issues with portal email invites and password resets and apologize for the inconvenience. We understand how frustrating this can be. We are no longer working with Microsoft on this issue and are now working with AWS to come up with a permanent solution. If you need to invite anyone to an Innovo app, you can use the Set Password feature and let them know what their password is. Once the issue has been resolved, you can send them a password reset from the portal or they can use the Forgot Password on the login screen of the app to reset. If you need to reset a users password that is already invited, we recommend you delete the user and re-invite them using the Set Password feature. We will keep you posted throughout the day with status updates.

We believe the email issue has been resolved. All portal emails will now be coming from no-reply@goinnovo.com. We will continue to monitor this throughout the day but if you run into any issues, please reach out to support@goinnovo.com. Thank you for your patience!

This incident has been resolved. All portal emails will come from no-reply@goinnovo.com. Thank you for your patience and support while we worked through it. If you have any questions, please let us know at support@goinnovo.com. Have a great night!

We are having another issue today sending out email invites and password resets from the portal. We are working with Microsoft on the issue and will send out an update as soon as we know more.

We are continuing to investigate this issue.

We are continuing to investigate the issue. If you need to invite anyone to an Innovo app, you can use the Set Password feature and let them know what their password is. Once the issue has been resolved, you can send them a password reset from the portal or they can use the Forgot Password on the login screen of the app to reset.

A fix has been implemented and we are monitoring the results. Email invites and password resets are currently working.

It looks like email invites and password resets are not working again. We will keep you posted as we investigate the issue.

Email invites and password resets are currently working. We will continue to monitor the issue throughout the day. Thank you for your patience!

This issue has been resolved. if you continue to have issues sending out email invites and password resets, please let us know at support@goinnovo.com. We really appreciate your patience!

We are currently experiencing an issue sending out invite emails and password resets. We are working with Microsoft on the issue and will post a status update as soon as we know more. If you need to invite anyone to an Innovo app, you can use the Set Password feature and let them know what their password is. Once the issue has been resolved, you can send them a password reset from the portal or they can use the Forgot Password on the login screen of the app to reset. Sorry for the inconvenience!

The issue has been resolved. We had to change the email the portal uses so all invite emails and password resets will now come from noreply@goinnovo.com. if you see any issues moving forward, please contact support@goinnovo.com.

We are currently investigating the issue.

The portal is back up and you can now log back into apps. Sorry for the inconvenience.

The portal issue today was due to GitHub returning a 500 error. We use GitHub for push notifications for OE Touch.

We will continue to monitor the issue and make sure it is fully resolved.

The issue has been resolved. Total downtime was about 10 min.

We are experiencing issues with Deliver Web and Route. We will provide updates as we investigate and work to resolve them.

The issue has been identified and we are working on a resolution.

A fix has been implemented and we are monitoring the results. It may take a short period for all Deliver Web and Route sites to be back online.

Deliver Web and Route are back online.

We are currently investigating this issue.

The Innovo Customer Portal is down as well as the ability to log in to any Innovo app. If you are currently logged into an Innovo app, the apps should work as normal.

The Innovo Customer Portal is back online. It was a DoS (Denial of Service) attack. The good news is our server can now withstand this style of attack moving forward. Thank you for your patience and we are sorry for the inconvenience.

This incident has been resolved.

Our DNS provider is currently experiencing an outage which is effecting our services. We are currently investigating this issue.

Our service provider has restored DNS service. All of our services are now operational.

Our phone service is currently down. We are working on resolving the issue. If you need to contact us, please send an email to support@goinnovo.com.

Our phone system is back up!

We are experiencing intermittent outages and are currently investigating the issue. All users currently logged into Innovo apps should not be experiencing the issue.

The Portal is back online. We will continue to monitor to make sure all issues are resolved.

Intermittent portal issues appear to be resolved.

We are currently investigating this issue.

A few weeks ago we sent out a communication regarding our cloud service provider and how their infrastructure changes were adversely affecting our services. At that time, we created a backup production environment using a new cloud provider. Today, our current provider is again experiencing issues so we have decided to officially move our service to that new cloud provider. Unfortunately that means Deliver will be down for the remainder of today. However, starting tomorrow Deliver will be running on a new cloud provider and we are hoping for better performance and less interruptions. We really appreciate your patience as we make this change.

This incident has been resolved.