Locomote outage history

In-App Support Chat Temporarily Unavailable

## What happened? A recent platform update caused our in-app support chat to stop loading for logged-in users. Customers attempting to use the chat widget would not have seen it appear. ## Who was affected? Any logged-in user for companies serviced directly by Webjet Business Travel with live chat support enabled. ## How long was it down? The issue was introduced approximately 11:12 PM AEDT on 26 February and resolved 9:54 AM AEDT on 27 February. ## What did we do? We identified the issue and rolled back the change. The support chat widget is now fully operational. ## Was any data lost? No. No customer data was affected. Chat history and account information remain intact. ## What if I need help? If you were unable to reach us via chat during this window and still need assistance, please contact our support team directly via the live chat widget.

The issue has been identified and a fix is being worked on.

Workflow loading issue has been resolved. We're investigating a secondary issue with reporting access for admins.

Reporting auth issue is now also resolved, we are monitoring for any further issues.

Reporting auth issue is now also resolved, we are monitoring for any further issues.

We apologise for any disruption this caused by our recent partial outage, which affected workflow and report loading. The root cause of the issue was preparatory work recently put in place for planned future upgrades, which revealed a configuration mismatch between our production and development/test environments that allowed this issue to bypass both manual and automated testing. We’ve since adjusted our test environments to align with the production configuration in question, and confirmed the issue is now correctly caught in automated tests. We are confident this class of problem is now remediated.

We've escalated and are awaiting a 3rd party API provider to resolve the issue.

3rd party has restored availability.

We've received reports of workflow loading failures and are investigating.

We've identified the cause of workflow loading errors and are rolling out a fix.

Workflows should now be functioning as normal again.

This morning engineers identified and resolved an issue with the removal of a downstream 3rd party logging system within the Management Platform. The Management Platform has been rolled back and has now resumed normal function.

This morning engineers identified and resolved an issue with the removal of a downstream system within the Management Platform. The Management Platform has been rolled back and has now resumed normal function. We apologise for the inconvenience caused by this outage. ## Issue detail Locomote has extensive test coverage of our entire application that is the cornerstone of our development confidence. The root cause of this bug slipped past detection in our test suite due to three separate issues aligning: * The cause of the bug was in code that we were deprecating ahead of future removal * The deprecation caused a false positive in a code quality tool, advising a parameter was no longer used at definition site * The removal of the parameter did not flag any test failures, because this specific code only runs in production The net result was that because the code quality tool didn’t account for call-site usage, whenever this codepath was invoked in production the parameter length now mis-matched, causing an exception. This exception was unhandled in this specific case, causing a 503 error for users accessing pages triggering this, most notably workflows. The nature of this particular change \(a deprecation\) didn’t invoke a need for production context testing, as we might otherwise do with specific production-dependent features. Our Engineering team made the choice to roll-back this particular deployment to resolve the issue as quickly as possible, and to give us time to further investigate and test the issue resolution before any future deployments. ## Remediation actions * We’re assessing our codebases for any further environment-dependent code to place extra review flags on changes in those contexts * We’re re-evaluating our out-of-hours escalation process to ensure on-call escalations happen correctly and promptly ‌ - Mario CTO Locomote

We are currently investigating this issue.

We've recovered availability but are still determining root cause.

System availability is stable, we continue to monitor and assess root cause.

We are continuing to investigate this issue.

Issue resolved, post-mortem notes to follow.

Issue resolved, post-mortem notes to follow.

Today the Locomote platform experienced a service interruption due to an SSL certificate expiring. During the outage, users received an "Invalid SSL Certificate" error when accessing the Locomote platform from their web browser. The tech team resolved the issue promptly and will ensure it does not reoccur in the future. We want to apologise for any inconvenience caused.

Currently our global distribution system Travelport are experiencing profile outages affecting Locomote customers. Case has been raised with Travelport and highest priority given. This is affecting some users updating profiles and completing some bookings. Please use live chat if you are experiencing any difficulties and we'll be able to support.

Some bookings are affected due to this issue with the Travelport Profile system. Unfortunately out of our control but we are working closely with our partners to ensure this is resolved ASAP. Affecting profile updates more severely in cases. Again if you are having issues please do jump on Live chat / contact support and our team will be there to help.

Travelport have made some config changes at the core that we believe will resolve our immediate issue. We are actively monitoring current performance.

After monitoring for an extended period of time, we believe the issue with our partner (Travelport) has been resolved.

28 October 2019 11:00am (AEDT): We are aware of an issue where there are no hotels being returned. This has been escalated to Travelport as a priority.

Hotel results are now being returned, we are monitoring and will resolve shortly.

Hotel results are now being returned. We apologise for any inconvenience this caused. If there are any further issues please report them to our Support team by emailing [email protected]

We are aware of an issue where some users may not be able to book Jetstar (JQ) instant purchase airfares and are receiving a payment error. You can however book Jetstar (JQ) airfares with Baggage. Our team is working to resolve this as soon as possible. We apologise for any inconvenience.

This incident has been resolved.

We are aware of an issue where some users may not be able to book a hotel with flights (when using On Account as payment) in the same trip. You can however book a hotel separate to the flights. Our team is working to resolve this as soon as possible. We sincerely apologise for any inconvenience.

The issue has been identified and a fix is being implemented.

The issue in which travellers were unable to book a hotel with flights has been Resolved. You will now be able to successfully book Flights, Hotels and Cars in the same booking.. We sincerely apologise for any inconvenience that this may have caused.

Locomote is aware of and is currently investigating an issue in which travellers are unable to progress past the check out page in order to complete their bookings. The traveller will receive an error message stating “Error unmarshalling message body: Expected” We advise that customers should contact their Travel Management Company in order to complete any bookings. We apologise for any inconvenience that this will cause and will continue to update users with the outcome.

We are continuing to investigate an issue on the Locomote Corporate Travel Platform in which travellers are unable to progress past the checkout page, after selecting a seat. The traveller will receive an error message stating “Error unmarshalling message body: Expected.” We advise that customers should contact their Travel Management Company if they require specific seat selection in the interim, or to book without seat selection in order to complete any bookings. We apologise for any inconvenience that this will cause and we will continue to update users with the outcome.

The issue in which travellers were unable to progress past the checkout due to an error “Error unmarshalling message body: Expected” has now be Resolved. You should be able to select seats at checkout as normal. We sincerely apologise for any inconvenience that this may have caused.

We are aware of an issue where some users may not be able to reset their password to the Locomote Corporate Travel Platform when using a web browser other than Internet Explorer. This is due to CAPTCHA not properly loading, which is required to validate the password reset. If you need to book a trip on Locomote in the interim, then please contact your Travel Arranger or Company Administrator to book on your behalf. Our team is working to resolve this as soon as possible. We sincerely apologise for any inconvenience.

We have now fixed an issue where some users were not able to reset their password to the Locomote Corporate Travel Platform when using a web browser other than Internet Explorer. This was due to CAPTCHA not properly loading, which is required to validate the password reset. We sincerely apologise for any inconvenience this may have caused.

We are aware of an issue where users may not be able access and login to the Locomote Corporate Travel Platform. We are currently investigating and are working to resolve this outage as soon as possible. We apologise for any inconvenience and appreciate your understanding.

We have resolved the outage where users may not have been able to access and login to the Locomote Corporate Travel Platform. This was due to a background maintenance job that slowed the system down. We apologise for any inconvenience this may have caused.

We are aware of an issue in which users are currently unable to complete travel bookings via the Locomote platform. Our team is working on a fix to resolve this as soon as possible. At this stage, we strongly suggest that users book offline via their Travel Management Company. We will continue to communicate any updates moving forward. We apologise for the inconvenience that this has caused

The issue where users were unable to complete travel bookings via the Locomote platform has now been resolved. We apologise for any inconvenience that this may have caused.

Our Support & Help Centre provider Zendesk is currently experiencing issues. During this time, you will be unable to view, update or lodge tickets either directly via email or via our Locomote Help Centre. From Zendesk: "Amazon AWS is reporting issues with their Relational Database Service affecting Support, Talk and Guide services on pod 15. We are working with them to resolve the issue. We will post more information as it becomes available." We direct you to use our chat platform Intercom for any issues, which is located on our Locomote platform. Our team will be available to answer any queries you may have. We apologise for any inconvenience.

The issues around our Support & Help Centre provider Zendesk have been resolved. You should now be unable to view, update or lodge tickets either directly via email or via our Locomote Help Centre. We apologise for any inconvenience this may have caused.

We are aware of an issue where some users may not be able to search or complete an online booking on the Locomote Corporate Travel Platform when using the latest mobile version of Google Chrome. Our team is currently working on a fix to resolve this as soon as possible. If you are experiencing issues, if possible please use the desktop version, another browser on your phone (eg. Safari for iOS, Firefox for Android) otherwise you can complete an offline booking. We apologise for any inconvenience.

We are still working on resolving the issue where some users may not be able to search or complete an online booking on the Locomote Corporate Travel Platform when using the latest mobile version of Google Chrome. If you are experiencing issues, if possible please use the desktop version, another browser on your phone (eg. Safari for iOS, Firefox for Android) otherwise you can complete an offline booking. We apologise for any inconvenience.

The issue where some users may not be able to search or complete an online booking on the Locomote Corporate Travel Platform when using the latest mobile version of Google Chrome has been resolved.

We are aware of an issue where some users may not be able to login to the Locomote Corporate Travel Platform when using Safari. We have identified the cause and our team is currently working on a fix to resolve this as soon as possible. If you are experiencing issues, please use another browser (eg. Google Chrome, Firefox). We apologise for any inconvenience.

The issue where some users may not be able to login to the Locomote Corporate Travel Platform when using Safari has now been resolved. We apologise for any inconvenience this may have caused.

We are aware of an issue in the Locomote platform in which users users are unable to complete updates to their profile, resulting in an inability to access online workflows to book and/or arrange travel on behalf of other travellers. Users may notice an error message when hovering over their online workflows, with instructions to contact Support. Users can still book/arrange travel by completing their request using an offline workflow via their travel consultant. We are currently working on fixing this as soon as possible and will update this message as we have more information. Please accept our sincerest apologies for any inconvenience caused.

We have now resolved the issue in the Locomote platform in which users were unable to complete updates to their profile, resulting in an inability to access online workflows to book and/or arrange travel on behalf of other travellers. We apologise for any inconvenience this may have caused during this time.

We are aware of an issue in which Locomote users may find that their workflows are unavailable to be selected. This is due to the "Alternative Title" section of profiles being marked as mandatory. A current workaround is to update your "Alternative Title" section of your profile to either Mr, Mrs or Miss, which will allow you to progress with bookings. We are working to resolve this as soon as possible. Please accept our apologies for the inconvenience.

We are continuing to investigate this issue.

We have now identified the root cause where Locomote users may find that their workflows are unavailable to be selected. This was due to the "Alternative Title" section of profiles being marked as mandatory. A current workaround is to update your "Alternative Title" section of your profile to either Mr, Mrs or Miss, which will allow you to progress with bookings. A fix is being developed, that we hope to release to production as soon as possible to resolve this issue. Please accept our apologies for the inconvenience.

The issue in which "Alternative Title" was affecting the availability of workflows for users, has been successfully resolved. You will still see the Alternative Title field within a profile, however this is now no longer mandatory unless your company configuration enforces it. Again we apologies for any inconvenience this has caused in the interim.

We are aware of an issue in which replying to "New Message" email notifications from your email client will result in extended delays in delivery. A workaround is to reply directly from the Locomote platform via the "Messages" module. We are working to resolve this as soon as possible. Please accept our apologies for the inconvenience.

We are continuing to investigate this issue.

All users are now able to fully utilise the messaging functionality with the Locomote platform without delays. If you have any further issues please contact Locomote Support. We appreciate your patience and understanding during this time.