Historia awarii Roadmunk

Status: We are currently investigating issues. Impact: Some users may experience degraded performance.

The incident has been resolved

We are currently investigating this issue

The issue has been identified and a fix is being implemented

A fix has been implemented and we are monitoring the results

The incident has been resolved

We are currently investigating this issue

We have identified the root cause for a partial outage affecting some users due to DNS resolution issues. We have implemented a fix and expect a rolling recovery as DNS caches refresh.

The incident has been resolved

Our team is aware of an ongoing issue causing authentication attempts to fail. This is related to a widespread outage reported by our provider. We are in contact with their support team and are monitoring the situation.

We have investigated and we are continuing to experience widespread issues due to an ongoing AWS incident with the US-EAST-1 Region ( https://health.aws.amazon.com/health/status ) . We will continue to provide update as the situation evolves.

AWS appears to have fixed the issue, and our services have recovered. We are monitoring the situation closely.

While our cloud provider (AWS) continues to recover from an earlier outage, our own observability platforms confirm that all services have returned to a stable state. We will continue to actively monitor all systems to ensure ongoing performance and stability.

1. **What happened -** The issue appeared to be a memory leak with the sync servers on our European deployment. 2. **Why it happened -** We do not deploy over the weekend which caused the server to not restart as it usually does. During this time period, the memory usage continued to rise on the machine until it ran out of available memory and thus created the incident. 3. **What was done to fix it -** We were able to identify the host having the issue and issued the reboot to fix the the host not being available. 4. **How will this prevented in the future -** * Begun an investigation of the memory usage of sync servers * Increased the memory and CPU on proxy servers * Added another proxy server for our European deployment * Updated our platform plans to include better alerting on this condition so that we can enable self-healing in future releases

This incident has been resolved.

We are continuing to investigate this issue.

A number of our EU customers are experiencing issues accessing Roadmunk. Our team is aware and currently looking into this issue.

A number of users on our EU deployment (eu.roadmunk.com) reported loading issues where they could not load past ~62%. Fortunately incognito access was a successful workaround during this issue. Thanks to a reboot of our proxy servers, the full solution was rolled out within the hour and logins are now successful.

When your product or service isn’t functioning as expected, let your customers know by creating an incident. Communicate early, even if you don’t know exactly what’s going on.

Empathize with those affected and let them know everything is operating as normal.

As you continue to work through the incident, update your customers frequently.

Let your users know once a fix is in place, and keep communication clear and precise.

Hi there, A fix is on it's way for our North American deployment! We've identified a solution to the issue you are experiencing and are working on rolling this out to everyone. We're almost there, Roadmunk Support

Hi there, Good news! We were able to make the necessary changes to restore expected functionality within the Roadmunk application. Everything is operating as normal. Thank you so much for your patience. Happy roadmapping, Roadmunk Support

Hi there, Things aren't working quite as expected on our North American deployment - our apologies! We're experiencing an elevated level of errors and are currently looking into the downtime issue. Thanks for your patience, we will certainly keep you posted! Roadmunk Support

AWS issues have been resolved and services moving back to normal. If you are still experiencing any issues, please feel free to out to someone from our team. Thank you.

Our host AWS is experiencing problems and we are now seeing these issues having an impact on our application. If you have any issues accessing Roadmunk, please reach out to our team (support@roadmunk.com).

Recovery has been completed for affected users within our App server. Our team is available for any follow-up questions. You can reach us at: support@roadmunk.com.

We are continuing to monitor for any further issues.

We are currently experiencing an issue with Portfolio roadmaps. At the moment, it may not affect all users within your Roadmunk account. During this time, we ask that you refrain from building any new Portfolio roadmaps until our team has investigated the issue in full. For more details, feel free to reach out to our team: support@roadmunk.com. Thank you for your patience and understanding.

We've identified an issue and are currently applying a fix that will result in 10-15 minute downtime between now and 10:15 EST. Thank you again for your continued patience.

We've applied a fix to help alleviate the issue. As of now, roadmap sources will not be removed from any new portfolio roadmaps you build. With that said, our team is still monitoring the situation and a new recovery plan is underway to help restore your sources back to their original portfolio roadmap. Any new information will be updated accordingly as we receive it.

Hi everyone, Updating a previous item, the recovery process is well underway. To ensure a smooth process, our team will be restoring accounts based on their respective Roadmunk server. Corresponding updates will be made to notify when each server has been completed. If you're unsure about which server you belong to, simply sign in to Roadmunk and check the URL in your browser. Any account outside of a private deployment will see (app, eu, apac) preceding '.roadmunk.com'. To start, recovery has been completed for all users affected within our APAC server. As always, please subscribe or check back here for any updates. For any additional questions, please don't hesitate to reach out to our support team (support@roadmunk.com). Thanks again for your continued patience.

Recovery has been completed for affected users within our EU server. More updates to follow.

This incident has been resolved.

Included below is a postmortem to the Portfolio roadmap issue we experienced. If there are any outstanding questions after reading, please feel free to reach out to our support team - support@roadmunk.com **What happened** - Some users experienced Portfolio roadmaps in which source roadmaps were erroneously removed.  **Why it happened** - A software defect was missed during testing and was released to production. This defect introduced a logic error that unfortunately removed source roadmaps from Portfolios when a permission change happened. **What was done to fix it** - When the extent of the problem became available Roadmunk’s engineering team investigated and found the root cause. The software was patched accordingly to alleviate the issue. In addition, Roadmunk ran a data restoration procedure for the affected roadmaps and restored them to their original state. All impacted roadmaps were successfully restored two days after the root cause was addressed.  **How will this be prevented in the future** - Moving forward, test cases have been added for this type of situation so it gets caught ahead of time. In addition, should the defect logic present itself again, we've also included safeguards to ensure roadmaps remain stable and return to a legacy state.

Hi there, Things aren't working quite as expected. We're experiencing connectivity issues on the platform and our team is investigating. If you are currently signed in, we ask that you refrain from logging out of the application. Thanks for your patience, we will certainly keep you posted! For any questions, please reach out to support@roadmunk.com Roadmunk Support

Included below is a postmortem to the connectivity issue we experienced yesterday. If there are any outstanding questions after reading, please feel free to reach out to our support team - [support@roadmunk.com](mailto:support@roadmunk.com) **What happened** - Customers that were not already logged in to Roadmunk were unable to log in and access the application. **Why it happened** - An invalid configuration change was made to Roadmunk’s authentication and released to production which caused the application to not run properly. Please note that at no time were any accounts vulnerable to unauthorized access. **What was done to fix it** - When the extent of the problem became available Roadmunk’s engineering team investigated and found the root cause. The team executed a manual rollback that deployed an earlier version of authentication to alleviate the issue and allow users the ability to log in. Then, the permanent solution was solidified and deployed to the current version. The issue was resolved within 1 hour.  **How will this be prevented in the future** - Moving forward, Roadmunk deployment automation has been enhanced to block deployment on this type of misconfiguration.  In addition, code and configuration review process improvements were made to catch such changes ahead of time.

Good news! We were able to make the necessary changes to restore expected functionality within the Roadmunk application. Everything is operating as normal. Thank you so much for your patience. Roadmunk Support

Good news! We were able to make the necessary changes to restore expected functionality within the Roadmunk application. Everything is operating as normal. Thank you so much for your patience. Happy roadmapping, Roadmunk Support

We are continuing to work on a fix for this issue.

Hi there, We've identified an issue and are currently looking further into a solution. Our sincere apologies about this. We will update you once a fix is identified. Appreciate your patience! Roadmunk Support

Hi there, Good news! We were able to make the necessary changes to restore expected functionality within the Roadmunk application. Everything is operating as normal. Thank you so much for your patience. Happy roadmapping, Roadmunk Support

Hi there, We've identified an issue and are currently looking further into a solution. Our sincere apologies about this. We will update you once a fix is identified. Appreciate your patience! Roadmunk Support

Thank you for your patience! Our team has resolved this issue and are currently monitoring to ensure that there are no residual effects or impact. If you are still seeing errors when trying to access Roadmunk following this fix, please try refreshing the page - any ongoing issues can be reported to our team via support@roadmunk.com. Our apologies again for any inconvenience this may have caused and we're happy to have things back up and running for you. Hope you have a wonderful day and happy roadmapping!

Our team is still in the process of resolving this issue and are anticipating restored service soon. We'll keep you posted with any further updates as they're brought to our team. Thank you for your continued patience and our apologies once again for this inconvenience.

We are currently in the process resolving an issue which has caused 504 errors to appear for users attempting to access the application. The root cause has been located and we will keep you posted with updates as our team continues their work to restore access. Our most sincere apologies for this roadblock and we hope to be back up shortly.

Hi there, Good news! We were able to make the necessary changes to restore expected functionality within the Roadmunk application. Everything is operating as normal. Thank you so much for your patience. Happy roadmapping, Roadmunk Support

Hi there, Things aren't working quite as expected! We're experiencing an elevated level of errors and are currently looking into the issue. Thanks for your patience, we will certainly keep you posted! Roadmunk Support

Hi there, We've identified an issue and are currently looking further into a solution. Our sincere apologies about this. We will update you once a fix is identified. Appreciate your patience! Roadmunk Support

Hi there, Good news! We were able to make the necessary changes to restore expected functionality within the Roadmunk application. Everything is operating as normal. Thank you so much for your patience. Happy roadmapping, Roadmunk Support

Thank you for your patience and apologies for any inconvenience! Our team has confirmed that all deployments are once again accessible and that this issue has been resolved. If you are still running into any issues, please don't hesitate to reach out to support@roadmunk.com and we will be happy to assist. Hope you have a wonderful weekend and happy roadmapping!

A fix has been rolled out and is propagating across deployments - we will update to resolved once all deployments have resumed expected behaviour.

We are currently in the process of investigating a site access issue which is blocking users on most deployments across Roadmunk. Our team will be updating users via the Status page with more details shortly as our engineering team continues their investigation.

We are currently investigating this issue.

We are continuing to investigate this issue.

This incident has been resolved.

We have resolved the issues related to authentication and SSO.

This incident has been resolved.

We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

We are continuing to investigate this issue.

We are currently investigating delays with authentication.

Performance has been restored

We are continuing to investigate this issue.

We are currently investigating issues where users are experiencing slow response times and loading times for roadmapping and authentication.