Teem 장애 기록

We are currently investigating an issue with Teem. We will update you when we have more information.

The issue has been identified and a fix is being implemented.

The issue with the Plugin has been identified and a fix is being implemented. We will post another update at 6 PM CST. If you're experiencing this issue, please make sure you have followed these steps https://knowledge.eptura.com/Teem/Production_Information/Announcement/New_authentication_implementation_for_Teem_Outlook_plugin From there please reach out to support for help as there is a current workaround.

The issue with the Plugin has been identified and a fix is being implemented. We will post another update tomorrow at 9 AM CST If you're experiencing this issue, please make sure you have followed these steps https://knowledge.eptura.com/Teem/Production_Information/Announcement/New_authentication_implementation_for_Teem_Outlook_plugin Once those steps have been followed if you try to login and are met with an infinite login loop please open an incognito window and log into your OWA instance and re-launch the Plugin. If you need assistance on how to do this or are encountering another error message please reach out to support for further troubleshooting.

The issue with the Plugin has been identified and a fix is being implemented. We will post another update at 12 PM CST If you're experiencing this issue, please make sure you have followed these steps https://knowledge.eptura.com/Teem/Production_Information/Announcement/New_authentication_implementation_for_Teem_Outlook_plugin Once those steps have been followed if you try to login and are met with an infinite login loop please open an incognito window and log into your OWA instance and re-launch the Plugin. If you need assistance on how to do this or are encountering another error message please reach out to support for further troubleshooting.

The issue with the Plugin has been identified and a fix is being implemented. We will post another update at 5 PM CST If you're experiencing this issue, please make sure you have followed these steps https://knowledge.eptura.com/Teem/Production_Information/Announcement/New_authentication_implementation_for_Teem_Outlook_plugin Once those steps have been followed if you try to login and are met with an infinite login loop please open an incognito window and log into your OWA instance and re-launch the Plugin. If you need assistance on how to do this or are encountering another error message please reach out to support for further troubleshooting.

The issue with the Plugin has been identified and a fix is being implemented. We will post another update at 9 AM MST If you're experiencing this issue, please make sure you have followed these steps https://knowledge.eptura.com/Teem/Production_Information/Announcement/New_authentication_implementation_for_Teem_Outlook_plugin Once those steps have been followed if you try to login and are met with an infinite login loop please open an incognito window and log into your OWA instance and re-launch the Plugin. If you need assistance on how to do this or are encountering another error message please reach out to support for further troubleshooting.

A fix has been implemented. We are moving into the Monitoring Phase for the next two hours. We appreciate your patience as our team worked diligently to provide a solution for this.

As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase. Thank you for all the patience as we worked through this issue.

We are currently investigating an issue with Teem. We will update you when we have more information.

Our engineering team has identified inconsistent time zone behavior for all-day bookings via Plugin and is working to validate the impact across all customers. Users can continue to make bookings via https://app.teem.com/ or directly from the device while we work on addressing the issue. We shall keep you informed as soon as more information becomes available. Thank you for your continued patience and understanding.

Our engineering team has identified that the issue is occurring only with bookings made via the Outlook plugin for whole-day events where impersonation calendars are in use. Users can continue to make bookings via https://app.teem.com or directly from the device while we work on resolving the issue. We will keep you informed as soon as more information becomes available. Thank you for your continued patience and understanding.

Our engineering team continues to work on implementing a solution occurring only with bookings made via the Outlook plugin for whole-day events where impersonation calendars are in use. Users can continue to make bookings via https://app.teem.com or directly from the device while we work on resolving the issue. We will keep you informed as soon as more information becomes available. We appreciate your patience and understanding as we work to resolve this issue.

We are actively investigating an issue affecting all-day bookings. Our Development team is working on a solution. We'll share more updates as soon as they become available. Thank you for your continued patience and understanding.

Our engineering team continues to work on the issue affecting all-day bookings. Users can continue to make bookings via https://app.teem.com or directly from the device while we work on resolving the issue. We will keep you informed as soon as more information becomes available. We appreciate your patience and understanding as we work to resolve this issue.

Our engineering team has developed a fix, and it is now being tested internally. Users can continue to make bookings via https://app.teem.com or directly from the device while we work on resolving the issue. We will keep you informed as soon as more information becomes available. We appreciate your patience and understanding as we work to resolve this issue.

Our QA team are continuing to test this issue internally. Users can continue to make bookings via https://app.teem.com or directly from the device while we work on resolving the issue. We will keep you informed as soon as more information becomes available. We appreciate your patience and understanding as we work to resolve this issue.

Our QA team continues to test this issue internally. Users can continue to make bookings via https://app.teem.com or directly from the device while we work on resolving the issue. We will keep you informed as soon as more information becomes available. We appreciate your patience and understanding as we work to resolve this issue.

A fix has been developed and applied to production. We are actively monitoring the situation to ensure continued stability and performance. If you’re still experiencing any issues, please reach out to our support team through your existing support case on the Customer Hub, or create a new one if none exists. We will continue to monitor the situation for the next 8 hours before marking it as resolved. Thank you for your patience.

We are continuing to monitor for any further issues.

This issue is currently resolved with a release that happened last night. After confirmation we are going to be closing out this Status Page. We appreciate all the patience with our customer base as we have worked diligently to get this update out.

We are currently investigating an issue with TEEM and the inability to access the platform. Our Engineering and Cloud Operations team is actively working to determine the root cause of the disruption and assess its impact. We will provide our next update by 9:48am CST. Thank you for your patience as we work to resolve this issue.

We are currently investigating an issue with TEEM and the inability to access the platform. Our Engineering and Cloud Operations team is actively working to determine the root cause of the disruption and assess its impact. We will provide our next update by 1:00 PM CST. Thank you for your patience as we work to resolve this issue.

The issue with performance has been identified and a fix is being implemented. We will post another update at 1PM CST.

A fix has been implemented. We are moving into the Monitoring Phase for the next 24 hours.

The issue with App.teem.com performance has been identified and a fix is being implemented. We will post another update at 10 AM CST.

A fix has been implemented. We are moving into the Monitoring Phase for the next two hours. If you have any issues please don't hesitate to reach out to support.

As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase. A Preliminary RCA will be posted in this incident in 2 business days. Please stay subscribed to the page to receive post automatically.

**Teem by Eptura detailed Root Cause Analysis | 3/4/2025**  **S1 – Teem Inaccessible**    We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.     **Description:**  **\(The Incident is logged in MST\)**  On March 4th, 2025, the production application encountered a performance issue due to a high server load. This issue lasted for approximately four days. Upon investigation, it was determined that multiple duplicate events were being generated, resulting in an excessive number of database entries and a slowdown in application response times. The situation escalated, ultimately leading to a system outage and the activation of a fire alarm.    **Type of Event:**  Outage    **Services/Modules impacted:**  [App.Teem.Com/Calendaring](http://App.Teem.Com/Calendaring)    **Timeline:**` `**The timeline is posted in MST.**  Around 7:34 AM. We received notice of some people not able to access [app.teem.com](http://app.teem.com). At 10:19 AM our Engineering team clearing tables to help alleviate performance. At 11:29 AM we moved the issue to an S2 due to website being up and accessible. At 10:17 PM we are finishing clearing tables meaning performance should now be at full. 3:49 AM website is working as intended. At 9:17 AM the issue was resolved and we entered the monitoring phase for two hours. At 11 AM we cleared the status page.    **Total Duration of Event:**  27 Hours    **Root cause:**     * Duplicate event generation: The event processing logic did not prevent the creation of redundant events.  * Delayed event processing: Recent codebase changes that integrated Graph API calls extended the event processing time.  * Push Callback Timing Issue:   * When an event is created in our database, a corresponding call is made to Outlook to create the event on their end.  * The increased processing time due to the Graph API changes caused push callbacks to arrive before the original event creation process was completed.  * This resulted in the original event remaining incomplete and missing updates from Outlook.  * A scheduled interval call exists to update missed events, but the frequent push callbacks led to the original event being updated only after multiple duplicates were created.      **Remediation:**  To address the issue, we introduced a locking mechanism by implementing a cache to prevent event creation by push callback before the original event creation flow is completed. Additionally, we handled the scenario where, in the event the locking mechanism fails, the original event is updated with the push callback data. We are also working on sending a unique payload key to Outlook to compare within push callbacks, ensuring more accurate and timely event updates.      **Preventative Action:**     To enhance system reliability, we have improved logging and monitoring for push callback processing, enabling early detection of any anomalies. Additionally, we have implemented a mechanism to prevent the first push callback for an event, ensuring that the original event creation flow is updated smoothly and without any hindrance

We have successfully completed maintenance to enhance security and compliance by transitioning to Microsoft Graph API for calendar services using Microsoft Exchange. We are aware of a small issue that occurred during the release we are addressing, which may cause calendars to produce duplicate meetings. Our team is actively managing the operational incident and working to resolve it as quickly as possible. If you are experiencing an issue with duplicate calendar events, you can temporarily remove the Location from the calendar to prevent duplication notices. Once we implement the update early next week, you can re-add the Location. Steps to remove the Location: Go to your calendar section on app.teem.com on the left side of the page. Select the proper account, then navigate to the calendar experiencing the duplication issue. Under the Location option click the red dot, this will remove the location from that calendar, halting the duplicate notifications. After our team implements the update early next week, you can re-add the Location. We will keep you informed once the update has been implemented. For any questions or to learn more about alternative options to Microsoft Graph API, please review the technical documentation on our Knowledge Center or contact support. We will update the status page as our Engineering team works through this issue on Monday March 3rd at 8 AM MST.

Important update on calendar syncing We recently completed maintenance to enhance security and compliance by transitioning to Microsoft Graph API for calendar services using Microsoft Exchange. We are aware of a small issue that occurred during the release, which may cause calendars to produce duplicate meetings. Our team is actively managing the operational incident and will release the fix today, March 3, 2025, at 10:30 PM MST. This fix will address known issues for Google Sync, as well as performance and O365 Calendaring issues. If you are experiencing an issue with duplicate calendar events, you can temporarily remove the Location from the calendar to prevent duplication notices. Once we implement the update, you can re-add the Location. Steps to remove the Location: Go to your calendar section on app.teem.com on the left side of the page. Select the proper account, then navigate to the calendar experiencing the duplication issue. Under the Location option click the red dot, this will remove the location from that calendar, halting the duplicate notifications. After our team implements the update early next week, you can re-add the Location. We will keep you informed once the update has been implemented. For any questions or to learn more about alternative options to Microsoft Graph API, please review the technical documentation on our Knowledge Center or contact support. This will be completed at 10:30 PM MST. This will fix known issues for Google Sync as well as performance issues and O365 Calendaring issues.

As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase. A Preliminary RCA will be posted in this incident in 2 business days. Please stay subscribed to the page to receive post automatically.

We are currently investigating an issue with the loading of app.teem.com. Our Engineering team is currently investigating to determine the cause of the disruption. Next update will be posted at 11 AM MST

We are currently still investigating an issue with app.teem.com and logging Our Engineering team is currently investigating to determine the cause of the disruption. Next update will be posted at 12:30 PM MST.

We are currently still investigating an issue with app.teem.com and logging Our Engineering team is currently investigating to determine the cause of the disruption. Next update will be posted at 2:30 PM MST.

A fix has been implemented. We are moving into the Monitoring Phase for the next hour. We will be closing this out at 2 PM MST

As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase. A Preliminary RCA will be posted in this incident in 2 business days. Please stay subscribed to the page to receive post automatically.

**Teem by Eptura detailed Root Cause Analysis | 2/25/2025**  **S1 – Duplicate Events causing downtime**    We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.     **Description:**  **\(The Incident is logged in MST\)**  On February 22, 2025, the production application encountered a performance issue due to a high server load. This issue lasted for approximately four days. Upon investigation, it was determined that multiple duplicate events were being generated, resulting in an excessive number of database entries and a slowdown in application response times. The situation escalated, ultimately leading to a system outage and the activation of a fire alarm.    **Type of Event:**  Outage    **Services/Modules impacted:**  [App.Teem.Com/Calendaring](http://App.Teem.Com/Calendaring)    **Timeline:**` `**The timeline is posted in MST.**  Around 8:58 AM. We received notice of some instances of slowed experience and inability to access [app.teem.com](http://app.teem.com). At 8:59 AM our Engineering team is reaching out to AWS team to address this. At 10:30 AM we moved the issue to an S1. At 10:45 AM the site became accessible again, but with slowness. At 12:52 PM we resolved this issue and went into a monitoring phase. At 2:02 PM we cleared the status page.    **Total Duration of Event:**  4 Hours    **Root cause:**     * Duplicate event generation: The event processing logic did not prevent the creation of redundant events.  * Delayed event processing: Recent codebase changes that integrated Graph API calls extended the event processing time.  * Push Callback Timing Issue:   * When an event is created in our database, a corresponding call is made to Outlook to create the event on their end.  * The increased processing time due to the Graph API changes caused push callbacks to arrive before the original event creation process was completed.  * This resulted in the original event remaining incomplete and missing updates from Outlook.  * A scheduled interval call exists to update missed events, but the frequent push callbacks led to the original event being updated only after multiple duplicates were created.      **Remediation:**  To address the issue, we introduced a locking mechanism by implementing a cache to prevent event creation by push callback before the original event creation flow is completed. Additionally, we handled the scenario where, in the event the locking mechanism fails, the original event is updated with the push callback data. We are also working on sending a unique payload key to Outlook to compare within push callbacks, ensuring more accurate and timely event updates.      **Preventative Action:**     To enhance system reliability, we have improved logging and monitoring for push callback processing, enabling early detection of any anomalies. Additionally, we have implemented a mechanism to prevent the first push callback for an event, ensuring that the original event creation flow is updated smoothly and without any hindrance

We are currently investigating an issue with Teem. We will update you when we have more information.

We are currently investigating an issue with Calendar Events using Google & O365 are not syncing, even when Force Re-Sync is used. Our Engineering team is actively working to determine the root cause of the disruption and assess its impact. We will provide our next update by 2:30 MST.

Our engineering team continues to investigate the root cause for calendar events not syncing. As a workaround, all calendars have been forced resynced on back end services. Doing this has allowed calendar events to sync for customers who have been affected. We will continue to force resync until full resolution. For any questions, please contact our support team. Our next update will be provided, Friday, 09/06/2024 at 7am MST.

We are continuing to investigate this issue.

Our engineering team has implemented mitigations while we continue to investigate the root cause. If you continue to experience any issues, please raise a support case. Our next update will be 12pm MST.

We appreciate your patience as our team is working diligently to resolve the issue with unpredictable calendar event synchronization for both O365 and Google users. We have identified an issue which could be causing these symptoms and are actively working to address it. In the meantime, we have implemented enhanced measures to mitigate the issue, including increasing the frequency of forced calendar syncs to four times a day. This should help improve the reliability of your calendar events until the full resolution is in place. Our team is dedicated to restoring normal calendar sync performance, and we will keep you updated with additional status updates as we continue to monitor and make progress towards a resolution. Our next update will be posted by Tuesday, Sept. 10 at 7am MDT. Should you experience any additional issues, please contact our support team for assistance. Thank you.

We have implemented a solution for the sync issue affecting Google (410 errors) and O365 (Missing Initialization errors). We will be monitoring the situation in the next several days to ensure stability and performance. Our Engineering team is overseeing the process to confirm that the issue has been fully resolved. The next update will be provided by Friday, 9am MST.

We have implemented a solution for the sync issue affecting Google (410 errors) and O365 (Missing Initialization errors). We will be monitoring the situation in the next several days to ensure stability and performance. Our Engineering team is overseeing the process to confirm that the issue has been fully resolved. The next update will be provided by Monday, 9am MST.

Monitoring will continue till tomorrow, Tuesday, September 17th 9am CST. We appreciate your continued patience as we worked to resolve this issue.

We are pleased to inform you that the issue with Calendar Events using Google & O365 are not syncing has been resolved. Our Engineering team has completed the necessary actions and verified that the service is now functioning normally. A Root Cause Analysis (RCA) will be conducted to understand the incident in detail and will be made available on our Status Page within 10 days. Thank you for your patience and cooperation throughout this process. If you have any further questions or concerns, please feel free to reach out.

**Teem by Eptura detailed Root Cause Analysis | September 5, 2024**  **S2 |Office365 & Google Calendar | Reservations not syncing**  ‌ We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.    **Description:**  Some TEEM customers experienced difficulties syncing calendar events with both Office365 and Google Calendar. Calendar events were not syncing automatically from the source calendar to TEEM, unless a manual Force Sync was initiated. In some cases, even using Force Sync did not resolve the issue, leading to confusion when using the reservation module. While both Office365 and Google Calendar showed similar symptoms of calendar syncing failures, the underlying causes were different for each service and required separate solutions to resolve.    **Type of Event:**  Functionality Issue    **Services/Modules impacted:**  Calendar Service/ Production    **Timeline:** September 5, 2024, Reported MDT   10:30 AM: Customers began to report the inability to sync their calendar events for Google \(410 errors\) and O365 \(Missing Initialization errors\). An internal Fire Alarm was raised, and all customers were notified that we are investigating the issue via the status page.   September 6, 2024, Reported MDT   11:53 AM: The engineering team has identified the issue and continues to work towards a resolution. Customers were notified that we have moved from investigating to an identified phase. In the meantime, engineers have implemented enhanced measures to mitigate the issue by running a script to manually sync calendars for all reporting customers, four times a day and should improve the reliability of calendar events till full resolution.  September 10, 2024, Reported MDT   9:34 AM: All customers were notified that a solution was implemented for the sync issue affecting Google \(410 errors\) and Office365 \(Missing Initialization errors\). To ensure stability and performance our engineering team is overseeing the process to confirm that the issue has been fully resolved. Monitoring will continue over the next several days.   September 17, 2024, Reported MDT   8:02 AM: All customers were notified that our engineering team has completed the necessary actions and verified that the service is now functioning normally. As no additional customers have reported specific issues in regard to Google \(410 errors\) and O365 \(Missing Initialization errors\). The status page was updated to a resolved state.    **Total Duration of Event:**  11 days, 21 hours, 32 minutes    **Office365 Root Cause:** The issue occurred due to two concurrent requests attempting to refresh the Office 365 access token at the same time. This created a situation where the system, under certain conditions, returned a null token \(None\), which was then passed to the API client. As a result, calendar syncing was interrupted. **Office365 Remediation:** We have updated the system to ensure that, when multiple requests are made, the current access token is used if it’s valid. This prevents the token from being set to None and ensures the API client always receives a valid token. **Office 365 Preventative Measures:** In addition to the fix, we’ve ensured that the system will no longer return a null token in any situation. We have also added logging to monitor the token refresh process closely, allowing us to better detect and resolve any future issues quickly.    **Google Calendar Root Cause:** The system was making multiple attempts to delete events from Google Calendar, even when the event had already been deleted. This caused a 410 error, indicating the resource was no longer available. The issue occurred because the system did not verify whether an event still existed before attempting to delete it. **Google Calendar Remediation:**  1. The system now checks if a Google Calendar event still exists before attempting to delete it, ensuring that deletion requests are only made when necessary.  2. We have introduced a locking mechanism to prevent duplicate or conflicting deletion requests from being made simultaneously.  **Google Calendar Preventative Measure:** To prevent similar issues in the future, Google watchers will be updated using dedicated cron jobs, ensuring synchronization happens in a controlled and consistent manner. The lock mechanism will also ensure that API calls are handled sequentially and without conflict.

We are currently experiencing issues with Calendar syncing. Our engineering team has been notified and we'll be updating this status page at 1:51 MST.

We are continuing to investigate this issue. We'll be updating the status page at 4:51 PM MST

We are continuing to investigate this issue. We'll be updating the status page at 8:51 PM MST

We are continuing to investigate this issue. We'll be updating the status page at 12:51 AM MST

We are continuing to investigate this issue. We'll be updating the status page at 4:51 AM MST

We have found this issue to be mitigated and is only affecting a small subset of customers that we are working diligently to support and find a resolution. If you are experiencing issues, please don't hesitate to reach out to support.

We are currently investigating an issue with Teem. We will update you when we have more information.

We are continuing to investigate this issue.

We have found this issue to be mitigated and is only effecting a small subset of customers that we are working diligently to support and find a resolution. If you are experiencing issues please don't hesitate to reach out to support.

**Teem by Eptura detailed Root Cause Analysis | August 20, 2024**  **S2 O365 Multiple Users Marked Inactive**    We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.    **Description:**  Customers using O365 were unable to access the Teem platform. When trying to login, customers were met with an error message.    **Type of Event:**  Functionality Issue    **Services/Modules impacted:**  Production/ Office 365     **Timeline** \(Reported MST\)**:**` `  On the morning of August 20, 2024, at approximately 8:20 AM MST, our support team began receiving reports from end users who were inadvertently marked as Inactive in the Teem platform. We promptly informed all Teem customers of the incident via our Status Page.  At 10:24 AM MST, we marked the Status Page as resolved. However, recognizing the importance of addressing this issue thoroughly, our engineering team continued to collaborate closely with our Support team to find a comprehensive solution. To prevent further impact, the engineering team implemented a nightly script to revert the status of users marked as Inactive.  The investigation continued until November 1, 2024, when our engineering team successfully released a HotFix to address the issue. Although the initial HotFix did not fully resolve the problem, the team enhanced our logging capabilities to better track and understand the behavior.  On November 11, 2024, the additional logs provided valuable insights, enabling our engineering team to resolve the issue in our QA environment. The final HotFix was released on November 14, 2024, and customers have confirmed the resolution.    **Total Duration of Event:**  83 Days    **Root Cause:**   The issue arose from the concurrent processing of multiple user batches. During this process, one thread completed its task earlier than expected and inadvertently deleted all cache keys, including the main Sync\_Key and associated batch\_keys. This led to subsequent threads receiving an empty batch list, which resulted in the deprovisioning or deactivation of users.    **Remediation:**  To resolve this issue, we have implemented database row-level locking for batch processing. This ensures that batch processing happens sequentially, avoiding conflicts. Key updates include:  1. Introduced a dedicated table to track batch process counts and the ID of the last processed batch.  2. Applied database row-level locks to manage synchronization safely and efficiently.  3. Updated the deprovisioning process to occur only after verifying that all batches are fully processed.     **Preventative Action:**    To prevent recurrence, we have made the following improvements:  *  Enhanced concurrency handling to ensure seamless user batch synchronization.  * Added extensive logging in CloudWatch to monitor and better understand process behavior.  * Streamlined the user synchronization process to ensure that all O365 users are successfully synced with the Teem directory.  These enhancements will significantly improve the reliability and performance of our system. Thank you for your patience and support as we continue to make these improvements.

We are currently investigating an issue with Teem. We will update you when we have more information.

This was an internal incident and has been resolved. Thank you for understanding.

We are currently investigating an issue with Teem. We will update you when we have more information.

This incident has been resolved.

We are currently investigating an issue with Insights not generating data. Our internal team is currently investigating to determine the cause of the disruption. The next update will be posted at 4:10pm CST.

We are continuing to investigate this issue.

Our engineering team continues to investigate the inability pull data from Insights. We appreciate your patience as we work through this.

We appreciate your patience and are pleased to share that our team has identified the source of the Insight data disruption and is implementing a fix. We are responding directly to open tickets with details on the resolution for specific customer environments and will continue to monitor overall Insights performance. For further questions, please reach out to our dedicated support team. Thank you.

We are currently investigating an issue with Teem. We will update you when we have more information.

This incident has been resolved.

We are currently experiencing an issue with the Teem application resulting in a 404 when trying to access the site.

A fix has been implemented. We are moving into the Monitoring Phase for the next 30 minutes.

As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase.

**Teem Detailed Root Cause Analysis | 4.05.2024** **404 Error Site Wide Outage** ‌ We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident. ‌ **Description:** On April 5th, 2024,  internal teams and customer support noticed issues happening with logging and being able to access sites. Reports of some Teem instances experiencing inability to login with a forward of a 404 error also were presented. This issue impacted a large scale of Teem Customers. This cause of this site outage was due to a migration to a new server that would allow easier rollbacks and data integrity, that unfortunately had a performance error and needed to be done early. Upon receiving notice of the site access issues, our dedicated team promptly took action to restart the migration process, ensuring its successful completion and resolving the issue. **Type of Event:** Site Access **Remediation:** Upon immediate notice of the login issues, our dedicated Teem team was able to restart the migration process to successful completion, resolving the issue. **Root Cause Analysis:** There was a production outage caused by a database that we were hosting on. We needed to then move the data base to our standby server and set up a new one. We had planned the migration for a few weeks later and were preparing for it but the server encountered errors before the planned maintenance happened forcing us to start the migration early. We started the migration server backed up the DB and re-established connection with services. **Preventative Action:** The Teem team was able to start a migration process and monitor it through successful completion and review with internal testing to ensure the issue was resolved for customers. This incident has been closed, but our team is dedicated to closely monitoring future updates as they are released to ensure the best customer experience.

We are currently investigating an issue with customers using Google Calendar Service. Our Engineering team is currently investigating to determine the cause of the disruption. The next update will be posted at 7:45pm MDT.

We are continuing to investigate this issue in regard to Google Calendar Sync. The next update will be posted at 11:45pm MDT.

The issue with Google Calendar Service. has been identified and a fix is being implemented. We will post another update at 1am CST.

A fix has been implemented. We are moving into the Monitoring Phase for the next 4 hours.

We are continuing to monitor for any further issues for next 4 hours.

As the fix implemented haven't resolved the issue completely, we have moved to the investigation phase. Our Engineering team is currently investigating the issue with Google Calendar Service to determine the cause of the disruption. The next update will be posted at 12 PM CST.

We are continuing to investigate this issue on priority. We will post another update at 4 PM CST.

As the previous fix implemented did not resolve the issue completely. We are continuing the investigation with Google Calendar Service and have determined the cause of the disruption and are working on a fix. The next update will be posted at 8 PM CST.

We are continuing to investigate this issue on priority. We apologize for the delay, next update will be at 3 am CST

A fix has been implemented. We are moving into the Monitoring Phase for the next 12 hours.

As the fix implemented haven't resolved the issue completely, we have moved to the investigation phase. Our Engineering team is currently investigating the issue with Google Calendar Service to determine the cause of the disruption. The next update will be posted at 7 AM CST.

We are continuing to investigate this issue on priority. We apologize for the delay, next update will be shared at 11 AM CST.

A fix had been identified and applied to optimize the performance of the Google Calendar Sync. We are moving into the Monitoring Phase for the next 4 hours and next update will be shared at 3 PM CST.

We are continuing to monitor for any further issues for next 12 hours.

As the fix implemented haven't resolved the issue completely, we have moved to the investigation phase. Our Engineering team is currently investigating the issue with Google Calendar Service to determine the cause of disruption. The next update will be posted at 11:30 AM CST.

We are continuing to investigate this issue on priority. We apologize for the delay, next update will be shared at 3:30 PM CST.

Our team is currently working to resolve an issue that is impacting sync times for customers using Google calendars. We want to assure you that our team is fully committed to resolving this issue as swiftly as possible. We recognize the importance of timely event syncing, and we apologize for any delays you may be experiencing. Restoring normal calendar sync performance is our top priority, and we will keep you updated with additional status updates as we make progress towards a resolution. Thank you.

We appreciate your patience as our team is working diligently to resolve the issue with unpredictable timing of calendar event synchronization. We have identified a potential issue which could be causing these symptoms and are actively working to address it. Currently, we have observed that the PgBouncer and PgBouncer_ro services will not run simultaneously on job managers. Due to the startup script, it is unclear which of the two services is running, and it seems that the "last to start wins" scenario occurs. In the event of an instance restart, a different service could potentially "win" and cause further inconsistency. To resolve this, we have worked on a solution where these services now have separate unix socket directories. By providing different unix socket directories to both services, they can run simultaneously and eliminate the inconsistency. This eliminated significant errors on jobmanagers. Our team is dedicated to restoring normal calendar sync performance, and we will keep you updated with additional status updates as we continue to monitor and make progress towards a resolution.

We appreciate your patience as our team is working diligently to resolve the issue with unpredictable timing of calendar event synchronization. We have identified an issue that was causing these symptoms, and have developed and implemented a fix to resolve the issue. Our team is dedicated to ensuring normal calendar sync performance, and we will continue to monitor the issue to ensure full resolution and provide additional status updates if needed. Thank you.

We deeply appreciate your patience as our team worked diligently to resolve the recent calendar event synchronization timing issue. We are pleased to inform you that we have successfully identified the root cause of the problem and have implemented a fix to resolve it. Our team is committed to ensuring normal calendar performance and will continue to monitor this issue closely to ensure the best possible customer experience. For any further questions and concerns, please reach out to our dedicated support team. Thank you.

**Teem by Eptura detailed Root Cause Analysis | April 11, 2024**  **S2 Google Calendar Service not Synchronizing**    We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.    **Description:**  Customers using the Google Calendar service experienced events that were not automatically synced. During this time, a workaround was provided to force a manual sync, updating the calendars.     **Type of Event:**  Functionality Issue    **Services/Modules impacted:**  Production/ Google Calendar Service    **Timeline** \(Reported MST\)**:**` `  On the late afternoon of April 11th, 2024, at approximately 3:50pm, multiple customers reported an issue with their Google Calendar Service not automatically syncing calendar events. Customers were provided a temporary workaround to manually force sync their calendars. All customers were made aware of the Severity 2 incident via Teem Status Page. The investigation continued through April 19, 2024, when the CloudOps team identified the root cause of the issue. On April 22, 2024, at approximately 11:08am, all customers were notified via Status Page that the fix had been implemented and we moved into a monitoring phase. After continuous monitoring, no additional reports for Google Calendar Events and customers confirming that their Calendar events were syncing automatically, the Severity 2 incident was marked as resolved on April 29, 2024, at 10:23am.    **Total Duration of Event:**  17 days, 18 hours, 33 minutes    **Root Cause:**   We observed that the PgBouncer and PgBouncer\_ro services will not run simultaneously on job managers. Due to the startup script, it is unclear which of the two services is running, and it seems that the "last to start wins" scenario occurs. In an instance restart, a different service could "win" and cause further inconsistency. We have also discovered that three of our Job Managers are running outdated code.    **Remediation:**  These services shared a unix socket directory. By providing different unix socket directories, the services both would run simultaneously and eliminate the inconsistency. This eliminated significant errors on the jobmanagers    **Preventative Action:**   Our team is dedicated to continuously improving the Google Calendar Service by enhancing our current processes and implementing robust monitoring systems. We appreciate your patience and cooperation during this disruption.

We are currently investigating an issue with Teem. We will update you when we have more information.

The issue with TEEM has been identified and a fix is being implemented. We will post another update at 11:00 CST.

We are continuing to work on the fix for this issue. We will post another update at 13:00 CST.

A fix has been implemented by the Engineering team. We'll be going into Monitoring for the next hour, an update will be given at 14:30 CST.

We have had confirmation from internal teams as well as customers of the issue being resolved. We appreciate everyone's patience as our engineers worked to resolve this.

**Teem Detailed Root Cause Analysis | 4.05.2024**  **404 Error Site Wide Outage**    We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.     **Description:**  On April 5th, 2024,  internal teams and customer support noticed issues happening with logging and being able to access sites. Reports of some Teem instances experiencing inability to login with a forward of a 404 error also were presented. This issue impacted a large scale of Teem Customers.  This cause of this site outage was due to a migration to a new server that would allow easier rollbacks and data integrity, that unfortunately had a performance error and needed to be done early.  Upon receiving notice of the site access issues, our dedicated team promptly took action to restart the migration process, ensuring its successful completion and resolving the issue.    **Type of Event:**  Site Access     **Remediation:**  Upon immediate notice of the login issues, our dedicated Teem team was able to restart the migration process to successful completion, resolving the issue.    **Timeline:**  ` `April 5  * \(4:14 AM\) – Internal teams were noticing issues with logging as well as site access  * \(4:21 AM\) – Customers also reporting the issue, at this point we have already notified the team  * \(4:24 AM\) – Investigation process has started with internal teams  * \(5:23 AM\) – Internal teams notify us of continued investigation but that the issue is close to being found  * \(6:43 AM\) – The issue has been found and a fix has been implemented with the rollback of the server to the new DB still needing around a few hours to finish.  * \(12:45 PM\) – The database has been moved and restored.    **Total Duration of Event:**  \(0 day/8 hours/30minutes\)     **Root Cause Analysis:**    There was a production outage caused by a database that we were hosting on. We needed to then move the data base to our standby server and set up a new one. We had planned the migration for a few weeks later and were preparing for it but the server encountered errors before the planned maintenance happened forcing us to start the migration early. We started the migration server backed up the DB and re-established connection with services.    **Preventative Action:**   The Teem team was able to start a migration process and monitor it through successful completion and review with internal testing to ensure the issue was resolved for customers. This incident has been closed, but our team is dedicated to closely monitoring future updates as they are released to ensure the best customer experience.

We are currently having issues with what seems to be mainly O365 SSO. We are investigating this issue and will make an update when we have gathered more information.

We are continuing to investigate this issue we will update at 8:00 AM MST

We are continuing to investigate this issue we will update at 1:00 PM CST

Our Engineering team is continuing the investigating to determine the cause of the disruption. Next update will be posted at 5:00pm CST

The issue with O365 SSO has been identified and a fix is being implemented. We will post another update at 8:00pm CST or earlier when testing is complete.

Our Engineering team has found a fix and implemented it. We have had some confirmation the issue is resolved. We will be in monitoring for the next hour while awaiting further confirmations.

The issue is now resolved. We appreciate your patience while we resolved this issue. We'll be posting the RCA by March 26th.

**Teem by Eptura detailed Root Cause Analysis | 3/11/2024**  **S2 – O365 SSO Login Issues**    We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.     **Description:**  **\(The Incident is logged in MST\)**  **On Monday on Monday March 11th Internal Teem members and customers started experiencing issues with logging into the Teem application via O365. At around 10:15 PM a fire alarm is pulled by Jared Collins Teem Support Manager**    **Type of Event:**  Outage    **Services/Modules impacted:**  SSO/Logging    **Timeline:**` `**The timeline is posted in MST.**  We received issues of logging in at 5:25 PM but it was still logging perfectly fine after a few tries. We confirmed logging worked over the next few hours and kept a close eye on things. At 10:06 PM A customer reaches out that logging is having issues, internally we are now having issues as well. 10:08 PM a Jira is started to be created for our Engineering team to jump on the issue. 10:15 PM a Fire Alarm is pulled. 10:21 PM Status page is updated with the status of issue at hand. 10:23 PM Engineering confirms they are working on the issue. 11:54 PM Status page is updated once more for the evening. 8:00 AM status page is updated. 12:00 PM Status page is updated again that the issue is currently still at hand. 2:37 PM Engineering has found a fix and implemented said fix to environment. 3:05 PM customers confirming that the issue is resolved. 4 PM the status page is taken down.    **Total Duration of Event:**  16 Hours    **Root Cause:**   The cause of the issue at hand was an outdated token living on a server that is older. The token was re-established and the issue was then fixed.    **Remediation:**  We have put Fire Alarms in to notify the team of SSO expiration a month before it happens so we can update the token and ensure that SSO has no issues going forward.    **Preventative Action:**   Having a system in place to notify of token expiration.

We are currently investigating an issue with Teem. We will update you when we have more information.

We are currently investigating an issue with TEEM. Our Engineering team is currently investigating to determine the cause of the disruption. The next update will be posted at 7 AM CST.

We are currently investigating an issue with TEEM. Our Engineering team is currently investigating to determine the cause of the disruption. The next update will be posted at 9 AM CST.

A fix has been implemented. We are moving into the Monitoring Phase for the next 2 hours. 10:00am CST

As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase. A RCA will be posted in this incident in 10 business days. Please stay subscribed to the page to receive post automatically.

**Teem by Eptura detailed Root Cause Analysis | 2/12/2024** **S1 – Inability to Access Teem**   We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.   **Description:** **On Monday February 12, 2024, around 7:02pm MST both internal and external customers experienced an inability to access Teem. At approximately 7:06pm MST, internal team members were alerted that Teem Login Page has failed a monitoring check. Internal teams immediately began to investigate the issue.**   **Type of Event:** Outage   **Services/Modules impacted:** All production.   **Timeline:** **The timeline is posted in MST.** **We received an alert at 7:02 PM. Our internal Engineering team was able to jump onto the issue. 7:16 PM. We continued investigating and jumped on a call with AWS engineering to get this issue resolved. 8:39 PM Our internal teams are still on a call with AWS engineering. 9:40 PM The call is continuing with AWS as our Engineering team and AWS engineering teams are working together to get database online. 3:02 AM Our Engineering team posted a status page as a Severity 1 letting our customers know the issues at hand 4:20 AM We are still investigating the issue. 5:59 AM Investigation continues 7:11 AM the issue has been resolved and we have moved Virtual Machines to allow hosting for our database. We then notified our customer base and put the status page into a monitoring state. 9:26 AM the issue has been confirmed resolved and the status page has now been updated to reflect.**     **Total Duration of Event:** 12 Hours   **Root Cause:** AWS hosting required an update to our Virtual Machine we are hosting our Database on. Our Engineering team wasn’t notified due to an email being sent to an old email address. We were notified of the outage right away due to failsafe's put in place. Investigation and attempts at resolution started immediately.   **Remediation:** We have moved to RDS for AWS and this should now no longer cause issues with downtime on server updates. We have also updated all emails and notification systems.   **Preventative Action:** Having the correct email in place as well as being on a hosting server allows for quick switchover without downtime.

We are currently investigating a calendar issue with Teem. Some customers may see an error on their calendars - "SSL: CERTIFICATE_VERIFY_FAILED]"

The issue with "Teem calendar error - SSL error "SSL: CERTIFICATE_VERIFY_FAILED]" has been identified and a fix is being implemented. We will post another update at 1:07pm MST or sooner.

A fix has been implemented. We are moving into the Monitoring Phase for the next 1 hour.

As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase. A Detailed RCA will be posted in this incident in 10 business days. Please stay subscribed to the page to receive updates automatically.

**Eptura Workplace Detailed Root Cause Analysis | Severity 2 | October 23, 2023**   **SSO Certificate Failed**  ‌ We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.    **Description:**  On October 23, at approximately 9:30am EDT, support received multiple reports of calendars not syncing events due to an SSL error. ‌ **Type of Event:**  Performance Degradation    **Services\\Modules Impacted:**  Production: Calendars    **Timeline:** On October 23, at approximately 9:30am EDT, support received multiple reports of calendars not syncing events due to an SSL error. After a short investigation support was able to replicate this issue and all customers were notified of the disruption via Status Page at 9:33am EDT. 11:07am EDT internal teams have identified the issue and began working on the fix. At 11:33am the fix was implemented, and the status page was updated to a monitoring phase. As not additional reports were received, and customers confirmed the resolution the status page was updated to a resolved phase at 1:56pm.  ‌ **Total Duration of Event:**  2hrs 3mins    **Root Cause:**   Internal teams found a certificate that expired on the exchange adapter for Teem.    **Preventative Action:**    Internal teams continue to monitor all systems and have created internal processes to ensure alerts are set up to prevent this from happening in the future.

We are currently investigating an issue with Teem. We will update you when we have more information.

We are continuing to investigate this issue.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

**Teem Detailed Root Cause Analysis – Severity 2 – 08/29/2023**   **Description:** On August 29, 2023, at approximately 4:45am EDT, customer support started to receive reports of Calendar Events not syncing at all. When customers are booking events, Teem and Source Calendars are not syncing causing disruption in customer workflow.   **Type of Event:** Service Disruption   **Services\\Modules Impacted:** Production – Calendars   **Timeline:** On August 29, 2023, at approximately 4:45am EDT, customer support began to receive reports of Calendar Events not syncing at all. When customers are booking events, Teem and Source Calendars are not syncing causing disruption in customer workflow. At approximately 10:26am EDT, the engineering team identifies the issue and begins to implement the fix. At 10:36am EDT all customers are made aware that the fix has deployed to production and that we are moving into a monitoring phase. Customers begin to confirm the fix and at 11:01 EDT, the status page is marked as resolved. ‌ **Total Duration of Event:** **6hrs 16mins** ‌ **Root Cause:** The team continues to narrow down the root cause. However, the DevOps team identifies an internal configuration issue and begins to apply a manual fix.   **Preventative Action:** Our internal teams are working on enhancing the observability to bring awareness to the appropriate teams.

We are currently investigating this issue.

Our team continues to investigate "S2 - Outlook and Google Calendars - Events are not Syncing from Teem to Source Calendar". We appreciate your patience as we analyze this.

The team continues to investigate this issue. We appreciate your patience as we work through this.

We are continuing to investigate this issue.

The team continues to work through this complex issue involving S2 - Outlook and Google Calendars - Events are not Syncing from Teem to Source Calendar. We are still in an investigating phase and will continue to provide updates throughout the day.

The issue with S2 - Outlook and Google Calendars - Events are not Syncing from Teem to Source Calendar has been identified. The team is working on a fix and we will continue to keep you posted on the progress. Thank you and we appreciate your continued patience.

We continue to work toward a resolution for the issue, S2 - Outlook and Google Calendars - Events are not Syncing from Teem to Source Calendar.

A fix has been implemented. We are moving into the Monitoring Phase. If you reported a support ticket, our team will be reaching out shortly to confirm the resolution. Thank you for working with us during this disruption.

Since the fix was implemented your external calendar events have been synced, we continue to monitor the calendar events update queue internally. You may experience a delay receiving new calendar events as the queue continues to process.

Calendar events that were created before the fix was implemented have been successfully processed. As new calendar events are being created, our engineering and dev ops team continue to monitor the calendar events updated queue.

As we continue to monitor the calendar events queue, we are seeing newer events processing in a timely manner. If you are seeing a small delay in new events syncing to source calendar, using the "Calendar Refresh" functionality in Teem will assist for quicker processing of newer calendar events.

As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase. A Detailed RCA will be posted in this incident in 10 business days. Please stay subscribed to the page to receive updates automatically.

This incident was resolved, Wednesday June 7, 2023 (17:15 MDT). The RCA will follow shortly.

**Teem Detailed Root Cause Analysis – Severity 2 – June 5, 2023**  **Outlook and Google Events are not Syncing from Teem to Source Calendar**    **Description:**  _On June 5, 2023, at approximately 11:43am MST, customers support received additional reports of “Outlook and Google Calendar Events are not Syncing from Teem to Source Calendar”. An event created in Teem should sync back to the source calendar and vice versa._    **Type of Event:**  _Performance Degradation_    **Services\\Modules Impacted:**  _Production \(Reservation Module\)_     **Timeline:**` `  _On June 5, 2023, at approximately 11:43am MST, customer support received additional reports of “Outlook and Google Calendar events are not Syncing from Teem to Source Calendar”. The initial ticket was escalated to a S2 incident at 11:43am MST and posted to our status page alerting customers. Internal teams continue to investigate this issue through June 6, 2023. Our internal team identified the root cause of the issue at 9:10am MST and began work on the resolution. The status page was updated from investing to an identified phase. At 3:30pm MST a hotfix was implemented, and the status page was moved to a monitoring phase. At 4:27pm MST after the hotfix was implemented, internal teams acknowledged that calendar events are slowly being processed through the queue. The internal teams continue to monitor the queue through June 7, 2023. At 6:56am MST, all prior events created before the hotfix was implemented have been cleared. Internal teams continue to monitor new events that were created after the hotfix was released. As monitoring continued, customers reported that all calendar events are syncing at about 4:02pm MST. At 5:15pm MST, the incident was marked as resolved._     **Total Duration of Event:**  _2 days 5 hours 32 mins_    **Root Cause Analysis:**  _Engineering teams found this was related to a release pushed out May 19, 2023._     **Preventative Action:**   _While the engineering team fixed the regression bug. They found that there was an internal process during the release roll out that needed improvement with QA._