Teem Status

As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase. Thank you for all the patience as we worked through this issue.

This issue is currently resolved with a release that happened last night. After confirmation we are going to be closing out this Status Page. We appreciate all the patience with our customer base as we have worked diligently to get this update out.

**Teem by Eptura detailed Root Cause Analysis | 3/4/2025**  **S1 – Teem Inaccessible**    We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.     **Description:**  **\(The Incident is logged in MST\)**  On March 4th, 2025, the production application encountered a performance issue due to a high server load. This issue lasted for approximately four days. Upon investigation, it was determined that multiple duplicate events were being generated, resulting in an excessive number of database entries and a slowdown in application response times. The situation escalated, ultimately leading to a system outage and the activation of a fire alarm.    **Type of Event:**  Outage    **Services/Modules impacted:**  [App.Teem.Com/Calendaring](http://App.Teem.Com/Calendaring)    **Timeline:**` `**The timeline is posted in MST.**  Around 7:34 AM. We received notice of some people not able to access [app.teem.com](http://app.teem.com). At 10:19 AM our Engineering team clearing tables to help alleviate performance. At 11:29 AM we moved the issue to an S2 due to website being up and accessible. At 10:17 PM we are finishing clearing tables meaning performance should now be at full. 3:49 AM website is working as intended. At 9:17 AM the issue was resolved and we entered the monitoring phase for two hours. At 11 AM we cleared the status page.    **Total Duration of Event:**  27 Hours    **Root cause:**     * Duplicate event generation: The event processing logic did not prevent the creation of redundant events.  * Delayed event processing: Recent codebase changes that integrated Graph API calls extended the event processing time.  * Push Callback Timing Issue:   * When an event is created in our database, a corresponding call is made to Outlook to create the event on their end.  * The increased processing time due to the Graph API changes caused push callbacks to arrive before the original event creation process was completed.  * This resulted in the original event remaining incomplete and missing updates from Outlook.  * A scheduled interval call exists to update missed events, but the frequent push callbacks led to the original event being updated only after multiple duplicates were created.      **Remediation:**  To address the issue, we introduced a locking mechanism by implementing a cache to prevent event creation by push callback before the original event creation flow is completed. Additionally, we handled the scenario where, in the event the locking mechanism fails, the original event is updated with the push callback data. We are also working on sending a unique payload key to Outlook to compare within push callbacks, ensuring more accurate and timely event updates.      **Preventative Action:**     To enhance system reliability, we have improved logging and monitoring for push callback processing, enabling early detection of any anomalies. Additionally, we have implemented a mechanism to prevent the first push callback for an event, ensuring that the original event creation flow is updated smoothly and without any hindrance

As we have not seen further service disruptions after the fix was implemented, we have moved to the Resolved Phase. A Preliminary RCA will be posted in this incident in 2 business days. Please stay subscribed to the page to receive post automatically.

**Teem by Eptura detailed Root Cause Analysis | 2/25/2025**  **S1 – Duplicate Events causing downtime**    We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.     **Description:**  **\(The Incident is logged in MST\)**  On February 22, 2025, the production application encountered a performance issue due to a high server load. This issue lasted for approximately four days. Upon investigation, it was determined that multiple duplicate events were being generated, resulting in an excessive number of database entries and a slowdown in application response times. The situation escalated, ultimately leading to a system outage and the activation of a fire alarm.    **Type of Event:**  Outage    **Services/Modules impacted:**  [App.Teem.Com/Calendaring](http://App.Teem.Com/Calendaring)    **Timeline:**` `**The timeline is posted in MST.**  Around 8:58 AM. We received notice of some instances of slowed experience and inability to access [app.teem.com](http://app.teem.com). At 8:59 AM our Engineering team is reaching out to AWS team to address this. At 10:30 AM we moved the issue to an S1. At 10:45 AM the site became accessible again, but with slowness. At 12:52 PM we resolved this issue and went into a monitoring phase. At 2:02 PM we cleared the status page.    **Total Duration of Event:**  4 Hours    **Root cause:**     * Duplicate event generation: The event processing logic did not prevent the creation of redundant events.  * Delayed event processing: Recent codebase changes that integrated Graph API calls extended the event processing time.  * Push Callback Timing Issue:   * When an event is created in our database, a corresponding call is made to Outlook to create the event on their end.  * The increased processing time due to the Graph API changes caused push callbacks to arrive before the original event creation process was completed.  * This resulted in the original event remaining incomplete and missing updates from Outlook.  * A scheduled interval call exists to update missed events, but the frequent push callbacks led to the original event being updated only after multiple duplicates were created.      **Remediation:**  To address the issue, we introduced a locking mechanism by implementing a cache to prevent event creation by push callback before the original event creation flow is completed. Additionally, we handled the scenario where, in the event the locking mechanism fails, the original event is updated with the push callback data. We are also working on sending a unique payload key to Outlook to compare within push callbacks, ensuring more accurate and timely event updates.      **Preventative Action:**     To enhance system reliability, we have improved logging and monitoring for push callback processing, enabling early detection of any anomalies. Additionally, we have implemented a mechanism to prevent the first push callback for an event, ensuring that the original event creation flow is updated smoothly and without any hindrance

**Teem by Eptura detailed Root Cause Analysis | September 5, 2024**  **S2 |Office365 & Google Calendar | Reservations not syncing**  ‌ We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.    **Description:**  Some TEEM customers experienced difficulties syncing calendar events with both Office365 and Google Calendar. Calendar events were not syncing automatically from the source calendar to TEEM, unless a manual Force Sync was initiated. In some cases, even using Force Sync did not resolve the issue, leading to confusion when using the reservation module. While both Office365 and Google Calendar showed similar symptoms of calendar syncing failures, the underlying causes were different for each service and required separate solutions to resolve.    **Type of Event:**  Functionality Issue    **Services/Modules impacted:**  Calendar Service/ Production    **Timeline:** September 5, 2024, Reported MDT   10:30 AM: Customers began to report the inability to sync their calendar events for Google \(410 errors\) and O365 \(Missing Initialization errors\). An internal Fire Alarm was raised, and all customers were notified that we are investigating the issue via the status page.   September 6, 2024, Reported MDT   11:53 AM: The engineering team has identified the issue and continues to work towards a resolution. Customers were notified that we have moved from investigating to an identified phase. In the meantime, engineers have implemented enhanced measures to mitigate the issue by running a script to manually sync calendars for all reporting customers, four times a day and should improve the reliability of calendar events till full resolution.  September 10, 2024, Reported MDT   9:34 AM: All customers were notified that a solution was implemented for the sync issue affecting Google \(410 errors\) and Office365 \(Missing Initialization errors\). To ensure stability and performance our engineering team is overseeing the process to confirm that the issue has been fully resolved. Monitoring will continue over the next several days.   September 17, 2024, Reported MDT   8:02 AM: All customers were notified that our engineering team has completed the necessary actions and verified that the service is now functioning normally. As no additional customers have reported specific issues in regard to Google \(410 errors\) and O365 \(Missing Initialization errors\). The status page was updated to a resolved state.    **Total Duration of Event:**  11 days, 21 hours, 32 minutes    **Office365 Root Cause:** The issue occurred due to two concurrent requests attempting to refresh the Office 365 access token at the same time. This created a situation where the system, under certain conditions, returned a null token \(None\), which was then passed to the API client. As a result, calendar syncing was interrupted. **Office365 Remediation:** We have updated the system to ensure that, when multiple requests are made, the current access token is used if it’s valid. This prevents the token from being set to None and ensures the API client always receives a valid token. **Office 365 Preventative Measures:** In addition to the fix, we’ve ensured that the system will no longer return a null token in any situation. We have also added logging to monitor the token refresh process closely, allowing us to better detect and resolve any future issues quickly.    **Google Calendar Root Cause:** The system was making multiple attempts to delete events from Google Calendar, even when the event had already been deleted. This caused a 410 error, indicating the resource was no longer available. The issue occurred because the system did not verify whether an event still existed before attempting to delete it. **Google Calendar Remediation:**  1. The system now checks if a Google Calendar event still exists before attempting to delete it, ensuring that deletion requests are only made when necessary.  2. We have introduced a locking mechanism to prevent duplicate or conflicting deletion requests from being made simultaneously.  **Google Calendar Preventative Measure:** To prevent similar issues in the future, Google watchers will be updated using dedicated cron jobs, ensuring synchronization happens in a controlled and consistent manner. The lock mechanism will also ensure that API calls are handled sequentially and without conflict.

We have found this issue to be mitigated and is only affecting a small subset of customers that we are working diligently to support and find a resolution. If you are experiencing issues, please don't hesitate to reach out to support.

**Teem by Eptura detailed Root Cause Analysis | August 20, 2024**  **S2 O365 Multiple Users Marked Inactive**    We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.    **Description:**  Customers using O365 were unable to access the Teem platform. When trying to login, customers were met with an error message.    **Type of Event:**  Functionality Issue    **Services/Modules impacted:**  Production/ Office 365     **Timeline** \(Reported MST\)**:**` `  On the morning of August 20, 2024, at approximately 8:20 AM MST, our support team began receiving reports from end users who were inadvertently marked as Inactive in the Teem platform. We promptly informed all Teem customers of the incident via our Status Page.  At 10:24 AM MST, we marked the Status Page as resolved. However, recognizing the importance of addressing this issue thoroughly, our engineering team continued to collaborate closely with our Support team to find a comprehensive solution. To prevent further impact, the engineering team implemented a nightly script to revert the status of users marked as Inactive.  The investigation continued until November 1, 2024, when our engineering team successfully released a HotFix to address the issue. Although the initial HotFix did not fully resolve the problem, the team enhanced our logging capabilities to better track and understand the behavior.  On November 11, 2024, the additional logs provided valuable insights, enabling our engineering team to resolve the issue in our QA environment. The final HotFix was released on November 14, 2024, and customers have confirmed the resolution.    **Total Duration of Event:**  83 Days    **Root Cause:**   The issue arose from the concurrent processing of multiple user batches. During this process, one thread completed its task earlier than expected and inadvertently deleted all cache keys, including the main Sync\_Key and associated batch\_keys. This led to subsequent threads receiving an empty batch list, which resulted in the deprovisioning or deactivation of users.    **Remediation:**  To resolve this issue, we have implemented database row-level locking for batch processing. This ensures that batch processing happens sequentially, avoiding conflicts. Key updates include:  1. Introduced a dedicated table to track batch process counts and the ID of the last processed batch.  2. Applied database row-level locks to manage synchronization safely and efficiently.  3. Updated the deprovisioning process to occur only after verifying that all batches are fully processed.     **Preventative Action:**    To prevent recurrence, we have made the following improvements:  *  Enhanced concurrency handling to ensure seamless user batch synchronization.  * Added extensive logging in CloudWatch to monitor and better understand process behavior.  * Streamlined the user synchronization process to ensure that all O365 users are successfully synced with the Teem directory.  These enhancements will significantly improve the reliability and performance of our system. Thank you for your patience and support as we continue to make these improvements.

This was an internal incident and has been resolved. Thank you for understanding.

This incident has been resolved.

The scheduled maintenance has been completed.

Maintenance completed: Action required We have successfully completed maintenance to enhance security and compliance by transitioning to Microsoft Graph API for calendar services using Microsoft Exchange. Action Required: If you are currently using O365 service accounts for calendaring, you must reauthenticate your calendar and provide consent for your service accounts before Monday, Feb. 24, to ensure uninterrupted calendar syncing, even if you recently reauthenticated or updated to the Delegate access method. View steps to reauthenticate you service account

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

At this time, we are closing this maintenance window. Further communications will be communicated as an incident due to degraded performance that some customers are reporting. Next update will be at 0700 Hours MST.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.