Historia awarii Cronofy

We are investigating an issue where Event Trigger notifications are failing to send in our US data center

A failing component in our US data center led to scheduled notifications for Event Triggers backing up and not sending. We've resolved the immediate cause, which will lead to notifications being delivered behind schedule. We are monitoring this and investigating any wider impact from the issue.

All pending notifications have now been flushed. Notifications scheduled since approximately 02:00 UTC on July 1st were failing to be sent. This affected customers in the US data center making use of the following API features: - Event Triggers; approximately 9000 notifications for calendar events starting/ending will have been delivered late - Scheduler Workflows; a small number of Scheduling Request lifecycle notifications will have been delivered late - Meeting Agents; a small number of scheduled Meeting Agents will not have arrived to their scheduled meetings A postmortem of the incident will take place and be attached to this incident in the next 48 hours.

Between approximately 02:25 UTC on July 1st and 10:11 UTC on July 7th 2026, scheduled notifications in our US data center failed to send. The affected notifications were queued but never picked up, and were flushed once the underlying cause was cleared. The impact was limited to customers in the US data center using notification-driven API features. Approximately 9,000 Event Trigger notifications for calendar events starting and ending were delivered late. A small number of Scheduler Workflows lifecycle notifications were also delivered late, and a small number of scheduled Meeting Agents did not arrive to their meetings. The root cause was a regression introduced on April 22nd, in an internal service responsible for sending scheduled notifications. The service uses leader election so that a single process handles distributing the imminent notifications at any one time, coordinated through a lock. The regression made that configuration fragile. If a leader did not release the lock cleanly, no other process would override it. When the leader stopped in the early hours of July 1st, it left the lock held, and with no process able to take over, notifications quietly backed up from that point. Nothing alerted us to this. The failure mode was an absence of activity rather than an error, and we had no alerting on notification throughput that would have caught the drop-off. During the incident, we manually cleared the stuck lock, which allowed the service to resume and flush the backlog within four minutes. ## Timeline All times are in UTC. * **April 22nd, 10:27** A change is merged that makes the leader election's lock-holding fragile. The fault is latent and has no immediate effect. * **July 1st, 02:25** The leader process in the US stops in a way that invokes the regression, leaving the lock held. No notifications are picked up from this point. * **July 2nd, 06:42** A support ticket reports an issue with Event Triggers. * **July 6th, 14:53** A second support ticket reports an issue with Event Triggers. * **July 7th, 09:38** The pattern across the two tickets is identified internally and raised. * **July 7th, 09:50** Telemetry is reviewed, showing a clear drop-off in notification sends. * **July 7th, 09:57** An internal incident channel is opened. * **July 7th, 10:05** The held lock is raised as the probable cause. * **July 7th, 10:07** The lock is manually cleared. * **July 7th, 10:07-10:11** The backlog of late notifications is flushed by the now-working service. * **July 7th, 10:12** The public incident is opened. * **July 7th, 10:40** Impact is reviewed and confirmed, including the other affected paths \(Scheduler Workflows, Meeting Agents\). * **July 7th, 11:12** The public incident is resolved. ## Retrospective We always ask the same three questions. ### Could the issue have been resolved sooner? Once we were looking at the right thing, resolution was fast. The probable cause was raised within eight minutes of opening the channel, and the backlog was flushed four minutes after that. We are satisfied with how the resolution itself went. ### Could the issue have been identified sooner? This is where we fell short, and badly. The failure was live for almost six days. Two things let it run that long. First, we had no alerting on notification throughput, so a complete stop in sending produced no signal. It is harder to alert on things _not_ happening, but that is exactly the case this warranted. Second, the first support ticket arrived on July 2nd and did not surface the pattern. It took a second ticket, four days later, for the connection to be made. Six days is too long for a customer-facing feature to be silently failing. ### Could the issue have been prevented? The regression was introduced in April and sat latent until a leader failed to shut down cleanly for the first time. The fragility was in the lock's release path, the case where a leader stops without releasing cleanly, which is precisely the kind of edge that is easy to miss in review and under-covered by tests. We did not catch it at either stage. ## Actions We have added alerting on notification throughput, so that a drop-off in sends is positively and quickly identifiable rather than depending on customer reports. We have fixed the code so that a lock held by a stopped leader is reliably released and picked up by another process, rather than leaving the work stuck. We have added tests around the configuration to ensure we cannot fall foul of the misconfiguration by accident, and integration tests to show that an orphaned lock can be obtained by another process. We are reviewing how we handle support tickets that describe an absence of expected behavior, so that a single ticket of this kind prompts a check of the relevant telemetry sooner.

We've been monitoring an issue on a cache cluster node since shortly after the maintenance at the weekend. As part of mitigation, customers may see increased errors from the API for a short time period. We will begin the mitigation momentarily.

The mitigation has been completed. Increased API errors were observed from 09:30:30 to 09:31:30 UTC.

We have seen that Google Meet Meeting Agents are being rejected from the meetings before being able to join and are investigating the issue. These meetings will fail to be recorded and transcribed.

We are rolling out a fix which has tested positively internally, and will be monitoring success rates

We are continuing to monitor for any further issues.

Google Meet Meeting Agents are making join attempts successfully since the fix release earlier. We will continue to monitor their performance.

After monitoring we've seen that the fix has improved performance but we're still seeing elevated cases of failures to join Google Meet Meetings, so are still working improving things further.

We are now rolling out a further fix which has tested well internally, and will continue to monitor its performance.

Google Meet Meeting Agents are attempting meeting joins successfully again after our further fix earlier today. We shall continue to monitor but consider this incident to be closed.

Google's API is returning an increased number of 503 errors when we attempt to sync calendars. This affects a small percentage of calendars, and some users may see delayed calendar syncs as a result. We're continuing to monitor.

We are continuing to see elevated errors when communicating with Google Calendar. Cronofy is operating normally - event changes are still syncing correctly. The only impact is that some changes may take slightly longer to be reflected in Google Calendars than usual. We are continuing to monitor the situation, but will only update this incident if there is a significant change in either direction.

Events continue to sync events into Google Calendars as normal. We are still seeing an elevated number of errors between Cronofy and Google Calendar, but are confident that our system are keeping events in sync correctly. The vast majority of events are still syncing almost immediately, while a minority of updates sync after a few seconds. This should have no impact on customers applications, so we are resolving the incident. We will continue to monitor the situation and will raise a new incident if performance declines to a noticeable level.

Between 03:08 UTC and 03:10 UTC we observed increased API error rates in our US data center. During this time, customers may have received 500 Internal Server Error responses. Since that time, error rates have returned to normal levels. These errors were caused by a read-only database server going through an automated recovery process.

We've seen that Google's API is returning an increased number of 503 errors when we attempt to sync calendars. This is affecting a small percentage of calendars, and some users may see delayed calendar syncs as a result. We're continuing to monitor and investigate.

We are seeing signs of improvement from Google's side -- error rates have improved a great deal but are not yet quite back to normal levels. We continue to monitor the situation.

Google have identified an issue within their infrastructure which they're tracking at https://www.google.com/appsstatus/dashboard/incidents/KamjizntSiNtRETcBL2Q From our side, we're now seeing very low enough error rates that we don't think that users will experience any noticeable negative impact. We'll keep watching Google's incident until it's fully resolved.

Google has closed their incident and we've seen error rates remain low so are likewise marking this as resolved now.

Since approximately 12:30 UTC a subset of calendar events failed to be synced to users' calendars. We've identified the source of the issue and released a fix, we are now assessing the impact and actions necessary to resolve the issue for previously impacted calendar events.

We have identified that affected events were those being written to Microsoft calendars and using the `integrated` conferencing option. The system is already retrying these failures and successfully writing them. We are looking to accelerate this process.

At approximately 12:30 UTC a change was released that prevented events being created in Microsoft calendars if integrated conferencing was being used. A fix was released and the last error observed at approximately 16:05 UTC. Since that time over 90% of the impacted events have been successfully synced due to standard retry mechanisms with the platform. We are working to enqueue additional attempts for all affected calendars to speed up this process and to be certain nothing has been missed.

At approximately 12:30 UTC a change was released that prevented events being created in Microsoft calendars if integrated conferencing was being used. A fix was released and the last error observed at approximately 16:05 UTC. Over 90% of the impacted events were successfully synced due to standard retry mechanisms with the platform by 17:00 UTC, we also ran a sweep to proactively retry all remaining events which completed by 17:30 UTC. This incident is now resolved and operations are back to normal. We will be conducting a postmortem and will publish the findings of that next week.

On Friday February 27th 2026 at 12:28 UTC, a change was released to our production environment that caused a bug preventing events using Microsoft Integrated conferencing from being synced to calendars. At 16:05 UTC, a change was released to fix the issue and full functionality was restored. By 17:35 UTC, all previously impacted events had been successfully retried and synced and the incident was closed. In total, we identified 2,254 events that had been affected by this bug, all of which have now been successfully synced. We always ask the questions: * Could the issue have been identified sooner? * Could the issue have been resolved sooner? * Could the issue have been prevented? In this instance, the issue could and should have been identified by us earlier. The change that introduced this bug was released at 12:28 UTC, but we only became aware of the issue when a customer reported issues to us at 15:27 UTC. We never want to rely on customers having to tell us about issues in our platform before we are aware, so this is an instance where gaps in our system monitoring meant the issue went undetected for too long. We deployed a fix for this issue at 16:05 UTC, which was 37 minutes after the bug report email was sent, and 13 minutes after we identified the extent of the issue and opened an incident at 15:52 UTC. Therefore, we don’t believe the bug could’ve been resolved any sooner, as we acted quickly to release a fix once we were fully aware of the problem. Along with a gap in our alerting, the bug should have been prevented through automated testing, which would’ve flagged the issue before the initial change was deployed at 12:28 UTC. In this instance, our testing failed to check the logic down the path that Microsoft Teams integrated conferencing followed, because a check had been mocked instead of being integration tested, leading to a blind spot. We have addressed this and are reviewing where we have used test mocks which could hide integration issues. Actions: * Further guidelines around coding principles to be provided to our Engineering team regarding our approach to automated testing * Improve our system monitoring to provide specific alerting around this particular area

We are aware of issues affecting our Google Chrome Extension preventing users from creating scheduling requests in Gmail. We are currently investigating and will provide an update as soon as possible.

We are still working to identify the exact root cause of the issue. We will provide further updates as soon as possible.

A recent Google Chrome update has changed a mechanism that our Gmail extension relies on. We are developing a fix that is compatible with this new mechanism. We will provide an update in a few hours or when we have an updated version of the extension ready.

We have submitted an updated version of the Chrome Extension to Google which contains a fix for this issue. The updated version will be available as soon as it has been checked and published by Google. We will provide an update once we confirmation of this from Google.

We have released version 0.22.25 of the Cronofy Chrome extension, fixing the issue. If you have any problems, please first check that both Google Chrome and the Cronofy Chrome extension are up to date. If you have any additional issues then please contact [email protected].

Between 12:52 and 12:55 we saw that calls to our API encountered a higher level of 500 Server Error responses than usual We have investigated this and it appears to relate to a period where we saw a large spike in traffic, a rapid drop and another spike in quick succession resulting in some temporary 500 errors. This is now resolved but we will continue monitoring for any further issues.

We are investigating an elevated level of failures for provisioning MS Teams Conferencing.

We are continuing to investigate - the issue is not impacting all customers but we are not yet seeing a pattern to the domains or events impacted.

We are still investigating; we are capturing additional logs to help us narrow down the failing population. Some conferences are being provisioned after automatically retrying, but still not as many as we'd expect.

We've found some evidence that other users of the Microsoft Graph API are experiencing the same issue; we're looking for anything that may let us mitigate the impact and we're reaching out to Microsoft for support.

We believe the root cause is a lack of generated URLs for a meeting when we call Microsoft Graph APIs. We are waiting for a response from Microsoft on the issue. Unfortunately the timing of this is out of our hands. We are also working on alternative ways to provision Teams meetings, but this will not provide a short term fix. We are continuing to work on finding a way to resolve, mitigate or provide an more reliable path to Teams provisioning. We will update this incident over the coming days as we make meaningful progress.

We have seen that our error rate provisioning MS Teams Conferencing has returned to normal, low levels since 16:10 UTC yesterday. Normal levels of service have resumed, and we're working to improve our monitoring so that similar issues are detected sooner in the future.

We are currently investigating degraded sync performance for Microsoft 365 as we are experiencing a much higher error rate than normal.

We believe this is related to a Microsoft incident (ID: MO1221364). We are keeping a close eye on how this progresses

We are continuing to monitor the situation, but have not seen any improvement so far

Between 22:00 and 01:15 UTC we saw elevated error rates when performing sync operations with Microsoft 365. Microsoft continues to work on a full resolution to the incident with ID MO1221364. However, the error rates we are now observing are back to normal levels.

We have chosen to initiate a database failover in our US data center. Because of this, customers may have received a 500 Internal Server error between 20:45:15 and 20:47:30. The vast majority of these errors were observed between 20:45:15 and 20:46:00. We are now monitoring the database post-failover.

We're continuing to monitor the situation. An additional spike in server errors was observed between 21:03:15 and 21:03:30

We are continuing to monitor post-failover

Between 20:45:15 and 20:47:30, and 21:03:15 and 21:03:30 UTC, we observed increased API error rates in our US data center. The first window of these was caused by a database failover, the second of these was caused by a reboot of the database cluster reader node. We took this action in an attempt to alleviate a small but persistent number of database errors that we had seen since the earlier maintenance. During the maintenance, we believed these errors were transient and that they would dissipate throughout the day. As this did not happen, we felt it was necessary to take action now, and that action was to failover, reboot, and then ultimately replace the affected node. Since the replacement, we have not observed the aforementioned database errors. If you have any questions, please email [email protected].

Between 16:55 UTC and 17:03 UTC we observed increased API error rates in our US data center. Error rates returned to normal by 17:03 UTC. We are monitoring and investigating the root cause.

Error rates remain at normal levels, and we are continuing to monitor.

Between 16:55 UTC and 17:03 UTC we observed increased API error rates in our US data center. During this time customers may have received 401 Unauthorized or 500 Internal Server Error responses. Since that time error rates have returned to normal levels. A postmortem of the incident will take place and be attached to this incident in the coming days.

On November 18th 16:47UTC we had several errors about our database cluster encountering a sync issue. This ultimately resulted in an automated recovery mechanism kicking in, resulting in about 90 seconds where most communication with the database was blocked. The errors indicate that the root cause was an issue with the storage layer that the database relies on. No data was lost during this procedure, but customers will have seen our APIs respond with HTTP 500 errors as the database was unreachable. We posted an incident page as soon as we had investigated and understood that there were no ongoing issues. We have set up further alerting on these specific errors going forward to provide more thorough monitoring coverage. We will also shortly be announcing a database upgrade, planned prior to this incident, which will include stability improvements for our database technology. * `16:47` - We get the first database log message relating to replication * `16:54` - Database replica lag begins to climb * `16:55` - We start to see more of the replication errors * `16:56` - Internal alerting highlights increased client and server errors * `16:57` - We get a message stating that the database replica has been disconnected and reconnected to automatically resolve the issue. * `16:58` - Database replica lag returns to normal * `17:14` - Internal incident channel opened for investigation * `17:37` - [Public Status Page incident opened](https://status.cronofy.com/incidents/nmhfk7kl9t9m) * `17:37` - Monitored the database to make sure error levels had returned to normal * `18:33` - Status Page incident closed ### Retrospective * Could it have bee identified sooner? * Yes * The database replication error is rare enough that it should’ve been worth being notified of. Had we been notified of this, we would’ve known there were issues. * Could it have been resolved sooner? * No * Even if we had known about this earlier, we couldn’t have done anything other than what ended up happening to resolve the issue. * Could it have been prevented? * No * With our current understanding, the issue appears to be related to a database issue that was out of our control. Once the issue started happening we wouldn’t have been able to take any action to resolve as the issue essentially self healed. ### Actions * Set up more alerting for specific database replication errors. * Conduct further internal reviews to solidify our understand on the above errors.

We are currently investigating the cause of Google calendar sync degradation

Google's API error rate has dropped again and has been at normal levels for the past 10 minutes. We'll continue to monitor and confirm this holds out.

Between 18:15 UTC and 18:28 UTC, we observed elevated error rates, and Google Calendar users may have experienced delays with calendar reads and writes. Since 18:28 UTC, all Google Calendar operations have been functioning as normal.

Customers using our Canadian data center may have experienced interruption of service when accessing both our API and application. This was caused by a recent change and will have presented as HTTP 502 - Gateway Timeout statuses. We have reverted the change and are in the process of assessing the exact scope of the issue.

Between 10:12 and 10:18 UTC a change to the way back-end resources are allocated led to a scenario whereby the services processing requests began being forcibly restarted. This was caused by the services rapidly exceeding predefined resource limits and being taken out of service. This resulted in a partial outage that affected a small number of requests. Our internal monitoring alerted us to the issue and the changes were reverted. We're continuing to monitor the situation but consider this incident resolved. Full service was restored at 10:18 UTC.

### **Summary** On October 23rd, between 10:11 and 10:18 UTC, we experienced a partial outage in our Canadian data center. During this time customers may have received server errors or experienced increased response latency when using the scheduler or our API. Background processing was not impacted during this incident. At 10:09 UTC, we released a change that increased how much work each individual web server process can handle at a time. This inadvertently caused those processes to hit resource limits after a short period of time, which then resulted in those processes being restarted. This then kept repeating until we reverted the change. The first errors happened at 10:11 UTC, and our engineers were alerted to this at 10:14. At 10:17 we reverted the change, and by 10:19 UTC the server errors had stopped and response latency had returned to normal. ### **Postmortem** We always ask the questions: * Could it have been resolved sooner * No, reverting the change happened very quickly after we were alerted to the errors. The errors stopped as soon as our deployment processed had rolled back * Could it have been identified sooner * No, alerts came through within minutes of the initial change being deployed. Our external monitoring tools also check the availability of our API and scheduler every minute. * Could it have been prevented * Yes * This was not tested in our test environment before deploying. Replicating production traffic loads in difficult, so had we deployed this in our test environment we may not have encountered the exact same issue. However, there may have been some signs of increase resource utilization. * Rather than increasing how much work each individual web server process can handle, we could have increased the number of processes running. This is known as scaling horizontally, rather than scaling vertically. * There was also an oversight on how resource utilization would be affected by this change. From this we have now learnt more about the nuance involved with making these specific changes. ### **Actions** * Make it harder to make this same mistake again by adding comments to the codebase to ensure if a similar change is made in the future that we also appropriately increase resource limits * We felt that we could have been quicker to communicate about the issue on our status page. To help with this in the future we will: * Update our Incident Playbook to ensure guidelines on communication timelines are clear to all * Give more engineers access to post and update incident on our status page

Since around 07:15 UTC our US data center has been severely impacted by an ongoing incident in AWS us-east-1, where it is hosted: https://health.aws.amazon.com/health/status AWS have identified a potential root cause for the issue and are working on multiple parallel paths to accelerate recovery. At 09:20 UTC we started seeing signs of recovery. As of 09:30 UTC all high priority tasks were cleared which are those with the most visible user impact. We are unable to scale additional capacity to clear lower priority tasks at present, which are more background processes such as polling calendars (Apple) and housekeeping style tasks. This backlog is being processed, but not at a pace that means we are clearing it as quickly as we would like. All other data centers are fully operational.

Since around 07:15 UTC our US data center has been severely impacted by an ongoing incident in AWS us-east-1, where it is hosted: https://health.aws.amazon.com/health/status At 09:15 UTC we started seeing signs of recovery. As of 09:30 UTC all high priority tasks were cleared which are those with the most visible user impact. Lower priority tasks were cleared by 10:20 UTC. AWS continue to work towards full resolution. We remain unable to reliably provision additional capacity in this data center. We have taken steps to ensure we retain the capacity we already have. Performance of the US data is now back to normal, we continue to monitor whilst the underlying AWS incident is active.

AWS continue to work towards a full resolution. We remain unable to reliably provision additional capacity in our US data center. However, since the previous message, we have managed to increase the available capacity. Performance of the US data center continues to be normal, and we will continue to monitor.

On October 20th, AWS's us-east-1 region experienced a significant incident impacting multiple underlying services. This led to the operation of Cronofy's US data center, which is hosted in this AWS region, being severely impacted. None of Cronofy's other data centers were affected at any time. The main impact was felt between 07:15 and 09:15 UTC where many critical components struggled to communicate with AWS's services. Performance was severely degraded throughout this period. We saw some recovery from 09:15 UTC, with the backlog of high priority tasks completed by 09:30 UTC, these are the tasks most likely to have a user-noticeable impact. Some issues remained, mainly the ability to provision additional servers which has been widely reported by other AWS customers. However, we had sufficient capacity to clear the backlog of lower priority tasks by 10:20 UTC. Performance of the US data was back to normal from this time, and we continued to monitor. Throughout the day we ran with an altered configuration to obtain as much capacity as we could from AWS to provide as smooth a service as possible. Our ability to provision additional capacity was restored at 19:15 UTC. While the underlying AWS incident is still open, we are considering this incident to be resolved as we have been able to resume normal operations. A postmortem of the incident will take place and be attached to this incident in the next 48 hours. If you have any queries in the interim, please contact us at [email protected].

### Summary On October 20th, AWS's US-East-1 region experienced a significant incident impacting multiple underlying services. This led to the operation of Cronofy's US data center, which is hosted in this AWS region, being severely impacted. None of Cronofy's other data centers were affected at any time. The main impact was felt between 07:15 and 09:15 UTC where many critical components struggled to communicate with AWS's services. Performance was severely degraded throughout this period. We saw some recovery from 09:15 UTC, with the backlog of high priority tasks completed by 09:30 UTC, these are the tasks most likely to have a user-noticeable impact. Some issues remained, mainly the ability to provision additional servers which has been widely reported by other AWS customers. However, we had sufficient capacity to clear the backlog of lower priority tasks by 10:20 UTC. Performance of the US data was back to normal from this time, and we continued to monitor. Throughout the day we ran with an altered configuration to obtain as much capacity as we could from AWS to provide as smooth a service as possible. Our ability to provision additional capacity was restored at 19:15 UTC. The AWS incident was resolved at 22:53 and we consider this to be closed. ### Postmortem We always ask the questions: * Could this have been resolved sooner? * No, we very much at the mercy of AWS as they worked to resolve the issue and remove throttling as the services started to come back to life. The lack of access to the AWS console also made it difficult for our team to make meaningful changes to mitigate impact at the time. * As a business we operate our Data Centres completely independently of each other. This was a decision we made in order to ensure our customer’s data remains exactly where they choose to store it and removes any concerns around cross-region data transfers. As a result of this, we did not have the option to fallback onto another AWS region at the time of the US-East-1 outage, which could’ve provided increased availability at the height of the incident. We believe our initial decision to not perform any cross-region data transfers is still in the best interests of our customers, and don’t believe changing this to provide extra resiliency to be a viable option. * Could this have been identified sooner * No, our PagerDuty alerts came through 10 minutes before AWS announced the incident, and an internal incident channel was opened shortly afterwards. * Our platform, however, showed enough signs of life that our external monitoring services were not reporting downtime. * Our customers could have been informed sooner in this particular instance: * Our third-party StatusPage \([status.cronofy.com](http://status.cronofy.com)\) was not available during the worst of the incident, only returning at 09:50 UTC. Due to the AWS issues, we could also not update our DNS records to point to a different location. * We had to make use of improvised channels \(LinkedIn, X, Facebook\) until we could set up a temporary page at [status.cronofy.com](http://status.cronofy.com) at 08:45 UTC. This resulted in delays in informing our customers of the issues we were facing. * Could this have been prevented * No, this issue was due to an AWS incident and was not in our control. * The previously mentioned Data Residency choices mean that falling back to a different region was not an option. ### Actions * Review our external StatusPage to ensure we have the best setup for informing customers of ongoing incidents. * Review backup StatusPage options: * Update our internal Incident Playbook to list backup notification platform options * Review deployment pipeline setup for added resiliency in such situations

Since 10:40 UTC we have observed an increased error rate for Apple calendar sync operations. We are still seeing more successes than failures but consider there to be enough failures to mark Apple as degraded. Our retry mechanisms appear to be doing their job, but changes to calendars may take longer to sync both to and from Apple as a result of the underlying errors. We are monitoring the situation.

Starting at 10:40 UTC we observed an elevated error rate when communicating with Apple's calendar service. Errors returned to regular levels by 11:35 UTC. Sync with Apple calendars may have taken longer both to and from Apple as a result of the underlying errors during this period.

We are currently investigating degraded sync performance with Microsoft 365

Between 15:46 and 16:16 UTC, we saw an increase in response times from the Microsoft 365 Graph API. Since then we have not seen a repeat of this and response times have returned to normal.

We are seeing a small number of 5XX errors being returned from our US data center. We are investigating and will update shortly.

Investigation has shown that between 15:19 and 15:27 UTC a database issue resulted in a partial loss of service. During this period requests may have returned a 5xx status and failed to process. The database issue has now been resolved and we're moving to monitor the environment.

We're monitoring the environment to ensure things continue to run as expected.

Normal operations have continued and we are marking this incident as resolved. A postmortem of the incident will take place and be attached to this incident in the next 48 hours. If you have any queries in the interim, please contact us at [email protected]

# Summary Between 15:19:38 UTC and 15:26:55 UTC on Saturday August 16th 2025, requests made to the API in our US data centre may have received a HTTP status of 500 - Internal Error. This was caused by a transaction on one of our database tables not releasing a lock in a timely fashion which, in turn, caused a bottleneck that led to service degradation as subsequent requests queued behind it. The root cause was identified during the incident and will be addressed by reviewing the way these transactions are handled with a view to preventing future locks. ## Timeline All times are UTC unless otherwise stated. 15:19 - Table lock begins 15:21 - On-call engineers alerted to issues processing requests as 5XX responses increase 15:22 - Initial investigation begins 15:27 - Database lock clears, backlog begins processing 15:28 - Wider environment confirmed to be functional. No undue load observed 15:30 - Lock on database table observed to be likely cause 15:32 - Lock on database table confirmed to be the cause 15:50 - We begin extended monitoring 16:28 - No further long-running locks observed. We mark the incident as resolved ## Retrospective We always ask the questions: * Could the issue have been resolved sooner? * Could be issue have been identified sooner? * Could be issue have been prevented? In this case, we feel that yes, we could have identified the root cause sooner as it has highlighted a gap in our database monitoring that we’ll be working to address. Similarly, it could have been resolved sooner if we were to opt to clear the lock ourselves. However, our preference is to avoid the need for intervention by addressing the way we approach these queries in the first place. ## Actions Our Site Reliability Engineers will be looking at addressing the pinholes in our monitoring of database transactions to improve visibility in the future, while we look to improve the approach we take to table locks and long running transactions.

We've observed an elevated rate of errors from Google's Calendar API and are investigating the scale and impact. Updates to and from Google Calendar profiles may see slight delays during this time. We are monitoring the situation.

Google is returning a higher rate of 5xx errors when we try to perform token refresh operations. Approximately 1/3rd of our attempts to refresh tokens are seeing this error. The increase in errors started approximately 14:57 UTC. It appears our normal mechanisms are allowing retries to be processed successfully, and limiting the impact to the wider service. Updates to and from Google Calendar profiles may see slight delays during this time. Google appear to have acknowledged the incident: https://www.google.com/appsstatus/dashboard/incidents/oFcAZTr4EVieF5Fr6Ee9

Google's API error rate has dropped again and held at a normal level for the past 10 minutes, and both read and write operations to Google Calendar profiles appear to be back to our normal level of performance. We'll continue to monitor and confirm this holds out.

API success rates have remained stable since our previous message, and Google have resolved their own incident: https://www.google.com/appsstatus/dashboard/incidents/oFcAZTr4EVieF5Fr6Ee9 Between the times of 14:57 UTC - 15:35 UTC, we saw elevated rates of errors, and Google Calendar users may have had slight delays to calendar reads and writes. Since 15:35 UTC everything has been functioning as normal.