Cigo Tracker Status

Microsoft has confirmed that all affected Azure services have been fully restored. Cigo Tracker systems were back to normal operation as of 16:59 EST. Following service stabilization, we have now reverted from our backup environment back to our primary infrastructure. All systems are operating normally, and performance remains stable. If you wish to receive the official Microsoft incident report once it is published, please email us at support@cigotracker.com , and we will make it available to you as soon as we receive it. Thank you for your patience and understanding throughout this Azure service disruption.

### Summary On **September 15th, 2025**, we experienced service disruptions affecting the Import Tool and Confirmation module notifications. The root cause was a misidentified SSL certificate dependency during the scheduled **Azure Database for MySQL maintenance**, which included the rotation of the MySQL server’s SSL certificates. ### Impact * The Import Tool was unavailable for a period of time. * Confirmation module notifications \(via SMS and email\) were disrupted. * Some other automated functions experienced degraded performance. ### Root Cause While we were aware of the scheduled SSL certificate rotation as part of Azure’s maintenance process, our evaluation process incorrectly identified which certificates were being used by certain subsystems. As a result, several services lost connectivity when the certificates were rotated. ### Mitigation and Resolution * Our engineering team corrected the certificate configuration for the affected subsystems. * The Import Tool and Confirmation module notifications were restored to full functionality. * Safeguards are being added to ensure that future SSL rotations do not cause similar disruptions. ### Contributing Factors At the onset of the incident, we initially believed the outages were linked to updates and fixes deployed on Thursday and Friday of the previous week. This led to rollbacks that did not address the issue. Although unrelated to the outage, some of those updates remain rolled back and will be re-published in the coming days. ### Next Steps * We are reworking our dependency system analysis to more accurately map SSL and package dependencies across all services. * We are improving our monitoring and review processes for certificate rotations to ensure full coverage across all subsystems. * We will re-publish rolled-back updates after additional validation. ### Conclusion We sincerely apologize for the operational problems this incident caused. We take these issues seriously and are committed to ensuring that similar outages do not reoccur in the future.

The web application outage has been resolved, and all systems are currently operating normally. While we've successfully isolated the scope of the issue, there are still some contributing factors that remain under investigation. Our team is continuing to analyze the event to determine the full root cause of today's outage and to implement measures that prevent recurrence. We sincerely appreciate your patience and apologize again for the disruption.

The issue has been fully resolved. We reverted the deployment that caused the disruption, and the related changes will be revised and rescheduled for a future release. Thank you for your patience and understanding throughout the investigation.

## 📣 Incident Summary – March 18–19, 2025 **Service Impact on Cigo Tracker due to Azure Regional Outage** On March 18, 2025, Cigo Tracker experienced intermittent service disruption due to a regional outage within Microsoft Azure’s East US data center region. Below is a summary of the root cause, impact, and the steps being taken to prevent future occurrences. ### 🕒 What Happened? Azure’s East US region suffered two separate impact windows: * March 18, 13:37 to 16:52 UTC * March 18, 23:20 to March 19, 00:30 UTC The incident was triggered by a **third-party fiber cut** during external drilling work, which caused reduced network capacity in one of Azure’s Availability Zones. A **tooling failure** during Azure's recovery efforts later reintroduced traffic prematurely, leading to congestion and a second round of intermittent connectivity issues. ### 🔍 Root Cause 1. **Fiber Cut**: A construction-related accident physically damaged fiber cabling serving the East US datacenter, degrading network capacity. 2. **Router Maintenance**: A key router in the same zone was already under repair, limiting redundancy. 3. **Tooling Error**: Azure’s automated recovery system failed to fully isolate damaged infrastructure, inadvertently reintroducing traffic too early. 4. **Congestion Spillover**: The unexpected traffic load caused congestion to spread beyond AZ03 into neighboring zones. ### 🎯 Impact on Cigo Tracker While the Azure issue only affected a subset of inter-zone traffic in East US, this included infrastructure we rely on, resulting in **intermittent connectivity issues** for some customers during the incident windows. Core services were restored once Azure manually completed isolation and fiber recovery work. ### 🛠 Resolution Timeline * **13:37 UTC, Mar 18** – Outage begins due to fiber cut * **13:55 UTC** – Initial mitigation starts; traffic rerouted * **16:52 UTC** – First impact window ends * **23:20 UTC** – Second outage begins due to tooling error during recovery * **00:30 UTC, Mar 19** – Final mitigation complete * **06:50 UTC** – Full restoration of all infrastructure ### ✅ What Azure is Doing to Prevent Recurrence * Fixing tooling failures that allowed reintroduction of unready capacity \(by May 2025\) * Accelerating a capacity upgrade for the East US datacenter \(by July 2025\) * Architecting better safeguards to prevent impact from spreading across zones \(by February 2026\) We apologize for the inconvenience caused. Please rest assured that our team is working closely with Azure and continuing to invest in the resiliency of our platform. If you have any questions or would like help designing a more resilient setup, feel free to reach out to our support team. Thank you for your continued trust

On November 14th, at approximately 10:15 PM, during a planned network maintenance session, we undertook a migration of core network routing resources. Unfortunately, this migration inadvertently impacted our DNS records, causing DNS Probe errors for some customers attempting to access our services. To address the issue, our operations team promptly remapped the affected DNS records and adjusted network restriction rules to restore connectivity. The total downtime was less than one hour. The root cause of this incident was an unforeseen interaction between the migration process and automatic changes to both networking rules and DNS records. While this was an unanticipated challenge, we’re grateful for the swift recovery efforts of our operations team. The issue has been fully resolved, and measures have been implemented to mitigate the risk of similar occurrences in the future. If you are still experiencing connectivity issues, we recommend: 1. Allowing additional time for DNS propagation to reach your network. 2. Performing a DNS flush on your network or operating system. If the issue persists, please contact us at support@cigotracker.com, and we’ll be happy to assist you. We sincerely apologize for the inconvenience caused and appreciate your patience and understanding during this time.

Incident Summary: On August 11th, a significant issue occurred with the Email Consumer in the Notification Service. The Email Consumer stopped functioning at approximately 8:00 PM EDT, leading to a series of cascading issues within the Notification Service. This incident did not impact the rest of the Cigo Tracker service. Timeline of Events: - 8:00 PM EDT: The Email Consumer ceased functioning. - 8:00 PM - 7:00 AM EDT: Messages began queuing at a significantly reduced rate. - 7:00 AM - 10:00 AM EDT: The rate at which messages were queued increased substantially. - 11:00 AM - 2:08 PM EDT: The Email queue reached its capacity, resulting in the rejection of new messages. During this period, both email and SMS message publishing were halted, despite the expectation that only the Email queue should have been affected. Impact: - The system experienced a complete halt in both email and SMS message publishing between 11:00 AM and 2:08 PM EDT, affecting communication and potentially leading to delays in message delivery. Root Cause: - There is a suspected code-logic error that may be causing incorrect checks on the queues, leading to the stoppage of both email and SMS publishing when the Email queue reaches its capacity. Actions Taken: - An alert was sent to the appropriate internal communications channel when the Email Consumer failed. However, due to a configuration issue, the notification was not received by the intended recipients at the time of the failure. - The issue was remediated at around 2:00 PM EDT by restarting the Email Consumer and fixing the Email queue. Adjustments have been made to the channel configuration to include additional team members and ensure quicker responses in the future. Next Steps: - Investigate the root cause of the Email queue failure. - Determine why reaching the maximum queue size for emails also impacted the SMS queue. - Ensure that the monitoring and alerting systems are fully operational and that all team members are notified promptly of any critical failures. - Conduct a thorough review of the queue management logic to identify and rectify any underlying issues. Conclusion: We are taking appropriate actions to prevent this issue from recurring. Our team is committed to ensuring the reliability of the Notification Service, and we will continue to monitor and improve our systems to provide the best possible service to our customers.

Microsoft Azure has claimed that the issue has been fully resolved. For details, you can review the complete Azure outage history here for today's partial outage: Mitigation Statement - Azure Front Door - Issues accessing a subset of Microsoft services (Tracking ID: KTY1-HW8) [ https://azure.status.microsoft/en-us/status/history/ ] Everything seems to be fully operational on our end now. If you are still experiencing any issues, please reach out to our support team at support@cigotracker.com

The incident has been resolved, and SMS and email notification services are now fully restored. We are currently analyzing the issue to prevent it from happening again. Thank you for your patience and understanding.

This incident has been resolved. We identified the root cause of the problem and will work on improving our system's health checks to ensure such issues are identified and mitigated more rapidly in the future. Once again, we apologize for the inconvenience this has caused and appreciate your patience throughout this process.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance is complete. If you encounter any connectivity issues that might indicate database problems, please contact us at support@cigotracker.com

The scheduled maintenance is finished, and we've successfully redirected traffic from the old versions of our mobile app and API integrations targeting cigotracker.com to the new domain, app.cigotracker.com. If you're facing any problems or uncertainties after this change, just drop us a line at support@cigotracker.com, and we'll help you out.

The scheduled maintenance has been completed.