Cigo Tracker 장애 기록

1. 2026년 7월 27일 PM 1:46 UTCinvestigating

   우리는 수입 도구를 통해 일자리를 가져올 때 지연 및 오류의 보고서를 조사하고 있습니다. 일부 사용자는이 시간에 실패하거나 느린 처리 수입을 경험할 수 있습니다.

2. 2026년 7월 27일 PM 1:47 UTCinvestigating

   9:15 AM EST — 조사 우리는 수입 도구를 통해 일자리를 가져올 때 지연 및 오류의 보고서를 조사하고 있습니다. 일부 사용자는이 시간에 실패하거나 느린 처리 수입을 경험할 수 있습니다. 9:41 오전 EST — 해결 이 사건은 해결되었습니다. 가져오기 도구 기능 정상에 반환 하 고 모든 지연된 수입 처리 되었습니다.. 우리는 어떤 불편든지를 위해 apologize. 소요시간: 26분

3. 2026년 7월 27일 PM 1:51 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2025년 10월 29일 PM 3:58 UTCinvestigating

   We are currently experiencing service disruptions due to a Microsoft Azure outage affecting Cigo Tracker’s infrastructure. Our team is actively monitoring the situation and working closely with Microsoft to restore full functionality as soon as possible. We will continue to provide updates as new information becomes available. Thank you for your patience and understanding.

2. 2025년 10월 29일 PM 4:09 UTCinvestigating

   We are continuing to investigate this issue.

3. 2025년 10월 29일 PM 5:10 UTCidentified

   We’re aware of a Microsoft Azure service disruption that began around 16:00 UTC (12:00 PM EST) and is currently affecting the availability of some Cigo Tracker services. The issue is related to Azure Front Door (AFD), which is also impacting access to the Azure Portal and other Azure-hosted systems globally. Microsoft has acknowledged the issue and is actively working on mitigation steps, including failover options for affected infrastructure. Their latest update was posted at 16:57 UTC (12:57 PM EST) on October 29, 2025, confirming that investigations are ongoing. Our team continues to monitor the situation closely and will provide additional updates as Microsoft releases more information or once services stabilize. For live updates directly from Microsoft, you can check the Azure Status Page here: 🔗 https://azure.status.microsoft/en-us/status Thank you for your patience and understanding.

4. 2025년 10월 29일 PM 7:02 UTCidentified

   To ensure our clients’ operations remain uninterrupted, we have activated our failover. You can access the Cigo Tracker platform directly at https://app.cigotracker.com . Please note that our main website (home page) will remain temporarily inaccessible during this period. This decision was made intentionally to prioritize platform uptime and maintain delivery operations for all clients. Microsoft is actively deploying their last known good configuration and has begun recovering nodes to restore full connectivity. While progress is ongoing, Microsoft has indicated that protective safeguards are extending the overall recovery time. We will keep the system status as Degraded until we have collected sufficient data to confirm full stability, at which point we will mark the system as Operational. You can monitor Azure’s live updates here: https://azure.status.microsoft/en-us/status All times are reported in Eastern Standard Time (EST). Thank you for your patience and understanding as we continue to monitor and provide updates in real time.

5. 2025년 10월 29일 PM 7:15 UTCidentified

   We are continuing to work on a fix for this issue.

6. 2025년 10월 29일 PM 8:59 UTCmonitoring

   We have successfully bypassed the Azure Front Door issue. While Azure continues to experience a global outage, Cigo Tracker remains fully operational through our failover. Please note that Cigo Tracker is currently accessible only via https://app.cigotracker.com . We will continue to closely monitor system performance and provide updates as the situation with Microsoft Azure progresses. Thank you for your continued patience and understanding.

7. 2025년 10월 30일 AM 1:24 UTCresolved

   Microsoft has confirmed that all affected Azure services have been fully restored. Cigo Tracker systems were back to normal operation as of 16:59 EST. Following service stabilization, we have now reverted from our backup environment back to our primary infrastructure. All systems are operating normally, and performance remains stable. If you wish to receive the official Microsoft incident report once it is published, please email us at [email protected] , and we will make it available to you as soon as we receive it. Thank you for your patience and understanding throughout this Azure service disruption.

1. 2025년 9월 15일 PM 6:12 UTCidentified

   We recently experienced unexpected issues with SSL certification that disrupted database connections for several of our internal services. The Import Tool was initially affected but has since been fully restored. During our investigation, we also identified that the issue impacted the notifications subsystem for the Confirmation module, which relies on SMS and email sending. While the Import Tool is now operating normally, we are still working to restore full connectivity for Confirmation module notifications, along with a few other automated functions that may currently be degraded. Our engineering team has identified the root cause and is implementing safeguards to prevent similar issues during future SSL certification rotations. We sincerely apologize for the disruption and will continue to provide updates until all affected services are fully restored.

2. 2025년 9월 15일 PM 6:52 UTCidentified

   We have resolved the issues affecting the Import Tool and the Confirmation module notifications. Both are now fully operational. Our team is currently working on finalizing the recovery of a few remaining automated functions that may still be degraded. We will provide another update once all affected services are fully restored.

3. 2025년 9월 15일 PM 8:24 UTCresolved

   The SSL certification issue that impacted several internal services, including the Import Tool and the Confirmation module notifications, has been fully resolved. All affected services are now fully operational. We have implemented safeguards to prevent similar issues during future SSL certification rotations. Thank you for your patience and understanding throughout this incident.

4. 2025년 9월 15일 PM 8:28 UTCpostmortem

   ### Summary On **September 15th, 2025**, we experienced service disruptions affecting the Import Tool and Confirmation module notifications. The root cause was a misidentified SSL certificate dependency during the scheduled **Azure Database for MySQL maintenance**, which included the rotation of the MySQL server’s SSL certificates. ### Impact * The Import Tool was unavailable for a period of time. * Confirmation module notifications \(via SMS and email\) were disrupted. * Some other automated functions experienced degraded performance. ### Root Cause While we were aware of the scheduled SSL certificate rotation as part of Azure’s maintenance process, our evaluation process incorrectly identified which certificates were being used by certain subsystems. As a result, several services lost connectivity when the certificates were rotated. ### Mitigation and Resolution * Our engineering team corrected the certificate configuration for the affected subsystems. * The Import Tool and Confirmation module notifications were restored to full functionality. * Safeguards are being added to ensure that future SSL rotations do not cause similar disruptions. ### Contributing Factors At the onset of the incident, we initially believed the outages were linked to updates and fixes deployed on Thursday and Friday of the previous week. This led to rollbacks that did not address the issue. Although unrelated to the outage, some of those updates remain rolled back and will be re-published in the coming days. ### Next Steps * We are reworking our dependency system analysis to more accurately map SSL and package dependencies across all services. * We are improving our monitoring and review processes for certificate rotations to ensure full coverage across all subsystems. * We will re-publish rolled-back updates after additional validation. ### Conclusion We sincerely apologize for the operational problems this incident caused. We take these issues seriously and are committed to ensuring that similar outages do not reoccur in the future.

1. 2025년 7월 18일 PM 4:28 UTCinvestigating

   Our web application is currently unavailable. We’re actively investigating the cause of the outage. Earlier today, we performed some database changes which have since been reverted. We're currently assessing whether these changes contributed to the issue and working to restore full functionality as quickly as possible. We sincerely apologize for the inconvenience this is causing and appreciate your patience while we work to resolve the problem. Further updates will follow shortly.

2. 2025년 7월 18일 PM 5:59 UTCmonitoring

   The web application is now accessible again at https://app.cigotracker.com. We are closely monitoring system performance to ensure stability and confirm that all services are functioning as expected. We were also seeing some issues accessing the Cigo Tracker landing page (https://cigotracker.com), but this was also just reestablished. Thank you for your continued patience. We’ll provide another update once the incident is fully resolved.

3. 2025년 7월 18일 PM 7:10 UTCresolved

   The web application outage has been resolved, and all systems are currently operating normally. While we've successfully isolated the scope of the issue, there are still some contributing factors that remain under investigation. Our team is continuing to analyze the event to determine the full root cause of today's outage and to implement measures that prevent recurrence. We sincerely appreciate your patience and apologize again for the disruption.

1. 2025년 6월 23일 PM 8:08 UTCinvestigating

   We are currently investigating an issue related to an ongoing system update that is impacting platform performance and availability. Users may experience slow response times or intermittent access issues. Our engineering team is working to identify the root cause and restore full functionality as quickly as possible. We will share updates as progress is made.

2. 2025년 6월 23일 PM 8:16 UTCidentified

   We’ve identified the cause of the issue. While platform performance has recovered, the underlying system update did not complete successfully. Our team is conducting a thorough review of the root cause and will provide further updates as we determine the next steps.

3. 2025년 6월 24일 AM 2:02 UTCresolved

   The issue has been fully resolved. We reverted the deployment that caused the disruption, and the related changes will be revised and rescheduled for a future release. Thank you for your patience and understanding throughout the investigation.

1. 2025년 3월 19일 AM 12:44 UTCmonitoring

   We want to inform you that we recently experienced network degradation affecting our services due to an ongoing issue within Microsoft's Azure infrastructure in the East US region. What Happened? According to Azure, between 13:09 UTC and 18:51 UTC, a fiber cut impacted network capacity in the region, leading to intermittent connectivity loss and increased latency. While Azure has since mitigated the issue, we observed disruptions in our own services between 7:20 PM and 8:27 PM (Eastern Time), specifically affecting connections between the Cigo Tracker web app and our Redis service. Current Status As of 8:27 PM UTC, network latencies have returned to normal, and service stability has been restored. However, to ensure a prompt and complete resolution, we have escalated this matter to Azure's Operations Support with a critical priority. We appreciate your patience and will continue monitoring the situation closely. If you experience any further issues, please reach out to our support team.

2. 2025년 3월 19일 AM 3:35 UTCresolved

   After monitoring the situation over the past few hours, we can confirm that the immediate impact of the outage has been fully mitigated. Our services are stable, and network performance has returned to normal. We will provide a more detailed post-mortem once we receive a conclusive report from Microsoft's Azure Operations Support (OSS) team. Thank you for your patience and understanding. We sincerely apologize for any inconvenience this may have caused.

3. 2025년 4월 11일 PM 3:40 UTCpostmortem

   ## 📣 Incident Summary – March 18–19, 2025 **Service Impact on Cigo Tracker due to Azure Regional Outage** On March 18, 2025, Cigo Tracker experienced intermittent service disruption due to a regional outage within Microsoft Azure’s East US data center region. Below is a summary of the root cause, impact, and the steps being taken to prevent future occurrences. ### 🕒 What Happened? Azure’s East US region suffered two separate impact windows: * March 18, 13:37 to 16:52 UTC * March 18, 23:20 to March 19, 00:30 UTC The incident was triggered by a **third-party fiber cut** during external drilling work, which caused reduced network capacity in one of Azure’s Availability Zones. A **tooling failure** during Azure's recovery efforts later reintroduced traffic prematurely, leading to congestion and a second round of intermittent connectivity issues. ### 🔍 Root Cause 1. **Fiber Cut**: A construction-related accident physically damaged fiber cabling serving the East US datacenter, degrading network capacity. 2. **Router Maintenance**: A key router in the same zone was already under repair, limiting redundancy. 3. **Tooling Error**: Azure’s automated recovery system failed to fully isolate damaged infrastructure, inadvertently reintroducing traffic too early. 4. **Congestion Spillover**: The unexpected traffic load caused congestion to spread beyond AZ03 into neighboring zones. ### 🎯 Impact on Cigo Tracker While the Azure issue only affected a subset of inter-zone traffic in East US, this included infrastructure we rely on, resulting in **intermittent connectivity issues** for some customers during the incident windows. Core services were restored once Azure manually completed isolation and fiber recovery work. ### 🛠 Resolution Timeline * **13:37 UTC, Mar 18** – Outage begins due to fiber cut * **13:55 UTC** – Initial mitigation starts; traffic rerouted * **16:52 UTC** – First impact window ends * **23:20 UTC** – Second outage begins due to tooling error during recovery * **00:30 UTC, Mar 19** – Final mitigation complete * **06:50 UTC** – Full restoration of all infrastructure ### ✅ What Azure is Doing to Prevent Recurrence * Fixing tooling failures that allowed reintroduction of unready capacity \(by May 2025\) * Accelerating a capacity upgrade for the East US datacenter \(by July 2025\) * Architecting better safeguards to prevent impact from spreading across zones \(by February 2026\) We apologize for the inconvenience caused. Please rest assured that our team is working closely with Azure and continuing to invest in the resiliency of our platform. If you have any questions or would like help designing a more resilient setup, feel free to reach out to our support team. Thank you for your continued trust

1. 2024년 11월 15일 AM 5:00 UTCresolved

   On November 14th, at approximately 10:15 PM, during a planned network maintenance session, we undertook a migration of core network routing resources. Unfortunately, this migration inadvertently impacted our DNS records, causing DNS Probe errors for some customers attempting to access our services. To address the issue, our operations team promptly remapped the affected DNS records and adjusted network restriction rules to restore connectivity. The total downtime was less than one hour. The root cause of this incident was an unforeseen interaction between the migration process and automatic changes to both networking rules and DNS records. While this was an unanticipated challenge, we’re grateful for the swift recovery efforts of our operations team. The issue has been fully resolved, and measures have been implemented to mitigate the risk of similar occurrences in the future. If you are still experiencing connectivity issues, we recommend: 1. Allowing additional time for DNS propagation to reach your network. 2. Performing a DNS flush on your network or operating system. If the issue persists, please contact us at [email protected], and we’ll be happy to assist you. We sincerely apologize for the inconvenience caused and appreciate your patience and understanding during this time.

1. 2024년 8월 15일 PM 12:51 UTCresolved

   Incident Summary: On August 11th, a significant issue occurred with the Email Consumer in the Notification Service. The Email Consumer stopped functioning at approximately 8:00 PM EDT, leading to a series of cascading issues within the Notification Service. This incident did not impact the rest of the Cigo Tracker service. Timeline of Events: - 8:00 PM EDT: The Email Consumer ceased functioning. - 8:00 PM - 7:00 AM EDT: Messages began queuing at a significantly reduced rate. - 7:00 AM - 10:00 AM EDT: The rate at which messages were queued increased substantially. - 11:00 AM - 2:08 PM EDT: The Email queue reached its capacity, resulting in the rejection of new messages. During this period, both email and SMS message publishing were halted, despite the expectation that only the Email queue should have been affected. Impact: - The system experienced a complete halt in both email and SMS message publishing between 11:00 AM and 2:08 PM EDT, affecting communication and potentially leading to delays in message delivery. Root Cause: - There is a suspected code-logic error that may be causing incorrect checks on the queues, leading to the stoppage of both email and SMS publishing when the Email queue reaches its capacity. Actions Taken: - An alert was sent to the appropriate internal communications channel when the Email Consumer failed. However, due to a configuration issue, the notification was not received by the intended recipients at the time of the failure. - The issue was remediated at around 2:00 PM EDT by restarting the Email Consumer and fixing the Email queue. Adjustments have been made to the channel configuration to include additional team members and ensure quicker responses in the future. Next Steps: - Investigate the root cause of the Email queue failure. - Determine why reaching the maximum queue size for emails also impacted the SMS queue. - Ensure that the monitoring and alerting systems are fully operational and that all team members are notified promptly of any critical failures. - Conduct a thorough review of the queue management logic to identify and rectify any underlying issues. Conclusion: We are taking appropriate actions to prevent this issue from recurring. Our team is committed to ensuring the reliability of the Notification Service, and we will continue to monitor and improve our systems to provide the best possible service to our customers.

1. 2024년 7월 30일 PM 12:24 UTCinvestigating

   We are currently experiencing issues accessing our services due to an outage reported by Microsoft Azure. Microsoft Azure has reported an issue impacting access to the Azure portal and Azure services in general. We suspect that this may be related to general DNS issues affecting their services, which in turn is impacting access to our website and services. Our team is actively monitoring the situation and working closely with Azure to resolve the issue as quickly as possible. We will provide updates as soon as we have more information. Thank you for your patience and understanding as we work through this issue. If you have any questions or need further assistance, please contact our support team at [email protected].

2. 2024년 7월 30일 PM 1:32 UTCidentified

   We have identified the issue as being related to Azure’s network infrastructure. Multiple engineering teams at Microsoft are engaged to diagnose and resolve the issue. Azure's latest update ( https://azure.status.microsoft/en-us/status ): "We are investigating reports of issues connecting to Microsoft services globally. Customers may experience timeouts connecting to Azure services. We have multiple engineering teams engaged to diagnose and resolve the issue. More details will be provided as soon as possible." This message was last updated at 13:13 UTC on 30 July 2024. We will provide updates as soon as we have more information. Thank you for your patience and understanding as we work through this issue.

3. 2024년 7월 30일 PM 3:45 UTCmonitoring

   A fix has been implemented by the Microsoft Azure team. As of their latest update: "We have implemented networking configuration changes and have performed failovers to alternate networking paths to provide relief. Monitoring telemetry shows improvement in service availability from approximately 14:10 UTC onwards, and we are continuing to monitor to ensure full recovery." This message was last updated at 14:54 UTC on 30 July 2024. On our end all connectivity issues seem to be resolved so far, but we are monitoring until Azure confirms full resolution.

4. 2024년 7월 30일 PM 6:07 UTCmonitoring

   The latest update from Microsoft Azure indicates that the majority of the issue should now be mitigated: "An unexpected usage spike resulted in Azure Front Door (AFD) components performing below acceptable thresholds, leading to intermittent errors, timeout, and latency spikes. We have implemented network configuration changes and have performed failovers to provide alternate network paths for relief. Our monitoring telemetry shows improvement in service availability from approximately 14:10 UTC onwards. As we investigate reports of specific services and regions that are still experiencing intermittent errors, we believe that our network configuration changes have successfully mitigated the impacts of the usage spike, but that these changes are causing some side effects to certain services. We are updating our mitigation approach to minimize these side effects, and applying these following Safe Deployment Practices - beginning in Asia Pacific regions and then expanding in phases. We will provide an update on our continued mitigation efforts by 19:00 UTC, or sooner if we have progress to share." This message was last updated at 17:59 UTC on 30 July 2024 You can refer to Azure's latest updates here: https://azure.status.microsoft/en-us/status

5. 2024년 7월 30일 PM 9:20 UTCresolved

   Microsoft Azure has claimed that the issue has been fully resolved. For details, you can review the complete Azure outage history here for today's partial outage: Mitigation Statement - Azure Front Door - Issues accessing a subset of Microsoft services (Tracking ID: KTY1-HW8) [ https://azure.status.microsoft/en-us/status/history/ ] Everything seems to be fully operational on our end now. If you are still experiencing any issues, please reach out to our support team at [email protected]

1. 2024년 5월 30일 PM 2:18 UTCinvestigating

   We are currently investigating an emerging issue with our SMS and Email notifications service. Our team is actively investigating the issue to identify the root cause and implement a solution. We will provide updates as we make progress. We apologize for any inconvenience this may have caused and appreciate your patience as we work to resolve this issue promptly.

2. 2024년 5월 30일 PM 2:29 UTCidentified

   We've identified the issue with our messaging service that started at 9:40 AM (Eastern Time). Rest assured, all scheduled and event notifications sent before this time were successfully delivered. Our team is actively working on a fix, and we will post an update as soon as the problem is resolved. Thank you for your patience.

3. 2024년 5월 30일 PM 2:51 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2024년 5월 30일 PM 3:04 UTCresolved

   The incident has been resolved, and SMS and email notification services are now fully restored. We are currently analyzing the issue to prevent it from happening again. Thank you for your patience and understanding.

1. 2024년 5월 17일 PM 2:35 UTCinvestigating

   We are currently experiencing an outage with our SMS service that began last night. Our team is actively investigating the issue to identify the root cause and implement a solution. We will provide updates as we make progress. We apologize for any inconvenience this may have caused and appreciate your patience as we work to resolve this issue promptly.

2. 2024년 5월 17일 PM 2:40 UTCidentified

   The issue has been identified, and a fix has been implemented. We determined that the system stopped sending SMS messages yesterday at 2:28 PM EDT. Our SMS notification system is now operational and is currently processing the backlog at a rate of 300 SMS per second. We will continue to monitor the system to ensure a full recovery.

3. 2024년 5월 17일 PM 2:54 UTCmonitoring

   The backlog of SMS notifications has been fully processed, and our SMS service is back to normal operation. We are continuing to monitor the system to ensure everything is working well.

4. 2024년 5월 17일 PM 3:20 UTCresolved

   This incident has been resolved. We identified the root cause of the problem and will work on improving our system's health checks to ensure such issues are identified and mitigated more rapidly in the future. Once again, we apologize for the inconvenience this has caused and appreciate your patience throughout this process.

1. 2024년 5월 7일 PM 2:13 UTCinvestigating

   As of 8:55 AM EDT today, we are experiencing an unplanned outage affecting our homepage (https://cigotracker.com). Our technical team is actively investigating the cause of this issue. Impact: Please note that this outage does not affect the overall availability of our web application. Users can continue to access our web application through the following link: https://app.cigotracker.com/site/login. The operator applications on Android and iOS, as well as all other components of our web application, are operating normally. Next Steps: We are working diligently to identify and resolve the issue. Updates will be provided as more information becomes available.

2. 2024년 5월 7일 PM 4:37 UTCinvestigating

   We are actively investigating the issue causing delayed loading on the homepage. As a temporary measure, we have redirected traffic from cigotracker.com to the login page of our web application. Meanwhile, we are thoroughly examining our operational logs and conducting trace analysis to pinpoint the cause of the outage on the landing page. Thank you for your patience.

3. 2024년 5월 7일 PM 8:35 UTCresolved

   This incident has been resolved.

1. 2024년 4월 29일 PM 7:56 UTCresolved

   On Tuesday, April 23rd, at approximately 6 PM EDT, we implemented a network configuration adjustment in our backend routing rules. This adjustment inadvertently affected Cigo Pay, specifically causing errors when users attempted to submit tip transactions. Although Cigo Pay remained technically online, users encountered difficulties with the final submission of tips. Reports of this error began to surface on Wednesday morning. Upon investigation, we identified that the network configuration change was the root cause of the issue. We promptly rectified the configuration error by 1 PM EDT on the same day. We sincerely apologize for any inconvenience this incident may have caused. To prevent similar occurrences in the future, we have implemented enhancements to our monitoring systems. Additionally, we regret the oversight in failing to create an incident report at the time of the problem. Thank you for your patience and understanding as we worked to resolve this issue. If you have any further questions or concerns, please don't hesitate to reach out.

1. 2024년 1월 21일 AM 6:40 UTCinvestigating

   We are currently experiencing extended database downtimes stemming from Microsoft Azure's database instances. This has been occurring intermittently since 9:45 PM (EST), and the issue has been recurring at a higher frequency since 12 AM (EST). Our team is actively investigating the root cause of this service disruption. We appreciate your patience as we work to resolve this issue promptly.

2. 2024년 1월 21일 AM 7:23 UTCidentified

   Our ongoing investigation into the connectivity disruption impacting a subset of our database servers and various Azure services has identified an issue on Microsoft's end. The Microsoft Operational Systems Support team has acknowledged the problem and is actively addressing it. We are diligently awaiting further updates from their team and will keep you informed as soon as new information becomes available. Your patience during this time is sincerely appreciated.

3. 2024년 1월 21일 AM 7:57 UTCmonitoring

   It appears that Microsoft Azure has successfully implemented a fix, and our database server connections are now operational. However, we are currently awaiting official confirmation from the Microsoft Operational Systems Support team to validate that the issue has been fully mitigated. Thank you for your continued understanding.

4. 2024년 1월 21일 AM 8:30 UTCmonitoring

   Our team is actively monitoring our database server instances to guarantee service availability. While things are looking positive with the recent improvements, we are awaiting official confirmation from the Azure team to ensure that the problem has been fully resolved.

5. 2024년 1월 21일 PM 1:24 UTCresolved

   The incident has been successfully resolved. We're now awaiting the Root-Cause Analysis report from Microsoft Azure's team. Once received, we'll compile a post-mortem of the event to provide you with a comprehensive overview.

6. 2024년 1월 29일 PM 3:42 UTCpostmortem

   We want to provide you with an update on the January 21st incident that impacted our services. Here's a breakdown of the situation: **Incident Summary:** On January 20th, 2024, at around 9 PM EST, an internal maintenance process by the Azure OSS team resulted in a configuration change to Azure Resource Manager. Unfortunately, this led to repeated failures of the Azure Resource Manager's node upon startup. **Root Cause:** The configuration change triggered a negative feedback loop, overwhelming the remaining Azure Resource Manager nodes and causing a rapid drop in availability. This, in turn, affected our backend storage, leading to random failures on data plane API calls. These failures, specifically, disrupted the functionality of our database server, leading to intermittent crashes, particularly during the timeframe of 12 AM to 2 AM. **Resolution:** The Azure engineering team worked to address the issue, and we’re able to fully resolve it around 4 AM EST on January 21st, 2024. **Preventive Measures:** To prevent similar incidents in the future, we are closely reviewing our internal processes and working collaboratively with the Azure OSS team to implement additional safeguards. We sincerely apologize for any inconvenience this may have caused, and we appreciate your understanding as we continue to enhance our systems to provide you with a more reliable experience.

1. 2023년 11월 22일 PM 8:14 UTCinvestigating

   We are investigating the cause of the intermittent platform availability issue. We will post an update once we have more information.

2. 2023년 11월 22일 PM 8:29 UTCidentified

   We experienced connectivity issues between our web platform and one of our caching database services. We have identified a potential cause and implemented corrective measures. The situation appears to have been resolved, and the platform is now stable. Our team is currently conducting a thorough analysis to understand the root cause. Further updates will follow as we gather more information.

3. 2023년 11월 22일 PM 10:05 UTCresolved

   We encountered a temporary glitch in domain name resolution, leading to connectivity issues with our caching database and causing a partial service outage. We've taken immediate measures to address the issue, and it also appears to be self-healing. The disruption occurred from 3:04 PM to 3:16 PM (ET) and briefly from 3:25 PM to 3:28 PM (ET). We apologize for any inconvenience this may have caused. Rest assured, we're proactively engaging with our cloud hosting vendor to conduct a thorough analysis and prevent future occurrences. Thank you for your understanding.

1. 2023년 9월 13일 PM 2:59 UTCinvestigating

   We are presently in the process of identifying the underlying reasons for the diminished performance of our web and mobile applications, which appears to be stemming from connectivity problems with our servers. Our initial examination has not uncovered any issues originating from our side. Consequently, we have initiated contact with our cloud hosting provider, Microsoft Azure, to collaborate with their Operations Support System (OSS) team in order to further investigate this matter.

2. 2023년 9월 13일 PM 5:39 UTCmonitoring

   We wanted to provide you with an update on the recent server connectivity issues that were impacting our web and mobile app performance. Based on our observations, it appears that the server connectivity issues have been resolved, and our systems are now showing signs of stability. However, we are still actively monitoring our systems to ensure that everything remains in good working order. We understand the importance of a comprehensive analysis to prevent future occurrences, and to that end, we are eagerly awaiting further information from the Microsoft Azure Operations Support System (OSS) team. We hope that their expertise will help us pinpoint the root cause of the issue, allowing us to take any necessary preventive measures going forward. We appreciate your patience and understanding as we continue to work on this matter, and we will keep you updated as soon as we receive more information from the Microsoft Azure OSS team. If you have any questions or concerns in the meantime, please don't hesitate to reach out to us.

3. 2023년 9월 13일 PM 8:42 UTCresolved

   We are pleased to inform you that we are closing this incident with the following important notes: 1. Platform Performance: Our platform's performance has returned to its normal levels since approximately 1:28 PM (Eastern Time). We've closely monitored the situation, and the intermittent connectivity errors that were affecting our services have now disappeared. 2. Ongoing Investigation: While the immediate issue has been resolved, we continue to work closely with the Azure Operations Support System (OSS) team to conduct a comprehensive root cause analysis. Our joint efforts aim to identify the underlying reasons for the incident. 3. Future Updates: As soon as we gather more information and insights from our collaboration with the Azure OSS team, we will provide a post-mortem update on this incident. This update will offer a detailed account of our investigation results and outline our plan of action to prevent a recurrence of this issue. We sincerely apologize for any inconvenience or disruption this incident may have caused to our customers' operations. Our team is committed to ensuring the reliability and performance of our services, and we appreciate your patience and understanding throughout this process. If you have any further questions or require additional information, please do not hesitate to reach out to us.

4. 2023년 9월 14일 PM 1:49 UTCpostmortem

   ### Incident Timeline **Incident Duration:** Approximately 6 hours, from 11:20 UTC to 17:30 UTC on September 13, 2023 ### Timeline of Events #### Incident Identification \(September 13, 2023\) * **11:20 UTC / 7:20 ET:** Customers started experiencing issues with degraded web and mobile app performance, including higher latency and disconnects. #### Incident Response \(September 13, 2023\) * **13:00 UTC / 9:30 ET:** We initiated an internal investigation into the performance degradation and suspected connectivity issues with our servers. #### Mitigation and Communication \(September 13, 2023\) * **17:28 UTC / 13:28 ET:** Full platform performance was restored, and intermittent connectivity errors disappeared. However, we continued monitoring the situation. #### Incident Closure and Ongoing Investigation \(September 13, 2023\) * **19:00 UTC / 15:00 ET:** The incident was officially closed as platform performance returned to normal levels. ### Root Cause Analysis The root cause of this incident was identified as a faulty device in Azure Frontdoor. This device continued transmitting traffic from the edge sites for an extended period of time, leading to congestion and packet drops. The prolonged transmission from the faulty device resulted in higher latency, disconnects, and failed service responses. ### Mitigation Microsoft Azure mitigated the issue by routing traffic away from the problematic device to a healthy one. This action restored normal service operations. ### Preventive Measures To prevent future occurrences, we are committed to implementing the following measures: 1. **Collaboration with Azure:** We will maintain a strong collaboration with Microsoft Azure's OSS team to ensure a proactive approach to identifying and addressing potential issues promptly. 2. **Traffic Monitoring:** Regular monitoring of traffic patterns will be implemented to detect anomalies and address them swiftly. 3. **Redundancy and Failover:** We will explore redundancy options and failover mechanisms to minimize the impact of similar incidents. ### Conclusion We sincerely apologize for the inconvenience and disruption this incident may have caused our customers during the impact window of 11:20 UTC to 17:30 UTC \(7:20 ET to 13:30 ET\) on September 13, 2023. We appreciate your patience and understanding throughout the incident resolution process. Our commitment to providing reliable and performant services remains unwavering, and we will continue to work diligently to improve our systems and prevent future incidents. If you have any further questions or require additional information, please do not hesitate to reach out to us. Thank you for your continued support.

1. 2023년 4월 20일 PM 5:36 UTCinvestigating

   We are currently investigating a partial outage that is causing our web services to be unavailable intermittently. When our service fails to load, some of our users may observe a 502 gateway error page. We apologize for the inconvenience and are working with Microsoft Azure's operational support systems team to resolve the problem as soon as possible.

2. 2023년 4월 20일 PM 5:56 UTCmonitoring

   It appears that the issue was caused by the infrastructure of our cloud vendor, which has since stabilized. Our team is awaiting confirmation from Azure's Operations Support Systems (OSS) team and we are closely observing the performance and availability of our web application.

3. 2023년 4월 20일 PM 8:53 UTCresolved

   The incident that took place from 12:48 PM to 1:31 PM EDT (UTC -4) has been successfully rectified and the networking infrastructure has now returned to a stable-state. While we await Microsoft Azure's OSS team to furnish us with a RCA (Root Cause Analysis) in response to additional logs and information provided by us, we take this opportunity to apologize for any operational disruption that may have been caused by the partial outage. We are grateful for your patience and understanding in this matter.

1. 2023년 1월 25일 PM 3:18 UTCresolved

   The Cigo Tracker platform and backend systems that feed the mobile app were impacted by an outage within Microsoft Azure's networking infrastructure. The outage also impacted Microsoft 365, Teams, and Outlook. Due to this widespread outage, our systems were either fully or partially unavailable from 2:05 AM to 3:45 AM EST (7:05 AM to 9:45 AM UTC). The root cause was identified by Azure, and mitigated. This seems to have primarily impacted our customers in Europe and in the Middle East during their core hours of operation. We apologize for any inconvenience this may have caused. You can read more about the incident (Tracking ID VSG1-B90) here: https://status.azure.com/en-us/status/history/ This outage also made it to numerous online publications: - https://www.reuters.com/technology/microsoft-teams-down-thousands-users-india-downdetector-2023-01-25/ - https://www.theguardian.com/technology/2023/jan/25/microsoft-investigates-outage-affecting-teams-and-outlook-users-worldwide - https://techcrunch.com/2023/01/25/microsoft-teams-outlook-service-outage

1. 2022년 9월 7일 PM 5:07 UTCidentified

   Our platform is currently experiencing a partial outage. A portion of our users are fully impacted and are unable to access the platform, for other response times are slow, and the rest of our users are unaffected. The report from the Microsoft Azure team is as follows (as of 2022-09-07 1:05 PM EDT): Connectivity issues We are aware of connectivity issues to the Azure Portal and customers using Azure Front Door. We will provide more information as it is known. This message was last updated at 17:00 UTC on 07 September 2022

2. 2022년 9월 7일 PM 5:22 UTCidentified

   New update from Azure. They are still investigating the issue and we remain on hold: Starting at 16:10 (UTC) on 07 Sep 2022, customers using Azure Front Door could be experiencing connectivity issues. This could also be impacting customers’ ability to access the Azure Management Portal. We are investigating a spike in traffic as a potential cause. Retries are likely to be successful.

3. 2022년 9월 7일 PM 6:23 UTCidentified

   The Microsoft Azure team has confirmed that they identified the potential cause as "a spike in traffic". They have further clarified the following: "While we are not currently observing any traffic spikes currently, we are working on remediating the residual impact. We are recovering a number of nodes that are showing intermittent connectivity issues. For customers who are experiencing connectivity issues, retries are likely to be successful. Most customers should be seeing recovery at this stage." From our monitoring systems, our DNS health seems to be recovering, but there is still a 5% fluctuation that may impact some customers in some regions.

4. 2022년 9월 7일 PM 7:07 UTCmonitoring

   Microsoft Azure's team has confirmed that they are working on a full system recovery following a large spike in traffic that disrupted their network infrastructure: "We are recovering intermittent connectivity issues. Traffic managed by Azure Front Door service is being recovered by systematically going through the regions where we are observing resource impact and enforcing traffic management on the same. Once the recovery process is completed, the service should be able to resume handling traffic normally." We are monitoring the health of our Front Door and are seeing a steady recovery.

5. 2022년 9월 7일 PM 8:01 UTCmonitoring

   Microsoft Azure's team hasn't reported any additional updates, but based on our monitoring, traffic seems to have returned to normal in the last 30 minutes (3:30 PM EDT). We are standing by for a full resolution confirmation to be confirmed by the Azure team. Thank you for your patience and understanding.

6. 2022년 9월 7일 PM 8:32 UTCresolved

   The problem on Microsoft Azure's end seems to have been mitigated. Our workloads have returned to normal.

1. 2022년 8월 31일 PM 5:15 UTCidentified

   The issue has been identified with one of our third party providers, and we are working with them to resolve it as soon as possible.

2. 2022년 8월 31일 PM 5:51 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2022년 8월 31일 PM 5:53 UTCmonitoring

   We are continuing to monitor for any further issues.

4. 2022년 8월 31일 PM 5:55 UTCresolved

   This incident has been resolved.