Todyl outage history

Due to an upstream outage at the LA PoP, we are seeing intermittent connectivity challenges across several routes to other PoPs.  We have identified the impacted routes and are working on options to improve performance.  Will update as soon as we have more information.

This incident appears to be directly related to the Cogent fiber cuts (https://ecogent.cogentco.com/network-status) that are impacting traffic in the region. We have seen improvements as traffic gets rerouted in the region, but it appears that the Aukland PoP is still impacted. We will continue to monitor and update as we have more information.

This incident has been resolved.

We are actively investigating an issue impacting the consistent display of data across dashboards, widgets, and integrations pages within SIEM. Data ingestion is confirmed to be ongoing; however, affected users may experience inconsistent visibility of that data in the noted components. We will update once we have identified the root cause of the issue.

We continue to experience above-average query volume which is resulting in intermittent delays with data loading into the SIEM. The problem has been identified and we are in the process of implementing changes to alleviate the resource demand. We will provide an update once the changes are implemented.

A fix for the issue impacting the display of data has been implemented. We will continue to monitor to see if there are any further challenges. We will resolve once fully confirmed.

This incident has been resolved. We have been monitoring overnight and throughout today and confirm normal operational state for display of data across dashboards, widgets, and integrations pages within SIEM.

We are currently investigating an issue where SIEM data is not loading in the Todyl portal. Data is still being ingested into SIEM, just not displayed in the UI. We will update once we have identified the root cause of the issue.

We are currently experiencing above-average query volume which is resulting in intermittent delays with data loading into the SIEM. Additional resources have been allocated to the system and conditions have improved; however, the service remains in a degraded state. We are continuing to investigate the root cause and will provide a follow-up update within 2-3 hours

Todyl Engineering team has identified timing and resource adjustments for this issue and we are rolling out the necessary changes in the coming hour. We will post another update here after the remediation steps are validated.

Implemented changes has significantly improved the degradation, and the system is on its way to normal operational state. Changing status to monitoring.

Todyl has been monitoring since implementing the changes a few hours ago. Everything is looking to be back to normal operational state, marking this issue as resolved.

We are currently investigating some replication delays on internal servers. These delays can possibly cause errors when creating or updating configurations in the Todyl portal. Will update here when we have identified a resolution.

We identified an issue creating some replication delays on internal servers. These delays can possibly cause errors when creating or updating configurations in the Todyl portal. Will update here when we have implemented a resolution.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Investigating intermittent reported SASE tunnel errors. Please let technical support know if you are experiencing any tunnel issues. We will update here with more information shortly.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring tunnels for any other issues. Please let us know if you are experiencing any further challenges.

This incident has been resolved.

We are currently investigating reports of SASE connection errors on Windows Server 2012 devices. We will update as we have more information.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently conducting an active investigation into an issue affecting our SIEM platform. Users may be experiencing delays when loading SIEM results. As a result of this delay, cases may also appear delayed in the portal. Our team is aware of the impact and is actively investigating the affected components. Further updates will be provided as more information becomes available.

We have identified the root cause of the delays affecting SIEM results loading. The issue has been traced to a third-party upstream provider, and we are actively working with them to implement a fix. Users may continue to experience delays in SIEM results loading and cases appearing in the portal in the interim. Data integrity is not affected. Further updates will be provided as progress is made.

The platform has returned to normal operation. We are continuing to monitor the situation to ensure full stability. Further updates will be provided if necessary.

This incident has been resolved.

We have identified a subsequent issue related to the previous SASE incident. We are implementing a fix, but some users could still have intermittent SASE connection issues. We will update as soon as we know more on the fix.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating performance degradation affecting the Secure Global Network (SASE). A subset of users connecting may be experiencing disconnect and connect loop. Our engineering team is actively investigating the root cause.

The issue has been identified and a fix is being implemented.

We have deployed a fix to stabilize the SASE environment and are continuing to investigate the underlying root cause. SASE connections are currently stable once established. As a precaution, tenant and policy updates remain temporarily paused while we complete validation and work toward full resolution. We will provide additional updates as new information becomes available.

The team is now re-enabling tenant and policy changes across all PoPs. We will provide another update once the rollout is complete and stability has been validated.

The team has re-enabled tenant and policy changes across all PoPs and are monitoring the results.

This incident has been resolved.

We are currently investigating an influx of Bash Shell Profile Modification alerts. These are all benign. Will update here as we know more.

The issue has been identified and a fix is being implemented.

The alert influx is impacting performance within the Todyl Management portal. Affected components include the portal experience and reporting widgets, which are experiencing slower than expected loading times. We are working to resolve this issue and will provide updates as the situation develops.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating reports of certain URLs intermittently failing to resolve correctly. Some domains, including those in the Google domain family, may be impacted. Our team is actively investigating and we will provide updates as more information becomes available.

A third party provider has implemented a fix that appears to have resolved the intermittent URL failure. We will continue to monitor for any other issues and update here as needed.

This incident has been resolved.

Microsoft is currently experiencing an incident affecting their M365 API. As a result, O365 data ingestion is delayed, temporarily impacting O365-sourced events. We are monitoring the upstream Microsoft incident and will update as it progresses.

This incident has been resolved.

We are currently investigating connectivity issues with the Bogota Point of Presence (PoP). Devices will automatically route to the next nearest and fastest PoP and Static IP traffic will egress from the configured secondary location. Will update here as we the investigation continues.

The connectivity issue was caused by a power outage from a third party provider. The provider is aware and currently working to restore power as soon as possible.

The connectivity issue with the Bogota Point of Presence (PoP) has been restored. We will continue to monitor to ensure no other issues.

This incident has been resolved.

This issue is currently under investigation, but possibly related to a known Microsoft outage in the region - https://downdetector.com.au/status/microsoft-365/ We will post more information here as soon as we have better insight.

It appears the 3rd party issues have been restored. Will continue monitoring until we are clear that it is fully resolved.

This incident has been resolved.

We are currently investigating connectivity issues with the Miami Point of Presence (PoP). Devices will automatically route to the next nearest and fastest PoP and Static IP traffic will egress from the configured secondary location. Will update here as we the investigation continues.

The connectivity issue was caused by a power outage from a third party provider. The provider is aware and currently working to restore power as soon as possible.

The connectivity issue with the Miami Point of Presence (PoP) has been restored. We will continue to monitor to ensure no other issues, but SASE, tunnels, and Static IP traffic should be connecting now.

This incident has been resolved.

Some partners may be experiencing intermittent latency when using the Todyl portal.  Currently there is a partial Cloudflare system error being reported, leading to the intermittent errors on the Todyl portal.  We are monitoring the situation closely and getting updates from Cloudflare.  We will be be providing regular updates. For more information, please also refer to the Cloudflare status page - https://www.cloudflarestatus.com/.

The issue has been identified and awaiting on a Cloudflare resolution. For more information, please also refer to the Cloudflare status page - https://www.cloudflarestatus.com/.

Cloudflare has successfully mitigated the majority of congestion and packet drops. We are continuing to monitor any possible impacts to the Todyl portal. For more information, please also refer to the Cloudflare status page - https://www.cloudflarestatus.com/.

majority of congestion and packet drops has been mitigated. Services are stable, with only minor residual impact remaining. A small number of intermittent errors may still be observed for services with origins in North America. We are putting this into a monitoring state.

Cloudflare has successfully mitigated the majority of congestion and packet drops. We are continuing to monitor any possible impacts to the Todyl portal. For more information, please also refer to the Cloudflare status page - https://www.cloudflarestatus.com/.

This incident is considered resolved as we see no impact on Todyl portal anymore, despite Cloudflare still continuing to monitor the issue.

We are currently investigating performance degradation affecting the Secure Global Network (SASE) - Houston, HOU region. A subset of users connecting through this region may be experiencing elevated latency or reduced throughput. Our engineering team is actively investigating the root cause.

Performance has returned to normal operation for the Secure Global Network (SASE) - Houston, HOU region. The system has self-recovered and affected users should no longer be experiencing elevated latency or reduced throughput. We are continuing to monitor the environment to ensure sustained stability.

This incident has been resolved.

We are currently investigating an issue affecting Management Portal sign ins. A subset of users may experience intermittent failures when attempting to load the portal. All other Todyl services are operating normally. Our engineering team is actively investigating the root cause.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We're investigating a delay in EDR policy updates affecting a small subset of endpoints. Impacted agents remain fully connected and continue enforcing current policies. Detection, prevention, and response functionality are unaffected.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Intermittent loading errors were seen on portal between 3PM MT and 4PM MT. The errors affected SIEM search, SIEM dashboards, Cases alert data and SOAR playbook execution. The errors have been addressed since.