Expedited Security outage history

We are seeing an elevated level of failures when reading or updating WAF settings, including IP allow and block lists, through both the Expedited WAF API and the Management Dashboard. Requests may fail or lists may appear empty. Traffic protection is not affected. All sites remain online and continue to be protected and served normally.

Our engineering team is continuing to investigate and is working to restore full settings management functionality. Traffic protection remains unaffected and all sites are online and serving normally.

We are continuing to work on restoring settings management. Traffic protection remains unaffected and all sites are online and serving normally.

Work continues on restoring full settings management functionality. Traffic protection is unaffected and all sites remain online and serving normally.

Our team is still actively working on this issue and restoring settings management. Traffic protection remains unaffected and all sites are online and serving normally.

Settings management functionality has been fully restored. Reading and updating WAF settings, including IP allow and block lists, is working normally again through both the API and the Management Dashboard. Traffic protection was unaffected for the duration of this incident and all sites remained online. If you made IP list changes between September 16, 6:30 PM UTC and September 17, 2:30 PM UTC, please verify they were applied and re-submit any that were not. We apologize for the inconvenience.

We've identified an issue with network interconnect services in our EU and Southeast US regions, preventing requests from being completed. Requests to sites outside of those regions are unaffected.

We are continuing to monitor the affected networks for further issues, but all operations have been restored.

We detected an account desync issue that is preventing access to the WAF Management Dashboard for some Expedited WAF users. The cause of the issue has been determined, and engineers are working on resolving it.

This incident has been resolved. We apologize for any disruption this may have caused. If you're continuing to see errors please contact [email protected]

Heroku is experiencing issues with their upstream DNS provider. As the WAF and CDN target Heroku DNS domains as their origins, our Edge Network can't connect to Heroku. If connections into Heroku are unable to be made, a HTTP status code of 502 is returned along with a brief HTML error page describing the issue. As Heroku resolves their upstream DNS issues sites will immediately come back online with no further intervention needed. For the most recent updates, please follow Heroku's Status Page at: https://status.heroku.com/

Heroku's issues with their upstream DNS provider have been resolved. Please continue to follow https://status.heroku.com for further updates from them.

This incident has been resolved.

A small number of accounts are experiencing timeout errors after SSO from their Heroku applications. We have traced the cause to an edge case in how accounts are provisioned and are actively working on a solution.

We are continuing to investigate this issue.

We have identified the source of the issue causing some accounts to error after logging in via SSO linking from their Heroku dashboards. Throughout this outage, the external WAF interface has continued to protect all sites.

This incident has been resolved. To check if you can access your WAF dashboard correctly now, click on the Expedited WAF or Expedited CDN add-on in your Heroku Application's resources tab. If you have any further issues please report them to support@expeditedsecurity We apologize for the inconvenience that this has caused and are taking steps to modify the underlying procedures and checks that led to the dashboard control plane becoming out of sync with the larger Edge Network.

Some clients began experiencing extended latency resulting in reaching origin servers as a result of an upstream bandwidth provider outage. Network services are monitoring the situation as traffic is moved to alternate providers.

Heroku is currently experiencing issues with their Platform API. This is preventing the creation of new WAF instances as we are dependent upon API responses to configure the WAF. WAFs in the process of being provisioned are also delayed as a result. Existing WAF clients are unaffected.

Heroku has resolved their issues with the Platform API and all issues are resolved.

We're currently investigating a delay in the provisioning of new WAF instances. This does not affect the normal throughput of traffic for existing WAF clients.

We have identified the issue and provisioning times have returned to normal limits.

This incident has been resolved.

A Webfont icon library used in the dashboard was failing to load due to network issues. Upon replacement, with a locally cached version, the issues resolved and the dashboard is loading normally.

We have received reports that a mobile ISP in Thailand has misconfigured their network routing. This has resulted in traffic timing out before it hits our Edge network. We have reported the issue but are unable to directly change their routing to fix the error.

After approximately 6 hours the network errors were resolved and connections stopped timing out.