Font Awesome outage history

Pro NPM appears to be down. We are currently investigating the issue.

This incident has been resolved.

Several Font Awesome components are offline. We are currently investigating the issue and will provide an update when we have more information.

All services are restored. This appears to have been an issue with our CDN.

Our network provider, Cloudflare, is having some issues this morning. We're monitoring the situation and working to mitigate what we can.

Cloudflare is in global recovery mode and we haven't seen any errors for the past half hour. We're going to consider this resolved but continue watching it until the outage is completely resolved on the Cloudflare side.

The team is looking at the issue

We identified the root cause and the issue has now been resolved.

We're looking into reports that downloads are failing from Kits.

We think we've got this resolved. We had an internal queue that had backed up and was rejecting new jobs. We've got plans to re-architect this system pretty soon. We'll keep an eye on this. Please let us know if you experience any issues by sending an email to [email protected].

After monitoring for awhile, this looks to be resolved.

We are investigating an issue with Kits npm packages failing to build.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

Our kits infrastructure is currently down. We are investigating the issue and hope to have this resolved as quickly as possible.

We discovered an issue with one of our cloud providers. We are working to get it rectified.

This incident has been resolved.

The fontawesome.com website and API service were offline for around 10 minutes due to a DDoS. Service was restored after the issue was mitigated by our CDN provider.

We are currently investigating this issue.

We've identified the issue and are working on a root cause fix. In the meantime, the site is in maintenance mode to keep pressure off our database server.

A partial fix has been implemented now and the site is fully operational. There is one final fix we need to make that will bring our new metered billing feature back online.

The root fix has been applied and metered billing is also available again. Things look to be stable and humming along fine now.

We've missed a detail in our deployment of some new features and token management is not currently functional. We are working on the fix now.

We've got this all fixed up.

We are currently investigating this issue.

This issue has been resolved. We had a Load Balancer that had a quickly changing IP that caused other systems to be out-of-sync with the Kit origin servers. We'll be working to address the root cause using a different infrastructure solution.

We have a small number of requests (less than 1%) that are returning either a 530 DNS Error or 1016 Origin DNS Error. We've spoken with Cloudflare on this issue and they have acknowledged the issue and are investigating. https://www.cloudflarestatus.com/ We'll keep this issue up-to-date as we learn more.

Cloudflare has a fix in place and they are monitoring.

No further issues have been found over the last 24 hours. We're going to consider this one resolved.

A database which stores information for serving Kits through our CDN had a failover event which elected a new primary. During this time there was no interruption in serving Kits through the CDN but unfortunately it started preventing Kits from being saved through our web management at fontawesome.com/kits. We'll begin investigating why this issue affected the service in this way and plan changes necessary to fix the root cause.

We are currently investigating this issue.

This incident has been resolved.

We are currently investigating this issue.

This incident has been resolved.

We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating this issue.

This service has been restored to operational and we are now investigating the cause.

This incident has been resolved.

On May, 30th we attempted to deploy some new features to our Kits service which required a multi-step deployment procedure we've been using for over 3 years. Our Kits service runs on Amazon EC2 instances. We have the service distributed globally in 7 regions. Those instances function as origin servers for our CDN service. We are using Cloudflare's Load Balancer product in order to serve traffic at the edge. When we upgrade the software we systematically take regions offline, upgrade them, and them bring them back online. For our customers this normally results in zero downtime and the process is unnoticeable and seamless. Up until recently our deployment procedure has been stable and we haven't had any major downtime related to the deployment process itself. However, we saw different behavior today that led to a significant degradation of the service. During the last phases of the deployment which usually takes about an hour we noticed that a region became overloaded during the transition from out-of-service to in-service. The load on the now in-service region jumped to unexpected levels well above the normal traffic patterns seen. This load caused the individual servers to become unresponsive which then led to load shedding to other regions. Unfortunately, this only compounded the issue as the pattern repeated in the fallback region. With the increased and surging loads in various regions a cycle of in-service, surge load, instance failure continued until the entire Kits service was unstable and no viable origin servers were available to service requests. During the Kit service failure we also began to see load increase on one of the database servers that is used by [fontawesome.com](http://fontawesome.com). The reason for this is unknown right now but we suspect there is some indirect tie that needs to be found and corrected. The additional load on the database caused the [fontawesome.com](http://fontawesome.com) website to stop responding to most requests that require database connectivity. At this point our mitigation strategy was two-fold: 1. Increase the number of origin servers to support the Kits service 2. Upgrade the database server to a larger instance Our team began performing the steps necessary to scale up to handle the surge load. After these steps were complete both the Kits service and [fontawesome.com](http://fontawesome.com) site began functioning as normal. We are still investigating the link between Kits service load and the [fontawesome.com](http://fontawesome.com) database server. We will also be working with Cloudflare to understand what changes might have contributed to this issue. Over the next few days and weeks \(however long it takes\) we will look at this issue as a team and determine what steps we need to take in order to prevent this type of failure in the future. We understand that our customers rely on us and the high availability-especially the Kits service. When we are down we lose trust and fail to provide the level of service that we've pledged to you. That’s not acceptable to us and we know it’s not acceptable to our customers either. If you have any questions please feel free to email us at [[email protected]](mailto:[email protected]).

We are currently investigating this issue.

A distributed attack was launched against the site that has been mitigated. We will continue to monitor the situation.

This incident has been resolved.

We've had a handful of reports that attempts to visit fontawesome.com fails with a Cloudflare error about DNS resolution to the origin server. We're looking into this but think this is just a temporary condition.

This issue with 1016 errors applies primarily to traffic routed through Cloudflare's São Paulo and Johannesburg datacenters. We're continuing to work with Cloudflare to find a resolution to the issue. Thank you for your patience.

This incident has been resolved. Our website and API service are once again fully operational worldwide.