Historia awarii Funraise

1. 7 sierpnia 2026 17:00 UTCmonitoring

   W piątek, 7 sierpnia 2026, począwszy od 8: 00 AM MDT, niektórzy klienci doświadczyli powolnej lub niespójnej wydajności platformy. Wpływ na przepływy pracy, takie jak listy załadunkowe, dodawanie zwolenników, import i wprowadzanie transakcji offline. Niektóre działania mogą mieć czas w przeglądarce, nawet jeśli żądanie server- side później zakończone, co mogło spowodować duplikaty rekordów, gdy akcja została ponownie wypróbowana. Próby wymeldowania PayPal mogły również w tym okresie spowodować okresowe awarie. W naszym śledztwie zidentyfikowano nadużycia w zautomatyzowanym ruchu, ukierunkowane na publiczny punkt kontrolny PayPal. Ruch ten spowodował ograniczenie wyższego poziomu stawki PayPal i zwiększenie obciążenia usług platformy, co przyczyniło się do zwiększenia opóźnień i błędów okresowych. Zastosowaliśmy dodatkowe filtrowanie ruchu i kontrolę stawek, a usługa wraca do zdrowia. Monitorujemy opóźnienia platformy, błędy i zachowania płatnicze. Przygotowujemy również aktualizację dotyczącą ochrony na poziomie aplikacji w celu dalszego zmniejszenia prawdopodobieństwa wystąpienia podobnego ruchu w zakresie tworzenia zleceń PayPal wpływającego na dostępność platformy.

2. 7 sierpnia 2026 20:07 UTCresolved

   This incident has been resolved.

Automatyczne tłumaczenie oficjalnej aktualizacji incydentu.

1. 9 stycznia 2026 20:25 UTCmonitoring

   We are currently experiencing an issue with our live chat functionality due to an ongoing outage with Intercom, our third-party chat provider. Intercom has confirmed a service disruption affecting Inbox, Messenger, Fin, and related developer features in the US region. As a result, live chat and in-app messaging may be unavailable or not function as expected at this time. Please note that this issue is isolated to Intercom and does not impact the core functionality of our platform. Intercom is actively investigating and working to restore service. We will continue to monitor the situation and provide updates here as more information becomes available. Thank you for your patience.

2. 9 stycznia 2026 21:03 UTCresolved

   Intercom has implemented a fix for the earlier outage affecting Inbox, Messenger, Fin, and related features in the US region. Live chat and in-app messaging should now be functioning normally. We are continuing to monitor the situation to ensure stability and will provide further updates if needed. Thank you for your patience while this issue was being resolved.

1. 20 października 2025 18:15 UTCmonitoring

   We’ve completed an initial investigation of the major AWS outage mentioned in the retroactive post (https://status.funraise.org/incidents/h5bvwz1fy8kv) and confirmed that Funraise’s core services remain operational and unaffected. However, we are seeing impact at the boundaries where third-party platforms are experiencing issues—specifically with integrations. Some data syncs to external systems are failing, most notably Salesforce, which is currently reporting an ongoing outage. We are actively monitoring these dependencies and will provide updates as third-party services recover.

2. 21 października 2025 17:55 UTCresolved

   The issues impacting Funraise services due to the AWS outage have been fully resolved. All systems are now operating normally. Thank you for your patience while we monitored and restored stability.

1. 20 października 2025 14:34 UTCresolved

   We’re aware of an ongoing AWS incident in the US-EAST-1 region that may impact some cloud services. We’re investigating potential effects on Funraise and will update this page as we learn more. What’s happening Amazon Web Services (AWS) is experiencing increased error rates and latency across multiple services in the US-EAST-1 region. AWS has reported connectivity issues and intermittent errors beginning around 12:11 AM PDT and continuing through the morning, with signs of gradual recovery. Impact to Funraise We have not yet confirmed a direct impact to core Funraise applications. Out of an abundance of caution, we are actively monitoring for customer-visible symptoms, including: Elevated error rates or slow page loads in the app or public donation pages Delays in background jobs (exports, webhooks, receipts, integrations) Delayed event processing or queue backlogs What we’re doing Reviewing service health and telemetry across our infrastructure that relies on AWS. Verifying redundancy and autoscaling behavior, and watching for queue backlogs. Background (from AWS public updates) 7:29 AM PDT: AWS confirms multiple services in US-EAST-1 had connectivity issues; early signs of recovery. 7:14 AM PDT: Significant API errors/connectivity issues confirmed across multiple services in US-EAST-1. Earlier updates indicate DNS issues affecting DynamoDB and downstream services, new EC2 instance launch errors in US-EAST-1, and backlog processing for services like Lambda/EventBridge/CloudTrail, with gradual recovery actions underway. We’ll keep this post updated as we learn more. If you’re experiencing urgent issues, please contact Funraise Support with timestamps and any error messages observed.

1. 11 grudnia 2024 17:44 UTCinvestigating

   We're investigating an issue that is causing the Facebook integration to be disconnected.

2. 11 grudnia 2024 21:35 UTCinvestigating

   We are continuing to investigate this issue and if it is related to the the global Meta / Facebook outage.

3. 13 grudnia 2024 18:24 UTCresolved

   We are happy to report that our Facebook integration has been fully restored following the recent Facebook outage. Our team has worked diligently to resolve all issues caused by the disruption, ensuring that everything is back to normal. If your Facebook integration was disconnected during the outage, you will need to re-enable it in your account settings. Once reconnected, please contact our support team to request a re-sync of any Facebook transaction reports. This will ensure your data is accurate and up to date. Thank you for your patience and understanding during this time. If you experience any further issues, please don’t hesitate to reach out to our support team.

1. 24 sierpnia 2023 18:33 UTCmonitoring

   It has been reported by users and verified by our internal team that the PayPal buttons are intermittently disappearing from donation forms. Users affected by this issue are temporarily unable to make transactions through PayPal while the buttons are not visible. Preliminary investigations indicate that the root cause of this issue resides on PayPal's end. PayPal is working on tracking the resolution on their incident page: https://www.paypal-status.com/incident/production

2. 24 sierpnia 2023 18:33 UTCmonitoring

   We are continuing to monitor for any further issues.

3. 25 sierpnia 2023 04:28 UTCresolved

   This incident has been resolved.

1. 19 lipca 2023 19:46 UTCresolved

   Since 2023-07-18 3:01 ET, there was an issue detected with an upstream payments processor causing intermittent donation failures for a subset of customers for a subset of donations. The upstream provider resolved the issue as of 20:00 ET.

1. 30 grudnia 2022 21:34 UTCidentified

   We've identified an issue with an upstream payments provider (Legacy Delegate Mode PayPal) which is causing intermittent failures for a small number of customers and a subset of PayPal donations. We are currently monitoring the upstream provider for restoration of service.

2. 30 grudnia 2022 21:34 UTCidentified

   We are continuing to monitor this issue.

3. 31 grudnia 2022 00:56 UTCidentified

   We are continuing to monitor this issue with our upstream payments provider. We're seeing intermittent failures with other payment methods.

4. 31 grudnia 2022 02:41 UTCresolved

   This incident has been resolved.

1. 5 lipca 2022 20:02 UTCinvestigating

   We've identified an underlying hardware fault at our data tier that is causing some requests to slow down. We're working to add capacity to restore performance.

2. 5 lipca 2022 20:04 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 5 lipca 2022 20:07 UTCresolved

   This incident has been resolved.

4. 5 lipca 2022 20:07 UTCpostmortem

   Underlying database hardware failed and was replaced by high availability fail-over. No loss of data occurred and normal response times were restored within minutes.

1. 21 czerwca 2022 06:18 UTCinvestigating

   We are currently investigating a system wide outage due to an upstream service provider.

2. 21 czerwca 2022 07:17 UTCidentified

   We have identified an issue with our upstream hosting service.

3. 21 czerwca 2022 14:03 UTCmonitoring

   A fix has been implemented and we are monitoring

4. 21 czerwca 2022 19:18 UTCresolved

   This incident is resolved.

1. 28 lutego 2022 20:08 UTCidentified

   We're currently investigating reports of some platform accounts not showing fully up-to-date donation data. Account data is still being stored and ingested and donations are processing correctly but just are not showing up correctly in lists at the moment. We're working on resolving this issue and expect no data loss.

2. 1 marca 2022 04:00 UTCmonitoring

   A fix has been implemented and is being rolled out across Funraise accounts.

3. 2 marca 2022 19:11 UTCresolved

   This incident has been resolved.

1. 22 grudnia 2021 17:27 UTCresolved

   A handful (~10) of donations using our new PayPal integration failed to process between 4:15am to 6:30am Pacific Standard Time due to a power loss at an Amazon data center. Amazon resolved the situation and donations are processing normally through that integration since 6:30am PST.

1. 20 grudnia 2021 17:27 UTCresolved

   Funraise for the does not use log4j for any of its native application code. Funraise identified one area of log4j use in a 3rd party component that we use for search engine functionality (ElasticSearch) that is hosted separately from the Funraise Platform core infrastructure. Funraise applied the appropriate initial patches and mitigation strategies on Dec 13 and continues to monitor the situation as new recommendations and patches are released and has found no evidence of exploit or attempted exploit in our application logs. Log4j was also identified in a 3rd party component powering our BI functionality. This functionality is also hosted separately from the Funraise core infrastructure and presents lowered risk of RCE. The vendor for that component patched and updated their infrastructure on Dec 16. Funraise continues to maintain a proactive security posture and undergoes annual analysis of our codebase and infrastructure by an independent security firm.

1. 7 września 2021 23:39 UTCresolved

   The Funraise platform suffered a downtime from 1:31:11am UTC - 2:05:07am UTC due to insufficient compute to spawn new application instances. Funraise has increased server capacity to mitigate this issue going forward.

1. 27 lipca 2021 20:39 UTCinvestigating

   We're currently investigating reports of some platform accounts showing no data in certain sections. Account data is still being stored and ingested and donations are processing correctly but just are not showing up correctly in lists at the moment. We're working on resolving this issue and expect no data loss.

2. 27 lipca 2021 20:39 UTCinvestigating

   We are continuing to investigate this issue.

3. 27 lipca 2021 22:01 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 28 lipca 2021 02:09 UTCresolved

   This incident has been resolved.

1. 26 lipca 2021 18:50 UTCresolved

   Funraise platform suffered a downtime between 11:06am-11:17am PDT. The cause was underlying infrastructure not being able to boot a new set of application servers in a timely manner. Our systems team responded quickly and restarted our application server formation which provisioned a new set of servers which started successfully.

1. 17 grudnia 2020 19:58 UTCresolved

   Payments (Next Gen) which powers our Free tier as well as some customers that are enabled for advanced fraud protection suffered an outage between 4:10 AM and 9:42 AM Pacific Time which affected an upper bound of 30 potential donors. Our engineering team is continuing to investigate and has put additional notification schemes and telemetry in place to help detect and respond to future issues if they should arise as we isolate the root cause. The vast majority of our payments traffic is serviced by our legacy payments infrastructure and was therefore unaffected. If you're on our next-generation payments infrastructure we're truly sorry for the inconvenience that this outage caused.

1. 1 grudnia 2020 19:11 UTCresolved

   A minor performance degradation to the platform search capability occurred between 10 AM-11 AM PST. Our engineering teams responded by scaling up the capacity of our search infrastructure. To ensure smooth donation processing for your donors, you might have experienced a minor loss of filtering capability within some filterable sections of Funraise. Rest assured there is no data loss and your donations are processing at hyper speed! Keep crushing it!

1. 22 października 2020 07:40 UTCinvestigating

   We are currently investigating email sending delays

2. 22 października 2020 17:36 UTCresolved

   This incident has been resolved.

1. 23 września 2020 18:58 UTCidentified

   Delay in our search services is preventing incoming donations and manually entered data to be visible in the Funraise platform. Payment processing and integration services are still operational at this time. While data may not be readily visible for viewing, there should be no loss of data.

2. 23 września 2020 19:11 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 23 września 2020 20:41 UTCmonitoring

   We are continuing to monitor for any further issues.

4. 23 września 2020 21:59 UTCresolved

   This incident has been resolved.