Blackthorn 장애 기록

Our team is aware of an issue impacting Blackthorn Messaging and SMS — some scheduled messages may be delayed or fail to send. We're actively working on a fix and will keep this page updated. If you have questions, contact our support team at https://community.blackthorn.io/s/support-request

We are continuing to investigate this issue and will continue reporting as more information becomes available. Thank you for your patience.

We're happy to report that this issue has been resolved. Scheduled emails and SMS messages are now sending as expected. If you are still experiencing problems, please reach out to our support team at https://community.blackthorn.io/s/support-request.

We're aware of an issue with Smart Scheduler that began around 9:30am ET today, June 25. The following are currently being affected: - Scheduled emails are not being sent - New emails cannot be created - SMS messages are not being sent - Push notifications are not being sent *Please note: Anything scheduled via Smart Scheduler as of this morning at 9:30am ET has not been sent. Our team is actively investigating this issue. We'll update this page as soon as we have more information or a resolution. If you have questions, contact our support team at https://community.blackthorn.io/s/support-request

RESOLVED *A REQUIRED BASE PACKAGE UPGRADE WILL BE REQUIRED TO RESTORE FULL FUNCTIONALITY* Please see below. Root Cause: The Base Connected App experienced a disconnection during Messaging configuration updates on the morning of June 25th. We've addressed this by provisioning a new Connected App and releasing an updated Base package. As part of the fix, we also improved the security of our Connected App. It now uses a more flexible approach that allows us to rotate secrets without causing downtime for customers. Resolution: Upgrade the Blackthorn Base package from the AppExchange. Only the Base package needs to be upgraded. How to Upgrade: To restore full functionality, please install the updated Base package using the link below. Once installed, open Smart Scheduler and click Authorize to reconnect. - Production: https://login.salesforce.com/packaging/installPackage.apexp?p0=04tdN000000OqDZ - Sandbox: https://test.salesforce.com/packaging/installPackage.apexp?p0=04tdN000000OqDZ Recommended Upgrade Steps: 1. Upgrade your sandbox first 2. Test Smart Scheduler in sandbox 3. Upgrade your production org Once installed, open Smart Scheduler and click Authorize to reconnect. Questions? Submit a support case: https://community.blackthorn.io/s/support-request

There is a Blackthorn Payments feature issue causing temporary, intermittent service disruptions in payment processing. During an intermittent disruption, some features may be unavailable or operate at suboptimal performance. We are in the process of deploying a fix and will provide additional updates.

We are continuing to investigate this issue.

There is an ongoing security incident related to a widespread malware attack affecting multiple organizations, including Blackthorn. As part of our mitigation efforts, we are deploying patched versions of our Payments apps, which has caused temporary disruption to Stripe payment processing. Processing has been restored for the majority of impacted customers, and we are continuing to roll out updates. We will provide additional updates as our investigation continues.

We are currently deploying patched fixes for all impacted organizations. The majority of impacted organizations have been fixed, and we are currently monitoring the results. Additional updates to follow. If you are still experiencing disruptions and have questions, please create a case with Support.

We have completed the deployment of patched fixes for any impacted organizations. Although this incident is resolved, we will continue to monitor results. If you are experiencing disruptions and have questions, please create a case with Support.

The Blackthorn Messaging and Navigator applications are currently experiencing service interruptions due to a widespread outage of several third-party platforms, including Salesforce, AWS, and Twilio. Our infrastructure team is actively working with these providers to monitor the situation and restore full functionality as soon as possible. We’ll continue to provide updates as the issue progresses. In the meantime, you can track the status of these services directly through the following links: Salesforce Status: https://downdetector.com/status/salesforce/ Twilio Status: https://downdetector.com/status/twilio/ AWS Status: https://downdetector.com/status/aws-amazon-web-services/ Thank you for your patience and understanding while we work through this incident. Blackthorn Support

We are continuing to investigate this issue.

The AWS services have been restored and are now fully operational. We are monitoring the situation, but please contact Blackthorn Support at https://community.blackthorn.io/s/support if you run into any issues going forward. To stay updated on AWS’s real-time status, visit: https://health.aws.amazon.com/health/status

We are currently experiencing issues due to a significant Cloudflare outage, which affects the service we use to power vanity domains and manage traffic routing across regions. Our team is actively working to monitor the situation and restore full app stability. We’ll provide updates as soon as we have more information. In the meantime, you can access your events using the default Blackthorn URL format: https://events.blackthorn.io/sample-event Instead of: https://events.yourdomain.com/sample-event Documentation: https://docs.blackthorn.io/docs/events-admin-tab-settings To stay updated on Cloudflare’s real-time status, visit: https://www.cloudflarestatus.com/ For broader context on this major Cloudflare outage, you can also refer to: https://nypost.com/2025/06/12/business/google-cloud-users-report-widespread-outages-across-us/ Thank you for your patience and understanding.

Cloudflare services have been restored and are now fully operational. We are monitoring the situation, but please contact Blackthorn Support at https://community.blackthorn.io/s/support if you run into any issues going forward. To stay updated on Cloudflare’s real-time status, visit: https://www.cloudflarestatus.com/ Thank you!

This incident has been resolved. We are continuing to monitor the situation, but please contact Blackthorn Support at https://community.blackthorn.io/s/support if you experience any further issues.

We are currently experiencing a system issue where the Tabbed UI attendee form is allowing registrations to proceed even when required fields are left unfilled. This may result in incomplete attendee data or registrations not being properly recorded in our system. Our engineering team is actively investigating the issue and working on a fix. We will share updates as soon as the issue is resolved. We strongly recommend using the Dynamic or Simple UI for all attendee registrations until this issue with the Tabbed UI is resolved. Thank you

We’ve identified an issue in the May ’25 Webapp release where registrations using the Tabbed UI could succeed without completing all required form fields. This resulted in errors during registration processing, as incomplete data cannot be processed in Salesforce. To prevent further impact on our users while our Engineering team works on this issue as our highest priority, we are rolling back the Webapp to the April ’25 release. This rollback will restore the proper required field validation behavior. We will share updates as soon as the issue is resolved. Thank you for your patience and understanding.

We've successfully rolled back the May ’25 Webapp release to resolve the issue with the Tabbed UI attendee form. Required field validation is now working correctly, and registrations are processing as expected. We’re continuing to monitor the system and investigate the root cause. A full post-mortem will be shared once the analysis is complete. Thank you for your patience and support.

We are currently investigating an intermittent issue with our Email Template Builder Tool. Our team is actively investigating this issue, and we will provide regular updates as more information becomes available. If you require urgent assistance with email scheduling during this time, please feel free to contact our Support team, and we will be happy to help.

We’re continuing to investigate the issue relating to our Email Template Builder page continuous loading and the BeeFree integration for creating/editing Email Templates. Our first recommendation is to safelist the following IP Addresses (from the “Network Access” section in the Setup menu) as this has allowed for Email Template creation in several impacted environments. These addresses are used within Beefree SDK for outbound communication: 54.78.90.148 52.16.171.98 52.58.39.32 18.158.188.143 If your team are unable to make those IP Address changes in the “Network Access” Set up menu, there are several email options available in the interim: Using Classic or Lightning email templates, we’d be happy to assist with configuring a flow to send your communications. Using your existing marketing tools, we can help generate a Salesforce list of Attendees/Event information to be imported into the tech stack and sent externally. Please let the Support team know if you need help walking through either path. We’ll be sure to provide an update as soon as we have more information about a resolution.

We're currently investigating an issue affecting the Email Template Builder page and the BeeFree integration used for creating and editing email templates. Some users are experiencing continuous loading when attempting to access the builder. While we work on a permanent fix, two workarounds have helped some users temporarily regain access. First, performing a hard browser refresh has been effective in some cases. On Windows, this can be done by pressing Ctrl + F5, and on Mac by pressing Cmd + Shift + R. Second, switching to an alternative browser such as Safari or Microsoft Edge has allowed some customers to access the builder successfully. We will provide further updates as soon as we have more information regarding the root cause and a long-term resolution. Thank you for your continued patience and understanding.

We made progress on the issue impacting the Email Template Builder functionality. Testing for the bug fix has been completed successfully. The leadership team is now coordinating with BeeFree, our email template plugin provider, to determine the best approach for delivering the solution to our customers. Our development team has identified a workaround that has shown the highest success rate so far. When prompted with a blank Email Template Builder screen, they recommend clearing your browser cache and performing a hard reload of the page through the browser console. To assist with this, we’ve included a short video demonstration outlining the exact steps. https://www.loom.com/share/ec7d529983d646b29e8736207ca09512 We appreciate your continued patience as we work toward a permanent resolution and will provide further updates as soon as more information becomes available.

The issue affecting the Email Template Builder functionality and loading times has been resolved. The fix is included in our April '25 release, which is now available. Users can install the updated version by visiting our Candyshop portal: https://candyshop.blackthorn.io/products Once there, select the Blackthorn Events product and click the "Product Upgrade". If you need any assistance with the upgrade, our Support team is ready to help. We appreciate your patience as our team worked through this issue, and we thank you for your continued trust and support.

**Root Cause:** Unanticipated updates in either the BeeFree editor or Salesforce infrastructure introduced a timing issue where the BeeFree script began executing before essential page elements were fully available, resulting in frequent failures of the Email Template Builder to load properly. **Resolution:** We modified the EmailEditor page to ensure the BeeFree script initialises only after all necessary page elements have finished rendering. This fix is now included in our April '25 release.

The Blackthorn Engineering team is currently investigating an issue with the Messaging Application's URL Shortening Service. Messages sent containing shortened URLs may experience deliverability issues. Until this issue is fully resolved, it is recommended to disable the "Enable Trackable Links" feature in the Messaging Admin page.

This incident has been resolved. Root Cause Analysis: A regression occurred on one of our resources that is responsible for processing shortened URLs in SMS messages that are sent through the Blackthorn Messaging app. The necessary measures to ensure that this does not happen again in the future have been implemented. We appreciate your understanding and apologize for any inconvenience caused.

Root Cause Analysis: A regression occurred on one of our resources that is responsible for processing shortened URLs in SMS messages that are sent through the Blackthorn Messaging app. The necessary measures to ensure that this does not happen again in the future have been implemented. We appreciate your understanding and apologize for any inconvenience caused.

On 30th October, several customers using the Authorize.net payment gateway reported an issue with capturing payments. The issue was identified and reported to Authorize.net. They are investigating the issue and continue to monitor the situation. We are actively communicating with Authorization.net to get more information. We will continue to monitor the situation and sincerely apologize for any disruption to your business and customers. You can monitor our status page using the below link for reference: https://community.blackthorn.io/s/system-status The issue seems to be resolved for many customers, but we are still monitoring the reported cases. If you are still experiencing the issue, you may try the following steps: 1. Navigate to your Payment Gateway -> Click Connect Gateway. This will reconnect your payment gateway. 2. If the issue persists, regenerate the transaction API key in authorize.net and re-authorize the Payment Gateway once more in Salesforce. 3. If you are still experiencing the issue, please feel free to contact Blackthorn Support. Thank you!

We are continuing to investigate this issue.

We are pleased to inform you that Authorize.net has resolved the issue and has implemented a fix to mitigate the issue that was affecting the oAuth service used to authenticate Application Interface (API) requests to Authorize.net. We will continue to monitor the situation actively and sincerely apologize for any disruption to your business and customers. You can monitor our status page using the below link for reference: https://community.blackthorn.io/s/system-status The issue seems to be resolved for many customers, but we are still monitoring the reported cases. If you are still experiencing the issue, you may try the following steps: 1. Navigate to your Payment Gateway -> Click Connect Gateway. This will reconnect your payment gateway. 2. If the issue persists, regenerate the transaction API key in authorize.net and re-authorize the Payment Gateway once more in Salesforce. 3. If you are still experiencing the issue, please feel free to contact Blackthorn Support.

The incident has been reportedly fixed. We are still monitoring for other occurrences.

On October 30th, several customers using the [Authorize.net](http://authorize.net/) payment gateway reported an issue with capturing payments.The [Authorize.net](http://authorize.net/) team identified that a recent certificate change in the upstream process flow caused this issue. The change has been rolled back by their team. This issue is now resolved. The incident was related to [Authorize.net](http://authorize.net/) API Errors, where merchants encountered the error “E00I24—The provided access token is invalid.”  To mitigate the impact on the gateway, the certificate migration was rolled back, and the application was restarted. We will continue to monitor the situation actively and sincerely apologize for any disruption to your business and customers. Thank you

Earlier this morning, several customers using the Auth.net payment gateway reported an issue where they are unable to capture the payments. This issue has been escalated to our engineering team as the highest priority, and they are investigating it right now and continue to monitor the situation. We will provide an update shortly. Thank you

Earlier this morning, several customers using the Authorize.net payment gateway reported an issue where they are unable to capture the payments. Authorize.net has been notified of this, and they are investigating the issue and continue to monitor the situation. We will provide an update shortly. Thank you

We are pleased to inform you that Authorize.net has resolved the issue and is working on determining the root cause. We will continue to monitor the situation actively and sincerely apologize for any disruption to your business and customers.

We are continuing to monitor for any further issues.

This incident has been resolved.

On 29th October, several customers using the [Authorize.net](http://authorize.net/) payment gateway reported an issue with capturing payments. The issue was identified and reported to [Authorize.net](http://authorize.net/). We are pleased to inform you that [Authorize.net](http://authorize.net/) has resolved the issue and is working on determining the root cause. We will continue to monitor the situation actively and sincerely apologize for any disruption to your business and customers.You can monitor our status page using the below link for reference: [https://community.blackthorn.io/s/system-status](https://community.blackthorn.io/s/system-status) The issue seems to be resolved for many customers, but we are still monitoring the reported cases.If you are still experiencing the issue, you may try the following steps: 1. Navigate to your Payment Gateway -> Click Connect Gateway. This will reconnect your payment gateway. 2. If the issue still persist, then regenerate the transaction API key in [authorize.net](http://authorize.net/) and then re-authorize the Payment Gateway once more in Salesforce. 3. If you are still experiencing the issue, please feel free to contact Blackthorn Support

Our Engineering team is currently investigating an ongoing issue impacting SMS deliverability. We will provide an update shortly.

The SMS deliverability issue has been successfully identified and resolved. We will provide a detailed Root Cause Analysis for this incident and apologize for this system outage with the Messaging application.

Our engineering team has identified the root cause of the SMS Application outage.  We experienced a brief service interruption on September 26, 2024, between 12:00 PM and 2:00 PM EDT, which impacted all of our messaging services. Our internal load-balancer auto-scaling was stuck, which required our Engineering team to modify the web server. During this time, customers were unable to send or receive messages, and updates related to Stripe failed as well. Services were fully restored within two hours, and we ensured that no data was lost by verifying all processes through our health-check systems and API. Going forward, our Engineering team will generate additional Fallback DNS configuration to pass requests to another resource for processing redundancy to avoid further disruption.

Earlier this afternoon, several customers reported an intermittent event URL display issue. This issue was related to a recent Events Webapp release. Our engineering team has resolved the error and is continuing to monitor the situation.

Our Engineering team is currently investigating an issue relating to the inability to send messages. We will provide updates here as the investigation continues. Thank you for your patience as we urgently work to restore SMS deliverability.

Messaging App service is being restored, and we will confirm when this is complete. We are continuing to monitor the situation and will also provide a Root Cause Analysis regarding this issue.

Our Blackthorn Messaging App service has now been restored. We are currently monitoring the situation but please reach out to Blackthorn Support at https://community.blackthorn.io/s/support-request if you run into any issues going forward.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

After further investigation, our engineering team has identified the root cause.  It appears the Database supporting production services reached it's limit preventing customers from being able to send messages. This service had to be expanded to restore messaging, and should continue working as expected. Moving forward, Autoscaling and monitoring tools will be enabled as part of the resolution so this shouldn’t happen again.

The Blackthorn Engineering team recently updated security settings related to the connected application signing keys for our webapps. If you are continuing to experience any issues with Blackthorn Paylink, DocumentLink, or Event Page URLs, we recommend re-authorizing the application to resolve the issue. DocumentLink: To resolve this issue, we recommend reauthorizing the DocumentLink. Please note, for the authorization to work properly, you can only be logged into one Salesforce org. Here are the instructions for reauthorizing: https://docs.blackthorn.io/docs/authorize-documentlink Paylink: To resolve this issue please reauthorize the PayLink by navigating to the "Blackthorn | PayLink" app. Events: The Blackthorn Events application can be successfully reauthorized with the following steps from our User Guide: https://docs.blackthorn.io/docs/configuration#authorize-the-app Please let our Support Team know if you have any questions or concerns.

The Blackthorn Engineering team recently updated security settings related to the connected application signing keys for our webapps. If you are continuing to experience any issues with Blackthorn Paylink, DocumentLink, or Event Page URLs, we recommend re-authorizing the application to resolve the issue. If you need additional help re-authorizing your applications or run into any issues, please create a system-impaired case using this link: https://community.blackthorn.io/s/support-request. Support is also available for live zoom support if your team needs additional help with re-authorization. Please request live support when submitting your support case if additional help is required. DocumentLink: To resolve this issue, we recommend reauthorizing the DocumentLink. Please note, for the authorization to work properly, you can only be logged into one Salesforce org. Here are the instructions for reauthorizing: https://docs.blackthorn.io/docs/authorize-documentlink Paylink: To resolve this issue please reauthorize the PayLink by navigating to the “Blackthorn | PayLink” app, clicking "Authorize" blue checkmark, then clicking the "Grant Access" button. Events: The Blackthorn Events application can be successfully reauthorized with the following steps from our User Guide: https://docs.blackthorn.io/docs/configuration#authorize-the-app Please let our Support Team know if you have any questions or concerns.

This incident has been resolved.

Recently, we enhanced our security protocols, specifically through the rotation of the signing key for our Salesforce connected app, which is a critical component our products use to securely link your Salesforce organization to our web applications \(Events, PayLink, DocmentLink, Storefront\). This need for reauthentication was a direct result of us updating our security protocol. While we understand this caused a temporary disruption, please allow me to explain why this step was both necessary and ultimately beneficial: **Enhanced Security:** Regularly updating signing keys is a critical practice in maintaining high security standards. This process helps in safeguarding against potential vulnerabilities and ensuring that the integrity of your data remains uncompromised. **Proactive Risk Management:** The longer a signing key remains unchanged, the higher the risk of it being compromised. Rotating these keys minimizes this risk, thereby protecting both your data and our system from unauthorized access. **Adherence to Best Practices:** This update aligns with industry best practices and regulatory requirements for digital security. It's part of our commitment to adhering to the highest standards and ensuring that we are always focused on data protection. **Our Apology and Commitment to You** We realize that the need for reauthentication might have caused unexpected disruptions in use of our applications. We deeply regret any inconvenience this has caused, and are taking steps to ensure that future updates are smoother, timely, and more transparent. **Support and Assistance** To assist with any lingering issues or concerns, our dedicated support team is on standby. Additionally, we have prepared resources and guides to help you navigate these changes:  **DocumentLink:** To resolve this issue, we recommend reauthorizing the DocumentLink. Please note, for the authorization to work properly, you can only be logged into one Salesforce org. Here are the instructions for reauthorizing: [https://docs.blackthorn.io/docs/authorize-documentlink](https://docs.blackthorn.io/docs/authorize-documentlink) **Paylink:** To resolve this issue please reauthorize the PayLink by navigating to the “Blackthorn | PayLink” app, clicking "Authorize" blue checkmark, then clicking the "Grant Access" button. **Events:** The Blackthorn Events application can be successfully reauthorized with the following steps from our User Guide:[ https://docs.blackthorn.io/docs/configuration#authorize-the-app](https://docs.blackthorn.io/docs/configuration#authorize-the-app) If you need additional help re-authorizing your applications or run into any issues, please create a system-impaired case using this link: [https://community.blackthorn.io/s/support-request](https://community.blackthorn.io/s/support-request). Support is also available for live zoom support \(please request when submitting case\) if your team needs additional help with re-authorization. **Looking Ahead** Rest assured, we are fully committed to continuously improving our services and the experience they provide. Prior to the next scheduled key rotation we’ll be implementing a more robust update plan and communications. Thank you for your being part of the Blackthorn community, please contact us directly with any questions. Stuart Croft, COO. [Stuart@blackthorn.io](mailto:Stuart@blackthorn.io) Andrea Adcock, Chief Product Officer, [Andrea@blackthorn.io](mailto:Andrea@blackthorn.io)

The Blackthorn Engineering team recently updated security settings related to the connected application signing keys for our webapps. This process is now complete. If you are continuing to experience any issues with Blackthorn Paylink or DocumentLink, we recommend re-authorizing the application to resolve the issue.

Our Engineering team is currently investigating intermittent reports of slower-than-expected loading times for Event URLs and display issues. We will provide regular updates as this investigation progresses.

This issue has been resolved and our Engineering teams are continuing to monitor the situation.

This incident was caused by a Platform Worker response error by one of our service providers and resulted in intermittent Event URL display issues and longer-than-average loading times for pages. This issue has been resolved.

The Blackthorn Events application experienced a brief service interruption resulting in Event URL page display issues. This issue has been resolved.

We are monitoring a server and platform performance issue on one of our service providers that is impacted by unusual traffic activity. This may result in Event URL display issues.

The incident regarding the performance issue originating from our service provider has been resolved.

This incident was caused by a Platform Worker response error by one of our service providers and resulted in temporary Event URL display issues. This issue has been resolved.