Betty Blocks outage history

Hi, We are writing to inform you that our platform is currently experiencing degraded performance. This means that applications running on the below-mentioned zones may not be able to access our services. We understand this is disruptive, and we sincerely apologize for any inconvenience this may cause. Our team is working with the highest priority on this to identify the root cause of the issue and restore services as quickly as possible._ The following zones and components are currently affected: Affected Zones: NL6 Affected Components: * WASM actions Our highest priority is to bring our services back online as quickly and safely as possible. We have dedicated engineers working to resolve this issue. We will keep you informed of our progress and provide an estimated time of restoration as soon as we have a clearer understanding of the situation. You can stay informed by checking our status page at Betty Blocks Status , or by subscribing to updates on this page. Thank you for your patience and understanding during this critical time. We appreciate your support. Kind regards, The Support Team

Hi, Our product department has increased the resources available for WASM to improve stability, as a result the WASM actions are now running as expected again on the NL6 server. We want to thank you for your patience and understanding in this matter. Kind regards, Support

We are writing to inform you about an incident which affected users making use of sending mails. During this time, you may have experienced degraded performance in sending mails originating from our MandrillApp services. The following zones and components are currently affected: Affected Zones: All zones were affected, but only customers that are/were making use of our MandrillApp mailing services were affected. Affected Components: Classic actions - Classic actions that are making use of a MandrillApp API key in order to send out mails would fail to deliver the mail. NextGen Actions - NextGen actions that are making use of a MandrillApp API key in order to send out mails would fail to deliver the mail. The issue was quickly located and a fix has been implemented for this, which allowed the incoming mails being sent to work as expected again. We would like to apologize for the degraded performance that that has been experienced, and want to thank you for your patience.

We are writing to inform you about a temporary period of degraded performance you may currently be experiencing on our platform. Our team is aware of this issue and is actively working to identify and resolve it as quickly as possible. During this time, you may experience slower loading times, occasional delays in processing, or intermittent responsiveness in certain areas of the platform. The following zones and components are currently affected: Affected Zones: - NL4 Affected Components: - Bettywebblocks - You may experience delays in the loading times of your page - Background actions - Background actions, or scheduled actions are being processed at a degraded performance. We understand that degraded performance can be frustrating, and we sincerely apologize for any inconvenience this may cause to your workflow. Our engineers are working diligently to pinpoint the root cause and implement the necessary fixes. As soon as more information is available, we will provide more updates on our status page.

We are pleased to report that we have implemented initial fixes that have resulted in improvements to the performance on the public NL4 zone. You will notice improvements in jobs resolving at regular speed, and not having to wait a long time for the services to provide a response back. We are still actively working on fully resolving the underlying issue and restoring optimal performance across all mentioned zone(s) and component(s). We will continue to monitor the situation closely and provide further updates as needed.

We are happy to inform you that the performance issues on the public NL4 zone has now been fully resolved. Our team has implemented the necessary fixes, and you should now be experiencing normal performance across all relevant services. Thank you for your patience and understanding during this period.

Hi, We are writing to inform you that our platform is currently experiencing an unexpected outage. This means that applications running on the below-mentioned zones are experiencing degraded performance, and you may not be able to access our services. We understand this is disruptive, and we sincerely apologize for any inconvenience this may cause. Our team is working with the highest priority on this to identify the root cause of the issue and restore services as quickly as possible._ The following zones and components are currently affected by this outage: Affected Zones: Nl6 Affected Components: * WASM actions Our highest priority is to bring our services back online as quickly and safely as possible. We have dedicated engineers working to resolve this issue. Thank you for your patience and understanding during this critical time. We appreciate your support. Kind regards, The Support Team

This incident has been resolved.

Hi, We are writing to inform you of an unexpected outage on our platform. Scheduled actions were not being picked up, meaning data processing within the scheduled actions flow was not carried out on the affected zones during this period. We understand this is disruptive and sincerely apologize for any inconvenience caused. Our team treated this with the highest priority, and services have now been restored. The following zones and components were affected by this outage: Affected Zones: NL3, NL4 and NL6 Affected Components: * Scheduled actions Thank you for your patience and understanding during this critical time. We appreciate your support. Kind regards, The Support Team

We are writing to inform you that our platform is currently experiencing an unexpected outage. This means that applications running on the below-mentioned zones are temporarily unavailable, and you may not be able to access our services. We understand this is disruptive, and we sincerely apologize for any inconvenience this may cause. Our team is working with the highest priority on this to identify the root cause of the issue and restore services as quickly as possible. The following zones and components are currently affected by this outage: Affected Zones:
NL6 Affected Components: - WASM actions Our highest priority is to bring our services back online as quickly and safely as possible. We have dedicated engineers working to resolve this issue. We will keep you informed of our progress and provide an estimated time of restoration as soon as we have a clearer understanding of the situation. You can stay informed by checking our status page at Betty Blocks Status , or by subscribing to updates on this page. Thank you for your patience and understanding during this critical time. We appreciate your support. Kind regards, The Support Team

We are continuing to investigate this issue.

We are pleased to report that we have implemented a fix that is showing improvements to the performance on the public NL6 zone. The WASM functionalities are working as expected again, and will be monitored, until the stability is maintained. We will continue to monitor the situation closely and provide further updates as needed. Thank you for your understanding.

We are happy to inform you that the performance issues on the public NL6 zone have now been fully resolved. Our team has implemented the necessary fixes, and you should now be experiencing normal performance across all zones and components. Thank you for your patience and understanding during this period. We appreciate your support. If you continue to experience any performance issues, please do not hesitate to contact our support team through the chat, or reach out to us at [email protected]

There was an issue on the Private Cloud 13 which resulted in a partial outage regarding the Actions-JS functionalities of our platform. During this time, you may have been unable to make use of our NextGen actions, unable to log in to the runtime of your application, or intermittent responsiveness in certain areas of the platform. Affected Zones: Private Cloud 13 - Users were unable to make use of NextGen action, preventhing them from logging into the platform, or make use of other actions related functionalities. Affected Components: Actions-runtime: The use of actions in the runtime of your application were experienced as unresponsive. Resulting in errors being returned and preventing any further use of the application. Our Product department has implemented the necessary fixes in order for these issues to be resolved. Thank you for your patience and understanding.

private zone 18 is being affected by server issues. We are currently investigating the cause.

A fix has been implemented and we are monitoring the results.

We see a spike in CPU usage that is causing degraded performance. We restarted the affected service as a mitigation. We are continuing to investigate the root cause

Currently monitoring

This incident has been resolved.

There was an issue on the Private Cloud 12 which resulted in degraded performance regarding the Actions-JS functionalities of our platform. We are writing to inform you about a period of degraded performance that was experienced on our platform. During this time, you may have experienced slower loading times, occasional delays in processing, or intermittent responsiveness in certain areas of the platform. Affected Zones: Private Cloud 12 - Users were experiencing slowness with the usage of Actions-JS functionalities. Affected Components: Actions-runtime: The use of actions in the runtime of your application could be experienced as slow, or unresponsive. Resulting in errors being returned. Our Product department has implemented the necessary fixes in order for these issues to be resolved. Thank you for your patience and understanding.

We are writing to inform you about a temporary period of degraded performance you may currently be experiencing on our platform. This is due to an outage regarding the Anthropic Claude services. Our team is aware of this issue and is actively working to identify and resolve it as quickly as possible. During this time, you may experience slower loading times, occasional delays in processing, or intermittent responsiveness in certain areas of the platform. All zones that have the option to make use of our Betty Genius may experience delays, or unresponsiveness in the creation and generation of your Betty Genius Pages. Affected Components: Betty Genius - The generation of pages based on prompts may be unavailable, or slow in response. We understand that degraded performance can be frustrating, and we sincerely apologize for any inconvenience this may cause to your workflow. The engineers are working diligently to pinpoint the root cause and implement the necessary fixes.

We are happy to inform you that the performance issues on all zones have been fully resolved. The Anthropic team has implemented the necessary fixes, and you should now be experiencing normal performance across all zones and components. Thank you for your patience and understanding during this period. We appreciate your support. If you continue to experience any performance issues, please do not hesitate to contact our support team via the chat, or reach out to us at [email protected].

A Degraded performance was discovered on the public NL3 zone. During this time, you may have experienced slower loading times, occasional delays in processing, or intermittent responsiveness in certain areas of the platform. The following zones and components were affected: Affected Zones: Public zone NL3: (Europe West Server Cluster) - Users in this geographic region may experience increased latency. Affected Components: Classic Backoffice grids - You may notice data taking longer than usual to load or refresh. Classic Endpoints - Reaching out to your endpoints may not have resolved, or resolved at a slower rate. Our Product department has taken the necessary measures to ensure the stability of the platform. In case any unexpected behaviour is still experienced, please do not hesitate to reach out to us through the chat bubble, or at [email protected]

We are writing to inform you about a temporary period of degraded performance you may currently be experiencing on our platform. Our team is aware of this issue and is actively working to identify and resolve it as quickly as possible. During this time, you may experience slower loading times, occasional delays in processing, or intermittent responsiveness in certain areas of the platform. The following zones and components are currently affected: Affected Zones: Public zone NL4 Affected Components: Classic actions: You may notice actions taking longer than usual to trigger, or finish its job. We understand that degraded performance can be frustrating, and we sincerely apologize for any inconvenience this may cause to your workflow. Our engineers are working diligently to pinpoint the root cause and implement the necessary fixes. We will provide updates on our progress and, if possible, give an estimated time to resolution as they become available. You can stay informed by checking our status page at https://status.bettyblocks.com, or by subscribing to updates on this page. Thank you for your patience and understanding as we work to restore full performance.

We are happy to inform you that the performance issues on the public NL4 zone have now been fully resolved. Our team has implemented the necessary fixes, and you should now be experiencing normal performance across all zones and components. Thank you for your patience and understanding during this period. We appreciate your support. If you continue to experience any performance issues, please do not hesitate to contact our support team via Intercom.

We are writing to inform you that our platform is currently experiencing an unexpected outage. This means that applications running on the below-mentioned zones are temporarily unavailable, and you may not be able to access our services. We understand this is disruptive, and we sincerely apologize for any inconvenience this may cause. Our team is working with the highest priority on this to identify the root cause of the issue and restore services as quickly as possible. The following zones and components are currently affected by this outage: Affected Zones: - Public zone NL6 Affected Components: Pagebuilder - It is unable to access the functionalities of this page. Datamodel overview - It is unable to access the overview of your datamodels, and other permission related data. Application Configurations - It is not possible to access the configurations that are defined within your application. ring this outage. Our highest priority is to bring our services back online as quickly and safely as possible. We have dedicated engineers working to resolve this issue. We will keep you informed of our progress and provide an estimated time of restoration as soon as we have a clearer understanding of the situation. You can stay informed by checking our status page at Betty Blocks Status , or by subscribing to updates on this page. Thank you for your patience and understanding during this critical time. We appreciate your support.

We are happy to inform you that the performance issues on the public NL6 zone have now been fully resolved. Our team has implemented the necessary fixes, and you should now be experiencing normal performance across all zones and components. Thank you for your patience and understanding during this period. We appreciate your support. If you continue to experience any performance issues, please do not hesitate to contact our support team via the chat, or reach out to us via [email protected].

Affected Components: All Betty Blocks endpoints were unreachable between 11:52 AM and 12:06 PM CET.

We are writing to inform you about a temporary period of degraded performance you may currently be experiencing on our platform. Our team is aware of this issue and is actively working to identify and resolve it as quickly as possible. During this time, you may experience slower loading times, occasional delays in processing, or intermittent responsiveness in certain areas of the platform. The following zones and components are currently affected: Affected Zones: NL3: West europe cluster - Users in this geographic region may experience increased latency. Affected Components: Expression resolving: Classic expressions - You may notice an increased amount of expressions trying to resolve themselves at a slower rate. We understand that degraded performance can be frustrating, and we sincerely apologize for any inconvenience this may cause to your workflow. Our engineers are working diligently to pinpoint the root cause and implement the necessary fixes.

We are happy to announce that the outage affecting the zone NL3 has now been fully resolved, and all services are back online. Thank you so much for your patience and understanding during this challenging time. We know this disruption was significant, and we truly appreciate your support. Our team is now conducting thorough checks to ensure the stability and performance of the platform. If you experience any further issues, please do not hesitate to contact our support team via the chat, or at [email protected]

We are writing to inform you about a temporary period of degraded performance you may currently be experiencing on our platform. Our team is aware of this issue and is actively working to identify and resolve it as quickly as possible. During this time, you may experience slower loading times, occasional delays in processing, or intermittent responsiveness in certain areas of the platform. The following zones and components are currently affected: Affected Zones: NL6: West europe cluster - Users in this geographic region may experience increased latency. Affected Components: Expression resolving: Classic expressions - You may notice an increased amount of expressions trying to resolve themselves at a slower rate. We understand that degraded performance can be frustrating, and we sincerely apologize for any inconvenience this may cause to your workflow. Our engineers are working diligently to pinpoint the root cause and implement the necessary fixes.

We are pleased to report that we have implemented some initial fixes that have resulted in improvements to the performance on public NL6 zone. You may notice improvement in speed regarding resolving expressions. However, we are still actively working on fully resolving the underlying issue and restoring optimal performance across all mentioned zone(s) and component(s). We will continue to monitor the situation closely and provide further updates as needed.

We are happy to inform you that the performance issues on the NL6 zone have now been fully resolved. Our team has implemented the necessary fixes, and you should now be experiencing normal performance across all zones and components. Thank you for your patience and understanding during this period. We appreciate your support. If you continue to experience any performance issues, please do not hesitate to contact our support team via the support chat, or at [email protected].

We are writing to inform you about a temporary period of degraded performance you may currently be experiencing on our platform. Our team is aware of this issue and is actively working to identify and resolve it as quickly as possible. During this time, you may experience slower loading times, occasional delays in processing, or intermittent responsiveness in certain areas of the platform. The following zones and components are currently affected: Affected Zones: Public zone NL6: - Users on this zone may experience issues with retrieving or uploading Assets to their application. Affected Components: Assets/Files: (Downloading/Uploading files, or images) - You may notice downloading, or uploading files/images taking longer than usual, or run into errors when retrieving these. We understand that degraded performance can be frustrating, and we sincerely apologize for any inconvenience this may cause to your workflow. Our engineers are working diligently to pinpoint the root cause and implement the necessary fixes. We will provide updates on our progress and, if possible, give an estimated time to resolution as they become available. You can stay informed by checking our status page at https://status.bettyblocks.com, or by subscribing to updates on this page. Thank you for your patience and understanding as we work to restore full performance.

Our team has identified the root cause of the issue and is currently working on providing manual fixes to applications that have reached out. Furthermore, we are currently working on implementing a permanent fix that will be rolled out to all applications on the public NL6 zone. Performance may still be impacted during this time, but we are working hard to restore full functionality as quickly as possible. We will provide another update within the next few hours. Thank you for your ongoing patience.

We are pleased to report that we have implemented some initial fixes that have resulted in improvements to the performance on the public NL6 zone. You may notice assets being retrieved again. However, we are still actively working on fully resolving the underlying issue and restoring optimal performance. We will continue to monitor the situation closely and provide further updates as needed. Thank you for your understanding as we work towards a complete resolution.

We are happy to inform you that the performance issues on the public NL6 have now been fully resolved. Our team has implemented the necessary fixes, and you should now be experiencing normal performance across all components. Thank you for your patience and understanding during this period. We appreciate your support. If you continue to experience any performance issues, please do not hesitate to contact our support team via Intercom.

We are writing to inform you about a temporary period of degraded performance you may currently be experiencing on our platform. Our team is aware of this issue and is actively working to identify and resolve it as quickly as possible. During this time, you may experience slower loading times, occasional delays in processing, or intermittent responsiveness in certain areas of the platform. The following zones and components are currently affected: Affected Zones: - NL6 Affected Components: [Actions-runtime]: (processing actions) - You may encounter errors, or actions that are not working as expected. We understand that degraded performance can be frustrating, and w apologize for any inconvenience this may cause to your workflow. Our engineers are working diligently to pinpoint the root cause and implement the necessary fixes. We will provide updates on our progress and, if possible, give an estimated time to resolution as they become available. Thank you for your patience and understanding as we work to restore full performance. Kind regards, The Support Team

We are happy to inform you that the performance issues have now been fully resolved. Our team has implemented the necessary fixes, and you should now be experiencing normal performance across all zones and components. Thank you for your patience and understanding during this period. We appreciate your support. If you continue to experience any performance issues, please do not hesitate to contact our support team via Intercom. Kind regards, The Support Team

Recently, a second malicious version of several npm packages were published under the campaign known as “Sha1-Hulud.” These packages were designed to steal credentials and spread further across the software supply chain. We want to emphasize that there is no impact on existing releases. All current releases of the Betty Blocks platform (public cloud, private cloud, and on-premise) have not been affected by the malicious npm packages. At Betty Blocks, we want to update you on how we are handling this situation. What Betty Blocks is doing - Version-pinned packages: Almost all npm packages used in our platform are version-pinned. This means updates are never automatic and only happen when explicitly approved by our engineers. - Verification before release: In the coming weeks, every platform component that depends on npm packages will be checked to ensure no infected versions are included before we release to testing, acceptance, or production environments. All packages not pinned to a specific version will be verified before used in the next release. - On-premise and private cloud deployments: Clients running Betty Blocks in on-premise or private cloud setups will only receive updates that we have confirmed to be free of the affected package versions. In addition to the above measures, we have analyzed 284 internal and public code repositories and determined none of these repositories were affected by both current sha1-hulud and the previous shai-hulud incidents. What clients should know If you are developing your own custom components within the Betty Blocks platform, please be aware: - You are responsible for the npm packages you include in your custom work. - We strongly recommend reviewing your package versions and checking them against the published list of affected libraries. - Betty Blocks does not actively monitor custom development performed by clients. If you have any questions about this topic, or would like guidance on how to check your own packages, please contact your Betty Blocks support representative.

We are writing to inform you about a temporary period of degraded performance which you may have experienced on our platform. Our team is aware of this issue and has taken immediate action to resolve this as soon as we could. During this time, you may have experienced slower loading times, occasional delays in processing, or intermittent responsiveness in certain areas of the platform. The following zones and components were affected: Affected Zones: - NL1 Affected Components: Classic Endpoints & Classic Backoffice records: Users are having trouble uploading or retrieving files or assets from their applications. We understand that degraded performance can be frustrating, and we apologise for any inconvenience this may have caused to your workflow. Thank you for your patience and understanding.

We are happy to inform you that the performance issues on the NL1 zone have been fully resolved. Our team has implemented the necessary fixes, and you should now be experiencing normal performance across all zones and components. Thank you for your patience and understanding during this period. We appreciate your support. If you continue to experience any performance issues, please do not hesitate to contact our support team via Intercom.