Störungsverlauf von Technolutions

We're observing increased disk latency for databases hosted on the MORA cluster. We will be initiating a failover to secondary infrastructure as we investigate further.

We initiated a failover at 12:30pm Eastern Time, the failover went smoothly, and databases are operating within expected performance ranges. We will initiate a failback later this evening following a restart of the primary service infrastructure. We will continue to monitor to ensure that performance levels remain within expected ranges.

We are investigating an issue that is affecting the availability of certain databases. We will provide updates as we have them.

The affected databases are online and available again. We are continuing to investigate the cause of this interruption.

The interruption in availability was caused by a loss of quorum in the failover cluster. We are continuing to investigate periodic network interruptions that have occurred following recent upgrades to the database infrastructure and operating systems. In the meantime, we have further hardened the cluster configuration to prevent unintended failovers and interruptions.

Databases on LUNA were unavailable earlier this hour as the high availability service initiated an unexpected failover and got caught in a failover loop. Database servers were upgraded this weekend to an updated version, and we're investigating what role that may be playing. We are disabling the automatic failover temporarily as a protective measure as we investigate further.

There have been no further issues. We will continue to operate with manual failovers for a bit longer as we validate the efficacy of the automatic failovers. We will also continue to monitor for any other conditions that may result from the database server upgrades.

We are investigating the intermittent availability of certain databases in the US region. Some databases are still recovering following a failover to secondary infrastructure and may be unavailable during the completion of the failover. We will provide updates shortly.

All affected databases recovered approximately a half-hour ago. We're continuing to address some internal connectivity issues, but we're not seeing any observable impacts at this time. There may be brief connection interruptions as failbacks complete later.

We are continuing to monitor for any further issues.

There have been no continued impacts. The incident, which primarily affected databases on the LIMA cluster, was caused by an automated failover to secondary infrastructure. A routine servicing update to the third-party database engine introduced a behavioral change that, under specific conditions, resulted in an exhaustion of worker threads. Upon failing over, some databases were slower to recover as a result of a related exhaustion of worker threads on this secondary node. We have disabled the functionality that resulted in this behavioral change, as advised by the vendor, and have increased the ceiling for worker threads to reduce the potential for reoccurrence. Everything remains stable at this time, and we will continue to monitor for any further issues.

We are currently investigating an intermittent issue that has affected the processing of certain exports and imports. We believe that this is due to an issue with an updated AWS SDK that interacts with a background storage layer. We have temporarily reverted to the previous version of this third-party tool. Imports that previously failed are being reprocessed now.

We have completed our review of the incident that affected the processing of certain exports and imports on 12/5. Slate uses a cluster of worker nodes to process background jobs. These nodes are periodically refreshed with the latest version of Slate, and on 12/5 they were updated to a build that included a newer version of a third-party AWS SDK. Worker nodes employ a shadow-copying mechanism that creates local cached copies of binary dependencies to avoid locking files on shared storage. This mechanism refreshes cached assemblies only when it detects a change in the assembly’s identity, which typically occurs during breaking or versioned updates. The updated AWS SDK introduced an internal structural change: a property was moved from a derived class to its base class. Although no public API signature changed, the compiled IL binding did. Because the assembly’s identity did not change, the shadow-copy cache was not refreshed. As a result, the worker nodes attempted to execute IL referencing the property on the derived class against the older cached version, causing a runtime “method not found” error. On the afternoon of 12/5, we reverted the worker nodes to the earlier AWS SDK version, restoring normal operation. We have since implemented enhancements to our worker-node architecture to provide stronger cache isolation and to ensure that updates to the AWS SDK and other dependencies are correctly recognized, even when internal changes occur without a corresponding change in assembly identity. Web nodes use a different shadow-copying mechanism and were not affected.

At approximately 3:08pm Eastern Time, connectivity to databases hosted on the LUNA cluster was interrupted as a result of a database server stack dump. While connectivity usually is restored within a matter of seconds, some databases on the LUNA cluster did not see an automatic restoration of connectivity. A manual failover was therefore initiated, and by 3:38pm Eastern Time, connectivity was restored to all databases on the LUNA cluster. Databases on other clusters were not affected.

Earlier this morning, at approximately 2:51am Eastern Time, AWS experienced a service interruption which impacted certain specific components of Slate. While most Slate services remained online, the Slate Reader, PDF generation, and other services using the AWS DynamoDB database experienced elevated error rates. We observed full resolution of the AWS event by approximately 5:19am Eastern Time.

Earlier this morning, between approximately 8am and 10am Eastern Time, the HTML message bodies on a subset of scheduled mailings may have appeared blank when sending. A configuration update applied to background worker nodes temporarily resulted in some of these HTML messages failing to render at send time. All background worker nodes are rendering all HTML message bodies properly at this time. No transactional messages (e.g., form or event confirmation messages) were affected but, due to the specific and isolated nature of this issue, we are unfortunately unable to determine which specific HTML bodies may have failed to render properly at send time. We have put in place additional safeguards in order to help prevent this type of an issue from re-occurring during configuration updates in the future.

We're observing increased latency on the ARES cluster and will initiate a service failover. Connections to databases on the ARES cluster may be briefly affected as they're rerouted.

The failover completed successfully and services are operating within normal parameters at this time. We'll continue to monitor this cluster to ensure continued performance within normal operating parameters.

No further instances of latency have been observed on the ARES cluster. As a precautionary measure, we'll reset any other infrastructure that has recently experienced similar conditions which may have contributed to the elevated latency observed here.

On Wednesday afternoon, beginning around 3:40pm Eastern Time, a routine load balancer re-synchronization resulted in an unbalanced traffic distribution that increased the latency of some connections and returned a TCP RST status for some others, while still successfully processing and handling yet other connections depending upon their original node affinity. We have determined that this unbalanced traffic distribution was the result of an issue in the load balancers that reset node statuses during a re-synchronization. We have resolved the issue with the load balancers to prevent this from reoccurring. By 4:00pm Eastern Time, traffic had been fully redistributed across nodes and connection performance had returned to expected levels.

An AWS-initiated overnight update to a series of Redis servers used for background service queue management has delayed background queue processing as the queues entered a stalled state. Delayed queues are processing now and will complete shortly.

As of earlier this afternoon, fewer than 1% of databases had ongoing overnight jobs still executing. The remaining databases have jobs that typically take hours overnight to complete, and with the accumulation of daytime jobs and blocking processes on their databases, are continuing to run. Separately, we have adjusted the background process monitoring to better recover from a scenario in which the Redis servers become temporarily unavailable due to hardware or software updates to those nodes.

No issues have been observed since the overnight background service queue delays on 2/6, and all overnight queues on 2/7 performed as expected. We will continue to monitor background services to ensure ongoing operation within desired ranges.

AWS Lambda, which is used for certain Slate functionality including document conversion and PDF rendering, is currently experiencing an availability incident for their entire us-east-1 region. AWS is aware of the issue and is working towards a full resolution, which is expected soon. Slate functionality that does not rely upon AWS Lambda remains online and operational at this time. We will update this post as more information is made available.

AWS has reported that they have resolved the incident affecting AWS Lambda services in the us-east-1 region, and we have observed normal operation for the affected components since approximately 4:37pm Eastern Time. This incident is now resolved.

We are currently investigating this issue.

Normal operations have been restored to databases on the ARES cluster. We continue to investigate the underlying cause of the earlier interruption and will provide an update in the near future.

Earlier this afternoon, at approximately 1:30pm Eastern Time, the ARES database cluster lost connectivity to its disks as a result of an intermittent network interruption in the AWS infrastructure. This resulted in databases on this cluster being unavailable while network connectivity was restored. All databases are operating within normal ranges at this time, and we are continuing to monitor the ARES cluster. Due to the ARES cluster hosting a primary configuration database for Slate, this event may have also impacted some operations with databases on other clusters. These other clusters already contain replicas of the configuration data, and we will be accelerating our scheduled plan to utilize these replicas more fully to minimize any potential for future impact.

This incident has been resolved.

We are investigating increased latency for databases on the LUNA cluster, which appears to be the result of increased disk latency in AWS for this particular node.

Earlier today, elevated latency was observed for databases on the LUNA cluster. This appears to have been the result of a temporary hardware/network issue within the AWS infrastructure that resulted in increased latency for transaction log writes to the storage infrastructure. Performance levels have been within normal ranges for the past hour, and we will continue to monitor the storage volumes. Should increased latency be observed again, we will transition the cluster to new hardware with AWS.

No further latency has been observed since the original event. Should increased latency be observed again, we will reopen this incident and take the additional measures described to move to new hardware.

Our systems monitoring has detected that connectivity to databases on the LUNA cluster was just interrupted by a localized outage in the AWS network. We are preparing to failover to secondary infrastructure to restore connectivity for these databases.

A localized outage on the AWS network resulted in a temporary connectivity loss to databases on the LUNA cluster. All databases have fully recovered at this time, and any residual latency or timeouts will quickly disappear as key database tables are read back into the cache.

We are aware of an issue that is preventing the return of the normal system configuration on multiple databases. Remediation is in progress and we expect to have all databases back to normal operation within the next few minutes.

System configurations are returning normal values on all production databases at this time, and these systems have recovered. System configurations for test environments are currently being updated. Environments remain accessible during this refresh.

On Friday afternoon at approximately 4:20pm Eastern Time, an engineer applied a system configuration update too broadly. This configuration update was intended to update a single configuration value across databases. Due to human error, the value was applied to other unintended configuration values. While Slate databases remained online and operational for end-users and staff that may have already been logged in, new staff and single sign-on logins may have received a "Invalid Security Provider" error message due to this invalid configuration. Login issues were first reported around 4:40pm, as configuration caches expired and more databases picked up incorrect system configurations, and by 5:15pm, remediations were underway to revert the configuration changes. By 5:30pm, databases were restored to the correct and expected configurations. We deeply regret this error, and we have begun an internal audit and review of our protocols to prevent this from happening and to ensure that other such routine configuration updates are always appropriately scoped.

We are currently investigating degraded performance for databases on the OPUS cluster.

Following the initial post, we throttled back some background jobs (rule execution and scheduled jobs) and have observed no further performance issues for the past two hours. We are slowly increasing the throughput of these background jobs to observe the effect on system performance. We do not have an identified root cause at this time and are exploring the possibility of an issue with underlying hardware or infrastructure.

No further performance issues have been observed, so we will proceed with resolving this incident. We will continue to monitor these systems and should any performance issues re-occur, we will proceed with additional mitigations we have prepared for an overnight window.

AWS is experiencing performance issues with their us-east-1 infrastructure today that is causing increased latency to parts of Slate. While Slate remains online and operational at this time, their issue, which is affecting all of their customers in this region, is creating some intermittent increased latency and error rates in certain areas of Slate, such as in the Slate Reader while displaying documents. Further updates are available at status.aws.amazon.com. We will continue to monitor their status page for any updates.

AWS is reporting that the performance issues in their us-east-1 region have now been resolved. While we are still detecting a degree of network latency, it appears that all services are operating successfully at this time.

At 11:26am Eastern Time, a stack dump occurred on the ARES database server, which is a rare occurrence and the result of the database server encountering a set of unstable conditions. Databases on ARES may have been intermittently inaccessible while the server recovered. As of approximately 11:42am Eastern Time, all databases were accessible and operating under normal conditions. We are currently investigating the source of this stack dump. No other database servers or databases were affected during this event.

We have been able to confirm that the stack dump was the result of a likely bug in the Microsoft SQL Server software that was triggered by unknown conditions. These events are exceptionally rare, and while it is unlikely this particular set of conditions would be encountered again any time soon, we will be preparing a bug report to share with Microsoft in the hopes that it may be resolved in a future update. We will continue to monitor databases on ARES to ensure that there are no reoccurrences.

Databases may experience brief and intermittent periods of unavailability while cluster failovers complete to allow for third-party security and reliability upgrades to occur on the host machines.

Maintenance has concluded for all database clusters with the exception of ARES, which will be handled as part of a separate maintenance process due to special considerations with that specific cluster.

Maintenance has concluded for all database clusters.