58 Starrez incidents · 2023年1月 — official updates, affected components, duration and resolution details.
澳大利亚东南部的客户目前遭遇服务中断 - 工程师们正在积极致力于补救这个问题。 - 下一次更新预计在60分钟内,或者根据事件变化的需要。 对不起,任何不便, 星报团队
这一问题已经确定,Star Rez工程师公司正在积极为这一问题制定缓解措施。 下一次更新预计在60分钟之内进行,或者根据事件变化的需要。 对不起,任何不便, 星报团队
一项固定措施已经执行,我们看到该地区恢复了稳定。 StarRez工程师公司将密切监测,确认恢复正常运行 下一次更新是因事件变化而有必要的。 对不起,任何不便, 星报团队
一项固定措施已经执行,我们看到该地区恢复了稳定。 StarRez工程师公司将密切监测,确认恢复正常运行 下一次更新是因事件变化而有必要的。 对不起,任何不便, 星报团队
这一事件已经得到解决.
自动翻译自官方事件更新。
THD客户使用SFTP服务器数据-03.datacenter.adirondocksolutions.com 正在经历服务中断. 工程师们正积极致力于补救这个问题。 下一次更新将在60分钟内完成,或者根据事件变化的需要.
工程师们正积极致力于补救这个问题。 下一次更新预计在60分钟之内进行,或者根据事件变化的需要.
这个问题已经查明,工程师正在补救。 下一次更新将在60分钟内完成,或者根据事件变化的需要.
一项措施已经执行,我们正在监测结果.
这一事件已经得到解决.
自动翻译自官方事件更新。
We're currently seeing a partial outage with outbound email for some customers. This is due to a heat-related incident at one of our upstream infrastructure providers, and their team is actively working on it. Who's affected: Customers using THD-hosted email servers may notice emails being delayed or failing to send. Who's not affected: Customers using their own email infrastructure or AWS SES — your email is sending normally. Next update expected within 60 minutes, or as warranted by a change of events.
Our upstream provider is continuing to investigate the issue.
The issue impacting the upstream provider has been identified, and a solution is currently being implemented. Next update expected within 60 minutes, or as warranted by a change of events.
While our upstream provider continues working to remediate the underlying issue, our engineers are taking parallel action to restore email delivery. We are actively migrating affected customers to SES to reduce the impact of the disruption and restore email processing as quickly as possible. Next update expected within 60 minutes, or as warranted by a change of events.
While our upstream provider continues working to remediate the underlying issue, our engineers are taking parallel action to restore email delivery. We are continuing to actively migrate affected customers to SES to reduce the impact of the disruption and restore email processing as quickly as possible. Next update expected within 60 minutes, or as warranted by a change of events.
Our upstream provider continues working to remediate the underlying issue, our engineers are taking parallel action to restore email delivery. We are actively migrating affected customers to SES to reduce the impact of the disruption and restore normal email processing as quickly as possible. While the primary impact of the upstream provider issue remains focused on email delivery, customers may also experience broader service degradation, including intermittent slowness or temporary disruptions. Our teams continue to actively monitor the environment and work to minimize customer impact.
Our upstream provider has restored services and we're now seeing the email backlog being processed. Engineers are currently reviewing to confirm email functionality is working as expected for impacted customers. Next update expected within 60 minutes, or as warranted by a change of events.
We are continuing to monitor for any further issues.
This incident has been resolved.
A subset of customers in Australia East are currently experiencing a service disruption -Engineers are actively working to remediate the issue. -Next update expected within 60 minutes, or as warranted by a change of events. Apologies for any inconvenience, StarRez Team
The issue has been identified, and sites are coming back online -Next update expected within 60 minutes, or as warranted by a change of events. Apologies for any inconvenience, StarRez Team
Our upstream vendor logs have confirmed hardware failure as the cause of our issue however outages are ongoing as hardware recovers. We have escalated the situation with our vendor. -Engineers are actively working to remediate the issue. -Next update expected within 60 minutes, or as warranted by a change of events. Apologies for any inconvenience, StarRez Team
We are seeing stability return to the region. Engineers will monitor closely to confirm a return to normal operation -Next update eas warranted by a change of events. Apologies for any inconvenience, StarRez Team
This incident has been resolved.
THD customers on the ND & Baltimore servers are experiencing a service disruption. Engineers are actively working to remediate the issue. The next update is due within 60 minutes, or as warranted by changes in events.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
College Pads customers are experiencing a service disruption -Engineers are actively working to remediate the issue. -Next update expected within 60 minutes, or as warranted by a change of events.
The underlying issues have now been resolved with all customer workloads now back online. Engineers will continue to monitor the situation.
We are continuing to monitor for any further issues.
This incident has been resolved.
美国西部地区的客户目前受到服务中断的影响 - 工程师们正在积极地努力纠正这一问题。 - 下一次更新预计在60分钟内进行,或根据事件变化的需要。 对不起,任何不便, 星报团队
我们已发现一个问题, 我们已同提供方一起使问题升级,以加快补救,并正在积极监测局势。 一旦获得更多信息,将分享进一步的最新情况.
上游服务供应商已证实美国西部地区存在网络基础设施问题。 我们正在积极监测局势。 一旦获得更多信息,将分享进一步的最新情况.
上游服务供应商已证实美国西部地区存在网络基础设施问题。 上游问题尚未解决. 我们正在积极遵循灾后恢复程序,以恢复受灾客户的可用性。 对不起,任何不便, 星报团队
上游服务供应商对美国西部地区的网络基础设施问题采取了补救措施。 随着固定装置的推出,客户站点功能将开始恢复 我们正在积极监测局势。 一旦获得更多信息,将分享进一步的最新情况.
这一事件已经得到解决.
** 概览** 2026年7月23日,StarRez遭遇了影响美国西部地区的StarRez Cloud客户的服务中断. 从协调世界时14:44开始,受影响的客户无法使用StarRez Cloud应用程序。 所有受影响的客户站点于协调世界时2026年7月23日19:24恢复全面运营. ** 循环原因** 中断的原因是上游服务提供商导致全区域网络中断。 这一变化造成受影响的客户环境暂时无法进入若干小时,而上游服务提供商则努力解决联网问题。 ** 决议** StarRez工程师迅速查明了这个问题,并立即采取行动确认停电范围,向上游供应商提出高强度的支持案例. 在与上游供应商确认停电后, 我们的工程师开始执行一项灾后恢复计划, 一旦上游供应商解决了潜在的网络问题,所有客户站点被恢复到他们在美国西部的已配置的云区,并在协调世界时19:24前确认全面运行,大约是在中断开始后4.5小时. ** 下一步** StarRez将继续监测区域基础设施的健康,并同上游提供者协调,以尽量减少未来任何中断的影响.
自动翻译自官方事件更新。
美国中部地区的客户目前出现服务中断 - 工程师们正在积极致力于补救这个问题。 - 下一次更新预计在60分钟内,或者根据事件变化的需要。 对不起,任何不便, 星报团队
这个问题已经确定,工程师正在补救,使网站重新上网。 - 工程师们正在积极致力于补救这个问题。 - 下一次更新预计在60分钟内,或者根据事件变化的需要。 对不起,任何不便, 星报团队
一项措施已经执行,我们正在监测结果.
我们正在继续监测任何其他问题.
这一事件已经得到解决.
** 概览** 2026年6月26日,StarRez遭遇了影响美国中部地区的StarRez Cloud客户的服务中断. StarRez Cloud应用程序从协调世界时05:47开始对受影响的客户不开放。 所有受影响的客户站点于协调世界时2026年6月26日06:06恢复全面运营. ** 循环原因** 中断的原因是对支持美国中部地区的StarRez Cloud的基础设施进行了配置改变。 这一变化导致受影响的客户环境暂时无法使用,而基础设施则趋于稳定。 ** 决议** StarRez工程师迅速找出了这个问题,并立即采取行动使受影响的基础设施恢复健康状态. 在协调世界时06:06,即中断开始后约19分钟,美中地区所有客户站点都确认全面运营. ** 下一步** StarRez正在围绕配置实施更多的护卫,以覆盖资源,防止今后再次发生这类问题.
自动翻译自官方事件更新。
College Pads customers are experiencing a service disruption -Engineers are actively working to remediate the issue. -Next update expected within 60 minutes, or as warranted by a change of events.
The underlying issues have now been resolved with all customer workloads now back online. Engineers will continue to monitor the situation.
This incident has been resolved.
Customers in all regions are currently experiencing a service disruption for AI features within the StarRez product. The features impacted are: - Write with AI - Rez 360 Intelligence - Summaries - Parcels AI scanning ( - Program Agent Beta - Rez 360 Intelligence : Ask Ai beta All other functionality within the StarRez product is not impacted at this time. -Engineers are actively working to remediate the issue. -Next update expected within 60 minutes, or as warranted by a change of events. Apologies for any inconvenience, StarRez Team
The issue has been acknowledged and identified with our upstream vendor who is actively working to restore services. -Engineers are actively working to remediate the issue. -Next update expected within 60 minutes, or as warranted by a change of events. Apologies for any inconvenience, StarRez Team
We are seeing small improvements in completion rates however errors are still occurring, and we expect timeouts and prolonged response delays whilst this incident is ongoing. Our upstream vendor is continuing they're efforts to mitigate and resolve this situation. -Engineers are actively working to remediate the issue. -Next update expected within 60 minutes, or as warranted by a change of events. Apologies for any inconvenience, StarRez Team
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are aware of a service disruption currently. We are investigating the cause and will provide additional updates as possible, or within the next 60 minutes.
We’ve identified an issue with an upstream service provider that is impacting availability for a number of clients. We have escalated the issue with the provider to expedite remediation and are actively monitoring the situation. Further updates will be shared as more information becomes available.
The upstream service provider has confirmed an issue and are in the process of resolving this for our West Central US region. Further updates will be shared as more information becomes available.
The upstream provider is continuing their investigation into this incident. Further updates will be shared as more information becomes available.
The service disruption is still ongoing within this region. StarRez Engineers are actively in the process of executing our Disaster Recovery plan for impacted customers. Customers are starting to come online as this process completes. - Updates within the next hour or as warranted. Apologies for the inconvenience StarRez
The service disruption is still ongoing within this region. StarRez Engineers are actively in the process of executing our Disaster Recovery plan for impacted customers. The upstream provider is continuing their investigation into this incident. Further updates will be shared as more information becomes available. - Updates within the next hour or as warranted. Apologies for the inconvenience StarRez
The service disruption is still ongoing within this region. StarRez Engineers are actively in the process of executing our Disaster Recovery plan for impacted customers. The upstream provider has identified the root cause and are working on a fix for the issue. - Updates within the next hour or as warranted. Apologies for the inconvenience StarRez
The upstream provider has completed the fix and we are seeing recovery across all impacted sites. StarRez Engineers are continuing to monitor the situation. - Updates within the next hour or as warranted. Apologies for the inconvenience StarRez
The issue with the upstream service provider has been resolved. All sites are restored to full function. Sites that had to undergo Disaster Recovery steps will have these finalized and restored to normal during the relevant maintenance windows for those sites. Apologies for the inconvenience StarRez
## StarRez Root Cause Analysis ### Overview On 11 May 2026, StarRez experienced a service disruption affecting StarRez Cloud customers in the West Central US region. StarRez Cloud applications were unavailable for affected customers beginning at approximately 15:56 UTC. All affected customer sites were restored to full operation by 03:03 UTC on 12 May 2026. ### Root Cause The disruption was caused by hardware failures at our upstream service provider's West Central US datacenter. The failures caused the database services underpinning StarRez Cloud applications in that region to become unavailable, taking a number of customer sites offline. ### Resolution Once confirmed as an upstream infrastructure failure, StarRez engineers initiated disaster recovery for the affected region, restoring affected customer environments in an alternate region. Affected sites were progressively restored beginning at approximately 22:23 UTC, with all sites confirmed fully operational by 03:03 UTC on 12 May 2026. Customers recovered via disaster recovery were moved back to the West Central US region during their scheduled maintenance windows later the same day. ### Next Steps StarRez will continue monitoring regional infrastructure health and coordinate with upstream providers to minimize the impact of any future disruptions.
Customers in the East US region are currently experiencing intermittent service disruption. -Engineers are actively working to remediate the issue. -Next update expected within 60 minutes, or as warranted by a change of events. Apologies for any inconvenience, StarRez Team
The issue has been identified, and engineers are currently working to resolve this issue. Customers will continue experiencing intermittent outages whilst the work continues. Next update expected within 60 minutes, or as warranted by a change of events. Apologies for any inconvenience, StarRez Team
The underlying issue has been mitigated, and customer sites are now coming back online and remaining stable. -Next update expected within 60 minutes, or as warranted by a change of events.
All sites are now back online and stable. We will continue to monitor before resolving. -Next update as warranted by a change of events.
This incident has been resolved.
We are aware of a service disruption currently. We are investigating the cause and will provide additional updates as possible, or within the next 60 minutes.
The team has identified the issue and has implemented a mitigation, with affected sites returning to service. We will continue to provide updates as we have them, or within the next 60 minutes.
The team has applied a mitigation, and all affected sites are in full service. We are continuing to monitor. Next update as soon as we have it, or within the next 60 minutes.
This incident has been resolved.
There was a temporary disruption affecting access to the THD VPN tunnel. Services have since been fully restored, and we are continuing to monitor the situation closely.
This incident has been resolved.
THD customers on the Newport server might experience a service disruption. Engineers are actively working to remediate the issue. The next update is due within 60 minutes, or as warranted by changes in events.
Service has recovered since applying the mitigation. We are continuing to monitor the situation. Next update in 60 minutes or sooner if we have more information.
We are continuing to monitor for any further issues.
This incident has been resolved.
We are investigating service degradation for Cloud Payments in the US Region. Engineers are working on mitigating the issue. Next Update in 60 minutes or sooner if we have more information.
Engineers have applied a mitigation to address the issue and are continuing to review the situation. Next update in 60 minutes or sooner if we have more information.
Service has recovered since applying the mitigation. We are continuing to monitor the situation. Next update in 60 minutes or sooner if we have more information.
This incident has been resolved
We are currently investigating an issue affecting Single Sign-On (SSO) authentication. Users may experience difficulties logging in via SSO. - Engineers are actively working to remediate the issue. - Next update expected within 60 minutes, or as warranted by a change of events.
We are continuing to investigate this issue.
Engineers have remediated the issue. We will continue to monitor and will close this incident out once stability has been confirmed.
This incident has been resolved.
**Overview** On 5 March 2026, following the enablement of Cloudflare for all customers, THD experienced a service disruption that resulted in an authentication loop during Single Sign-On \(SSO\). **Root Cause** The SSO Service Provider configuration did not correctly handle session validation when operating behind the Cloudflare proxy, resulting in authentication sessions being invalidated and causing a login loop. **Resolution** The issue was initially mitigated by reverting the Cloudflare change, restoring normal SSO functionality. Configuration updates were then made to the SSO Service Provider to ensure compatibility with proxied traffic, after which Cloudflare was incrementally re-enabled.
We are aware of an ongoing Microsoft 365 outage that is affecting email delivery to Outlook mailboxes. This is a downstream issue within Microsoft 365 and is separate from StarRez Cloud. As a result, some customers may experience delays or failures in receiving MFA-related emails. We are monitoring the situation closely and will provide updates as more information becomes available. Further details are available via Microsoft’s status page: https://status.cloud.microsoft/ Apologies for any inconvenience caused, StarRez Team
We are aware of an ongoing Microsoft 365 outage that is affecting email delivery to Outlook mailboxes. This is a downstream issue within Microsoft 365 and is separate from StarRez Cloud. As a result, some customers may experience delays or failures in receiving MFA-related emails. We are monitoring the situation closely and will provide updates as more information becomes available. Further details are available via Microsoft’s status page: https://status.cloud.microsoft/ Apologies for any inconvenience caused, StarRez Team
We are continuing to monitor for any further issues.
This incident has been resolved.
We are investigating an issue affecting all regions. Next Update in 60 minutes or sooner if we have more information.
We are aware of an issue affecting access to StarRez Cloud as a result of a global outage of an upstream provider. We are assessing the impact and escalating with the provider as required
The upstream provider has resolved the issue, with all customer workloads back online. Engineers will continue to monitor and will close this incident out once stability has been confirmed.
This incident has been resolved
A subset of customers in the WestUS2 region are currently having a service disruption -Engineers are actively working to remediate the issue. -Next update expected within 60 minutes, or as warranted by a change of events. Apologies for any inconvenience, StarRez Team
The issue has been identified. Services are now coming back online -Next update expected within 60 minutes, or as warranted by a change of events. Apologies for any inconvenience, StarRez Team
The underlying issues have now been resolved with all customer workloads now back online. Engineers will continue to monitor and will close this incident out once stability has been confirmed. Apologies for any inconvenience, StarRez Team
This incident has been resolved.