35 Splunk incidents · 2023年6月 — official updates, affected components, duration and resolution details.
斯普伦克正在调查目前影响斯普伦克基地的"斯普伦克基地. splun克.com"的问题. 因此,用户可能无法访问斯普伦克基地并可能收到"500"出错. 用户也可能无法通过UI下载或安装Sprunk Cloud中的应用程序. 我们知道这对你们的行动可能产生的影响,并正在努力解决这一问题。 我们将在1小时后再次提供最新情况,如果获得更多信息,则更早提供.
我们正在继续调查这一问题.
斯普伦克在splunkbase.splunk.com解决了影响斯普伦克基地的问题. 用户现在应该能够按预期通过UI访问Sprunkbase,下载应用程序,并安装Sprunk Cloud中的应用程序. 我们为干扰道歉 并感谢你的耐心 当我们努力恢复服务.
自动翻译自官方事件更新。
As part of a recent maintenance activity, some environments with ITSI may experience a rolling restart of the environment. Customers may temporarily see restart notifications or reload prompts within the UI while these activities complete.
Update: Rolling restart activities may be occurring at midnight local time for the applicable cloud provider region for some environments with ITSI. Customers may continue to observe restart notifications or reload prompts within the UI while these activities complete. Customer environments that may see restart notifications include: Splunk Cloud Platform Splunk Cloud Platform for FedRAMP Moderate Splunk Cloud Platform for FedRAMP High Splunk Cloud Platform for IL5 No customer action is required.
We have identified the cause of the rolling restart activity affecting some environments with ITSI and continue to monitor impacted environments. Customers may continue to observe restart notifications, reload prompts, or intermittent search disruptions while rolling restart activities complete. To provide a more controlled customer experience, remaining restart activities are being coordinated to occur at midnight local time for the applicable cloud provider region. Customer environments that may see restart notifications include: Splunk Cloud Platform Splunk Cloud Platform for FedRAMP Moderate Splunk Cloud Platform for FedRAMP High Splunk Cloud Platform for IL5 No customer action is required.
We continue to monitor the rolling restart activities affecting some environments with ITSI. Communications have been sent to customers whose environments may experience potential impact. Customers may continue to observe restart notifications, reload prompts, or intermittent search disruptions while rolling restart activities complete. Remaining restart activities continue to be coordinated for midnight local time in the applicable cloud provider region to provide a more controlled customer experience. Customer environments that may see restart notifications include: Splunk Cloud Platform Splunk Cloud Platform for FedRAMP Moderate Splunk Cloud Platform for FedRAMP High Splunk Cloud Platform for IL5 No customer action is required.
This incident has been resolved.
Some customers are experiencing ingestion issues due to a DNS synchronization issue affecting HEC dash DNS records using the format http-inputs-${stack}.splunkcloud.com. HEC ingestion using the current dot-format DNS records, http-inputs.${stack}.splunkcloud.com, is functioning correctly. As a mitigation, customers using the dash-format HEC endpoint should update their upstream systems to use the dot-format endpoint: http-inputs.${stack}.splunkcloud.com. Our engineering teams are working to restore the affected DNS records through Zone DNS synchronization. We will provide another update once DNS synchronization has completed or if there is a meaningful change in impact.
Some customers experienced traditional HEC ingestion issues due to a DNS synchronization issue affecting HEC dash DNS records using the format http-inputs-${stack}.splunkcloud.com. HEC ingestion using the current dot-format DNS records, http-inputs.${stack}.splunkcloud.com, is functioning correctly. This issue was resolved for affected customers by 07:00 UTC.
We’re investigating an issue affecting some customers hosted in ap-northeast-1, eu-central-1, and us-east-1. Impacted Splunk Cloud Platform customers in these regions may experience access issues. Our teams are actively working to restore service, and we’ll share updates as more information becomes available
A mitigation has been successfully applied across the fleet. Engineering teams are actively monitoring to confirm service has been fully restored and to ensure no further interruptions occur.
We identified an issue caused by an internal change on May 26, 2026, that impacted ingestion through a PrivateLink. A fix has been identified, released and is currently rolling out. Some customers may continue to experience impact while the fix is being applied. We’ll provide another update once rollout is complete. Our teams are actively working to restore service and will provide another update as soon as more information is available
A fix has been implemented and we are monitoring the results.
The incident has been resolved. Our investigation determined that an internal configuration change began impacting service at 20:34 UTC on May 27, 2026. A fix was developed and rolled out beginning at 23:45 UTC, with full remediation completed by 03:45 UTC on May 28, 2026. We apologise for any inconvenience this caused and appreciate your patience while our teams worked to restore service.
We are investigating an issue indicating one of our suppliers experiencing in us-east-1. As a result, the Splunk Cloud Platform environment may be inaccessible for limited number of customers in the specified region. Our teams are working with the provider to resolve this issue. Your patience is greatly appreciated and we will provide more updates upon resolution.
We are investigating an issue indicating one of our suppliers experiencing outage in us-east-1. As a result, the Splunk Cloud Platform environment may be inaccessible for limited number of customers in the specified region. Our teams are working with the provider to resolve this issue. Your patience is greatly appreciated and we will provide more updates upon resolution.
We continue to investigate an issue indicating one of our suppliers experiencing in us-east-1. As a result, the Splunk Cloud Platform environment may be inaccessible for limited number of customers in the specified region. Our teams are working with the provider to resolve this issue. Your patience is greatly appreciated and we will provide more updates upon resolution.
As teams continue to investigate we will tentatively provide our next update by 05:00 AM UTC, or sooner if new information becomes available.
Investigation is in progress, we will provide our next update by 07:00 AM UTC, or sooner if new information becomes available.
Investigation is in progress, we will provide our next update by 09:00 AM UTC, or sooner if new information becomes available.
Investigation is ongoing, and remediation efforts are in progress. We will provide our next update by 10:00 UTC, or sooner if new information becomes available.
Investigation is ongoing, and remediation efforts are in progress. We will provide our next update by 11:00 UTC, or sooner if new information becomes available.
Investigation remains ongoing, and we continue to make progress toward recovery. Some affected infrastructure has stabilized, but some customers may still experience impact while remediation continues. Based on current recovery efforts, full restoration is expected to take several more hours. We will provide our next update by 13:00 UTC, or sooner if new information becomes available.
Update still investigating and recovery efforts remain in progress. Some impacted environments have shown improvement, but some customers may continue to experience service impact while remediation and validation continue. We will provide our next update by 15:00 UTC, or sooner if new information becomes available.
Update still investigating and recovery efforts remain in progress. Some impacted environments have shown improvement, but some customers may continue to experience service impact while remediation and validation continue. We will provide our next update by 17:00 UTC, or sooner if new information becomes available.
Update still investigating and recovery efforts remain in progress. Some impacted environments have shown improvement, but some customers may continue to experience service impact while remediation and validation continue. We will provide our next update by 19:00 UTC, or sooner if new information becomes available.
We are still investigating and recovery efforts remain in progress. Some impacted environments have shown improvement, but some customers may continue to experience service impact while remediation and validation continue. We will provide our next update by 21:00 UTC, or sooner if new information becomes available.
We are still investigating and recovery efforts remain in progress. Some impacted environments have shown improvement, but some customers may continue to experience service impact while remediation and validation continue. We will provide our next update by 23:00 UTC, or sooner if new information becomes available.
Service restoration is progressing steadily following the implementation of our mitigation efforts. While most environments have recovered, some customers may still experience intermittent impact as we finalize remediation and validation. Our team is actively monitoring all systems to ensure full stability.
Service restoration is progressing steadily following the implementation of our mitigation efforts. While most environments have recovered, some customers may still experience intermittent impact as we finalize remediation and validation. Our team is actively monitoring all systems to ensure full stability.
Recovery continues following mitigation efforts. Most environments have recovered, though some customers may still experience intermittent impact while final remediation and validation are completed. Our team is actively monitoring to confirm full stability.
Recovery continues to progress following our mitigation efforts. Most environments are operating normally, and we’re seeing stable performance across the majority of systems. A small number of customers may still experience intermittent impact as we complete final remediation and validation. Our team remains actively engaged and monitoring closely to confirm full stability.
This issue has been resolved. Teams will now begin their root cause analysis. If you believe that you are still impacted by this issue please contact customer support and reference CINC-66292.
We are currently addressing a major service disruption affecting search performance, which began at 16:34 UTC. Users may experience degraded search performance and intermittent UI errors. Our engineering teams have identified the root cause and are actively working to deploy a resolution. We are monitoring the situation closely and will provide further updates as they become available.
We are observing service recovery for the majority of our customers. Our engineering teams are actively investigating and addressing the remaining isolated cases where service has not yet been fully restored. We will continue to monitor the environment closely until all services are operating at normal capacity.
We have implemented a fix for this issue. During this time we are monitoring the results to confirm the resolution and will continue to provide any additional updates once available.
We are continuing to monitor for any further issues.
We are continuing to monitor the restoration of services for the majority of our impacted customers. Our teams are working diligently to resolve remaining issues for any users who have not yet seen a full return to normal service. We will continue to monitor the environment closely until all services are fully operational.
This incident has been resolved.
We are currently experiencing an internal IT service disruption. As a result all planned maintenance activities scheduled are being paused/canceled and we will provide further updates as more information becomes available.
The issue is currently stable, and teams are continuing to monitor service behaviour closely. We will continue to provide updates until the issue is fully resolved.
This incident has been resolved.
Our engineering team has identified an issue with the KVservice that is currently impacting search performance for ad-hoc, saved, and scheduled searches. Additional symptoms may include, but not limited to: - Inability to save searches within the Enterprise Security (ES) application. - Errors when attempting to update application settings, such as credentials. This is an active investigation. We are working to resolve this as quickly as possible and will provide status updates as more information becomes available. We apologize for any inconvenience this may cause.
Our engineering teams have narrowed the scope of the issue: Any search utilizing the KV Services were failing. This impact extended to the Mission Control and Analyst Queue dashboards. Additionally, we have identified the root cause and are actively mitigating the impact. Once mitigation is complete, we will continue to monitor the environment to ensure service stability for all users. We will continue to provide status updates as more information becomes available. We apologize for the inconvenience and appreciate your patience.
Our engineering team has successfully implemented the fix for the KVservice issue. We have monitored the environment and can confirm that there are no recurring errors. All searches and dashboards within the Enterprise Security (ES) application are now fully operational. If you continue to experience any issues related to this incident, please contact us by submitting a support case. We apologize for the inconvenience this may have caused and appreciate your patience as we worked to restore service.
Cisco is aware of an ongoing cloud service provider outage in the Middle East affecting multiple cloud offers and services. As a result, Login, Search and Ingest may be impacted for customers in the affected region. Cisco and Splunk are exploring what options are available for affected customers and will contact those affected customers directly. We will continue to closely monitor this issue as it unfolds.
Cisco has confirmed that the impact of the cloud service provider outage in the Middle East has been mitigated. Cisco engineers have successfully transitioned affected services out of the region. We have identified all affected customers and will be reaching out to each of you individually to discuss specific next steps and any necessary remediation. A proactive support case has been opened on your behalf to coordinate this effort. If you have any immediate concerns, please reach out to your account team or simply reply to your support case. Cisco appreciates your patience and cooperation on this matter.
We are currently experiencing a KVService outage impacting services. Our teams are actively working to resolve the issue. We will provide an update within the next 60 minutes.
We are currently experiencing a global outage impacting multiple Splunk services including Vault, KVService, Identity and Access Control (IAC), Splunk Mobile app, VictorOps, and Ingest processor, Edge processor. Customers may experience service disruptions such as login failures, lookup access issues, and degraded performance. We acknowledge the impact and are actively working with our partners to restore full service.
We are still investigating the issue. Customers may experience service disruptions such as login failures, lookup access issues, and degraded performance. We acknowledge the impact and are actively working with our partners to restore full service.
Splunk continues to monitor and remediate the services affected by this issue. Several services remain impacted; however, they are currently in a recovering state. We do not have an ETA for full recovery at this time, but our teams are working to restore services as quickly as possible. We appreciate your continued patience.
Several Splunk services are still in a recovering state. Our teams continue to monitor progress and work through remaining backlogs to fully restore functionality. We will provide further updates as recovery advances.
All impacted Splunk services are back up, with a few still in recovery and monitoring. Our teams continue to work through remaining backlogs and validate full functionality across all components.
All impacted Splunk services are back up, with Ingest Processor still in recovery and monitoring. Our teams continue to work through remaining backlogs and validate full functionality across all components.
All impacted services have been fully remediated. Systems are operating normally, and no further impact is expected. We will continue to monitor to ensure full service stability. Thank you for your patience throughout this incident.
We are currently investigating the issue with ACS (Admin Config Service) requests causing failures for all Splunk Cloud Platform customers, resulting in the error message "service temporarily unavailable, please try again later" when accessing self-service operations.
Our teams are currently investigating the issue with ACS (Admin Config Service) requests causing failures for all Splunk Cloud Platform customers, resulting in the error message "service temporarily unavailable, please try again later" when accessing self-service operations.
This issue has been identified and fix is being implemented.
The issue affecting ACS has been identified. Our team is currently testing a potential fix to ensure it resolves the problem. Service functionality may remain limited during this time. We will provide updates as testing progresses.
The issue affecting ACS has been identified. Our team is continuing testing a potential fix to ensure it resolves the problem. Service functionality may remain limited during this time. We will provide updates as testing progresses.
We have identified the root cause of the earlier disruption and applied a workaround to restore full functionality. All ACS services are now operating normally, and our monitoring indicates requests are being processed successfully. We will implement a permanent fix during a scheduled maintenance window, and no customer impact is expected during that time. Thank you for your patience as we worked to resolve this issue.
All ACS services are now operating normally, and our monitoring indicates requests are being processed successfully. We will implement a permanent fix during a scheduled maintenance window, and no customer impact is expected during that time. Thank you for your patience as we worked to resolve this issue.
We are aware that some customers are currently experiencing issues logging in to the Splunk Cloud Platform. Our technical teams are actively engaged and investigating the root cause. We are working to resolve this as quickly as possible and will provide further updates as more information becomes available.
We are continuing to investigate this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This issue has been resolved. Teams will now begin their root cause analysis. If you believe that you are still impacted by this issue please contact customer support and reference CINC-63161.
We are investigating a potential issue with KVService, that may impact several Splunk cloud platform customers in eu-central-1. As a result of this issue, customers are experiencing lookups failing to complete. We will provide an update within the next 30 minutes.
We continue to investigate the issue with KVService, that may impact several Splunk cloud platform customers in eu-central-1. As a result of this issue, customers are experiencing lookups failing to complete as well as errors within Splunk applications that depend heavily on KVService.
We are continuing to investigate this issue.
The issue has been identified and a temporary fix is implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved after applying the fix.
We are investigating an issue indicating an AWS outage in us-east-1. As a result, the Splunk Cloud Platform environment may be inaccessible for customers in the specified region. Our teams are working with the provider to resolve this issue. Your patience is greatly appreciated and we will provide more updates upon resolution.
We are continuing to work with service provider on a fix for this issue.
We are aware of recent issues impacting our services due to an AWS incident in the US-EAST-1 region. AWS has applied mitigations and is making progress toward full resolution. At this time, our systems are stable, and we are closely monitoring for any further impact. We will continue to monitor the situation and provide another update by 7:00 AM PDT or sooner if new information becomes available.
We have confirmed that multiple AWS services experienced network connectivity issues in the US-EAST-1 Region. We are seeing early signs of recovery for these connectivity issues and AWS is continuing to investigate the root cause. AWS has applied mitigations and is making progress toward full resolution. Thank you for your patience.
We are aware of an outage impacting multiple AWS services in the US-EAST-1 region. This is affecting several Splunk Cloud Platform environments that rely on these AWS services. Splunk users may experience delays or errors in search execution, data ingestion, or app functionality for stacks hosted in the affected AWS region (US-EAST-1). According to AWS, they have narrowed down the source of the network connectivity issues to an internal subsystem responsible for monitoring the health of their network load balancers. AWS has implemented throttling for new EC2 instance launches to aid recovery and is actively applying mitigations. https://health.aws.amazon.com/health/status?path=open-issues Splunk is continuing to monitor the situation closely and will provide updates as AWS progresses toward full service restoration.
We continue to track the ongoing AWS operational issue in the US-EAST-1 (N. Virginia) region that is causing intermittent service degradation across multiple AWS services. This outage continues to affect several Splunk Cloud Platform environments hosted in the impacted region. Customers hosted in US-EAST-1 may continue to experience latency, delayed search results, or reduced availability of certain Cloud functions. All other Splunk Cloud regions remain fully operational. https://health.aws.amazon.com/health/status?path=open-issues Splunk is continuing to monitor the situation closely and will provide updates as AWS progresses toward full service restoration.
Splunk is actively monitoring an ongoing operational issue in the AWS US-EAST-1 (N. Virginia) region that is causing intermittent service degradation across multiple AWS services and impacting several Splunk Cloud Platform environments. While steady recovery is being observed across Splunk's cloud infrastructure, customers hosted in US-EAST-1 may still experience latency, delayed search results, or reduced availability of certain Cloud functions until AWS infrastructure fully recovers. All other Splunk Cloud regions remain fully operational. Splunk will continue to provide updates as AWS progresses toward full service restoration.
As AWS continues its recovery efforts in the US-EAST-1 region, Splunk Cloud Platform customers in this region may still experience intermittent issues. We are closely monitoring AWS updates and will provide additional status information as it becomes available. All other Splunk Cloud regions remain fully operational.
As AWS continues its recovery efforts in the US-EAST-1 region, Splunk Cloud Platform customers in this region may still experience intermittent issues. We are closely monitoring AWS updates and will provide additional status information as it becomes available. All other Splunk Cloud regions remain fully operational.
We are continuing to monitor for any further issues.
AWS has reported that the earlier issues impacting services in the US-EAST-1 (N. Virginia) region have been fully resolved as of 3:01 PM PDT. All impacted services have returned to normal operations. Users may continue to observe minor latency or delays as AWS and dependent services complete backlog processing and return to standard performance levels. We are closely monitoring our systems and will provide further updates if any new issues arise. Thank you for your patience.
AWS has reported that the earlier issues impacting services in the US-EAST-1 (N. Virginia) region have been fully resolved as of 3:01 PM PDT. All impacted services have returned to normal operations. Users may continue to observe minor latency or delays as AWS and dependent services complete backlog processing and return to standard performance levels. We understand the criticality of your services and appreciate your patience.
We have identified an issue with the Splunk Mobile app not being able to connect to Splunk Secure Gateway. This issue ONLY affects the mobile apps, and not customer's Splunk Enterprise/Splunk Cloud instances. Your patience is greatly appreciated and we will provide more updates as we investigate and work on a fix for the issue.
The issue has been identified and a fix is being implemented. Please note that the issue ONLY affects the Splunk mobile app, and not customer's Splunk Enterprise/Splunk Cloud instances.
A fix has been identified and is being currently implemented. Please note that the issue ONLY affects the Splunk mobile app, and not customer's Splunk Enterprise/Splunk Cloud instances.
A software fix has been provided for release to consumers. iOS version 2.44.0 is ready and available for customer download via the App Store. Android version 2.43.1 is awaiting review and will be released in the upcoming days. Affected customers should update to the fixed app version as soon as it is available. Please note that the issue ONLY affects the Splunk mobile app, not customers' Splunk Enterprise/Splunk Cloud instances.
We are investigating a potential issue with search and ingest that may impact customers on Splunk Cloud Platform for FedRAMP High and Splunk Cloud Platform for IL5. As a result of the issue, customer search and indexing may experience issues completing successfully. We will provide an update within the next 30 minutes.
We have identified the source of the issue and are currently working on implementing a fix.
We are starting to see recovery on some of the impacted instances. Teams continue to work on implementing a fix for this issue.
We see recovery on some of the impacted instances. Teams continue to work on implementing a fix for this issue.
We continue working to implement a fix for this issue.
A fix has been implemented and we are monitoring the results.
We are continuing to monitor for any further issues.
This incident has been resolved.
We are aware of a service provider outage on GitHub that lasted from 12:42 PM PT to 1:22 PM PT. As a result, changes such as firewall rule modifications or updates to the IP allow list made via the Admin Config Service (ACS) may have failed during this time. Further details may be found at https://www.githubstatus.com This issue has been resolved by the provider. If you believe that you are still impacted by this issue please contact customer support and reference CINC-61287.
We are investigating a reported issue for all stacks hosted on GCP. As a result, the Splunk Cloud Platform environment may be inaccessible for customers. Please refer to the current live updates from GCP here - https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW Our teams are monitoring the status of the outage. Your patience is greatly appreciated and we will provide more updates upon resolution.
We are continuing to work on a fix for this issue.
A fix has been implemented. We are monitoring to confirm full service recovery. Additional details can be found here: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW
This incident has been resolved.
Splunk is aware of an issue currently affecting SOAR customers. As a result of the issue, users will not be able to save playbooks after making edits. Error message: "Unexpected keyword argument 'code_names' in function call" We understand the impact this has on your operations and are working to resolve the issue as quickly as possible. We will provide more information when it becomes available. No actions are needed from customers at this time. Thank you for your patience and understanding.
We have implemented a fix for this issue. During this time we are monitoring the results to confirm the resolution and will continue to provide any additional updates once available.
This issue has been resolved. Teams will now begin their root cause analysis. If you believe that you are still impacted by this issue please contact customer support and reference .
Hello, Splunk Cloud customers using the SOAR platform may be impacted by Playbooks appearing to load but backend logic failing. We have identified the source of the issue in version 6.4.1 and are currently working on implementing a fix. At this time, there is no further action needed from customers. There is an active workaround to mitigate impact: - Open the playbook in the Visual Playbook Editor (VPE). - Manually review and adjust the join settings (check/uncheck as needed). - Save the playbook to apply the changes Your patience is greatly appreciated and we will provide more updates as we implement the fix.
Splunk has identified an issue in SOAR version 6.4.1 where Playbooks may appear to load correctly but failing in backend execution. A fix is in progress, and no immediate action is required from customers. As a workaround, to mitigate the impact: - Open the playbook in the Visual Playbook Editor (VPE) - Manually review and adjust the join settings (check/uncheck as needed) - Save the playbook to apply the changes We appreciate your patience and will share further updates as the fix is implemented.
This fix will be deployed to all cloud stacks on Thursday, June 5, 2025, between 1:00 AM and 3:00 AM local stack time. On-premises customers will receive the updated release for download on June 5, 2025 (PDT). If you wish to apply the workaround before the scheduled fix is deployed, you can do so by following the steps below: - Open the affected playbook in the Visual Playbook Editor (VPE). - Manually review and adjust the join settings (check/uncheck as needed). - Save the playbook to apply the changes. Teams will now begin their root cause analysis. If you believe that you are still impacted by this issue please contact customer support and reference CINC-61075.
