Sparc Media Hub 장애 기록

At 9:22AM PST, Text Groove had received reports of increased error rates on their platform affecting users in our region. Their engineering team is investigating. Updates will be shared as they are available. Thank you for your patience.

At 10:12AM PST, Text Groove have implemented a fix and are currently monitoring. We are seeing services slowly being restored. Thank you for your continued patience.

This incident has been resolved.

The team have identified an issue where emails are not being sent from the SPARC platform. This is directly linked to a third party vendor issue and the team are working to implementing a fix.

This incident has been resolved. Emails will now send without issue.

The team have identified an issue where emails are not being sent from the SPARC platform. The team are working to implement a fix.

The SPARC team have implemented a fix for this issue. The issue was directly related to an outage with a third party provider. We apologize for the inconvenience. Emails should be sending from the SPARC Platform. Any issues, please reach out to support.

As of 3:48AM PST/6:48 AM EST, SPARC Media Hub's DNS provider have been experiencing a global outage across all their services. Those outages have impacted platform access to SPARC. While we've seen a decrease in the outage errors, we continue to monitor and work with Cloudflare Engineers to restore service fully. We apologize for the inconvenience.

Cloudflare believe a final fix has been implemented and are starting to see full service restored. We will continue to monitor to ensure connectivity to the Platform is maintained.

This incident is considered resolved. A full investigation will be conducted to determine the full cause.

Since 12:11am PST, Amazon Web Services are experience degraded performance of services in our region. We're monitoring the outage with Amazon Engineers and will post updates as they become available. During this time, parts of the platform may become unavailable such as file uploads or text message processing. We apologize for the inconvenience.

Amazon Engineers have resolved this outage.

We're currently investigating an issue where winners may receive a "404 Page Not Found" error when visiting the link. The error seems intermittent and not persistent.

A fix has been implemented and we are monitoring services for waiver links. Links already generated should begin to work as per normal.

This incident has been resolved. All services restored fully as of 3:13pm EST

We're currently investigating an outage with the platform and are working to resolve the issue as soon as possible.

A fix has been implemented and we are currently monitoring as service appears to be restored.

This incident has been resolved.

We are currently investigating an issue that has stopped access to the platform. We are working fast to restore service.

Service has been restored. We are monitoring to ensure services remain up.

This incident has been resolved. We're seeing service fully restored across all servers. We apologize for the inconvenience.

We are currently investigating an error that prevents users from accessing the platform.

A fix has been implemented. The issue has been identified as an issue with our DNS / Security Firewall service. Support teams have escalated the issue internally with the service provider to prevent issue from happening again.

This incident has been resolved.

We are currently experiencing instability with our network. We’re currently investigating and engaging our relevant partners to resolve and increase stability.

We’ve identified the issue as a global networking issue with our server partner. We’re actively working with our partners to fully restore services. We apologize for any inconvenience.

Our partner has notified us that they’ve implemented a fix for the networking issues. We will continue to monitor to ensure services are fully restored.

We’ve classified this issue as being resolved. Any additional issues, please reach out to support.

Our texting partner is currently experience a system wide outage that affects SPARC Media Hub texting services. Text messages are not being delivered. We will actively work with them to restore services as soon as possible. We will provide updates as they are available. Thank you for your patience.

TextGroove has identified the issue to be connected to their Global Security and web-app routing provider. They are in communication with their team and continue to work on restoring services as soon as possible. Thank you for your patience.

Services are now fully restored. This issue is now fully resolved. If you continue to experience any issues, please reach out to support. If we receive additional details from their team, we will publish those details accordingly. Thank you for your patience and sorry for any inconvenience this may have caused.

SPARC has been made aware of an issue that is affecting file uploads within the platform. Our third-party partner who handles all storage are experiencing network issues that may cause users to experience a "400 Bad Request" error when uploading files within SPARC. We've engaged with our partner's engineering team to request a prompt fix and will provide updates as they become available. We apologies for the inconvenience and are working hard to restore services fully.

Our partner have informed us that the networking issues have been resolved.

Texting engineers are currently investigating increased error rates with message delivery. At this time, messaging may experience issues or delays. Thank you for your patience. We will provide further updates as they become available.

Engineers have identified this issue as being related to a system issue currently being experienced by our primary carrier partner. Relevant carrier engineers are investigating. Impact: Messaging outage (inbound & outbound) on Canada Long Code numbers.

Carrier-side engineers continue to investigate this issue. Thank you for your continued patience. Impact: Messaging outage (inbound & outbound) on Canada Long Code numbers.

Carrier-side engineers have identified the root cause of this issue as an issue with their cloud messaging database and have engaged their cloud provider's engineering team to work towards a resolution. This is being treated with the highest priority for all teams involved. Thank you for your continued patience.

Carrier-side engineers have fixed the root cause of this issue and we are seeing messages being sent through successfully. We are going to continue to monitor while the carriers queues are being processed. Thank you for your continued patience.

Texting services have been fully restored. Any issues, please reach out to support.

We've been notified of an issue with our Content Delivery Network. Our partner is currently investigating and we will provide updates as they become available. Some users may experience an error when uploading files to SPARC, specifically Commercials within the Creative Dashboard and FTP Import Feature. We apologize for any inconvenience this may cause.

Our partner has implemented a fix, and should see normal activities with uploading files within SPARC. We will continue to monitor the situation to confirm this incident is fully resolved and will post an update soon.

This incident has been resolved.

Our texting provider has been informed of an issue causing latency on message delivery with some carriers / operators by our primary Canadian Longcode aggregator. At this time we are unaware of the full impact. Relevant carrier teams are investigating. We will post additional information as it becomes available. Thank you for your patience.

We’ve been notified that this issue has been resolved and messages are flowing normally.

Some users may experience some errors when viewing files within SPARC. Users will be presented with a “Your connection is not private” error. The files are stored correctly, and will display correctly after a propagation period passes. We will continue to monitor until the issue has been resolved.

A fix has been implemented and we are monitoring the results.

This issue has been resolved.

We are currently investigating a Platform outage, we will provide an update as soon as one becomes available. We apologize for the inconvience.

The team have implemented a fix for the platform outage. We will continue to monitor the situation and provide updates as they become available.

This issue has been determined to be resolved.

On January 10th, 2023 at approximately 7:45 AM EST, SPARC Media Hub experienced a complete Platform outage which prevented customers from accessing the workflow system. At 7:46 AM EST, SPARC Media Hub deployed a team to immediately start investigating the issue. When customers would visit the platform, customers would see “No route to database”. Which prompted engineers to investigate the connection to our database servers. SPARC Media Hub utilizes an internal private network which technology that significantly improves performance between servers without sacrificing security. At 7:52 AM EST, the team identified the connection to the client application servers and the database servers were not communicating through the private network which caused the errors to appear. The team began troubleshooting steps such as restarting the servers and ensuring all software versions were up-to-date. Upon updating the private network software to the latest version, connection resumed and the platform server was restored at approximately 8:04 AM EST. The total downtown was 19 minutes.

Our testing provider is aware of an issue affecting a small subset of customers. Some inbound messages may be queued or delayed. Engineers are investigating and will provide an update as soon as possible.

Our texting provider engineers have resolved this issue and queues have cleared. Messages are now delivering in real time. Thank you for your patience.

Our Engineering team has identified an issue with networking. At this time, users may be experiencing site timeouts, and/or slowed website performance. Our team is working with our networking partner to mitigate the impact and we will share an update once we have more information.

Our networking partner have implemented a fix for this incident, and believe this incident is resolved.

Our texting provider is aware of a carrier-side issue affecting message delivery on Canadian Longcode number. Relevant carrier teams have been engaged and are investigating. Thank you for your patience.

Carrier teams have identified the issue and have implemented a fix. Message queues are clearing at this time.

We are seeing messaging traffic in real time and are considering this issue now resolved. Thank you for your patience.