Störungsverlauf von Telerivet

Telerivet's servers are currently unable to send email due to an issue with our transactional email service provider.

Telerivet has implemented a workaround and emails are being sent now.

Outgoing emails were delayed between 01:08 and 01:55 UTC due to a DNS change made by Telerivet's transactional email service provider (SendGrid), which was incompatible with a configuration setting on Telerivet's servers. This issue has been resolved by updating a configuration setting on Telerivet's servers.

Telerivet is currently experiencing delays updating search indexes for contacts and messages. After adding or updating contacts, the updated contacts may not appear in the Telerivet web app immediately. Searching for messages may not return new results.

Telerivet has implented an update to prioritize indexing of contacts. New and updated contacts are now appearing in search without a significant delay.

Telerivet has deployed a fix to improve performance of indexing new messages for search. New messages are currently being indexed without a delay.

Telerivet is currently investigating an issue with database connectivity.

We are continuing to investigate this issue.

The issue has been identified as a particular query causing Telerivet's database server to crash. Telerivet has disabled scheduled message functionality to avoid this issue and is continuing to investigate the issue.

We are continuing to work on a fix for this issue.

The issue was caused by a corrupted database table storing scheduled messages and other scheduled events, which caused the database server to crash when triggering scheduled events. The corrupted database table has been repaired, and scheduled messages are being sent now. All Telerivet functionality should be working normally.

Telerivet has not detected any further issues with database connectivity or data integrity, and has verified that no scheduled messages or other data were lost due to this issue.

Telerivet is currently experiencing intermittent issues with database connectivity which could impact message delivery and service processing. We are currently investigating this issue.

The issue with intermittent database connectivity appears to have been triggered by a software update. Telerivet has reverted this update and is continuing to investigate the issue.

Telerivet has not encountered any issues with database connectivity since reverting the software update yesterday. All systems are operating normally.

Messages sent via Nexmo routes are currently encountering errors due to an ongoing issue with the Nexmo (Vonage) service. For more information, see https://www.nexmostatus.com/incidents/6p4nlld7n4br . Non-Nexmo routes are operating normally.

Nexmo has reported that all services have recovered. For more information, see https://www.nexmostatus.com/incidents/6p4nlld7n4br .

Messages sent via Nexmo routes are currently encountering errors due to an ongoing issue with the Nexmo service. For more information, see https://www.nexmostatus.com/incidents/ykf63np3bq89 . Non-Nexmo routes are operating normally.

Nexmo has reported that all services have recovered. For more information, see https://www.nexmostatus.com/incidents/ykf63np3bq89 .

Users with routes provided by Nexmo have been reporting that messages are currently not being delivered with the error "Unroutable". Telerivet is currently working with Nexmo to resolve this issue.

SMS messages sent via Nexmo are being delivered again now. The issue was limited to clients using Telerivet's Nexmo account to send SMS messages in the United States and Canada. Clients using their own Nexmo account were unaffected. Nexmo disabled Telerivet's SMS routes in the US and Canada at 7:25 AM PDT, without notifying Telerivet. Nexmo restored these routes at 2:43 PM PDT. Telerivet is continuing to communicate with Nexmo to determine the reason why Nexmo disabled Telerivet's SMS routes in the US and Canada, and to prevent this issue from reoccurring.

On October 17, one of Nexmo's US partners identified un-permitted content in outgoing SMS and advised Nexmo to stop that traffic immediately. As a result, Nexmo temporarily blocked all messages sent from that route, which included all messages sent by users with virtual numbers in the US and Canada under Telerivet's Nexmo account. Nexmo has not indicated whether any Telerivet user was responsible for sending un-permitted content. Nexmo has reported that they are planning to improve their internal processes to avoid a situation like this from happening again. Telerivet has already been updated to require manual approval of each user who requests to send messages via Telerivet's Nexmo account, to reduce the risk of Telerivet users sending un-permitted content. Telerivet has already credited affected users with the cost of messages that were not delivered due to this issue, as well as 25% of their monthly service plan fee.

SMS messages sent via Nexmo in the US and Canada may not be received at this time due to an issue with Nexmo. For more information, see https://www.nexmostatus.com/incidents/jzgfpqrh4dqs .

This issue has been resolved by Nexmo, and messages that queued during the incident have been delivered. For more information, see https://www.nexmostatus.com/incidents/jzgfpqrh4dqs .

Users sending SMS via Nexmo have been receiving a "HTTP 504" (Gateway Timeout) error due to an outage with Nexmo. For more information, see https://www.nexmostatus.com/ .

The Nexmo API outage was resolved about 5 hours ago. For more information, see https://www.nexmostatus.com/incidents/qdl3bhsq1cbz

One of Telerivet's API servers experienced errors connecting to other Telerivet services at 00:22 UTC, for about 1 minute, causing some API requests to fail. Telerivet's services are currently operational. We are continuing to monitor for additional issues.

This incident has been resolved.

At 19:40 UTC, one of Telerivet's message queue servers failed, causing errors when sending messages via the Telerivet web app and REST API. Telerivet's automated failover systems resolved the issue within 45 seconds, and all systems are currently operational.

This incident has been resolved.

A load balancer serving the Telerivet REST API stopped routing external traffic to Telerivet's servers at 02:29 UTC, causing all REST API requests to fail. Telerivet updated the DNS entries at 02:37 UTC to bypass this load balancer, and the REST API should now be working again. Telerivet is continuing to monitor networking issues within our primary datacenter.

The networking problem with the REST API load balancer has been resolved by Google Compute Engine. No further networking issues have been observed or reported for the past 3 hours.

Due to an outage in Amazon S3, Telerivet is currently unable to receive or display MMS (multimedia messages) or play voice calls with recorded audio. Telerivet's SMS functionality is not affected by the S3 outage, and voice calls can play text-to-speech or an audio file from a URL outside of Amazon S3.

The Amazon S3 outage has ended, so MMS and recorded audio are now working in Telerivet. We are planning to integrate with an additional storage provider to mitigate the impact of future S3 outages on Telerivet's service.

We are currently investigating intermittent errors with the message queue.

The errors with the message queue appear to have been resolved, however we are still investigating the root cause.

It seems likely that the intermittent errors were caused by a new setting we were testing for certain message queues, including some queues that were processing a high number of messages when the errors occurred. We have disabled this setting for these queues and will continue monitoring to see if the errors recur.

One of Telerivet's message queue servers experienced slow response times and elevated error rates from approximately 2:12 AM PDT until we restarted it at 2:28 AM PDT. We will continue to monitor the message queue and investigate the root cause of the incident.

This incident has been resolved.

Due to a DNS outage in Nexmo's infrastructure, Telerivet is currently encountering errors when sending messages to Nexmo. We are working to resolve this issue.

We have temporarily hardcoded the IP addresses of Nexmo's servers on Telerivet's servers, so Nexmo should be working again.

The outage of Nexmo on Friday was related to the large distributed denial of service attack against Dyn (https://en.wikipedia.org/wiki/October_2016_Dyn_cyberattack). No additional DNS outages have been observed since Friday.

Telerivet is currently observing long response times and intermittent errors with the message queue, and we are currently working to resolve the issue.

The message queue returned to normal at approximately 00:59, and messages queued during the delay have been sent. We are continuing to investigate the root cause of the delays and intermittent errors with the message queue to prevent the issue from happening again.

At approximately 00:10 UTC, one of our servers running RabbitMQ (the software that Telerivet uses internally to queue messages and other tasks) began experiencing very high CPU usage, very slow response times, and intermittent errors when queueing or dequeueing messages. During this time, messages were still able to be queued (with intermittent errors), and only 2% of API requests failed; however, the slow response times and intermittent errors from RabbitMQ caused the worker processes dequeuing messages to gradually fall further and further behind. Switching to a standby server in our RabbitMQ cluster did not resolve the issue. Eventually, we restarted the RabbitMQ process, at which time the CPU usage returned to normal, the intermittent errors stopped, and the worker processes quickly caught up. At this time, Telerivet has not yet identified a particular bug or configuration issue with RabbitMQ that caused this issue. In the next few days, we will be upgrading RabbitMQ to the latest release, as well as performing additional testing to try to reproduce the behavior in RabbitMQ outside of Telerivet's production environment.

Telerivet's primary datacenter in Google Compute Engine is currently experiencing network connectivity issues, so we have activated Telerivet's secondary datacenter in Amazon Web Services. Most services should now be functional, although search may be unavailable.

At 02:09 UTC, Google Compute Engine began experiencing severe network connectivity issues worldwide ( https://status.cloud.google.com/incident/compute/16007 ). In preparation for this type of outage, Telerivet has a standby datacenter from another hosting provider (Amazon Web Services), which we activated at 02:19 UTC. Most Telerivet services were operational at that time, except for search and some SMPP shortcode connections. SMPP connections were reactivated over the next half hour. Google resolved the issue with Google Compute Engine at 02:27 UTC, and subsequently we reactivated our primary datacenter in Google Compute Engine. Search indexes for each project have gradually been coming back online. Search is now available for most active projects, and should become available in all projects over the next several hours.

This incident has been resolved.

Telerivet's primary data center is again experiencing DDoS attacks ( http://status.linode.com/incidents/5ryq5w4l2mfj ), so we have relocated Telerivet to another data center. Some functionality (full-text search, dynamic groups, and SMPP connections) is currently unavailable in the new data center, and we are working to restore those services.

All services are currently operational in Telerivet's secondary data center. Search, dynamic groups, Android real-time connections, and SMPP connections are now working, with the exception of a small number of SMPP connections where the mobile network or aggregator requires a particular IP address in our primary data center. As the primary data center remains under DDoS attack at this time, Telerivet will remain in the secondary data center for now.

Linode has published an update about the DDoS attacks at http://status.linode.com/incidents/mmdbljlglnfd

Due to DDoS attacks in the secondary data center, Telerivet has returned to the primary data center, and most services are operational. Search and dynamic groups are temporarily unavailable while we rebuild the index. All SMPP connections should be working now.

Search and dynamic groups are now available, and all Telerivet services are operational.

This incident has been resolved.

Telerivet's primary data center is currently experiencing DDoS attacks ( http://status.linode.com/incidents/rknrs83pgjxv ), so we have relocated Telerivet to another data center. Some functionality (full-text search, dynamic groups, and SMPP connections) is currently unavailable in the new data center, and we are working to restore those services.

Network connectivity at Telerivet's primary data center has apparently returned to normal, so we have returned Telerivet to that data center, and all services are currently operational.

This incident is marked as resolved, because no network interruptions have been observed in the past several hours. In the meantime, we're actively working on improving our systems to increase reliability and reduce downtime in case of further DDoS attacks targeted at Linode.