53 Avochato incidents · март 2020 г. — official updates, affected components, duration and resolution details.
Телефонные операторы видели повышенные 502 ошибки при отправке сообщения, которые могут появиться в Avochato с ошибкой: «Невозможно создать запись ошибки сервера (502)». Наша команда обострила этот вопрос и теперь он решен: https://status.twilio.com
Автоматический перевод официального обновления инцидента.
Телефонные операторы видят повышенные 502 ошибки при отправке сообщения, которые могут появиться в Avochato с ошибкой: «Невозможно создать запись Server error (502)». Наша команда обострила эту проблему, чтобы отслеживать сроки решения. Например, https://status.twilio.com показывает проблемы с подключением для SMS и MMS.
Автоматический перевод официального обновления инцидента.
В настоящее время Avochato изучает проблему, в которой некоторые пользователи могут столкнуться с проблемой входа в систему.
Этот инцидент был урегулирован
Автоматический перевод официального обновления инцидента.
Amazon Web Services reported outages globally this morning which may impact services Avochato relies on for text messages and other services. Our team will be monitoring the status and providing updates here. https://health.aws.amazon.com/health/status https://www.cnn.com/business/live-news/amazon-tech-outage-10-20-25-intl
Amazon shared systems are recovered from the outage. Avochato is monitoring.
This incident has been resolved.
Periodic latency has been observed starting at 8am PST. This may lead to longer than expected load times in inboxes across different organizations. Our team has been investigating the root cause with Amazon Web Services and will report back with findings.
The team is continuing to investigate this issue while working on options with Amazon Web Services. In the meantime, we recommend not refreshing an inbox. Once the inbox is loaded, messages will be received instantly and can be responded to in real-time. Refreshing the inbox may lead to longer delays.
Our service provider Amazon Web Services has not yet identified the root of the issues on Friday. However, the Avochato team is continuing with the investigation while having performed several actions to remediate the latency issues. As of 10:55am PST on Monday June 9, latency improved for those organizations experiencing issues when refreshing inboxes. Avochato's team switched to new server instances on Monday, and will continue Tuesday with a rollout of a permanent solution.
Latency in the product has been observed today. This may lead to longer than expected load times in inboxes across different organizations.
Avochato is monitoring the latency improvements as of Monday 8am PST.
This incident has been resolved.
Avochato is investigating an issue with the AWS cloud provider, which may lead to latency on inbox loading. Please stay tuned for updates as the team resolves this issue asap.
Currently seeing improvements to load speeds, however the team is continuing to monitor and work with our cloud provider as the impacted servers may still be impacting certain organizations and inboxes.
Avochato is now monitoring the improvements across impacted inboxes with latency issues, which began 9:30am and resolved by 10:40am PT. Cloud provider resources are still under investigation but all impacted customers should now see normal inbox load times.
The issue was resolved at 10:40am PT. Avochato continues to monitor and investigate the root case for the latency.
We are working on an issue with importing, saving, and creating new contacts in inboxes that use custom fields, as well as creating new fields in your inbox. As a temporary workaround, removing custom field columns from CSV uploads and/or API requests will resolve the issue. App load times may be slower than usual while we resolve the underlying issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
## What Happened An underlying database table hit a primary key limit that had not properly been migrated to 8 bytes. Today, we hit the limit on the maximum primary key for that table. This resulted in inboxes using Custom Fields feature see intermittent failures with various features that tried to update the value of custom fields on their contacts. This included creating contacts via API, uploading contacts to broadcast audience lists, editing contacts in the app, and handling answers to survey questions if the surveys stored data on custom fields. Inboxes that did not use the Custom Fields feature were not impacted. Sending messages with $custom\_field embeded in the message were not impacted. Contact uploads for that did not include custom field column names were not impacted even if they had custom fields, and should have finished accordingly. ## Resolution We have fully migrated the impacted table to use the correct primary key size limit and existing uploads continued as usual and reindex the results. We were able to do this without much disruption, though load times for inboxes with many contacts with custom fields may have been slower than usual during the day. Contact uploads that failed were retried after our database migration completed, so if you had a broadcast audience upload that failed, please double check your audience, as it should have all the contacts \(and their custom fields\) accounted for. Impacted surveys will have properly continued down their execution path. ## Next Steps All our new tables for the past few years already use 8 bytes to represent primary keys where appropriate. However, we plan on migrating all existing integer primary keys to use 8 bytes to avoid this issue in the future for any other tables. Thanks for your patience and trusting Avochato with your contact data, Christopher
Avochato is investigating an issue with calls and texts that may be impacting your inbox.
Carrier call issues have been intermittent with resolutions rolling out across many numbers. Avochato is monitoring the resolution.
Avochato continues to monitor the carrier resolution for calls across inboxes.
We've identified a connectivity issue with a third-party vendor we use for visualizing inbox data. This may impact viewing graphs in the Avochato dashboard and the ability to download CSVs of those tables. Legacy analytics views are not impacted, we will share updates on this status page as a resolution is investigated by our third-party vendor.
Avochato has had several conversations today with Google Cloud, and the issue has been escalated with the highest urgency. Google is still investigating the issue on their end. We will continue to share updates as soon as they become available.
We have dug into the underlying connection issue with the Google team and patched the servers. Please contact our support team if you're still having issues accessing the analytics in your Avochato account.
Some Avochato desktop web users have observed screen-sizing issues on wide screens and in wide-mode inbox views, leading to message readability issues. Avochato is currently investigating the issue and recommends using Classic or Compact views if you are experiencing issues reading messages. Mobile web and app views are unaffected. Rest assured that all content sent and received to end-users via SMS is being delivered over SMS, as this issue is specifically a front-end web issue with specific screen widths.
Avochato is monitoring a fix released for select views in the inbox.
This issue is now resolved.
Around 10:05am PT Avochato noticed delays in the loading of messages. This is being investigated with a resolution expected shortly.
We are continuing to work on a fix for this issue.
Avochato is monitoring the results of resumed normal load times in the inbox.
We are continuing to monitor for any further issues.
This incident has been resolved.
AT&T’s network went down for many of its customers across the United States Thursday morning, leaving consumers unable to place calls, text or access the internet. Although Verizon and T-Mobile customers reported some network outages, too, they appeared far less widespread. T-Mobile and Verizon said their networks were unaffected by AT&T’s service outage and customers reporting outages may have been unable to reach customers who use AT&T. https://www.cnn.com/2024/02/22/tech/att-cell-service-outage/index.html
This incident has been resolved.
Delays have been observed in retrieving data from Avochato's cloud service provider. Engineering teams on both sides are actively investigating to identify the cause and a resolution.
On Friday 12/15, Avochato's cloud provider confirmed the issue has been identified on their end. A temporary solution was provided until the investigation is completed. Avochato will update this issue as additional information is received.
We are experiencing MMS/SMS delivery failures to Verizon and AT&T networks in the US over a subset of long-codes. Our engineers are working with our carrier partner to resolve the issue. We expect to provide another update in 30 minutes or as soon as more information becomes available.
We have confirmed that MMS/SMS short code is not affected by the ongoing incident. We are experiencing MMS/SMS delivery failures to Verizon and AT&T networks in the US over a subset of long-codes. We are actively investigating the and will provide another update in 30 minutes or as soon as more information becomes available.
SMS/MMS providers are actively working on a resolution to delivery failures on Verizon and AT&T Networks in the US for a subset of Long Codes. Experts are on a bridge with a downstream telecom partner to work toward resolution. Customers may be experiencing delivery failures with a 30007 error for any traffic toward Verizon and AT&T. Avochato will continue to provide updates as they are received from providers. More details can also be found on: https://status.twilio.com.
Providers are observing successful MMS/SMS delivery to Verizon and AT&T networks in the US. Avochato will continue to monitor to ensure full service recovery and will provide additional updates.
As of 1pm PT, providers were not experiencing MMS/SMS delivery delays when sending messages to Verizon and AT&T. Avochato has been monitoring results in the meantime and will share all known findings on the incident once that information is provided. This incident is now resolved.
Avochato has identified known issues with US carriers regarding A2P SMS/MMS traffic. Some Avochato customers that are fully approved and established on A2P may be experiencing deliverability errors due to unexpected carrier issues. Avochato is working with carriers on a solution and will post updates here.
Providers have identified that some Avochato phone numbers have been continuing to experience deliverability issues even with a fully verified A2P application. This is due to carrier issues with the phone number itself, not the A2P application nor an Avochato-specific issue. This remains escalated directly with carriers and Avochato will be sharing updates as they become available by carriers.
This incident has been resolved.
We are investigating an issue impacting outbound message delivery failures on the carrier network.
Outbound messages for certain Avochato phone numbers are still experiencing issues sending outbound texts due to a network outage with our carrier partners.
We are observing significantly reduced errors connecting to our carrier partner but are awaiting confirmation that the incident has been resolved.
This incident has been resolved.
# What Happened A brief outage with Twilio’s REST API caused outbound SMS and WhatsApp messages to fail to send. Also, some other actions, including syncing Use-Cases to inboxes, would to fail intermittently and need to be re-attempted. Avochato partners with Twilio to send and receive calls and text messages for some customers. For more information on this particular incident, please see [https://status.twilio.com/](https://status.twilio.com/) ## Impact Some customers may have seen a specific 20503 error code when sending an outbound message via Avochato. _These specific outbound failed messages were not automatically re-tried._ Avochato inboxes continued to receive incoming messages during the impact period \(appx between 5:30-6:30 PST on 8/18/23\), and outbound messages continued to deliver as intended as soon as the issue was resolved. # Next steps Our team will continue evaluating the resiliency of the inbox in case of temporary outages, as well as mechanisms to retry sending these types of failed messages. Meanwhile, customers leveraging other carrier partners, as well as Avochato live chat, were unaffected. Thank you to the Twilio team for communicating status updates for this issue. To our customers, we apologize for any inconvenience during the impact period, and thank you for using Avochato.
US carriers have observed issues with MMS delivery towards US recipients using a US long code number. These issues may have impacted your Avochato number(s) on May 5. Our providers have released a patch to resolve this issue. We are in touch and expecting to receive updates on the status of the MMS issues.
Carriers have resolved the issues with MMS as of 13:41 PDT.
Avochato continues to investigate the incident.
We are experiencing a platform outage and Avochato is currently in read-only mode. We are working to resolve the issue as quickly as possible.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring results. Our backlog of messages is being processed -- messages that were scheduled or sent during the impact period will deliver over the next hour. Thanks for your patience.
This incident has been resolved.
We've noticed a significant spike in errors from our 3rd party service used to validate and choose new phone numbers. We're working to resolve this impact in your Avochato account.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.