58 Retreaver incidents · Ekim 2016 — official updates, affected components, duration and resolution details.
Son 15 dakika boyunca raporlama arayüzlerimizde (RTB Dashboard) gösteren RTB kayıtlarında bir gecikme tespit ettik. Ekibimiz bunu restore etmeye çalışıyor.
RTB raporlaması tamamen operasyonel olarak geri döndü.
Resmî olay güncellemesinden otomatik olarak çevrilmiştir.
RTB talepleri yaparken kullanıcıların küçük bir alt kümesi hataları görüyor
Geliştiricilerimiz sorunu buldular ve bunu çözmeye çalışıyorlar.
RTB şimdi Operasyonel
Geliştirme ekibi RTB hata oranında bir artış olduğunu fark etti. Bir incelemeden sonra bir düzeltme uygulandı. Alınan RTB taleplerinin%6'sı etkilendi. Olay çözüldüğünden ve daha fazla istek etkilenemez.
Resmî olay güncellemesinden otomatik olarak çevrilmiştir.
Calls are operational
Bu konuyu araştırmaya devam ediyoruz.
Bu olay çözüldü
Resmî olay güncellemesinden otomatik olarak çevrilmiştir.
00 UTC'de, RTB trafiğinin bir yüzdesi geri döndü. Genel trafik hacmi etkilenmedi, ancak bazı yayıncılar olay sırasında zaman kesinti yanıtları almış olabilir. Sorun 22:30 UTC tarafından tamamen çözülmüştü.
Resmî olay güncellemesinden otomatik olarak çevrilmiştir.
Aramalar ve diğer hizmetler iyi çalışıyor, sadece kredi kartında yüksek miktarda hata görüyoruz, böylece bozuldu.
Bir düzeltme uygulandı ve sonuçları takip ediyoruz.
This incident has been resolved.
Resmî olay güncellemesinden otomatik olarak çevrilmiştir.
RTB Inbound Request Gecikme Paneldeki RTB talepleri şu anda daha yüksek çözünürlüklü gecikmeler yaşıyor. gecikme genellikle bir dakika altında, ancak şu anda güncellemek için önemli ölçüde daha uzun sürüyor.
Bu olay çözüldü.
Resmî olay güncellemesinden otomatik olarak çevrilmiştir.
API uç noktalarımızla bazı servis sorunlarını araştırmak
Ev sahibimizle ilgili sorunlar yaşıyoruz ve şimdi hesap portalımızı ve JavaScript'imizi etkiliyor. Mühendislerimiz her şeyi online olarak geri getirmek için çalışıyor ve yakında bir güncelleme sağlayacaktır
API hizmeti, bu durumu kapatmadan önce sunucularımızı izlemeye devam ediyoruz.
Bu olay çözüldü.
Resmî olay güncellemesinden otomatik olarak çevrilmiştir.
We are investigating an issue with our RTB servers.
Our engineering team is continuing to work on restoring RTB to full performance.
Service to RTB is improving and we are continuing to monitor it.
We are continuing to monitor for any further issues.
RTB service has been stable for the past 2hrs, marking as resolved.
We are currently investigating this issue.
This incident has been resolved.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
We detected some our API servers went offline, our engineers responded, and service has been restored. We will continuing monitoring for a period of time before marking this as resolved.
After restoring service we detected another regression in API servers and are investigating.
API service is back online and we are monitoring
This incident has been resolved.
We are working on resolving an issue with our AI model vendor. For the time being disposition analysis and AI chat features on learn.retreaver.com are currently offline. We will mark this as resolved once Retreaver AI services are restored.
This incident has been resolved.
Requests that were send through the night might have returned HTTP 504. If some ingest processes are dependent on this please be advised that these requests will have to be send again once the incident is resolved.
Improvements have been deployed and clients should not be experiencing this.
The issue is currently under investigation. Please use Call 1.0
A third party service involved in the reporting are experiencing issues. Working with them on a resolution.
Incident should be resolved. We are monitoring and confirming.
This incident has been resolved.
Over the last three days, we have observed a recurring industry-wide performance dip between 11:30 AM and 12:30 PM EST. During this window, several external platforms and agencies are experiencing significant RTB latency. Because Retreaver processes these incoming requests and routes them to buyers, this upstream slowdown is resulting in occasional “heavy load” alerts or timeouts. To mitigate this, Ping Shield is actively scaling its protection to stabilize the system. These bursts typically last for 10–15 minutes before service returns to normal. Based on the data we will release a few additional improvements to Ping Shield to help protect against them.
The incident has been resolved. It is occurring for a limited amount of time.
A 'carrier' which is used by another 'call tracking provider' which isn’t Retreaver or Twilio is currently experiencing an incident.
A fix has been implemented and we are monitoring the results.
Carrier marked as operational
There has been and instance of a report with degraded performance.
The incident has been resolved the benchmark report ran in a satisfactory amount of time
We are currently experiencing service disruptions across several product lines due to an ongoing AWS outage in their US1 Region. You can track AWS’s official status here: https://health.aws.amazon.com/health/status This incident is impacting many of our connected services and telephony providers, resulting in disruptions across multiple areas of our platform. The issue stems from an ongoing AWS outage in their US1 Region and is the same as the one from a few hours ago.
Monitoring the results.
Monitoring the results.
We see calls not connected. We've identified that this is caused by the same issue experienced across the board with Twilio (https://status.twilio.com/) and AWS (https://status.aws.amazon.com/)
We see no issues on our side. At the same time AWS reports that services are still impacted.
This incident has been resolved.
We are currently experiencing service disruptions across several product lines due to an ongoing AWS outage in their US1 Region. You can track AWS’s official status here: https://health.aws.amazon.com/health/status This incident is impacting many of our connected services and telephony providers, resulting in disruptions across multiple areas of our platform. The issue stems from an ongoing AWS outage in their US1 Region.
We are continuing to investigate this issue.
Call are processing again. Continuing monitoring
We see no issues on Retreaver at the moment despite the fact the that there are 64 services with degraded performance on AWS.
You may occasionally see an error message with “We’re sorry, something went wrong.” We will be deploying a fix shortly.
We've identified an issue impacting a very limited set of users. A solution is being implemented. If you come across an error message when accessing our web portal and it's a showstopper, please get in touch with us and we'll find a way to help.
We detected another spike in errors for a subset of visitors to our website. Our team is investigating.
Website access has now been restored, we will keep monitoring for any further issues
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are aware of some RTBs not showing up in our RTB Log reporting section. These RTBs are still operational and our engineers are working on resolving the issue so they appear properly in the Log section.
RTB log has caught and all the requests are shown
This incident has been resolved.
