19 Pagertree incidents · Juli 2023 — official updates, affected components, duration and resolution details.
2026-04-30 18:51:31 UTC Operational [ Resolved ] 2026-04-30 15:30:00 UTC Operational [ Identified ] All voice calls are currently coming from our Plivo provider +18175926665 Systems are operational, but you will notice calls coming from this phone number.
2025-11-19 21:21:57 UTC Operational [ Resolved ] 2025-11-18 14:45:56 UTC Operational [ Monitoring ] We are seeing restored connectivity to all services. Cloudflare has posted "A fix has been implemented and we believe the incident is now resolved. We are continuing to monitor for errors to ensure all services are back to normal" We continue to monitor. https://www.cloudflarestatus.com/incidents/8gmgl950y3h7 2025-11-18 14:37:00 UTC Partial outage [ Identified ] We continue to see intermittent connectivity issues with the web application, API, and marketing site. Cloudflare has posted "We've deployed a change which has restored dashboard services. We are still working to remediate broad application services impact". We continue to monitor. https://www.cloudflarestatus.com/incidents/8gmgl950y3h7 2025-11-18 14:23:20 UTC Outage [ Identified ] We continue to experience a complete outage affecting access to the web application, API, and marketing site. Cloudflare has posted "We are continuing to work on a fix for this issue". We will continue to monitor. https://www.cloudflarestatus.com/incidents/8gmgl950y3h7 2025-11-18 13:58:43 UTC Outage [ Identified ] We continue to experience a complete outage affecting access to the web application, API, and marketing site. Cloudflare has posted "We are continuing working on restoring service for application services customers". We will continue to monitor. https://www.cloudflarestatus.com/incidents/8gmgl950y3h7 2025-11-18 13:39:52 UTC Outage [ Identified ] We continue to experience a complete outage affecting access to the web application, API, and marketing site. Cloudflare has posted that the issue has been identified and a fix is being implemented. We will continue to monitor. https://www.cloudflarestatus.com/incidents/8gmgl950y3h7 2025-11-18 13:10:31 UTC Outage [ Identified ] We are experiencing a complete outage affecting access to the web application, API, and marketing site. Cloudflare has posted that the issue has been identified and a fix is being implemented. We will continue to monitor. https://www.cloudflarestatus.com/incidents/8gmgl950y3h7 2025-11-18 12:59:37 UTC Partial outage [ Identified ] We continue to see intermittent connectivity issues with the web application, API, and marketing site. Cloudflare has posted that they have started remediation efforts, and we continue to monitor the situation. https://www.cloudflarestatus.com/incidents/8gmgl950y3h7 2025-11-18 12:29:27 UTC Partial outage [ Identified ] We are still seeing intermittent connectivity issues with the web application, API, and marketing site. Cloudflare has posted that they have started remediation efforts. https://www.cloudflarestatus.com/incidents/8gmgl950y3h7 2025-11-18 12:09:22 UTC Partial outage [ Identified ] Web Application, API, and Marketing Site are all affected by this outage. We continue to monitor Cloudflare for updates. https://www.cloudflarestatus.com/incidents/8gmgl950y3h7 2025-11-18 11:59:25 UTC Partial outage [ Identified ] Web Application, API, and Marketing Site are all affected by this outage. We continue to monitor Cloudflare for updates. https://www.cloudflarestatus.com/ 2025-11-18 11:52:14 UTC Problem detected [ Investigating ] We are actively investigating the incident. https://www.cloudflarestatus.com/
2025-10-21 12:24:05 UTC Operational [ Resolved ] 2025-10-20 13:14:25 UTC Operational [ Monitoring ] We continue to monitor the multi service outage. At this moment, most services have recovered. https://health.aws.amazon.com/health/status https://status.twilio.com/incidents/2mjg0mr2pf29 https://status.flyio.net/ 2025-10-20 11:08:31 UTC Degraded performance [ Monitoring ] We are also seeing some errors in upstream providers for many services (Twilio, OneSignal) due to an upstream AWS outage. You might see notifications like SMS, Voice, and Push are delayed or are in the error state. https://status.twilio.com/incidents/2mjg0mr2pf29 https://health.aws.amazon.com/health/status 2025-10-20 10:55:52 UTC Degraded performance [ Monitoring ] We are continuing to monitor the incident with the upstream provider. The majority of the intermittent connectivity loss could be seen between 10/19 23:30 and 10/20 02:20 2025-10-20 07:20:00 UTC Degraded performance [ Identified ] Our PaaS provider is experiencing IPv6 connectivity issues (due to a faulty network switch), which can cause problems while connecting to the PagerTree services. https://status.flyio.net/ 2025-10-20 06:30:00 UTC Problem detected [ Identified ]
2025-10-08 18:19:18 UTC Operational [ Maintenance complete ] Service upgrade complete. 2025-10-08 17:00:00 UTC Outage [ In maintenance ] We are currently upgrading our services; it should take up to 2 hours. 2025-10-03 23:00:00 UTC Operational [ Maintenance scheduled ] We will be upgrading our database on Oct 8, 2025, at 1 p.m. Eastern / 10 a.m. Pacific, requiring up to 2 hours of downtime for all services.
2025-06-20 17:36:12 UTC Operational [ Resolved ] Docs site is now operational. 2025-06-20 15:55:08 UTC Partial outage [ Investigating ] Our docs site is currently not working and resulting in a JavaScript error. Docs can currently be reached at https://pagertree.gitbook.io
2025-05-16 15:22:22 UTC Operational [ Resolved ] The root cause of this issue was CPU throttling by our PaaS provider. We changed CPU configurations to prevent this in the future. 2025-05-16 14:47:10 UTC Operational [ Monitoring ] We have identified the root cause of the backup. (CPU Saturation) During the incident, critical queues were only slower than normal. Alert processing and notification delivery were delayed up to 15 minutes, but nevertheless delivered. Our search queue was backed up to 100 minutes. 2025-05-16 14:06:13 UTC Problem detected [ Investigating ] Inbound Queue Processing has been backed up since 12:45 UTC. We are investigating...
2025-05-15 16:00:00 UTC Operational [ Maintenance complete ] Upstream provider critical network maintenance complete. 2025-05-15 14:00:00 UTC Degraded performance [ In maintenance ] An upstream provider is performing critical network maintenance in SEA, it should take up to 2 hours. 2025-05-12 17:15:00 UTC Operational [ Maintenance scheduled ] An upstream provider is performing critical network maintenance in SEA, from 14:00 UTC (07:00am PDT local time) to 16:00 UTC (09:00am PDT local time). You may experience a short total loss of connectivity for up to 15 minutes within the scheduled maintenance window hours.
2025-02-20 23:05:00 UTC Operational [ Resolved ] Delayed Notification & Workflows - 23:05- 23:15 UTC - Queues were backed up. Currently investigating the root cause. Services should be normal now.
2025-02-05 18:07:31 UTC Operational [ Maintenance complete ] Database upgrade complete. 2025-02-05 17:35:00 UTC Operational [ In maintenance ] We are currently upgrading our database; it should take less than one hour. Customers could see slower than usual performance and intermittent 5xx errors briefly during the planned maintenance. 2025-02-05 16:35:00 UTC Operational [ Maintenance scheduled ] We will be upgrading our database in an hour. Customers could see slower than usual performance and intermittent 5xx errors briefly during the planned maintenance.
2024-10-24 22:54:17 UTC Operational [ Resolved ] The Android app is available again. You can find it here: https://play.google.com/store/apps/details?id=com.pagertree.app&pcampaignid=MKT-Other-global-all-co-prtnr-py-PartBadge-Mar2515-1 2024-10-24 13:38:21 UTC Problem detected [ Resolving ] We are aware that the Android app is currently unavailable from the Play store. We are working with Google to get it relisted. Thank you for your patience.
2024-09-02 01:33:56 UTC Operational [ Resolved ] 2024-09-01 20:23:46 UTC Operational [ Monitoring ] Fly.io has pushed a fix. All PagerTree services are now available. We will continue to monitor. 2024-09-01 20:06:22 UTC Outage [ Monitoring ] We are aware of our service provider network outage affecting all PagerTree services. We are monitoring https://status.flyio.net/
2024-06-20 20:05:58 UTC Operational [ Resolved ] We have not observed anymore 502 error codes for the last 1h. Marking this as resolved. 2024-06-20 19:05:38 UTC Degraded performance [ Identified ] We have identified Cloudflare is performing system maintenance. Access can be intermittent depending on where you are connecting from. We will continue to monitor. We apologize for the inconvenience. https://www.cloudflarestatus.com/ 2024-06-20 18:58:38 UTC Problem detected [ Investigating ] Our developers are currently investigating the issue. We will provide more updates in 30 minutes or as we discover more information.
2024-05-31 19:20:35 UTC Operational [ Resolved ] 2024-05-31 18:52:58 UTC Degraded performance [ Monitoring ] A combination of Fly.io networking issues (https://status.flyio.net/incidents/y0hghbxzzfh9) and Cloudflare issues (https://www.cloudflarestatus.com/incidents/0qhbf6p8xcsg) have caused intermittent network outage and resolution failures. Teams from both organizations have addressed the issues. We are monitoring the results. Web & API access were affected. We will continue to investigate the impact. 2024-05-31 18:37:34 UTC Problem detected [ Investigating ] We are still investigating network issues. We will post another update in 30 minutes. 2024-05-31 17:59:14 UTC Problem detected [ Investigating ] We are currently experiencing networking issues. We are investigating
2024-04-27 02:26:47 UTC Operational [ Resolved ] 2024-04-27 00:52:47 UTC Operational [ Monitoring ] We see recovered connectivity from our upstream provider. We will continue to monitor our services and our upstream provider(https://status.flyio.net/). We will resolve this incident when our upstream provider has called this resolved. 2024-04-27 00:17:13 UTC Operational [ Monitoring ] We can confirm the issue is with our upstream provider, fly.io (https://status.flyio.net/). Fly.io has implemented a fix, and we are monitoring the results. We will provide updates as they come in. 2024-04-27 00:10:19 UTC Degraded performance [ Identified ] We can confirm the issue is with our upstream provider, fly.io (https://status.flyio.net/). Fly.io has implanted a fix and we are monitoring the results. All app, API, and background processing services are affected. This includes incoming HTTP and notifications. We will provide updates as they come in. 2024-04-27 00:00:27 UTC Outage [ Identified ] We can confirm the issue is with our upstream provider, fly.io (https://status.flyio.net/). Fly.io continues to investigate the issue. All app, API, and background processing services are affected. This includes incoming HTTP and notifications. We will provide updates as they come in. 2024-04-26 23:53:45 UTC Problem detected [ Identified ] We can confirm the issue is with our upstream provider, fly.io (https://status.flyio.net/) All app, API, and background processing services are affected. This includes incoming HTTP and notifications. 2024-04-26 23:51:07 UTC Problem detected [ Investigating ] We are observing elevated errors and connectivity issues across PagerTree. We suspect the issue is with our upstream provider Fly.io. (https://status.flyio.net/) We will provide updates as we find out more.
2024-04-24 21:30:34 UTC Operational [ Resolved ] Our upstream provider (https://status.flyio.net/) has marked this issue as resolved. 2024-04-24 20:55:05 UTC Operational [ Monitoring ] We continue to see improved connectivity from our upstream provider. We will continue to monitor our services and our upstream provider(https://status.flyio.net/). We will resolve this incident when our upstream provider has called this resolved. 2024-04-24 20:11:57 UTC Operational [ Monitoring ] We've seen improved connectivity and no errors for the last 45 minutes. We will continue to monitor our services and our upstream provider(https://status.flyio.net/) 2024-04-24 19:46:04 UTC Operational [ Monitoring ] We've seen improved connectivity and no errors for the last 15 minutes. We will continue to monitor our services and our upstream provider(https://status.flyio.net/) 2024-04-24 19:25:48 UTC Degraded performance [ Monitoring ] A fix has been implemented by our upstream provider (https://status.flyio.net/) and we are monitoring the results. 2024-04-24 19:08:49 UTC Problem detected [ Investigating ] We can confirm the issue is with our upstream provider fly.io (https://status.flyio.net/) All app, API, and background processing services are affected. This includes incoming HTTP and notifications. 2024-04-24 19:05:35 UTC Problem detected [ Investigating ] We are observing elevated errors and connectivity issues across PagerTree. We suspect the issue is with our upstream provider Fly.io. (https://status.flyio.net/) We will provide updates as we find out more.
2023-11-14 03:03:23 UTC Operational [ Resolved ] Our upstream provider has restored network connectivity and declared the incident resolved. https://status.flyio.net/ 2023-11-14 02:34:05 UTC Degraded performance [ Monitoring ] We are seeing improved connectivity with all PagerTree services up and operational. We continue to monitor the situation. You can follow along here: https://status.flyio.net/ 2023-11-14 02:07:45 UTC Partial outage [ Identified ] Our upstream provider has identified a segment of hosts are affected by an upstream networking issue which has resulted in downtime for Fly.io services. We are monitoring the situation.
2023-07-31 18:28:41 UTC Operational [ Resolved ] We have published a postmortem here: https://pagertree.com/blog/fly.io-migrate-to-v2-postgres-stuck-in-read-only-mode 2023-07-30 01:43:02 UTC Operational [ Resolved ] Our database is up in a read/write state. All services are now fully functional. A postmortem will be written on Monday, summarizing the issue, root cause, and remediation. Thank you for your patience and understanding. 2023-07-30 01:16:46 UTC Operational [ Monitoring ] Our database is back up in a read/write state. We are monitoring at the moment. Thank you for your patience. 2023-07-30 00:59:57 UTC Partial outage [ Identified ] Still working on the database read only issue. We will post another update in 30 minutes 2023-07-30 00:35:27 UTC Partial outage [ Identified ] Inserts in the database will be blocked for approximately 10 minutes. Incoming requests will fail. Read requests will succeed. 2023-07-30 00:31:03 UTC Problem detected [ Identified ] We are currently working through a database upgrade.
2023-07-28 22:40:00 UTC Operational [ Maintenance complete ] Database upgrade complete. 2023-07-28 22:30:00 UTC Partial outage [ In maintenance ] We are currently performing a database upgrade. We expect services to be down for approximately 10 minutes. 2023-07-28 21:48:00 UTC Operational [ Maintenance scheduled ] We will perform a database upgrade on July 28, 2023 at 22:30 UTC. We expect services to be down for approximately 10 minutes.
2023-07-27 03:40:00 UTC Operational [ Maintenance complete ] Database upgrade complete. 2023-07-27 03:30:00 UTC Outage [ In maintenance ] We are currently performing a database upgrade. We expect services to be down for approximately 5 minutes. 2023-07-26 13:51:56 UTC Operational [ Maintenance scheduled ] We will perform a database upgrade on July 27, 2023 at 03:30 UTC. We expect services to be down for approximately 5 minutes.