52 Doxyme incidents · 2020年8月 — official updates, affected components, duration and resolution details.
透过doxy. 有些邀请可能被推迟或未送达。 这是一个已知的问题 与中断 曼德里尔, 我们的第三方电子邮件提供商。 我们的团队和曼德罗的团队都在积极努力解决这个问题。 Doxy.me仍然可以使用,你可以继续以其他方式分享你的房间链接. 我们将随着我们了解更多的情况提供最新情况.
我们已确认, 影响使用邀请Via发送的电子邮件邀请的问题, 两个小组都在积极努力恢复电子邮件发送.
曼德瑞已经做了一个修复。 我们正利用邀请Via来监测发送的电子邮件邀请.
我们正在继续监测任何其他问题.
曼德瑞的干扰已经解决了 正在按预期发送使用邀请处发送的电子邮件邀请函.
自动翻译自官方事件更新。
我们正在调查造成部分服务中断的问题。 虽然网站仍然可以访问,但一些功能没有达到预期效果,包括认证和登录. 我们的团队正同我们的上游提供者积极合作,以查明根源并尽快恢复全部功能。 我们将随着我们了解更多的情况提供最新情况.
我们确认,由于AWS CloudFront事件不断发生,这一问题影响了我们的认证提供者。 网站可能仍可访问,但有些用户可能无法登录或使用某些平台功能. 我们的供应商正在积极实施减缓措施,以绕过受影响的云纹服务,我们正在密切监测其进展。 一旦获得更多信息,我们将提供另一个最新情况.
服务正常运行。 我们继续监视平台 确保一切稳定.
这一事件已经得到解决.
自动翻译自官方事件更新。
We're investigating a system-wide performance degradation impacting many users.
Service is returning to normal. We are continuing to monitor the situation.
The issue has been resolved. The doxy.me system is functioning as normal. Once our investigation is complete, a full report will be provided.
**Summary of event:** On March 23, doxy.me experienced intermittent service disruptions during a planned system upgrade. A combination of internal issues and a third-party service issue led to degraded performance throughout the day. **Incident Classification:** Service disruption caused by a combination of internal system changes and a third-party service issue. This was not a security breach. **Business impacts:** Some users experienced dropped calls, difficulty accessing History and Analytics, and issues with Custom Waiting Rooms. **Data Impact:** No evidence of unauthorized access, disclosure, or loss of customer data or PHI. Some data was temporarily unavailable during the incident but has been fully restored. **Root cause:** The disruption was caused by overlapping issues, including a third-party service not updating correctly, delays in loading History, and a migration issue affecting some settings. **How the issue was resolved:** Our team deployed fixes throughout the day and paused the upgrade to prevent further impact. Service returned to normal after all issues, including the third-party issue, were resolved. **Preventative next steps:** We are improving our system upgrade process, strengthening safeguards around third-party services, and enhancing monitoring to respond more quickly in the future.
We are currently investigating a reported system outage that's impacting provider log in.
Provider login is returning back to normal. We are continuing to monitor the situation.
The issue impacting provider login has been resolved. The doxy.me system is functioning as normal. Once our investigation is complete, a post-mortem will be provided.
We are currently investigating a reported system outage that's impacting provider log in.
The ability to login is returning back to normal. We are continuing to monitor the situation.
The issue preventing provider login has been resolved. The doxy.me system is functioning as normal. Once our investigation is complete, a post-mortem will be provided.
We are currently investigating an issue affecting some provider logins on doxy.me. Users may experience a blank or unresponsive login page. Initial investigation indicates that the issue originates from our authentication system, which is currently experiencing a global outage related to CDN configuration. We’re actively monitoring the situation and working to ensure service stability. Components Affected: • Provider Sign-In • Authentication Service
Fix for the issue with auth CDN provider is applied, and login services on doxy.me are now recovering. We are continuing to monitor performance and accessibility across all regions to ensure full stability. Components Affected: • Provider Sign-In • Authentication Service
This incident has been resolved. All login and authentication services are now operating normally.
We’re investigating an issue causing file uploads (like check-in photos) to fail and preventing some new sessions from starting. Ongoing calls are working normally. The issue appears related to an AWS outage and we’re monitoring their progress.
A fix has been implemented by our AWS partner, and affected doxy.me services are recovering. File uploads and new session starts are operating normally, but we’re continuing to monitor to ensure full stability.
A doxy.me partner, AWS, is experiencing an issue that is now impacting some provider sign-ins, site functionality, and file uploads. We’re monitoring AWS and will share updates as they become available.
The ongoing AWS issue continues to affect some doxy.me services. We’re closely monitoring the situation and will share updates as they are available.
The AWS issue has been resolved and all doxy.me services are now functioning normally.
We are investigating an issue with our SMS partner affecting custom SMS delivery. Default SMS continues to work as expected.
The issue has been identified with our SMS provider and work is underway on a long-term solution. Custom SMS appointment invites are currently sending, but there may still be disruptions.
Custom SMS appointment invites are sending reliably and we’re continuing to monitor.
After a period of monitoring, we have confirmed SMS appointment invites are sending normally and the issue is resolved.
We are currently investigating a reported system outage that's impacting provider log in.
The ability to login is returning back to normal. We are continuing to monitor the situation.
The issue preventing provider login has been resolved. The doxy.me system is functioning as normal. Once our investigation is complete, a post-mortem will be provided.
We're currently investigating a reported system outage that's impacting patient check-in and provider account access.
The system is returning to normal. We are continuing to monitor the situation.
The system issue preventing patient check-in and provider account access has been resolved. The doxy.me system is functioning as normal. Once our investigation is complete, a post-mortem will be provided.
**Summary of event** On April 28, our monitoring system detected a sharp slowdown in platform performance, preventing some users from accessing doxy.me. This disruption lasted for approximately 35 minutes before service was restored by increasing system capacity. **Business impacts** During the incident, users were unable to access doxy.me for telehealth sessions, leading to delayed or missed appointments and interruptions in patient care. **Root cause** A recent system update unintentionally lowered the number of active servers. Because this deployment occurred after our daily resource allocations, the platform did not have sufficient capacity to handle the level of traffic, resulting in degraded performance. **How the issue was resolved** Once the root cause was identified, our engineering team increased the number of active servers. This promptly reduced system load and restored access for users. **Preventative next steps** * Update deployment procedures to appropriately size server capacity at the time updates are launched * Improve intra-day automated resource allocations and real-time monitoring
We're actively investigating an outage affecting Apps, such as File Transfer and Whiteboard.
We've identified the cause of the outage. Our team is working on a fix, and we’ll share updates as we make progress.
Monitoring - App functionality has been restored. We’ll continue to monitor the situation.
The issue causing the app outage has been fixed and the system is now fully operational. All apps are functioning normally. A post-mortem report will be provided in the coming days.
We're currently investigating a reported system outage that's impacting patient check-in and provider account access.
System is returning back to normal. We are continuing to monitor the situation.
A fix has been implemented and we are monitoring the results.
We recently encountered an issue that prevented patients from appearing in their provider's waiting room queue. This issue has now been resolved, and patient check-in is functioning as expected. A post-mortem report will be provided in the coming days.
We are currently investigating this issue.
The system is returning to normal. We are continuing to monitor the situation.
The system issue preventing patient check-in and provider account access has been resolved. The doxy.me system is functioning as normal. Once our investigation is complete, a post-mortem will be provided.
We’re aware that patients are not appearing in their provider’s waiting room queue, and we are actively investigating the issue. We will provide updates as soon as we have more information. Thank you for your patience.
System is returning back to normal. We are continuing to monitor the situation.
We recently encountered an issue that prevented patients from appearing in their provider’s waiting room queue. This issue has now been resolved, and patient check-in is functioning as expected. We will provide a post-mortem report in the coming days.
**Summary of event**: On October 24, patients who successfully checked in did not appear in the Waiting Room queue due to an incident from our real-time partner, PubNub. PubNub provides the technology that enables instant updates to [doxy.me](http://doxy.me) when a patient has joined a provider's Waiting Room and allows us to start a video call. This led to an outage on [doxy.me](http://doxy.me) approximately between October 24, 7:15 AM ET - 10:00 AM ET and October 24, 8:15 PM ET - October 25, 6:00 AM ET. **Business impacts:** Although providers were able to log in, their Waiting Room queue remained empty, preventing a session from starting between the patient and provider. **Root cause:** Our real-time partner, PubNub, encountered an unusual surge in traffic from sources other than [doxy.me](http://doxy.me), that temporarily overwhelmed their system, preventing it from responding quickly enough. As a result, PubNub’s communications to [doxy.me](http://doxy.me) were interrupted, causing patients to not appear in the Waiting Room queue. **How was the issue resolved:** PubNub blocked the source of surging traffic and made changes to handle the increased demand. Once PubNub made this update, [doxy.me](http://doxy.me) was fully operational again. [Doxy.me](http://Doxy.me) did not make any changes. **Preventative next steps:** [Doxy.me](http://Doxy.me) will collaborate with PubNub to ensure PubNub can manage traffic surges from its other customers and maintain scalable infrastructure, minimizing any potential impact on [doxy.me](http://doxy.me) services and customers.
We are currently investigating this issue of patients not being able to check in and start call. We will update this status as soon as we have more information
Our real-time service provider has identified the issue and is working on a fix. We will continue to monitor and work towards restoring our services.
Our real-time service provider is seeing improvements with the patient check-in and appointment start issues, and we will continue to closely monitor the recovery process
Resolved - The issue affecting patient check-ins and starting doxy.me appointments has been resolved and will continue to be monitored to ensure stable performance
We recently encountered an issue that prevented patients from checking in on doxy.me 2.0. This issue has now been resolved, and patient check-in is functioning as expected. We will provide a post-mortem report in the coming days.
**Summary of event**: On October 19, [doxy.me](http://doxy.me) encountered an issue that prevented patients from checking in on [doxy.me](http://doxy.me) 2.0. This outage resulted in 8 hours and 28 minutes of unplanned downtime between 8:01 AM ET and 4:30 PM ET. **Business impacts:** Patients were unable to complete the check-in process. After entering their name and clicking the Check-In button, they remained stuck on the check-in screen instead of joining their provider’s waiting room. While providers were able to log in during this incident and receive patient check-in notifications, patients didn’t appear in their waiting room queue. This prevented a provider from starting a video call with the patient. **Root cause:** Security improvements implemented in our real-time messaging system and [doxy.me](http://doxy.me) 2.0 platform led to an issue that arose approximately 48 hours after the release. The continuous release cycle of [doxy.me](http://doxy.me) prevented the issue from appearing in our non-production environments used for pre-release testing. **How was the issue resolved:** Engineering identified real-time messaging as the root cause of the outage and updated accordingly while maintaining the security enhancements implemented. **Preventative next steps:** We’re enhancing our testing framework to include additional scenarios that will help identify unexpected real-time issues.
We are aware of an outage currently affecting doxy.me. Our team is currently investigating at this time. We will post an update as we know more.
We are continuing to investigate this issue.
The issue has been identified and a fix is being created. We are continuing to monitor the situation and have seen normal site functionality return.
This incident has been resolved.
We are currently investigating this issue.
We are continuing to investigate this issue.
Between 12:37 PM and 1:13 AM EST, our partner AWS experienced elevated error rates for request serviced by the CloudFront Origin Shield and Regional Edge Cache in the US-EAST-1 region. The issue has been resolved and service is operating normally.
We are aware of an outage currently affecting doxy.me. This appears to affect sign-in, call connecting, and various other system services. Our team is currently investigating at this time. We will post an update as we know more.
We are seeing our system stabilizing. We are continuing to look into the incident and are going to continue to monitor the situation.
System levels have remained stable overnight. We are continuing to monitor the system performance and will provide another update by 12pm ET.
Doxy.me services continue to be stable. We are considering this incident resolved and will provide a post mortem within the few business days.
**Summary of event:** On May 25th, [Doxy.me](http://doxy.me/) experienced a database issue that resulted in 27 minutes of unplanned downtime from 2:03pm to 2:30pm ET. **Business Impacts:** Sessions in progress were not impacted. The incident resulted in providers being unable to log in for their sessions. In addition, patients who were not already in the queue were unable to check in to their providers waiting room. **Root cause:** A data import file was processed internally. While it was being processed, a series of long-running secondary server processes were triggered. Unfortunately, these processes built up in number, consuming additional server resources until the servers were backed up. **How was the issue resolved:** The file was able to finish processing and the system self recovered. **Preventative next steps:** We are optimizing the the long-running processes which will help prevent this scenario from happening again. In the short term, we are no longer running large data imports during peak business hours.
We are currently investigating an issue where older versions of web browsers (Safari, Chrome, Firefox, Edge) are not allowing doxy.me to load. Updating your browser to the latest version will allow doxy.me to load. Browsers able to load doxy.me: Safari 15.4 and newer Chrome 93 and newer Firefox 92 and newer Edge 93 and newer If you or your client is impacted at this time, our help article will help guide you to update.
We have determined the root cause for this incident and are currently testing a fix. At this time, we expect the fix to be deployed early tomorrow, between 3:00am and 7:00am EST (between 08:00 and 12:00 UTC).
We have deployed a fix restoring functionality to the out of date browsers. We recommend keeping your devices and browsers up to date in general, but do apologize for any disruption this incident may have caused. We will share the post mortem within the next few business days.
**Summary of event:** On January 10th, an update caused out-of-date browsers to not load [doxy.me](http://doxy.me/). **Business Impacts:** Patients or providers using the impacted browsers were unable to access [doxy.me](http://doxy.me/). **Root cause:** One of the software packages used in our code was updated on January 10th, 2023 at 08:20 AM UTC. The updated version was incompatible with and no longer supported older browser versions, including any versions older than the following: * Chrome: 93 \(released August 31st, 2021\) * Firefox: 92 \(released September 7th, 2021\) * Safari: 15.4 \(released March 15th, 2022\) * Edge: 93 \(released August 3rd, 2021\) **How was the issue resolved:** [Doxy.me](http://doxy.me/) released a fix that reverted the update to the software packages on January 11th, at 8:03 AM UTC. **Preventative next steps:** We will be updating our internal system requirements and our supported browser policies. Once approved, we will increase our communication efforts with our customers to make it easier for them to recognize when they are using an out-of-date browser and how to update it. We will also shore up our testing coverage to better detect these problems in advance. Thank you for your patience.
