23 Medable incidents · Meurzh 2017 — official updates, affected components, duration and resolution details.
Durante este tempo, as aplicações web estavam intermitentemente disponíveis devido a um problema de provedor de rede upstream
Traduzido automaticamente da atualização oficial do incidente.
From 9:30pm EDT to 10:05pm EDT we observed that web applications in China may have been intermittently available, due to a configuration issue. The service was stabilized quickly as we proceeded to investigate and fix the root cause. Impact was intermittent for about 35 minutes.
We sustained impact to the Medable Cortex file upload functionality from 3:20am to 4:40am EDT on 10/20/2025 due to unavailability of the AWS SQS service in the US-East1 Datacenter.
We encountered a total of 17 min of downtime, during a 40 minute period of general instability in our EU1-Prod region on 8/27 between 11:10am EDT and 11:50am EDT. This was due to a database maintenance activity which was performed incorrectly and was not expected to cause impact.
We are currently investigating an issue where SMS notifications are not being delivered to US phone numbers.
Our email delivery provider stopped delivering emails for 4 hours due to an unforeseen account related issue. This has been resolved.
On 11/8/2022 from 1:18am PST to 2:28am PST the Web Application proxy service went down rendering all web applications unavailable, Including Study Manager and Patient App Web. Upon investigation, action was taken to restore the service and all applications were available shortly thereafter.
We had a breif (5 min outage) on our US prod environment due to database maintenance. This was not expected to cause impact, but unforeseen factors required us to redeploy the API services after we finished the DB maintenance activity. We do not expect this to be repeated.
We are currently investigating an issue affecting our Development environment in North America. More updates to follow shortly
All services are operational.
This incident has been resolved.
The issue has been identified and a fix is being implemented.
This incident has been resolved.
We are currently investigating this issue.
This incident has been resolved. A review of todays outages is underway and updates and RCA will be provided soon.
We are currently investigating this issue.
This incident has been resolved.
We are currently investigating this issue.
This incident has been resolved.
We are currently investigating this issue.
This incident has been resolved.
From 19:00 UTC - 19:20 UTC today, access to files on the Medable platform in the US development environment were unavailable. This impacted any user attempting to upload or download files via the cortex api and it also impacted web applications that were served up from this environment as well. Medable identified this issue immediately and worked with our infrastructure providers to restore functionality. We apologize for the disruption this may have caused our customers in their development environments and we are working with our infrastructure providers to ensure this issue cannot happen again.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
Platform availability was restored at 19:18 UTC. We've been monitoring the fix and will continue to monitor.
We experienced an API outage with our production environment on 13Apr2020. Our teams identified this at 18:21 UTC and worked to resolve the issue. This was a Severity 0 - Critical \(according to the Medable Service Level Agreement\) issue and was resolved by 19:18 UTC. We are currently completing our Incident Management Documentation and a full root cause analysis \(RCA\). Once the RCA is complete, we will be updating this space with the details as well. We apologize for the inconvenience and appreciate your patience while we resolved the issue.
We are investigating an issue where some endpoints are having difficulty establishing TLS handshake.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating an issue impacting the cortex web apps in prod and dev. The Cortex API remains available. Customer end-user applications are not impacted.
We are continuing to investigate this issue.
The issue has been identified as a networking issue with our infrastructure provider's load balancers that is leading to 502 errors being returned to clients.
We are monitoring our infrastructure provider's response to the issue and will update accordingly.
The issue has been resolved. We are continuing to monitor our infrastructure provider's response to the issue and will provide updates here.
We are investigating an outage across dev and production. We will report back with more details as we have them.
All services have been restored. We are working with our infrastructure partners to identify the root cause and steps to prevent the issue in the future.
We are currently investigating this issue.
The development api is experiencing an outage. We are investigating now and will update with more details as we have them.
Dev API restored. We will update with details on the cause shortly.
#### Cause A routine update in our deployment layer resulted in network connectivity issues between nodes. This resulted in services that were functioning normally to appear down to load balancing and proxy services. This ultimately resulted in the api being inaccessible externally. #### Resolution Upon identifying the cause, we worked to re-establish connectivity internally, restoring load balancing and proxying services. #### Prevention New maintenance processes have been put in place for the nodes in question. Preventive maintenance measures will take place on non-production nodes that will allow for testing and verification of the updates before they can impact production services. In production, these updates will then be applied during scheduled maintenance so that the impacts can be closely monitored.
Medable is experiencing an outage in the production api environment. The issue is under investigation, currently.
The outage has been resolved. All production services are operational. We will report back with an analysis of the situation shortly.
#### Cause A routine update in our deployment layer resulted in network connectivity issues between nodes. This resulted in services that were functioning normally to appear down to load balancing and proxy services. This ultimately resulted in the api being inaccessible externally. #### Resolution Upon identifying the cause, we worked to re-establish connectivity internally, restoring load balancing and proxying services. #### Prevention New maintenance processes have been put in place for the nodes in question. Preventive maintenance measures will take place on non-production nodes that will allow for testing and verification of the updates before they can impact production services. In production, these updates will then be applied during scheduled maintenance so that the impacts can be closely monitored.
