Pdfgeneratorapi 장애 기록

We're experiencing an elevated level of API errors and are currently looking into the issue.

We have identified the root cause of the issue and are applying a fix.

We have resoled the issue.

We're experiencing an elevated level of API errors and are currently looking into the issue.

We are continuing to investigate this issue.

This incident has been resolved.

We are currently experiencing service disruptions due to a major AWS (Amazon Web Services) outage which is affecting our systems and functionalities. We sincerely apologize for the inconvenience this may cause. Our team is actively monitoring the situation and working on mitigation steps where possible. Thank you for your patience and understanding. We will keep you informed as we learn more and as the situation evolves.

We have resolved the issue affecting our asynchronous document generation service, and all systems are now operating normally. The service disruption began at approximately 1:00 AM EEST and was fully resolved by 7:50 AM EEST. The incident was caused by a failure in our queue database system, which prevented a replica node from being promoted to handle new processing requests.

Our team is currently addressing a service degradation impacting our document generation feature. We understand the importance of this service to your workflow, and our teams are working diligently to restore full functionality. What is happening? Starting at approximately 12:00 AM UTC, we identified an issue causing intermittent failures for document generation requests. Our monitoring indicates that approximately 50% of these requests are currently failing. Our investigation has traced the root cause of performance issues within our Redis cache layer in our primary AWS Region. What are we doing? Our DevOps and Engineering teams are actively working on a resolution. We are currently implementing remediation steps to restore cache stability and bring the service back to full health. We are confident in our team's ability to resolve this matter promptly. We sincerely apologise for the disruption this has caused and appreciate your patience.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

### Summary On August 13, 2025, between approximately 02:30 and 08:44 \(UTC\+3\), the PDF Generator API service experienced a severe degradation. During this period, a significant percentage of API requests failed. The issue was resolved by failing over a component of our caching infrastructure, and the service has been stable since. ### Impact * **Service:** PDF Generator API * **Impact:** A high rate of failed API requests for all users. * **Duration:** Approximately 6 hours and 15 minutes of severe degradation. ### Timeline of Events \(UTC\+3\) * **August 12, 22:00:** A low volume of connection errors to our caching layer was first detected by our monitoring systems. The overall service health remained stable with no noticeable impact on users. * **August 13, 02:30:** A sharp increase in connection failures occurred, leading to a severe degradation of the PDF Generator API service. The on-call engineering team was engaged. * **August 13, 02:30 - 08:44:** The engineering team actively investigated the issue. During this time, the service continued to experience a high failure rate. * **August 13, 08:44:** A successful manual failover of a key component in our caching infrastructure was completed. Service functionality was immediately restored. ### Root Cause Analysis The direct cause of the incident was a widespread failure of our application servers to establish connections with our primary caching instance, which runs on Redis in an AWS cluster. Our investigation confirmed that the Redis server itself was healthy, responsive to direct connections, and processing data normally. All internal metrics for the server appeared normal throughout the incident. The issue was isolated to the network path between the application and the cache. Evidence, including a concurrent spike in reported issues for Amazon Web Services \(AWS\) on third-party monitoring sites like Downdetector, suggests the incident may have been related to a transient issue within the underlying cloud infrastructure. ### Resolution The immediate fix was to redirect all application traffic from the primary cache instance to a secondary hot-standby replica. This action successfully bypassed the connectivity problem, and full service functionality was restored upon completion of the failover. ### Action Items & Lessons Learned To prevent recurrence and improve our response, we are taking the following actions: * **Enhance Monitoring:** We are implementing more granular, application-level alerts to track connection success rates for our caching tier. This will allow for faster detection of this specific failure scenario. * **Automate Failover:** We will develop and test an automated failover procedure for the caching tier. This will significantly reduce recovery time \(RTO\) if a similar incident occurs in the future. * **Review Incident Protocols:** We will review and update our incident response protocols to formalise the process for investigating and escalating potential issues with our cloud provider. We apologise for the disruption this incident caused our users. We are committed to learning from this event and improving the reliability of our services.

We're experiencing an elevated level of API errors and are currently looking into the issue.

A weekly release window to deploy updates and fixes. The update can cause temporary downtime up to 5 seconds. If the deploy includes customer facing updates, we will also publish release notes here: https://support.pdfgeneratorapi.com/en/category/release-notes-uouwx9/

We're experiencing an elevated level of API errors due to cache issues. The team is working on solving the issue.

We are continuing to work on a fix for this issue.

A fix has been implemented and we are monitoring the results.

Root cause: Redis ran out of memory when a huge base64-encoded PDF got stuck in the document-delivery queue, pushing usage over 90%. The team temporarily upgraded the Redis node to a bigger instance, which cleared the alert, and then deleted the oversized queue key to free up about 1.2 GB.

Our document generation processes are taking longer time than usual. We have identified the root cause and working on solving the issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

A configuration issues caused downtime in Enterprise Deployment EU. We have found the root cause and are updating the deployment configuration.

A fix has been implemented and we are monitoring the results.

The issue affected Enterprise EU customers using API v4 for document generation. The problem was caused by the misconfiguration of the Generator API endpoints required by the latest release updates to generation logic. We are looking into how we can improve the configuration management to mitigate risks in the future.

Our document generation processes are taking longer time than usual. We have identified the root cause and working on solving the issue.

This incident has been resolved.

We're experiencing an elevated level of API errors and are looking into the issue. The API returns correct response content, but the status code is returned as 500, which can cause false negative results in customer applications.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

This incident has been resolved.

We are investigating Redis connection issues on Enterprise Deployment (EU)

Redis connection issues resolved.

Our document generation processes are taking longer time than usual. We have identified the root cause and working on solving the issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Today, we experienced our longest downtime in the ten years we have provided our document generation service. We apologize to all of our customers who were affected by the downtime. The saddest part of this incident is that we could have avoided it. We are improving our processes to avoid repeating our mistakes. **CHRONOLOGY OF THE INCIDENT** * Logged to Slack monitoring channel: 18:17 20.06.2024 UTC * Reports from customers via Crisp: 01:00 21.06.2024 UTC * Incident seen by technical team member: 08:00 21.06.2024 UTC * Issue resolved: 08:36 21.06.2024 UTC * Issue closed: 08:40 21.06.2024 UTC **IMPACT OF THE INCIDENT** The Cloud US Generator API service downtime affected all Cloud Service users, and none of the customers could generate documents during the incident. **THE ROOT CAUSE** The root cause of the incident was a misconfiguration of nginx. We synced configuration changes in ArcoCD on 20.06.2024 but didn't restart all the PODs. When new PODs were added or existing ones restarted, they failed to come up and caused the downtime. The existing alerting system logged issues to our Slack monitoring channel, but the health check considered the service running and didn't send out SMS notifications to the technical team. **LESSONS LEARNED** * After syncing configuration changes, we must restart all PODs and validate if everything is working as expected. This would have saved us from the incident.  * The US-based support team should call the European-based technical team if they validated the incident outside of the EU's working hours. This would have allowed us to act much faster and solve the incident. * All health checks need to include sub-systems and microservices on which they depend. This would have notified the technical team much faster and reduced the downtime.

Document Generation Service Major outage

Our document generation processes are taking longer time than usual. We have identified the root cause and working on solving the issue.

This incident has been resolved.

We are currently investigating this issue.

The issue has been identified and a fix is being implemented.

This incident has been resolved.

We had issues with our database instances, which prevented some users from using the service. We are investigating the root cause if the problem and working on improving reliability. The incident lasted from 7 a.m. to 8:30 a.m. EEST.

We are currently investigating this issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are experiencing issues with the API endpoints. We have identified the issues and will deploy a fix as soon as possible.

We have identified the root cause of the issue and the team is solving the issue.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.