Knak 장애 기록

1. 2026년 10월 21일 오후 1:28 UTCinvestigating

   우리의 공학 및 보안 팀은 PHP2SHELL에 응답하고, 최근에는 구성 요소 Knak를 포함하여 널리 사용되는 웹 소프트웨어에 취약점을 공개했습니다. 우리는 적극적으로 우리의 시스템을 모니터링하고 보호 조치를 적용하고있다. 데이터를 보호하는 것은 먼저 온다. 우리는 더 많은 행동을 필요로하는 위협을 볼 경우, 우리는 약간의 사전 통지로 Knak를 오프라인으로 복용 할 수 있습니다. 우리는 어떤 혼란이 당신의 팀의 능력에 영향을 미칩니다. 우리는 inconvenience를 위해 사과하고 우리의 상태 페이지에 갱신을 유효하게 할 것입니다.

2. 2026년 10월 21일 오후 2:27 UTCinvestigating

   우리는 패치를 작동

3. 2026년 10월 21일 오후 2:28 UTCinvestigating

   우리는 패치를 작용합니다.

4. 2026년 10월 21일 오후 3:12 UTCinvestigating

   우리는 패치를 적용하고 현재 Knak를 온라인으로 가져 오기 전에 테스트하고 있습니다. 우리의 예비 조사는 Knak가 손상 된 표시가 없습니다. Knak가 돌아왔습니다.

5. 2026년 10월 21일 오후 3:32 UTCidentified

   문제는 식별되고 수정이 구현됩니다.

6. 2026년 10월 21일 오후 3:45 UTCmonitoring

   수정이 구현되어 결과를 모니터링하고 있습니다.

7. 2026년 10월 21일 오후 4:03 UTCmonitoring

   우리는 더 많은 문제를 모니터링하고 있습니다.

8. 2026년 10월 21일 오후 7:05 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 7월 16일 오후 7:17 UTCidentified

   문제는 식별되고 수정이 구현됩니다.

2. 2026년 7월 16일 오후 10:11 UTCmonitoring

   수정이 구현되어 결과를 모니터링하고 있습니다.

3. 2026년 7월 17일 오전 3:33 UTCresolved

   이 사건은 해결되었습니다.

공식 인시던트 업데이트를 자동 번역했습니다.

1. 2026년 6월 23일 오후 2:39 UTCidentified

   The issue has been identified and a fix is being implemented.

2. 2026년 6월 23일 오후 2:40 UTCidentified

   We are continuing to work on a fix for this issue.

3. 2026년 6월 23일 오후 2:42 UTCmonitoring

   A Claude outage is degrading Knak AI's performance. We are monitoring the provider's status updates.

4. 2026년 6월 23일 오후 4:14 UTCresolved

   This incident has been resolved.

1. 2026년 5월 8일 오후 1:26 UTCmonitoring

   A fix has been implemented and we are monitoring

2. 2026년 5월 8일 오후 3:37 UTCresolved

   This incident has been resolved

1. 2026년 3월 24일 오후 5:12 UTCinvestigating

   We are currently investigating the issue

2. 2026년 3월 24일 오후 9:07 UTCidentified

   The issue has been identified

3. 2026년 3월 24일 오후 9:07 UTCidentified

   The issue has been identified and a fix has been deployed and we are currently monitoring

4. 2026년 3월 24일 오후 9:08 UTCresolved

   This incident has been resolved.

1. 2026년 3월 6일 오후 2:40 UTCinvestigating

   We are currently investigating.

2. 2026년 3월 6일 오후 2:43 UTCidentified

   The issue has been identified and we are implementing the fix.

3. 2026년 3월 6일 오후 3:03 UTCmonitoring

   A fix has been implemented and we are now monitoring

4. 2026년 3월 6일 오후 3:05 UTCmonitoring

   We are continuing to monitor for any further issues.

5. 2026년 3월 6일 오후 3:09 UTCresolved

   This incident has now been resolved. Assets synced between 9:00 AM and 10:04 AM EST will have to be resynced.

1. 2026년 1월 28일 오후 3:51 UTCinvestigating

   We are currently investigating an issue with global UTM input field not visible

2. 2026년 1월 28일 오후 3:51 UTCidentified

   The issue has been identified and we are working to release a fix

3. 2026년 1월 28일 오후 4:25 UTCmonitoring

   A fix has been implemented and we are monitoring the release of the fix

4. 2026년 1월 28일 오후 7:09 UTCresolved

   This incident has been resolved.

1. 2026년 1월 26일 오후 10:38 UTCinvestigating

   We are currently investigating the issue.

2. 2026년 1월 27일 오전 1:11 UTCresolved

   A fix has been applied and we are monitoring

1. 2026년 1월 19일 오후 9:39 UTCinvestigating

   We are currently investigating an issue with the Knak tracking pixel

2. 2026년 1월 19일 오후 10:40 UTCmonitoring

   A fix has been implemented and we are monitoring

3. 2026년 1월 20일 오후 10:22 UTCresolved

   This incident has been resolved.

4. 2026년 1월 21일 오후 3:24 UTCpostmortem

   ### **Quick Summary** Between **January 9th and January 19th**, Knak experienced an issue with our analytics tracking pixel affecting data collection for a subset of clients. This was caused by a system upgrade intended to improve performance and remove rate limits. While the upgrade successfully improved traffic capacity, it introduced a data formatting error that prevented tracking data from being saved to our database. **The tracking pixel itself continued to function for end-users \(email recipients\), ensuring no broken images or errors were visible.** We have since rolled back the change, restored partial data, and implemented new safeguards to prevent this from recurring. ### **What This Means** For clients who have the Knak Tracking Pixel enabled, **we were unable to collect performance data between January 9th at 10:32 AM EST and January 19th at 5:26 PM EST.** While your emails were delivered successfully and the pixel image loaded correctly for your recipients, the "open" event data was not stored in our database. Consequently, analytics reports for campaigns sent or active during this specific window will not reflect the actual open rates or engagement metrics. This did not impact downstream reporting from marketing automation platforms or any performance data Knak collects from those systems. This issue only affected assets with the “Knak Tracking Pixel” enabled in settings. No action is required from Knak clients. ### **Root Cause** In December, we identified that our previous tracking proxy service had a strict request limit that throttled data collection during high-volume campaigns. To support higher traffic and remove these limits, we migrated to a new, more scalable proxy service on January 9th. The root cause of the incident was a data formatting mismatch between this new proxy service and our data processing software. * **The Mismatch:** The new service delivered tracking information \(such as timestamps\) in a different structural location than the previous service. * **The Failure:** When our processing software could not locate the timestamps in the expected place, it defaulted the values to the text string "Unknown." * **The Data Loss:** Our analytics database is designed to store timestamps strictly as numbers. When the system attempted to save the text "Unknown" into a number-only field, the database rejected the incoming data batches. This resulted in a scenario where the tracking pixel successfully loaded for the end-user \(the email recipient saw no errors\), but the data was silently rejected by our database backend. ### **Timeline** _All times are in Eastern Standard Time \(EST\)_ * **January 9, 10:32 AM:** The infrastructure upgrade to the new proxy service is deployed to remove rate limits. Data collection stops due to the formatting mismatch described above. * **January 19, 3:38 PM:** Our team identifies a gap in success logs for tracking jobs and immediately begins an investigation. * **January 19, 4:06 PM:** The engineering team confirms that while requests are hitting the server, data is not persisting into the analytics database. * **January 19, 5:26 PM:** **Service Restored.** We roll back the infrastructure to the previous proxy service. Valid data collection resumes immediately for all new requests. * **January 20, 12:30 PM:** The team identifies the specific database rejection error \(String vs. Integer type mismatch\). * **January 20, 1:54 PM:** A code fix is deployed to ensure timestamps can never default to an invalid format, preventing this specific failure mode in the future. * **January 20, 2:01 PM:** Engineering began investigating methods to recover and backfill any partial data available from the outage window. ### **Learnings & Actions Taken** To prevent a recurrence, Knak is implementing the following changes to our engineering and monitoring protocols: 1. **Eliminating Silent Failures:** We will update our code to prioritize error visibility. While the tracking pixel will always load for the user \(to ensure the recipient experience is never impacted\), our backend systems will soon aggressively log and flag when default values are used instead of real data. 2. **Volume-Based Alerting:** We are implementing new alerts that trigger specifically when data flow into our database drops unexpectedly. Previously, our alerts focused on _server errors_ \(crashes\). Since the server was technically "running" \(just processing bad data\), no alarm sounded. The new alerts will focus on _data throughput_. 3. **Strict Data Validation:** We have added stricter validation logic to our ingestion pipeline to ensure that data types \(like timestamps\) are verified before they reach the database, preventing a single formatting error from rejecting a batch of data. 4. **Partial Data Restoration**: We have now began the process to partially restore some data between January 19th and January 20th

1. 2025년 11월 19일 오후 7:35 UTCinvestigating

   We are currently investigating an issue of PDFs not being able to download

2. 2025년 11월 19일 오후 8:16 UTCresolved

   This incident is resolved PDFs can be downloaded again

1. 2025년 10월 20일 오후 2:22 UTCinvestigating

   We are currently investigating this issue.

2. 2025년 10월 20일 오후 2:33 UTCmonitoring

   We have been experiencing a temporary outage due to the ongoing AWS outage https://health.aws.amazon.com/health/status. We are currently stable but are closely monitoring the situation.

3. 2025년 10월 21일 오전 11:31 UTCresolved

   This incident has been resolved.

1. 2025년 10월 8일 오후 7:12 UTCidentified

   The issue has been identified and a fix is being implemented.

2. 2025년 10월 8일 오후 7:12 UTCidentified

   We are continuing to work on a fix for this issue.

3. 2025년 10월 8일 오후 7:18 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 2025년 10월 9일 오전 12:11 UTCresolved

   This incident has been resolved.

5. 2025년 10월 9일 오후 3:16 UTCpostmortem

   ### Timeline On October 8th, at 1:27PM EST we deployed a package upgrade to our backup rendering service which is responsible for generating previews of pdfs and pngs. At 2:06PM EST we routed our production traffic to our backup rendering service. During which time it was monitored noting that the service was succeeding users were able to download pdfs and pngs At 3:01PM EST Knak Engineers were informed that the pdfs and pngs being download were blank pdfs and pngs. At 3:18PM EST Knak Engineers reverted the traffic back to the primary rendering service, which they then confirmed that pdfs and pngs were correctly downloading ### Root Cause An underlying package that our rendering service relies on has implemented a breaking change. It was believed that the package upgrade followed standard semantic versioning which would mean that this was minor change which would not impact production, it was later discovered it does not follow semantic versioning. This was not noticed in our non production environments as functionally from an integration standpoint our rendering service was taking in code and rendering a pdf/png. ### Action Taken * We have rolled back the change and went back to testing the package upgrade * We have documented how to manually end to end test our rendering service * We are working on implementing a fully automated end to end test for our application/rendering service

1. 2025년 6월 12일 오후 6:39 UTCidentified

   The issue is believed to be caused by a potential sub-processor outage.

2. 2025년 6월 12일 오후 8:42 UTCmonitoring

   The issue has been resolved and is being monitored.

3. 2025년 6월 12일 오후 10:49 UTCresolved

   This incident has been resolved.

4. 2025년 6월 19일 오후 6:09 UTCpostmortem

   On June 12, 2025 \(EDT\), our AI-powered features experienced a temporary service disruption. Our team immediately identified the cause as an outage with one of our sub-processors and monitored the situation until service was restored by the external provider. **Root Cause:** The issue was caused by a temporary service outage on the Google Cloud Vertex AI platform, an external service that our AI features depend on to function.

1. 2025년 4월 4일 오후 2:10 UTCresolved

   This incident has been resolved.

2. 2025년 4월 4일 오후 7:38 UTCpostmortem

   On April 3rd at 7:26 AM EDT, our application experienced an incident resulting in unresponsiveness and the inability to serve requests. Upon immediate awareness, remediation efforts were initiated, and a temporary fix was implemented by 8:25 AM EDT, at which time resolution of the underlying cause was believed to be achieved.  On April 4th at 8:48 AM EDT, the application again became unresponsive. A swift response facilitated the deployment of another temporary fix by 9:03 AM EDT on April 4th. Subsequent investigation revealed the root cause to be the unintended accumulation of temporary server operating logs, leading to disk space exhaustion on the application server. A permanent fix has since been implemented to prevent further log accumulation, and no additional space consumption has been observed. No action is required from our customers regarding this incident. **Root Cause:** During runtime our logging system was failing to send logs to our external logging tools this prompted the server to fail back to logging within the server, the logging directory was ephemeral storage so a new deployment would temporarily fix the issue. It was not until April 3rd at 7:26AM EDT that our temporary storage had completely filled causing the server to not have any disk swap space; rendering the server to be unable to respond to request **Actions:** 1. We have since fixed the issue with our logging so that we are not failing back to logging on the server 2. We have also added strict alerting on our temporary storage on our server to ensure that we are alerted if we are soon going to run out of space

1. 2025년 4월 3일 오후 12:09 UTCinvestigating

   We are currently investigating this issue.

2. 2025년 4월 3일 오후 12:29 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

3. 2025년 4월 3일 오후 12:57 UTCmonitoring

   We are continuing to monitor for any further issues.

4. 2025년 4월 3일 오후 1:30 UTCresolved

   This incident has been resolved.

5. 2025년 4월 4일 오후 7:36 UTCpostmortem

   On April 3rd at 7:26 AM EDT, our application experienced an incident resulting in unresponsiveness and the inability to serve requests. Upon immediate awareness, remediation efforts were initiated, and a temporary fix was implemented by 8:25 AM EDT, at which time resolution of the underlying cause was believed to be achieved.  On April 4th at 8:48 AM EDT, the application again became unresponsive. A swift response facilitated the deployment of another temporary fix by 9:03 AM EDT on April 4th. Subsequent investigation revealed the root cause to be the unintended accumulation of temporary server operating logs, leading to disk space exhaustion on the application server. A permanent fix has since been implemented to prevent further log accumulation, and no additional space consumption has been observed. No action is required from our customers regarding this incident. **Root Cause:** During runtime our logging system was failing to send logs to our external logging tools this prompted the server to fail back to logging within the server, the logging directory was ephemeral storage so a new deployment would temporarily fix the issue. It was not until April 3rd at 7:26AM EDT that our temporary storage had completely filled causing the server to not have any disk swap space; rendering the server to be unable to respond to request **Actions:** 1. We have since fixed the issue with our logging so that we are not failing back to logging on the server 2. We have also added strict alerting on our temporary storage on our server to ensure that we are alerted if we are soon going to run out of space

1. 2024년 9월 27일 오후 3:52 UTCinvestigating

   We are currently investigating this issue.

2. 2024년 9월 27일 오후 3:53 UTCidentified

   The issue has been identified and a fix is being implemented.

3. 2024년 9월 27일 오후 3:57 UTCresolved

   This incident has been resolved.

1. 2024년 9월 4일 오후 6:30 UTCresolved

   At 10:10AM EDT today, a release of Knak was published which caused issues with custom column sizing. As soon as Knak was made aware of the issue, we rolled back the release (at 11:20AM EDT). Any email, landing page, theme, or module opened during that time (10:10AM-11:20AM EDT) that had custom column sizing would have been affected. For emails and landing pages opened during that time that have issues with column sizing, the recommendation is to use the version history feature in the info tab (https://help.knak.io/en/articles/4449817-email-landing-page-version-updates) to roll the asset back to a version prior to 10:10AM EDT this morning. For themes and modules that are affected, please communicate with support so that they can roll those versions back in your instance.

1. 2024년 7월 25일 오후 1:18 UTCidentified

   The issue has been identified and a fix is being implemented. A workaround for the moment is to disable uBlock Origin for the Knak site specifically

2. 2024년 7월 25일 오후 3:55 UTCresolved

   The incompatibility with uBlock Origin in today's release has been resolved. Note that the issue was only affecting users that were using privacy blockers such as uBlock Origin.

1. 2024년 4월 5일 오후 1:45 UTCresolved

   The application was unavailable from 8:16AM to 9:00AM.

2. 2024년 4월 5일 오후 1:47 UTCpostmortem

   **Root Cause:** During the last application build, a failure transpired while attempting to write the JavaScript assets to the S3 bucket essential for our CDN functionality. Consequently, when the deployment took place, our main application was referencing assets that were absent. **Actions:** 1. We are reordering our build sequence to ensure that the copying of assets to S3 transpires prior to the push to ECR, thereby implementing a fail-safe mechanism. 2. We are enhancing our rollback procedure to simplify its execution for all parties, enabling headless operation. 3. We will be reviewing and addressing observability gaps for builds, and release monitoring.

1. 2023년 7월 13일 오후 12:37 UTCresolved

   Images edited in the builder before December 12th, 2022 used a service that would generate a URL based on the transformations applied to the images. Going forward, any new images edited will not use this service. As part of our security reviews, it was found that we needed to add trust hosts in order to validate that requests were being serviced from the right DNS name(s). Unfortunately, in tightening the restrictions and creating an allow list that included enterprise.knak.io, adding the edited-images.knak.io was missed and requests to this legacy service started failing.

2. 2023년 7월 13일 오후 12:37 UTCpostmortem

   * We will be formalizing and streamlining our incident response process since some of the initial response to the issue could have been more prompt.  Formalizing the process also makes it so that we don't skip steps \(i.e. putting up a status page for the partial outage\) * We are making changes to our observability platform to ensure that these types of issues are addressed in a more timely manner before customers bring it up.