54 Integrateio incidents · maj 2022 — official updates, affected components, duration and resolution details.
Wprowadzono rozwiązanie i monitorujemy wyniki.
Kontynuujemy monitorowanie dalszych kwestii. Do tej pory praca jest w porządku.
Automatyczne tłumaczenie oficjalnej aktualizacji incydentu.
Obecnie badamy tę kwestię.
Incydent został rozwiązany.
Automatyczne tłumaczenie oficjalnej aktualizacji incydentu.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
The incident has been resolved.
Obecnie badamy tę kwestię
Incydent został rozwiązany.
Automatyczne tłumaczenie oficjalnej aktualizacji incydentu.
We are currently investigating the issue.
The issue has been identified and it's due to our upstream provider and the affected scope is for newly created clusters. New cluster creation and job runs should now be working. We are working on unblocking clusters that are stuck on Creating.
The incident has been resolved.
Issue - There’s a Salesforce service disruption. During a service disruption, end users can’t access the service. Please refer to more on Salesforce status - https://status.salesforce.com/incidents/20004038 Affected integrate.io jobs having the Salesforce connectors (Source/destination) are failing with an error message : Error Occurred While Authenticating: { "error" : "system_down", "error_description" : "system may be currently unavailable" }
Please be advised that the Salesforce incident has been fully resolved. You can view the official status update here: https://status.salesforce.com/incidents/20004038 We kindly ask that you check your jobs utilizing Salesforce connectors; all integrations should now be fully functional. Should you have any further questions or continue to experience issues, please do not hesitate to reach out to us at [email protected]
We are currently aware of job failures on Postgres destination with a non-empty schema field due to a backwards compatibility issue. Please run the jobs on a newly created cluster and it should fix the issue. We apologize for the inconvenience caused.
The incident has been resolved. We will be following up with an RCA soon.
## Summary On **April 22, 2026**, between **02:20 UTC** and **07:33 UTC** \(~5 hours 13 minutes\), Postgres Destination jobs across [Integrate.io](http://Integrate.io) failed. The root cause was a partial rollback: our application layer rolled back successfully, but the deployment to our data processing engine did not, leaving the two components on mismatched versions. A corrective deployment at 07:33 UTC restored service. ## Customer impact * **Scope:** All Postgres Destinations, across all customers and all operation types. * **Window:** April 22, 2026 — 02:20 UTC → 07:33 UTC. * **Symptom:** Postgres Destination job runs failed at execution time. * **Status:** Fully resolved as of 07:33 UTC. Jobs succeed on retry or their next scheduled run. ## Timeline \(UTC\) * **02:20** — A rollback intended to address an earlier Postgres Destination issue deployed successfully to our application layer but silently failed to deploy to our data processing engine. The two components were left on incompatible versions. * **From 02:20 onward** — Postgres Destination jobs began failing for all customers. * **07:33** — Corrective deployment completed. Application and data processing engine are back in sync and Postgres Destination jobs resumed. ## Root cause Our application layer and data processing engine coordinate closely during job execution, so their deployed versions must match. A rollback deployed earlier today was split across both components; the pipeline reported the release as complete when only the application-layer half had actually deployed. For the duration of the skew, every Postgres Destination job failed. The underlying gap is in a **new CI/CD pipeline we recently introduced**, which did not correctly treat a partial deployment as a failure. ## Resolution A corrective deployment at 07:33 UTC brought the data processing engine in line with the application layer. Once versions matched, Postgres Destination jobs resumed normal operation. ## Next steps * Harden the new CI/CD pipeline so that any partial deployment is treated as a failed release and automatically reverts to the previously-matched versions rather than leaving components on mismatched builds. * Add pre-release validation that runs Postgres Destination jobs against every new version pair before promotion. * Publish learnings from this incident internally to inform future deployment-pipeline work. We apologize for the disruption and thank our customers for their patience.
We've updated the credentials for our Google integration as part of a security rotation. If you have a Google Ads, Google Analytics, Google Sheets or Google Drive connection, you may need to reconnect it: - Go to your Connections page - Open the affected connection - Click Reconnect and follow the Google authorization prompt If you have any trouble, our support team is here to help.
This incident has been resolved. If you encounter any issues related to Google Ads, GA analytics, or Google Sheets/Drive, please follow the steps suggested in this incident.
We are currently investigating this issue
The issue has been identified and a fix has been implemented. Please rerun the jobs on a newly created cluster.
The incident has been resolved.
The issue has been identified with our upstream provider and a fix is being implemented.
A fix has been implemented by our upstream provider and services are starting to recover. We are monitoring the results.
The incident has been resolved.
We have identified an issue causing job failures with the BigQuery V1 connector. This was introduced by a recent library upgrade that resulted in a regression affecting the connector. We are actively working on a fix. In the meantime, customers may migrate to the BigQuery V2 connector by following our documentation or by reaching out to our support team for assistance. We apologize for the inconvenience and appreciate your patience.
A fix has been implemented, and we are actively monitoring the results to ensure stability. Please proceed with running the jobs on a newly created cluster. This issue impacts packages that include the ExecuteSQL component and BigQuery V1. We apologize for the inconvenience and appreciate your patience.
This incident has been resolved.
We are currently investigating the issue.
The issue has been identified with the upstream provided and we are continuing to work with them to fix.
We’re seeing an improvement in our applications and currently working on restoring processing engine workloads.
We are continuing to recover the processing engine workloads and jobs should now be working moving forward.
The incident has been resolved. We will be writing a post-mortem report soon.
We are currently investigating the issue.
A fix has been implemented and we are monitoring the results.
The incident has been resolved. The root cause was an issue with our upstream provider, which prevented clusters and jobs from triggering as expected. We are taking steps to improve our monitoring systems to ensure earlier detection in the future. We sincerely apologize for the inconvenience.
Jobs are partially failed to be executed on the clusters due to an issue from our upstream providers. The issue was resolved at Nov 21, 07:30 UTC.
Due to AWS's incident the following components are still impacted, SFTP To Go's audit logs and Cron To Go's processing engine For more details please see below AWS status- https://health.aws.amazon.com/health/status
This incident has been resolved.
We are currently investigating the issue.
The incident has been resolved. The root cause was an issue with our upstream provider, which prevented clusters and jobs from triggering as expected. We are taking steps to improve our monitoring systems to ensure earlier detection in the future. We sincerely apologize for the inconvenience.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results. Job workloads aren't affected and should still run as normal.
The incident has been resolved.
We are currently investigating the issue.
The issue has been identified and a fix has been implemented. We are currently monitoring.
The incident has been resolved. The issue was caused by an issue on our upstream provider which caused clusters and jobs to not trigger. We will be working on improving our monitoring systems so we can detect the issue early. We apologize for the inconvenience caused.
We are currently investigating the issue.
A fix has been implemented and we are monitoring the results. We apologize for the inconvenience caused.
The incident has been resolved.
Hi All, We are experiencing an issue with our Reverse SSH connections, and the jobs are failing. Action: Investigating the issue
The issue has been identified, and please find the fix below to resolve the connectivity issue. Please delete/refresh the entry on your known_hosts file. You can do either of the following: Delete the specific line on ~/.ssh/known_hosts file which contains virginia-tunnel.xplenty.com. Make sure to save. Execute ssh-keygen -R "[virginia-tunnel.xplenty.com]:50683" on the tunnel instance
RCA: It looks like an issue one of our upstream providers. Follow the steps shared to resolve the issue. Detailed RCA will be shared soon.
RCA: Initial findings: There seems to have been an issue with our upstream provider, which we're working with them to identify why. The above solution resolves the issue.