51 Ironio incidents · October 2016 — official updates, affected components, duration and resolution details.
We are currently investigating issues with Iron dashboard (hud-e). IronMQ, IronWorker and IronCache services should work fine. If you have any questions - write us to [email protected]
This incident has been resolved.
We are experiencing issues with upstream provider. Currently in touch with them.
A fix has been implemented and we are monitoring the results.
We are continuing to monitor for any further issues.
This incident has been resolved.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results
This incident has been resolved.
Our engineering team is actively resolving the situation, and undertaking necessary steps to recover the system. We expect to post an update momentarily.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
We are continuing to monitor for any further issues.
This incident has been resolved.
Iron has completed a review of its systems and the Log4j security issue does not affect Iron's services.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
IronWorker customers may experience delays while pushing the tasks. We are currently investigating the issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
This incident has been resolved.
IronWorker customers could experience issues while pushing tasks to Public Cluster on March 14 between 12:00 am UTC and 6:30 am UTC. We've resolved the issue. Our development team is monitoring the situation.
Amazon performed an upgrade of our PostgreSQL DB version. This DB stores logins, passwords, tokens and other information about our customers. While upgrading they shutted down the database instance, performed the upgrade, and restarted the database instance. As a result, you could see HTTP 401 errors in your logs between 09:28 am UTC and 09:45 am UTC. Here is more information: https://forums.aws.amazon.com/ann.jspa?annID=8176
On May 1, at 11 am UTC we identified the issue with our autoscale functionality: there were errors while starting our IronWorker service on new instances but it was working fine on existing ones. After investigating into the errors thrown, we found the issue was happening while pulling our docker images from DockerHub. The root cause was problems on dockerhub side information about which you can find on their status page: https://status.docker.com/ In order to resolve the issue, we have manually launched our service on new instances and temporarily disabled the autoscale functionality.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
This incident has been resolved.
Due to a database upgrade issue, a portion of our IronWorker customers are experiencing issues with certain API commands. We've identified the issue and are in the process of resolving.
We are continuing to work on a fix for this issue.
Migration is still in progress. This is taking more time than expected but we're monitoring it closely.
The migration has completed and service has returned to normal.
**Overview** On May 13th, at 03:29 UTC, we began routine database upgrades. During the upgrade process we noticed errors in our logs indicating certain queries weren’t able to complete successfully. **What went wrong** After investigating into the errors thrown, we found data anomalies in our Production data set that didn’t exist in our Staging data set. This difference resulted in slow queries and errors that cascaded into service interruptions for a subset of our customers. **What we're doing to prevent this from happening again** Moving forward we’re taking steps to ensure our Staging data set is 100% up to date with our Production data set. If the copies of the data were exact, this would have been caught in Staging and wouldn’t have caused a disruption in service. **Resolution time** The incident was resolved at 11:49 UTC
Amazon has scheduled for maintenance one of the instances where our scheduler is running. They say that the instance will be unavailable for 2 hours: on November 7, from 12:00 am to 2:00 am UTC. We're going to run another scheduler on another instance at that period of time.
This incident has been resolved.
The issue has been identified and a fix is being implemented.
This incident has been resolved.
We are currently investigating this issue.
We are continuing to investigate this issue.
This incident has been resolved.
**Overview** On August 6th, at 15:07 UTC, we noticed connectivity issues across our network. These connectivity issues caused IronMQ to degrade into an unhealthy state which rendered the service un-usable. **What went wrong** At 12:49 AM PDT, the vendor who we rely on for DNS \(AWS Route 53\) experienced issues. In-network connectivity was broken and many components of our network were unable to communicate with each other. When the vendor issue was resolved at 1:04 AM PDT, the issue persisted within our network due to caching and TTL issues. **What we're doing to prevent this from happening again** * We identified the places within network that could have caused this issue and reviewed their caching strategies and TTL times. Multiple cache times were too aggressive and we’ve increased timeouts in the necessary places. We’re testing various failure scenarios within our staging network to confirm the validity of these timeout values. * We’re currently discussing backup DNS strategies as a team and will be posting updates on our blog about our strategy moving forward, and, continued progress. **Resolution time** The incident was resolved at 16:04 UTC.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.