50 Noko incidents · July 2014 — official updates, affected components, duration and resolution details.
We're seeing a delay on some background tasks due to a network issue in our data center and are working on fixing the issue.
We've moved background task processing from the affected servers temporarily to other servers while we work on fixing the underlying issue.
This incident has been resolved.
We are currently investigating this issue.
We've identified an issue with a stuck background worker and have restarted it. Noko is currently catching up; there should be no longer an impact on performance. We will continue monitoring.
Some low-priority background tasks are still catching up (which has no impact on using Noko). Everything will be completely back to normal in several hours!
Noko's Back up, sorry about the hiccup!
We're looking into Noko being very slow to load or timing out completely.
We've implemented a fix and we're monitoring the response times to make sure everything's squared away. Thanks so much for your patience!
It looks like the fix has stuck; but we're still going to be monitoring it
Noko's hosting provider is encountering an issue: https://status.digitalocean.com/incidents/xkpsn9hbnnlz. We're monitoring their status updates and will keep you posted.
Our monitoring systems are showing green again, but we're keeping a close eye and will keep you updated.
We're still monitoring the situation, and will open a new one if needed
We're aware of Noko being very slow to load or timing out completely, due to an outage from a major Internet backbone provider, which is affecting our hosting. We're monitoring the situation, and will keep you updated
It looks like our hosting provider was able to re-route traffic, but we're keeping an eye on it!
We're looking into Noko being very slow to load or timing out completely.
Sorry about the timeouts! The servers look like they're back to normal, but we're monitoring them to make sure everything's working.
Our hosting provider is looking into networking issues in their datacenter; there may be some intermittent packet loss and network latency. We'll update this incident as soon as they give the all-clear.
This incident has been resolved.
We're looking into Noko being very slow to load or timing out completely.
Sorry about the timeouts! A fix has been implemented and we're monitoring the servers to make sure everything's working.
We're looking into Freckle being very slow to load or timing out completely.
Sorry about the timeouts! A fix has been implemented and we're monitoring the servers to make sure everything's working.
We're looking into Freckle being very slow to load or timing out completely.
We've solved the timeouts and slowdowns, and will continue to monitor the app.
Freckle is down and not loading. We're identifying the root cause right now and will keep you posted.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Freckle's hosting provider is currently experiencing network connectivity issues. They have implemented a fix, and we are monitoring the status of their fix. We'll let you know as soon as the issue has been resolved!
This incident has been resolved.
We're seeing 500 errors and are investigating.
Unfortunately our hosting provider is doing further emergency maintenance which affects connectivity to our main database, which in turn causes 500 errors in the web app.
This incident has been resolved.
Freckle is currently down, we're investigating.
Freckle is down due to emergency hardware maintenance of our hosting provider. We will post updates as we receive them.
Our hosting provider has finished maintenance. Freckle is all up and running, we will monitor.
This incident has been resolved.
Freckle's server infrastructure will be going through some upgrades. Freckle should be generally available, but will be offline for a short period of time (up to 10 minutes)
This incident has been resolved.
We're looking into Freckle being very slow to load or timing out completely.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We're looking into Freckle being very slow to load or timing out completely.
We've implemented a fix to improve Freckle's overall performance, and are monitoring the results.
This incident has been resolved.
Freckle is down and not loading. We're identifying the root cause right now and will keep you posted.
We've been informed by our hosting provider that there's an issue with their hosting environment and they're working on resolving it. We'll keep posting updates here.
Our hosting provider's engineering team is continuing to work on fixing the issue. We'll update here as soon as we get more information.
We're still waiting for an update from our hosting provider, and will update our status page as soon as possible.
We're still waiting for an update from our hosting provider, but our monitoring tools show the web servers are online and accepting requests. However, the database is still in a degraded state and we're waiting for it to return to normal/actively monitoring it.
Our hosting provider still shows the outage, but our monitoring systems show Freckle as online and our engineering team is actively monitoring Freckle's status. Thanks to everyone for your patience as we've been dealing with this outage, and we're continuing to monitor Freckle (as always!)
Our hosting provider has resolved the underlying issue. We'll keep monitoring just in case.
This incident has been resolved.
We just got alerted by our monitoring that Freckle is down and we're investigating.
We've been informed by our hosting provider that there's an issue with their hosting environment and they're working on resolving it. We'll keep posting updates here.
We're in contact with our hosting provider's engineering team and working to identify and resolve the problem. The issue specifically affects our web servers only; the database/backend is functioning normally (but can't be accessed without the web front-end).
Our hosting provider's engineering team is continuing to work on fixing the issue. We'll update here as soon as we get more information.
Our hosting provider are right now replacing defective hardware on the web host—we'll keep you posted on the progress and update here as soon as we know more (and hopefully have an ETA for when Freckle will be back up soon!).
And Freckle is back! Yay! We'll keep monitoring and post updates as needed.
We've been given the A-OK from our hosting provider that the hardware issue has been fixed. All systems are working normally now. We'll of course continue to monitor (as we always do!). Happy weekend!
Due to an outage with DocRaptor, the 3rd party service we use to generate PDFs, PDF exports of reports and invoices are unavailable. We are actively monitoring the 3rd party service and will notify you when the outage is over.
PDF generation is working again! We'll be keeping an eye on it, though!