49 Digital Pigeon incidents · March 2016 — official updates, affected components, duration and resolution details.
We are investigating this issue.
We are continuing to investigate this issue.
The issue has been identified and a fix is being implemented.
A cache server had issues. A restart fixed the problem. Everything seems to be back to normal. Watching for the next 15 minutes to verify.
This incident has been resolved.
We are currently investigating an issue affecting the ability to load upload details.
We have identified and mitigated the issue and are currently monitoring the situation.
This incident has been resolved.
We are currently investigating an issue that is impacting recipients from accessing files that are sent to them.
We have identified that the issue is only affecting customers that are using Apple Push Notifications, and are working on a fix. To mitigate the issue customers can temporarily disable 'Mobile Alert' notifications in Digital Pigeon | Management | Profile | Notifications.
A fix to the Apple Push Notification issue has been deployed and we are monitoring the results.
This incident has been resolved. Apple Push Notifications for events within Digital Pigeon are now functioning correctly.
After further investigation, we have narrowed down the issue and understand it only occurs when file downloads have been disabled by the sender. We are aiming to deploy a fix shortly.
This incident has been resolved.
We are currently investigating this issue.
This incident has been resolved.
We are currently investigating an where customers that have set up full custom domains are unable to access uploads.
Closing the issue as monitoring confirms the issue have been solved.
We are investigating an issue that's causing potential delays with accessing files
We have now solved the issue, we are still monitoring for now. Some initial access to an upload pickup page may have failed but then worked after a browser refresh.
An update to the application this morning has caused issues which we are identifying now.
We have rolled back the update from this morning to resolve the issues. Reloading or restarting the desktop application will be required if you have issues forwarding files.
Attempted access to files on Oceana servers are currently being blocked by Google chrome and incorrectly flagged as ‘phishing’ which is not the case. We are investigating with google and working on mitigation now. It is safe to use the 'visit website' under the 'More Details' section. Current workarounds right now are: - Upload to another region (ie US West) - Enable 'Cloudfront Optimized Routes Downloads' in settings if possible
We are continuing to investigate this issue.
We have now been delisted by Google, we're still monitoring the situation but warnings have ceased.
This incident has been resolved.
We are currently investigating the cause.
A fix has been implemented and we are monitoring the servers.
Issues have been resolved, application functioning normally.
We are investigating an issue that's preventing email delivery to recipients from initial upload. WORKAROUND: If you have recipients who have not received a notification email you can click 'Resend' from the recipients tab and the email will be delivered. For new uploads you can upload and hit send first and then add recipients afterwards to ensure they are emailed.
We have identified the issue and it's limited to accounts with Apple Push Notifications enabled. We are working on a fix now.
We have a fix ready to deploy and will be ready within the hour
Servers have been updated with the fix for both email delivery and recipient pickup page time-out for users with iOS push notifications enabled.
This incident has been resolved.
We are investigating an issue prevent access to the application
This incident has been resolved.
We apologise for the inconvenience caused by this issue. We tracked the problem down to a configuration change that caused unexpected network issues that were intermittently affecting some customers. We have updated the configuration to prevent this issue going forward and will add further checks to prevent this situation happening again.
We are currently investigating this issue.
We have identified and rectified a server issue that caused slow response times and in some cases loss of access to the application. We are currently closely monitoring the servers but access is restored.
Incident is resolved, we will provide a triage report soon.
An incident at AWS resulted in the majority of the application servers to be taken out of action. While the remaining application servers took up the load and were able to response to most requests, with elevated response times, a configuration issue meant it took an 45 minutes before the replacement application servers were available. We will be deploying infrastructure fixes this weekend which will avoid this scenario from repeating again in the future.
We are currently investigating this issue.
We have restored service to the application now and are monitoring the situation carefully.
This incident has been resolved.
We are currently investigating this issue.
The app is responding again. We are continuing to investigating the potential root cause of the issue.
A fix has been implemented and we are monitoring the results.
We are closing the incident as the app has been stable for the last hour.
We are currently investigating this issue.
The app is responding again, we are continuing to investigate the underlying cause.
We believe we have identified the source of the issue and we are monitoring the fix.
We are continuing to monitor for any further issues.
This incident has been resolved.
We are currently investigating issues with app stability.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Our hosting provider Amazon AWS is reporting wide spread issues with the Oceania (Sydney) data center which are affecting file related tasks like Previews, Zip files, etc. We are monitoring the incident with Amazon AWS (https://status.aws.amazon.com/) and will keep this incident updated as we get more info. In the mean time we recommend temporarily switching to the 'USA (West)' or 'S.E. Asia (Singapore)' file server regions which are unaffected. To switch regions head to 'Manage' -> 'Account' -> 'Defaults', select your desired region then click 'Save Changes'. Note that this change only applies to new uploads.
Amazon AWS has provided a brief update indicating that they expect the issue to be resolved within the next 2 hours. The Oceania Digital Pigeon file processing servers are still operating but at a reduced capacity which is causing delays processing previews, zip, etc.
Amazon AWS is reporting the restoration of services in it's Sydney data center after a major issue with a component that controls the data center networking layer. The file task backlog has been processed and all our system seem to be functioning as expected. We are continuing to monitor while Amazon AWS holds their incident open. Please revert any temporary changes you made to your accounts default file server region as a result of this incident. To switch regions head to 'Manage' -> 'Account' -> 'Defaults', select your desired region then click 'Save Changes'. Note that this change only applies to new uploads.
This incident has been resolved.
Continuing from the previous incident (https://digitalpigeon.statuspage.io/incidents/r886w5d2t611) which was prematurely closed based on feedback from Amazon AWS. We are continuing to work with Amazon AWS to resolve the issue. A minority of users report that uploads, download and previews are failing. Investigation suggests that the effected users are experiencing issues due to problems resolving Domain Name Server* (DNS) queries to Amazon S3 services (our hosting provider). DNS is a core component of the internet and is usually provided and configured auto-magically by your ISP when you connect to internet. Unfortunately being such a low level part of the internet puts it completely outside of our control. As a temporary work-around for the issue we recommend switching to either the Cloudflare or Google DNS servers which are confirmed working (and are generally faster than those provided by your ISP). > Cloudflare MacOS: https://developers.cloudflare.com/1.1.1.1/setting-up-1.1.1.1/mac/ > Cloudflare Windows: https://developers.cloudflare.com/1.1.1.1/setting-up-1.1.1.1/windows/ > Google MacOS: https://developers.google.com/speed/public-dns/docs/using#mac_os > Google Windows: https://developers.google.com/speed/public-dns/docs/using#windows * DNS converts human readable host names (e.g. digitalpigeon-dp-us-west.s3.amazonaws.com) into a series of numbers that computers understand. Whenever your computer attempts to make a connection on the internet the first thing it does is contact your configured Domain Name Server and attempt to resolve its host name into an actual network address. If this step fails then, which is what's happening in this case, the connection will fail.
Amazon AWS reports that there service teams are actively investigating the issue. Unfortunately due to the complexity of the issue they are unable to provide an ETA. To illustrate the complexity of the issue check out: https://www.whatsmydns.net/#A/digitalpigeon-dp-us-west.s3.amazonaws.com
Amazon AWS is continuing to implement fixes to mitigate the issue... Amazon have also suggested a code change to help mitigate the problem. We are currently working on implementing and testing that change and hope to have it deployed within the hour.
We are deploying the code change suggested by Amazon AWS to help mitigate the issue.
We have deployed the code change as suggested by Amazon AWS. The change adds extra region information to all our Amazon S3 requests which according to Amazon will help mitigate the issue. We still recommend switching to either the Cloudflare or Google DNS servers which are confirmed working and are generally faster and apparently are a lot more reliable than those provided by your ISP. For more info on Cloudflare's DNS service check out https://www.cloudflare.com/learning/dns/what-is-1.1.1.1/. For setup instructions see: > Cloudflare MacOS: https://developers.cloudflare.com/1.1.1.1/setting-up-1.1.1.1/mac/ > Cloudflare Windows: https://developers.cloudflare.com/1.1.1.1/setting-up-1.1.1.1/windows/ > Google MacOS: https://developers.google.com/speed/public-dns/docs/using#mac_os > Google Windows: https://developers.google.com/speed/public-dns/docs/using#windows We will continue to provide updates as we get info from Amazon AWS.
The code level change we made seems as suggested by Amazon AWS appears to have mitigated the issue. However, Amazon AWS is still reporting that the issue is under investigation. We are considering all system as operational at this point, however we will continue to monitor the situation and provide updates. If you still experiencing issues please let us know.
We are considering this issue resolved as we have had no new reports of problems in the last 12 hours.
We have had small number of users report that uploads are failing immediately with the message 'Upload issues detected'. Investigation suggests that the uploads are failing due to problems resolving Domain Name Server (DNS) queries. A Domain Name Server is a core component the internet and is usually provided and configured auto-magically by your ISP when you connect to internet. The Domain Name Server converts human readable host names (e.g. digitalpigeon-dp-us-west.s3.amazonaws.com) into a series of numbers that computers understand. Whenever your computer attempts to make a connection on the internet the first thing it does is contact your configured Domain Name Server and attempt to resolve its host name into an actual network address. * Temporary Workaround * As a temporary work-around for the issue we recommend switching to either the Cloudflare or Google DNS servers (which are confirmed working). Instructions: Cloudflare MacOS: https://developers.cloudflare.com/1.1.1.1/setting-up-1.1.1.1/mac/ Cloudflare Windows: https://developers.cloudflare.com/1.1.1.1/setting-up-1.1.1.1/windows/ Google MacOS: https://developers.google.com/speed/public-dns/docs/using#mac_os Google Windows: https://developers.google.com/speed/public-dns/docs/using#windows
A ticket has been raised with our hosting provider Amazon AWS. They are currently investigating the issue...
Please see http://status.digitalpigeon.com/incidents/s3yhqmfk687j