31 Tettra incidents · Genver 2018 — official updates, affected components, duration and resolution details.
We are experience server issues with our hosting service Heroku (Salesforce). We are actively investigating the issue, and will update as soon as we have identified the issue. Customers will have issues logging in and using Tettra.
We have been able to identify the issue caused by heroku, and make a work around to get tettra accessible once again
We are currently investigating a major outage impacting our application. This is due to an upstream infrastructure failure with our hosting provider, Heroku. Users may be unable to access any part of the application at this time. We’ve reached out to Heroku support and are closely monitoring their incident updates.
We’ve confirmed that the outage is linked to a broader Salesforce incident that began at approximately 06:17 UTC and is impacting multiple services, including Heroku—our infrastructure provider. Salesforce has identified ongoing issues with multi-factor authentication (MFA) across several of their clouds. While Heroku is affected, the root cause remains under investigation. They are coordinating with a third-party cloud provider and reviewing recent changes to the Heroku environment. As a result, customers will continue to be unable to access app.tettra.co until Salesforce resolves these issues. We are actively monitoring the situation and will update customers as soon as we have more information.
We're continuing to investigate the Heroku outage and are looking for a solution. Our web app continues to be unreachable. We’ll provide an update in 30 minutes or sooner if additional information becomes available.
We're continuing to work on a fix. We have no other update right now. We’ll provide another update in 30 minutes or sooner if additional information becomes available.
Heroku has identified the likely cause of the issue and is working on a solution. We’ll provide an update when additional information becomes available.
The app is up and running, with degraded performance while Heroku fully fixes the issue. Tettra will be accessible, but expect some slower response times until the issue is fully resolve
We are currently investigating an issue casuing a partial outage to the Tettra app. The Tettra app itself still loads, but pages are not loading properly some of the time.
We are continuing to investigate this issue.
We have identified and fixed the issue. We are currently monitoring for on-going problems.
We are investigating an issue with our billing system that is defaulting some paid teams to the billing page.
We have identified the issue with our billing system that is blocking some paid teams from accessing their customers. We are currently in the process of fixing it.
We have restored access to all the affected accounts and are monitoring the situation.
The issue has been identified and a fix is being implemented.
This incident has been resolved.
Read-only users are not able to view pages or the sidebar right now. We're working on rolling back the deploy the caused this issue.
A fix for this issue has been deployed.
This incident has been resolved.
This incident has been resolved.
We were having issues with our SSO provider integration. This has now been resolved.
Our hosting provider Heroku is currently experiencing DNS issues, which is affecting the Tettra app’s ability to properly load. We are currently working to identify a potential workaround. https://status.heroku.com/incidents/2453
A workaround for the DNS issue has been implemented and the app is accessible as per normal
The Tettra webapp is not responding to requests. We are investigating the issue.
Our hosting provider, Heroku, is experiencing an outage.
Connectivity to the webapp is recovering.
We are continuing to monitor for any further issues.
This incident has been resolved.
Our transactional email service provider, Mailgun, is experiencing API issues causing our email delivery to fail. We are monitoring the situation. https://status.mailgun.com/incidents/0hgvxpzzhvcq
This incident has been resolved.
Slack is currently experiencing system-wide issues. Tettra users may be unable to sign in using Slack or use the Slack integration within Tettra. We are monitoring the situation in case there is anything we can do to mitigate these issues with Slack.
We are continuing to monitor Slack. You can see their own status page, here: https://status.slack.com/
Slack is starting to come back online, so we hope all issues with our integration will resolve soon.
We believe slack's outage is now contained to parts of Slack that do not impact Tettra. Please let us know if you see any further issues with the Slack integration.
All emails from our application have been suspended since Dec 24 approximately 22:24 UTC due to a spam attack. This includes all automated notification emails, invitation emails, and page share emails. We have identified and suspended the malicious account(s) and are in the process of re-enabling emails.
Links in existing/delivered emails are also disabled while we continue to work on a fix.
Email delivery and links are operational again. The 'share' page via email functionality has been temporarily disabled while we mitigate further spam attempts.
We've identified an issue with our content embedding provider that is causing content embedding in pages to fail. We'll update this issue once the problem is resolved.
Embedly is still working on a fix. At this time there is no ETA as to when content embedding will be available again. We will update here as we know more. Original incident: http://status.embed.ly/incidents/tdkr78g2qq5x
Content embedding functionality has been restored. Thanks for your patience.
From approximately Friday 4/17 5pm EST to Saturday 11am EST, readonly users were not able to access Tettra pages on the webapp.
app.tettra.co is unavailable due to problems with our service provider. We are investigating the issue.
We are continuing to monitor the app as it recovers.
Access to the webapp has returned to normal. We will continue to keep an eye on the underlying Heroku incident that caused this and the outage earlier today: https://status.heroku.com/incidents/1974
Visitors to app.tettra.co are seeing a Cloudflare error. We are investigating the cause.
Our service provider Heroku is experiencing widespread platform issues. We will update as we have more information.
The app is loading normally again. We'll keep monitoring until we hear from our hosting provider.
We are continuing to monitor for any further issues.
This incident has been resolved.
We are getting alerts of degraded performance to the point where some users are unable to view their content or log in. We are investigating.
We have narrowed down the issue to problems connecting to where we store data. We have contacted our service provider's support team but are also further investigating ourselves.
We have successfully mitigated the connection issues, and will work to assess what we can do to prevent such issues in the future.
We're aware of intermittent 500 errors being caused by issues with our billing provider. We will update you again when the problems have been resolved.
Our billing provider has recovered. Please let us know if you see any more 500 errors.
We are currently investigating an issue with our DNS provider Cloudflare.
We are continuing to investigate this issue.
Cloudflare has implemented a fix for the network issue and is currently monitoring the results. The application is available again. We are continuing to monitor the situation.