Populi 장애 기록

Our primary datacenter is experiencing network latency so Populi may appear to load slowly... we're in contact with their ops team and will post updates.

Out primary datacenter is weathering a large DDOS attack... they're actively working to mitigate it and we'll continue to monitor.

Our datacenter successfully mitigated the attack and all systems are back to normal. Thanks for your patience!

For ~20 minutes during a deployment that was supposed to be zero downtime, some customers experienced issues with CSS and LTI. To the best of our knowledge, these issues are resolved and we are monitoring.

Our main datacenter is currently experiencing network issues - we will post status updates when we understand the scope of the problem.

Our primary datacenter is dealing with a large-scale DDOS attack but we're hopeful they will have it mitigated shortly!

Our datacenter appears to have mitigated the DDOS attack, but we'll monitor for a while...

Everything seems to be back to normal so we're resolving this issue - thanks for your patience!

A misbehaving server caused some customers to be unable to access Populi a few minutes ago - it's been removed from rotation so everything should be back to normal but we'll continue to monitor.

Error rates have been normal for a while so we're closing this incident.

Our main datacenter is currently performing network maintenance and some users are not able to connect to Populi...

We are continuing to investigate this issue.

The site appears to be available again for all users... we'll monitor for a while.

We haven't seen any issues for a while so we're marking this incident resolved - thanks for your patience!

Populi is currently unavailable from several larger network providers such as Comcast... this appears to be a widespread Internet problem and our datacenter is fine (just unreachable for many users). We are continuing to investigate and will post updates here...

The issue appears to have been with an Internet connection into Populi's primary datacenter - failover has occurred and we're seeing usage statistics return to normal. We'll continue to monitor....

Traffic from all internet providers looks good but we will continue to monitor for a while before resolving this incident... thanks for your patience!

Everything continues to look good, so we're closing this incident.

At 9:35am Pacific this morning, a critical caching server crashed and Populi was unavailable to the majority of our users for about 6 minutes as we failed over to another server. Sorry about that!

From roughly 4:20am - 4:50am Pacific time a large subset of Populi users were unable to access the service. This was because a weekly database consistency check started running at roughly 3am and eventually locked up a critical database server, leading to widespread system failure. We eventually managed to restart the database and restore service, and we'll be taking a close look at that check today to prevent it causing issues in the future!

We are currently investigating this issue.

Daylight savings time is tricky and apparently broke logins for a bit - everything appears to be back to normal but we're keeping an eye on things...

This incident has been resolved - it was a timezone bug and we'll do our best to make sure it doesn't happen next year!

From roughly 11:30am to 12:30pm, some requests to Populi experienced delayed network connections or timed out - our apologies for the inconvenience! We've reconfigured some things and raised load-balancing limits to allow more traffic during peak times.

Today we've noticed a large number of delivery errors for outgoing SMS messages - we're working with phone carriers to get this resolved, but unfortunately we can't predict how long that will take...

We are continuing to work on a fix for this issue.

The carriers seem to have resolved this issue!

Populi uses Amazon Web Services to encode media and host files, and unfortunately Amazon is having some issues today... as soon as they recover completely our media/document encoding queues should start to catch up, but it may take several hours. Sorry about that!

We are continuing to monitor for any further issues.

Thanks for your patience - Amazon appears to be back to normal and our media encoding queues have all caught up!

From around midnight to 6:30am Pacific time, a subset of Populi users experienced a number of problems viewing and saving data. A routine data integrity tool that verifies data is replicated properly across our server clusters locked several tables and failed to unlock them as it normally does, which caused many writes to fail. However, because the vast majority of writes were still succeeding, we didn't get notified by our monitoring platforms as we normally do. The issue was reported to us and resolved within 30m. We're very sorry for this disruption to your morning, and are adding safety checks and additional monitoring conditions to make sure this doesn't happen again!

Starting May 28th, we've noticed large numbers of SMS messages being incorrectly flagged as spam by several carriers and subsequently not delivered. We're working with our third party SMS provider to resolve this issue, but because the problem appears to be on the carrier filtering side, unfortunately we don't have a timeline yet for when it will be resolved.

SMS messages have been flowing normally again for several hours, so resolving this incident.

We're having some performance issues in search... investigating.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

Maintenance is now complete. All systems are functioning normally.

We are currently switching over to alternative hardware and reporting should be back up shortly (though data may be slightly delayed).

Failover is complete and all systems should be operational

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

This morning CenturyLink (one of the backbone internet providers) is experiencing outages, which means that some users may not be able to access Populi and/or many other sites. In our testing we've been able to access Populi from multiple locations, but depending on where you are and what ISP you use, you might not be able to. Unfortunately there's not a lot we can do to resolve this issue since it's a widespread internet outage, but we'll be monitoring the situation closely.

Due to increased usage of Populi's online learning features right now, we hit a limit with our email provider that led to messages being marked as "rejected" so we retried sending them.... however, when they raised our limit the retires are flooding in and sending multiple copies of each message... sorry for the hassle! We're working with them to halt the flow of repeat email as soon as possible....

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved... there are a handful of backed up emails we'll be sending throughout the day, but all new email sent during the last several hours looks good.

Sorry about that, everyone! We’re in the middle of upgrading our email sending providers and adding DMARC support, and unfortunately the drastic usage spike we’ve been experiencing lately caused us to unexpectedly run up against a daily email sending cap with our new email provider \(Amazon\). That’s why emails were suddenly being delayed, and when we managed to work with Amazon to get our usage limit raised, a bug in our retry sending code \(that’s supposed to retry sending email in case a provider goes down, sporadic network failure, etc\) resulted in duplicate emails being sent to many users. This bug was a tricky one because it only manifested when we bumped up against the usage cap AND had production levels of email flowing through the system - and unfortunately that means we didn’t catch it during testing. We’ve temporarily fallen back to sending through our old email provider, and after we purge duplicates from the new queue and get any messages that should be sent flushed out, we’ll confirm the retry logic bug is fixed and carefully begin transitioning to Amazon again, after working with them to raise our sending limit yet again to give us lots of headroom this time. Thanks for your patience as we continue to deal with abnormally high usage and still try to push the product forward at the same time!

We pushed an update to a low-level encryption library that isn't playing nicely with API token validation... we're reverting and should be back to normal in a few minutes. Sorry for the hassle!

We are continuing to work on a fix for this issue.

This incident has been resolved.