Learnosity 中断历史

1. 2026年10月26日 UTC 18:05investigating

   截至协调世界时17:50,我们正在所有区域遇到影响项目池的问题。 学习支持和系统工程小组正在积极调查这一问题,并将尽快进行更新和解决.

2. 2026年10月26日 UTC 18:34investigating

   从协调世界时18:30开始,我们在创建和编辑项目Pools时正在调查退化的性能. 我们还在调查一个未经证实的 无关紧要的问题 可能影响本地设备评估软件包的创建和编辑 作者,评估,分析堆栈中的所有其他API都正常运行. 学习支助和系统工程小组正在继续积极调查这一问题,并将尽快进行更新和解决.

3. 2026年10月26日 UTC 19:01investigating

   从协调世界时的7:00起,我们在创建或编辑项目 Pools orx 本地设备评估包时继续调查退化的性能. 作者,评估,分析堆栈中的所有其他API都正常运行. 学习支助和系统工程小组正在继续积极调查这一问题,并将尽快进行更新和解决.

4. 2026年10月26日 UTC 19:31monitoring

   截至协调世界时19:30,项目Pool和当地设备评估的创建,正常的吞吐量约为30分钟. 我们现在将进入一个额外的监测期,以确保稳定。 学习支助和系统工程小组正在继续积极调查这一问题,并将尽快进行更新和解决.

5. 2026年10月26日 UTC 20:10resolved

   截至协调世界时20:10,我们没有看到关于项目Pool或本地设备评估软件包的创建或更新的进一步问题。 我们正在解决这一事件,并将继续调查退化性能的根本原因. 学习支持和系统工程小组将在我们完成根源分析并最后确定今后所需任何步骤或预防措施后进行尸检。 有问题或问题请联系.

自动翻译自官方事件更新。

1. 2026年10月20日 UTC 14:16investigating

   从UTC下午13点20分开始 我们开始在美东一地段的 强化数学计分服务中 发现评分延迟 没有观察到数据丢失,所有会话最终都被打入分数. 作为一种敲打效应,Data API的分分站点正经历间歇性可用性问题. 没有影响作者或评估的交付。 学习支持和系统工程小组正在积极调查这一问题,并将尽快进行更新和解决.

2. 2026年10月20日 UTC 15:04investigating

   从协调世界时15:00起,我们仍在调查我们增强数学分分数服务的短期退化性能,但还没有看到一个多小时的重大错误或延误。 积压的等待评分的会议迅速清理完毕,提交的评分仍然正常。 所有其他API都不受影响. 学习支助和系统工程小组正在继续积极调查这一问题,并将尽快进行更新和解决.

3. 2026年10月20日 UTC 15:51monitoring

   截至协调世界时15:45,缓解措施已成功保持提交队列清晰,积分继续正常运行. 在退化的绩效期间收到的材料被收集,没有丢失任何数据。 所有其他API仍在运作。 我们正在继续调查其根源,一旦调查完成,我们将分享一起死后事件。 与此同时,我们正在进入一个额外的监测期,以确认在结束事件之前继续保持稳定.

4. 2026年10月20日 UTC 18:17resolved

   从协调世界时18:00开始,经过长时间的额外监测,没有再出现进一步的问题,我们解决了这一影响美国-东一区增强数学分数的事件. 学习支持和系统工程小组将在我们完成根源分析并最后确定今后所需任何步骤或预防措施后进行尸检。 有问题或问题请联系.

5. 2026年10月27日 UTC 13:27postmortem

   □ 受影响的系统和区域 2026年10月20日,Learnsosity在US-East-1经历了积分延误和间歇性请求失败影响增强数学积分. 客户撞击开始于协调世界时约13:06,结束于协调世界时13:47,持续时间约41分钟. 受影响的打分请求经历的时间比正常的处理时间长,一些直接向Data API打分端点请求返回出错. 对于通过项目 API 保存的会话, 受影响的打分请求被排队并重试, 导致分数可用性延迟, 而不是分数失败 。 其他区域没有数学问题和服务的评估没有受到影响。 □ 调查 我们的调查查明了打分请求的具体模式,导致出乎意料的高处理时间。 这削弱了处理其他打分请求的能力,造成延误和间歇性暂停。 问题在于如何处理某些评分投入,而不是要求总量意外增加。 □ 解析度 协调世界时13:47分分分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分出分 在协调世界时15:12,学习工程师实施额外的交通隔离,以防止受影响的积分工作量影响对其他客户的积分业务. □ 预防 学习能力已实施改革,以减少不必要的资源消耗并改进受影响评分请求的恢复。 正在进一步改进,以加强对要求的评分验证,改进处理保障并避免个人评分工作量影响更广泛的服务。 在完成这些改进并核查其有效性的同时,将继续采取进一步的隔离措施.

自动翻译自官方事件更新。

1. 2026年10月8日 UTC 13:40investigating

   从协调世界时下午1:00起,我们正面临问题,将日落API v2024.1.LTS版本重新映射到所有区域最新的LTS版本. 这只影响到极少数仍被钉入v2024.1.LTS的顾客,后者于2026年3月到达了"生命终结". 没有其他API版本受到影响. 学习支持和系统工程小组正在积极调查这一问题,并将尽快进行更新和解决.

2. 2026年10月8日 UTC 13:47investigating

   至13:40> UTC时,我们正重定向 v2024.LTS 版本,随着缓存更新,API正在变得可访问. 学习支助和系统工程小组正在继续积极调查这一问题,并将尽快进行更新和解决.

3. 2026年10月8日 UTC 14:20investigating

   从UTC下午14:00起,v2024.LTS成功再映射出AMER地区最新的LTS版本,包括US-East-1(VA)和US-West-2(OR). 在这些地区,所有API都应无障碍。 我们目前正在重新规划欧洲经济区和亚太区域行动方案。 仍然可以直接访问活跃的API版本。 这个问题现在仅会影响EMEA和APAC区域一对微集客户的版本再映射,被固定在被日落的v2024.1.LTS API版本上,于2026年3月被EOL所达到. 学习支助和系统工程小组正在继续积极调查这一问题,并将尽快进行更新和解决.

4. 2026年10月8日 UTC 15:15resolved

   截至协调世界时15:00,v2024.LTS在所有区域都成功进行再规划. 使用此 EOL/ sunset 版本字符串应访问所有 API 。 学习支持和系统工程小组将在我们完成根源分析并最后确定今后所需任何步骤或预防措施后进行尸检。 有问题或问题请联系.

自动翻译自官方事件更新。

1. 2026年10月2日 UTC 18:08investigating

   从协调世界时16:45起,我们开始在美东一地段的强化数学分分数服务中遇到非常少的分数延迟. 所有会议都打入分数,没有丢失数据。 没有任何其他API受到影响. 我们正在调查持续拖延 仅限于这个服务。 学习支持和系统工程小组正在积极调查这一问题,并将尽快进行更新和解决.

2. 2026年10月2日 UTC 18:30identified

   从协调世界时18:30起,我们已经为增强数学分数服务确定了一个分数问题,并正在努力解决这一问题。 所有会话继续被捕获而不丢失数据,所有其他API正常运行. 学习支助和系统工程小组正在继续积极调查这一问题,并将尽快进行更新和解决.

3. 2026年10月2日 UTC 19:10monitoring

   从协调世界时的7:00起,我们一段时间没有看到增强数学分数服务出现延误,现在将进入监测阶段。 在解决这个问题之前,我们将继续监测30分钟。 学习支助和系统工程小组正在继续积极调查这一问题,并将尽快进行更新和解决.

4. 2026年10月2日 UTC 19:52resolved

   从协调世界时19:45起,我们已经解决了影响美东一区增强数学分数的问题. 所有API和服务都正常运行. 学习支持和系统工程小组将在我们完成根源分析并最后确定今后所需任何步骤或预防措施后进行尸检。 有问题或问题请联系.

5. 2026年10月27日 UTC 01:24postmortem

   □ ** 受影响的系统和区域** 2026年10月2日,Learnosity在US-East-1中经历了间歇分数错误和超时影响数学问题类型. 客户撞击开始于协调世界时约16:34,结束于协调世界时18:47,持续时间约2小时13分. 受影响的请求在失败前可能出现长达40秒的延误。 其他区域没有受到影响。 □** 调查** 事件发生在对数学评分服务的需求超过现有能力时。 虽然在早些时候的事件之后,已对能力和规模进行了改进,但这项服务在网络配置方面又遇到一个限制,无法提供更多情况。 由于计算时限在沉重的负荷下无法可靠地执行,因此计算缓慢使影响更为严重。 这些条件使服务无法充分应付需求。 □ ** 决议** 学习工程师通过将分数服务移动到具有额外分数能力的网络配置并增加分配给每个实例的计算资源来恢复服务. 这些更改使得服务得以恢复,在协调世界时18:47前清出评分出错. □ ** 预防** 学习力更新了计算时间限制执行机制,以可靠地终止重载下缓慢的计算. 健康检查也从计算处理中分离出来,以防止不必要的病例替换. 该服务现在以额外的计算和网络能力运作。 正在进一步改进建筑,以防止网络能力限制今后的规模,我们正在评估较短的计算时限,以降低异常缓慢计算的影响.

自动翻译自官方事件更新。

1. 2026年9月11日 UTC 18:19investigating

   截至协调世界时17:00,我们正在调查一些小间歇性问题,美国-东部-1地区得分。 学习支持和系统工程小组正在积极调查这一问题,并将尽快进行更新和解决.

2. 2026年9月11日 UTC 18:30investigating

   从协调世界时18:30起,所有API仍在运行,我们正在监测评分效率。 学习支助和系统工程小组正在继续积极调查这一问题,并将尽快进行更新和解决.

3. 2026年9月11日 UTC 19:42identified

   从协调世界时19:30起,我们确认这仅仅是三个强化数学问题类型评分的零星问题. 这些评分尝试在重试中成功,没有其他API受到影响. 学习支助和系统工程小组正在继续积极调查这一问题,并将尽快进行更新和解决.

4. 2026年9月11日 UTC 20:53monitoring

   从协调世界时20:45起,我们改进了对初始EMS评分请求和复试的处理,现在将进入60分钟的监测阶段. 所有API继续正常运行. 学习支助和系统工程小组正在继续积极调查这一问题,并将尽快进行更新和解决.

5. 2026年9月11日 UTC 21:44resolved

   截止 UTC21:45,增强数学分数问题类型回复通常在US-East-1地区评分,所有API都正常运行,我们正在结束这一事件. 学习支持和系统工程小组将在我们完成根源分析并最后确定今后所需任何步骤或预防措施后进行尸检。 有问题或问题请联系.

6. 2026年10月27日 UTC 01:23postmortem

   □ ** 受影响的系统和区域** 2026年9月11日,Learnsosity经历了间歇性得分错误和超时,影响美国-东方-1的数学问题类型. 客户撞击从协调世界时17:00左右开始,至协调世界时21:45结束,持续时间约4小时45分. 其他问题类型继续正常得分,而其他区域则未受影响。 多数失败的请求在重审时成功 。 □** 调查** 事件发生的原因是,在前几周需求增加之后,数学评分服务能力不足。 服务达到了所配置的最大实例数量,无法进一步缩放以容纳负载. 此外,每个案例的计算资源有限,计算过程缓慢,使案件超过预期时间。 在沉重的负荷下,计算时限没有得到可靠执行,繁忙的情况可能无法进行健康检查并被取而代之,从而进一步降低了现有能力。 □ ** 决议** 学习工程师通过增加运行实例的数量并提升最大实例限制来恢复服务. 正常打分业务于协调世界时21:45恢复,其余所有排队打分请求均由协调世界时23:17处理. 9月14日,部署了额外的配置改变,以增加计算资源,改进自动缩放,并减少不必要的实例替换。 □ ** 预防** 学习能力提高了评分能力,提高了实例限制,并根据处理器利用情况改进了自动分级。 健康检查已从计算处理中分离出来,以防止工作繁忙的情况被错误地确定为不健康。 我们还改进了计算时限的执行,并扩大了对资源利用、反应时间和评分错误的监测和警报。 目前正在开展进一步的工作,以加强网络规模化能力并评估较短的计算时限.

自动翻译自官方事件更新。

1. 2026年7月24日 UTC 12:16investigating

   我们意识到,美国-西部-2(OR)地区正在面临一个影响美国-西部-2(OR)地区的问题,这可能会在进入学习授权环境时造成间歇性连接问题。 我们正积极监测局势,并将随时提供最新情况。 此时,对学习顾客的影响是短暂和断断续续的,仅限于作曲. 评估和其他API行为正常。 学习支持和系统工程小组正在积极调查这一问题,并将尽快进行更新和解决.

2. 2026年7月24日 UTC 12:28monitoring

   从协调世界时11:02起,Learnsosity开始在访问Learnsosity作者站点时看到短命,间歇性超时的监控警报. 没有其他功能受到影响,在协调世界时11:11停止了学习警报. 协调世界时11:40,AWS报告在US-West-2(OR)的服务退化. 在协调世界时11:56,AWS报告其系统正在恢复. 学习支助和系统工程小组正在继续积极监测这一问题,并将尽快进行更新和解决.

3. 2026年7月24日 UTC 12:35monitoring

   在协调世界时12:30,AWS报告说,他们查明并纠正了这个问题,恢复工作仍在继续. 学习支助和系统工程小组将监测这一问题再达30分钟,并尽快更新和解决.

4. 2026年7月24日 UTC 13:02resolved

   从协调世界时下午1:00起,我们要求解决这个问题。 AWS系统正在恢复,学习写作未受影响50分钟. 学习支持和系统工程小组将在我们完成根源分析并最后确定今后所需任何步骤或预防措施后进行尸检。 有问题或问题请联系.

5. 2026年7月24日 UTC 13:03postmortem

   受影响的系统和地区 2026-07-24日,由于[https://health.aws.amazon.com/health/status?eventID=arn:aws:health:us-west-2:event/MULTIPLE SERVICES/AWS MULTIPLE SERVICES OPELATILE ISSUE/AWS MULTIPLE SERVICES OPELATIPRAAL ISSUE 83A36 24C89B493A6],学习系统经历了短暂的间歇性超时,影响进入学习作者网站. 学习客户撞击开始于协调世界时约11:02,结束于协调世界时11:11. AWS随后于协调世界时12:12完成了补救并正式结束了他们的区域事件. 对学习客户可能产生影响的总持续时间约为9分钟。 调查 学习在协调世界时11:02收到内部警报,OR地区的服务受到影响并立即开始调查. 初步调查没有发现学习服务退化的证据。 进一步调查发现,间歇性超时影响进入学习作者网站。 学会与AWS确认,已经确定了影响该区域网络基础设施的一个问题并已开始进行补救。 决议 影响Learnsosity作者站点的中断超时于协调世界时11:11停止,AWS在协调世界时11:15前有效解决了这个问题. AWS报道了一次短暂的再侦察事件,于协调世界时11:59解决,并于协调世界时12:12正式结束事件. 预防 没有学习系统促成这一事件。 内部监督和警报行为如所料.

自动翻译自官方事件更新。

1. 2026年4月17日 UTC 16:06investigating

   As of 16:00 UTC, we are currently experiencing a slowdown in availability of newly submitted submitted session results in the us-east-1 region. There is no data loss, all saves and submissions are being queued as designed, and the assessments stack remains operational. Delays may currently be experienced using the Data API to fetch session results, and the Reports API to display results. Other workflows that rely on the synchronous use of Data API results may also be affected. Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

2. 2026年4月17日 UTC 16:21identified

   As of 16:20 UTC, Learnosity has increased database performance and we are processing queued sessions rapidly. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.

3. 2026年4月17日 UTC 16:28monitoring

   As of 16:25 UTC, we have resolved the slowdown in processing session results and all sessions are being scored normally. We will continue to monitor for a further 30 minutes before closing this incident report. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.

4. 2026年4月17日 UTC 17:58resolved

   As of 16:55 UTC, we are have resolved the degraded performance issue affecting availability of session data in us-east-1. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required. Please reach out if you have any questions or concerns.

5. 2026年5月4日 UTC 17:23postmortem

   ### Affected Systems and Regions On 2026-04-17, Learnosity experienced a service degradation impacting availability of newly submitted session results for a small subset of customers in US-East-1. The issue began at approximately 13:54 UTC and was resolved at 16:24 UTC. The total duration of customer impact was approximately 2.5 hours. ### Investigation Service degradation was detected following the accumulation of submitted sessions in the async scoring queue, This was caused by elevated CPU utilization when heavy load coincided with a background data migration task that had been running for several days on the affected EC2 instance. This reduced message processing throughput, leading to queueing of sessions.  ### Resolution Learnosity CloudOps engineers responded by scaling session processing node EC2 instances, and corresponding database proxies, to maximum allowable capacity. The migration task was also stopped, which further reduced CPU pressure and allowed message processing throughput to recover. The session scoring backlog was fully cleared by 16:24 UTC. ### Prevention Learnosity is implementing the following measures to mitigate: * Pre-flight health checks and safeguards are being introduced to prevent migration tasks from running during busy periods of production usage. * Enhanced monitoring is being applied to detect backlog growth and database contention, and apply remedial action more quickly

1. 2026年4月16日 UTC 14:08investigating

   As of 13:30 UTC, we are currently experiencing slow downs affecting the Live Progress report in the us-east-1 region. Atypical use of eventbus may also be affected, such as custom reports/implementations. Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

2. 2026年4月16日 UTC 15:23resolved

   As of 14:30 UTC, the Live Progress report event degradation issue in the us-east-1 region has been resolved. Additional capacity processed the event load and these systems are operating normally. Brief additional latency during this scaling may have been introduced for users of Learnosity's premium Firehose feature, but we've seen no evidence that this affected customers. This, too, was resolved once scaling was complete. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required. Please reach out if you have any questions or concerns.

3. 2026年5月4日 UTC 14:39postmortem

   ### Affected Systems and Regions On 2026-04-16, Learnosity experienced a service degradation impacting the Live Progress report for a small subset of customers in the us-east-1 region. The issue began at approximately 13:12 UTC and was resolved at 14:40 UTC. The total duration of customer impact was approximately 88 minutes. ### Investigation The issue was detected following elevated error rates on the load balancer serving the eventbus service. Investigation determined that an unhealthy condition within the EC2 instances led to elevated CPU utilization and memory pressure within the Auto Scaling Group, resulting in application instability and repeated service restarts. Inconsistent recovery caused traffic to concentrate on a subset of hosts, further elevating error rates. A secondary effect of the instability was increased request pressure on downstream dependencies. ### Resolution Service was restored by stabilizing the affected EC2 instance and restoring consistent application availability across the Auto Scaling Group. Load distribution normalized once all instances returned to a healthy state. ### Prevention Learnosity is implementing the following measures to mitigate: * Improve service startup and dependency handling to ensure consistent recovery behavior * Review resource thresholds to reduce the likelihood of similar instability * Enhance monitoring to detect and respond more quickly to similar conditions

1. 2026年2月4日 UTC 18:34identified

   As of 18:30 UTC, we are currently experiencing an issue affecting assessment delivery in VA (us-east-1). Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

2. 2026年2月4日 UTC 18:55monitoring

   As of 18:52 UTC, the Learnosity System Engineering team has put a fix in place. We will continue to monitor for some time to ensure a stable recovery, but all services are operational again. Learnosity Support and Systems Engineering teams will continue to actively monitor the issue, and will follow on with an update and resolution as soon as possible.

3. 2026年2月4日 UTC 19:15resolved

   As of 19:12 UTC, traffic has returned to normal and everything remains stable. We have resolved the incident affecting assessments in VA (us-east-1). Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required. Please reach out if you have any questions or concerns.

4. 2026年2月10日 UTC 17:08postmortem

   ### **Affected Systems and Regions** On 2026-02-04, Learnosity experienced an availability incident impacting Questions API access for customers in the AMER region. The issue began at approximately 18:02 UTC and was formally declared resolved at 18:51 UTC. The total duration of customer impact was approximately 49 minutes. ### **Investigation** The issue was first detected when the Items API attempted to initialize the dependent Questions API. The incident was traced to a misconfiguration within the content delivery network servicing the AMER region, causing a SSL certificate error.. During a certificate management task, the distribution responsible for serving [_questions.learnosity.com_](http://questions.learnosity.com) was configured with an SSL certificate that did not match the _\*.learnosity.com_ domain nomenclature. This caused browsers and client systems to reject connections to the affected subdomain. ### **Resolution** Once the issue was identified, the response team updated the affected distribution to use the correct SSL certificate. Service was fully restored after the change was completed. ### **Prevention** Learnosity is strengthening certificate deployment and validation processes to ensure that domain mappings are verified prior to rollout. Additional safeguards have been added to prevent cross-domain certificate assignments and to improve monitoring for sudden traffic drops that may indicate certificate-related failures.

1. 2026年1月12日 UTC 17:06investigating

   As of 4:30 UTC, we are currently investigating an issue that may result in temporary session unavailability for a small number of US-only customers. We can confirm that no session data has been lost, and we are working to restore the affected sessions as quickly as possible. Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

2. 2026年1月12日 UTC 17:52identified

   As of 5:30 UTC, we've identified and corrected a load balancing configuration affecting database reads and writes for a very small number of US-only customers with consumers in the US-East-1 region. - All new sessions will behave as expected. - Read requests seeking data on sessions created prior to 23:00 UTC yesterday will also now behave correctly. - We are working on restoring sessions recently affected While all APIs continue to function normally, this update identifies service areas that may temporarily have limited access to recently saved or submitted sessions. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.

3. 2026年1月12日 UTC 19:10resolved

   As of 7:00 UTC, we've concluded our monitoring and confirmed that all new requests are behaving as expected. We are closing this record and will continue to work with affected customers directly to confirm that all recent sessions are accessible. Please reach out if you have any questions or concerns.

1. 2025年10月20日 UTC 14:53monitoring

   Learnosity services in VA are continuing to be partially impacted by the major AWS outage. AWS first reported the incident at 7:11 UTC As of 14:50 UTC they are continuing to work on its full resolution, but have again set their impact level to "Degraded". The AWS Health Status dashboard where updates are being posted by the AWS team can be found here: https://health.aws.amazon.com/health/status Overall, the error rate has fallen and has stayed below what we saw earlier in the day and we have been successful in slowly scaling up the infrastructure, but we're still seeing some issues stemming from the the ongoing AWS outage as we attempt to bring online some new instances, and we have begun to see an uptick in errors. These have included errors associated with Data API, Feedback Aide, and the delivery of adaptive assessments, delays in scoring and video encoding, as well as other smaller spikes across different systems. While the overall error rate remains low, Learnosity Support and Systems Engineering teams are actively monitoring the outage and manually intervening where possible to mitigate the issues we're facing. In spite of our best efforts it is likely that the APIs may not be fully operational in VA until AWS's service is fully restored to normal operation, but we are taking every step within our power to try to minimize the disruption to our services.

2. 2025年10月20日 UTC 15:30investigating

   As of 15:25 UTC, we are seeing increasing error rates. Data API is the most affected service at the moment and we are seeing an increase in scoring delays, but we are continuing to score incoming messages to the best rate our current capacity allows. AWS is still reporting a severity of "Degraded", but now with an increased count of 81 affected services. Learnosity Support and Systems Engineering teams are continuing to monitor the issue, and will continue our mitigation efforts wherever possible.

3. 2025年10月20日 UTC 18:13monitoring

   As of 19:10 UTC, we are beginning to see errors rates diminish and some Learnosity services are beginning to scale again. AWS is still reporting a severity of "Degraded" with problems to 104 of their services in us-east-1. Learnosity Support and Systems Engineering teams are continuing to monitor and manually intervene when possible to mitigate the effects of the outage.

4. 2025年10月20日 UTC 19:02monitoring

   As of 20:00 UTC, the error rate has continued to drop, systems remain scaled and the backlogs of scoring messages continues to drop. AWS is still reporting 109 impacted services and a severity of "Degraded" Learnosity Support and Systems Engineering teams are continuing to monitor the issue and are manually adjusting the infrastructure where needed to ensure stability.

5. 2025年10月20日 UTC 19:32monitoring

   As of 19:30 UTC, the error rate has fallen to 0.1% and continues to fall, we continue to steadily process the backlog of scoring messages. As of their latest update AWS has downgraded the severity of outage to "Impacted" but still list 109 services as affected. Learnosity Support and Systems Engineering teams are continuing to actively monitor the issue, and will follow on with an update and resolution as soon as possible.

6. 2025年10月20日 UTC 20:44monitoring

   As of 20:40 UTC, the error rate remains low, the infrastructure is scaled to meet demand, and the scoring queues are empty and new submissions are being processed as they come in. All services are operational and we are processing the remaining backlog of messages from the incident. AWS still lists 93 affected services and a severity of "Impacted". Learnosity Support and Systems Engineering teams are continuing to monitor the issue, but at the moment all systems are fully operational.

7. 2025年10月20日 UTC 21:35resolved

   As of 21:30 UTC, all systems keep operating normally, so we are resolving the status page about the AWS outage affecting Learnosity Services in the VA (us-east-1) region. Learnosity Support and Systems Engineering teams will continue to monitor the systems closely while AWS continues to work on the issues on their side. Please reach out if you have any questions or concerns.

1. 2025年10月20日 UTC 10:11monitoring

   AWS is going through late-stage recovery of a major outage in US-East-1 bringing down most of their services. Learnosity APIs were stable and unaffected in isolation, but some impact cascaded to Learnosity (such as availability of assets from CloudFront). The full extent of the incident is unknown at this time. NOTE: Up until the time of this writing (10:05 UTC), this included impact to sign-in services that prevented us from accessing our StatusPage. We immediately posted banner notifications in our Support Portal, and those banners will remain visible until we've been able to verify that service has been restored for sufficient time. Incident History: As of 7:00 UTC, AWS experienced a major outage. While Learnosity APIs were fully operational, some Learnosity services use AWS and were affected. These included access to our Support ticketing system (the Learnosity-hosted Help guides were unaffected), and availability of CloudFront-hosted assets, but other systems may have been impacted as well. As of 10:00 UTC, AWS has restored services and access to Learnosity-adjacent features is becoming available relatively quickly. Learnosity Support will monitor AWS for a further 30 minutes before calling this issue resolved. We will follow on with an update and resolution at that time, or sooner.

2. 2025年10月20日 UTC 11:23resolved

   As of 11:20 UTC, AWS services appear to have been stable for approximately an hour. Learnosity APIs--including services affected by the AWS outage (such as CloudFront)--appear to be fully operational. At this time we are labeling this issue resolved. Learnosity Support and Systems Engineering teams will continue to investigate impact from this incident and follow up with a post mortem once we hear from AWS. Please reach out if you have any questions or concerns.

1. 2025年10月2日 UTC 21:34investigating

   As of 21:15 UTC, Learnosity APIs are being impacted by a temporary outage from our Math partner, Desmos. Authoring or Editing content with Desmos questions or features, or initializing assessments that contain Desmos questions or features, can prevent the Author API or Items API from initializing. Specific reports that display question content, such as the Session Detail by Item report, can also fail to initialize Learnosity APIs are not otherwise impacted, so content that does not include Desmos features are not affected. Until we learn more, we will list APIs that may experience knock-on effects from this temporary Desmos outage. However, as most customers will not experience any issues, we will not list these APIs as down. Desmos and Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

2. 2025年10月2日 UTC 22:14monitoring

   As of 22:12 UTC, Desmos services have been restored and Learnosity APIs are initializing without issue. Desmos and Learnosity Support and Systems Engineering teams will continue to monitor for a further 30 minutes before declaring this issue resolved. We will follow on with an update and resolution as soon as possible.

3. 2025年10月2日 UTC 22:43resolved

   As of 22:42 UTC, all Learnosity APIs are continuing to function as expected in all regions, and we are declaring this issue resolved. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required. Please reach out if you have any questions or concerns.

1. 2025年8月19日 UTC 16:11investigating

   As of 3:40 UTC, we are experiencing degraded performance in our analytics stack affecting the availability of session data in US-East-1 for a subset of customers. This is affecting the Reports API and Data API. Neither authoring nor assessment stacks are affected, and there is no data loss. Submitted sessions are queueing for processing. Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

2. 2025年8月19日 UTC 16:35investigating

   As of 4:30 UTC, we are continuing to investigate degraded performance in our Data and Reports APIs. Only availability of recently submitted session data is affected. Historic session data, as well as all other API stacks, remain unaffected. New submissions are persisting correctly with no data loss and these submission are being queued for scoring. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.

3. 2025年8月19日 UTC 17:34identified

   (Note: We're correcting cited UTC times to the 24 hr format and will include both forms in this update only.) As of 5:30pm/17:30 UTC, we've identified a possible contributing cause for the degraded performance in our analytics stack. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.

4. 2025年8月19日 UTC 18:42identified

   As of 18:30 UTC, initial remediation efforts have tripled the rate of queued session processing and we are continuing to work toward a full resolution. Access to recently submitted session results via the Data API and Reports API remains the only affected part of the Learnosity ecosystem. New submissions continue to be safely queued for scoring while the degraded performance remains. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.

5. 2025年8月19日 UTC 19:56identified

   As of 19:30 UTC, we are now processing queued sessions rapidly and more than half of the backlog has already cleared.. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.

6. 2025年8月19日 UTC 20:35identified

   As of 20:30 UTC, the scoring queue backlog is almost empty and new sessions will soon be processed without delay. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.

7. 2025年8月19日 UTC 21:05monitoring

   As of 21:00 UTC, all sessions have been cleared from the scoring backlog queue and all submissions are being processed normally. Learnosity Support and Systems Engineering teams will monitor this situation for a further 30 minutes before calling it resolved.

8. 2025年8月19日 UTC 21:31resolved

   As of 21:30 UTC, we are have resolved the issue affecting availability of session data in US-East-1. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required. Please reach out if you have any questions or concerns.

9. 2025年9月19日 UTC 19:14postmortem

   ### Affected Systems and Regions On 19 August 2025, Learnosity experienced degraded performance in our analytics stack, affecting session data availability in US-East-1 for a subset of customers. This affected the Reports API and Data API, with no other stacks affected, and no data loss.  ### Investigation Monitoring detected a rapid increase in unprocessed and retried messages, along with elevated lock contention in the sessions database. The root cause was traced to a customer implementation issue generating an extraordinarily high number of submissions. This drove excessive retries, magnifying actual traffic volume.  The use of time-ordered v7 UUIDs for session IDs, normally handled without issue, became problematic under this contention. Uniqueness checks on each session ID required more resources and triggered a succession of temporary deadlocks. These deadlocks would usually self-resolve, but the amplified traffic prevented recovery, turning a minor issue into a sustained queue blockage. ### Resolution Once the issue was identified, Learnosity moved the customer to a dedicated, isolated sync queue, preventing cross‑tenant impact while we investigated. We applied targeted rate limits for the isolated service to protect the database, and drained the backlog. Where safe, long‑running queries were terminated to free locks and allow forward progress.  To support faster diagnosis, Learnosity enabled detailed deadlock logging and expanded metrics around message retries, abandonment, and per‑session activity. Learnosity also worked with the customer to adjust implementation settings, reducing combined saves and submits by two orders of magnitude. Session IDs were also switched to v4 UUIDs which simplified uniqueness checks further preventing deadlocks. Immediately after these changes were put into use, queues began to rapidly recover, and normal processing resumed. Most sessions for the subset of affected customers saw short delays, while the most significantly delayed session took ~6 hours before final persistence. Throughout, we identified no data loss. ### Prevention To prevent recurrence, we are: * Implementing targeted load tests and contention simulations to replicate high-parallelism patterns. * Reviewing customer identifier schemes for session IDs and auditing usage across all customers \(initial checks confirm none of our Top 50 customers currently use v7 UUIDs\). * Analyzing adoption of per-tenant fair-use queues \(or equivalent fair-share policies\) to cap burst throughput from a single tenant and protect shared infrastructure.

1. 2025年8月8日 UTC 13:55investigating

   As of 13:40 UTC, we began investigating a possible issue affecting manual update jobs to session response scores, sessions statuses, and session metadata in the US-East-1 (VA) region. We saw a brief buildup in the queue for these jobs, the queue processed as expected, and is now empty. However, we're investigating further, out of an abundance of caution, to see if there are any unexpected causes for the short queue buildup. Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

2. 2025年8月8日 UTC 13:57investigating

   As of 13:55 UTC, all manual update jobs have been executed and the jobs queue is empty. Requests to manually update session response scores, session statuses, and session metadata should execute as expected. We are still investigating possible reasons for the blip. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.

3. 2025年8月8日 UTC 14:19resolved

   As of 14:10 UTC, we've found no negative behavior in the manual update processes and attribute the short burst of jobs to uncharacteristic testing of the new manual grading workflow. The jobs queue functioned as expected, ensuring no loss of requests, and the jobs were processed in a timely fashion. Learnosity will continue to monitor this activity for a further 60 minutes, but we are calling this resolved at this time. We apologize for the unwarranted alert. Please reach out if you have any questions or concerns.

4. 2025年8月8日 UTC 14:19postmortem

   At 13:40 UTC on 8 August 2025, we noticed a brief backup of asynchronous jobs affecting manual session updates in the US-East-1 \(VA\) region. Possible jobs including manual updates to response scores, session statuses, and session metadata. The queue mechanism worked as expected, and all requests were processed in short order. Out of an abundance of caution, we raised an alert while systems remained operational, but found nothing of concern during our investigation. We hope the unwarranted alert was not a significant distraction.

1. 2025年3月3日 UTC 13:48investigating

   As of 13:45 UTC, we are experiencing delays in availability of session information in US-East-1 Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

2. 2025年3月3日 UTC 14:00investigating

   As of 14:00 UTC, we are still investigating an issue affecting the availability of session information in US-East-1. Both Data API and Reports API are experiencing extended delays and intermittent failures in returning session results. Authoring and Assessment APIs appear to be unaffected at this time and all session submissions are being successfully queued. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.

3. 2025年3月3日 UTC 15:06investigating

   As of 15:00 UTC, we are continuing to investigate the issue. Data and Reports API are still impacted. Assessment and Authoring APIs remain unaffected. Assessment submissions are being queued and will be processed as the system works through any backlog queue. Note that if the Data API or Reports API is used in a preliminary/synchronous assessment delivery workflow, this will likely have a knock-on affect when initializing assessments. Initializing any assessment API directly is unaffected at present. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.

4. 2025年3月3日 UTC 15:20monitoring

   As of 15:15 UTC, all services have been restored. All queued messages have been processed and session information should now be available. Users of the Learnosity Firehose service will still see slight delays as messages are sent out for recently processed events. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.

5. 2025年3月3日 UTC 15:54resolved

   As of 15:45 UTC, after a further 30 minutes of monitoring, we are resolving this issue. All services remain fully operational. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalised any next steps or preventative measures required. Please reach out if you have any questions or concerns.

6. 2025年3月14日 UTC 16:31postmortem

   ### Affected Systems and Regions On March 3, 2025, we experienced a temporary service slowdown impacting the availability of session information in the US-East-1 region. The issue began at 13:19 UTC and was resolved by 15:15 UTC. The Data API and Reports API experienced delays and intermittent failures when returning session results. Authoring and Assessment APIs were unaffected with no data loss. ### Investigation We identified a recently introduced inefficient database query was causing unnecessary strain when handling extremely large data sets. This led to delayed responses to the Data API sessions endpoints. This further led to delays or unavailability in select reports via the Reports API. Additionally, some timeouts and safeguards that should have triggered earlier did not activate as expected. ### Resolution To quickly restore performance: * We optimized the affected queries, improving efficiency and reducing database load. * We scaled up the impacted systems to immediately process the backlog. * We fine-tuned system timeouts and connection limits to prevent similar issues in the future. Following these fixes, all services resumed normal operations. ### Prevention To prevent future occurrences: * We have permanently updated our database handling methods to ensure more efficient query execution. * We are enhancing automated monitoring to detect and respond to database slowdowns before they impact customers. We appreciate your patience and remain committed to delivering a seamless experience.

1. 2025年1月9日 UTC 13:33investigating

   As of 13:30 UTC, We are currently investigating a possible performance degradation affecting Reports API and Data API results in US-East-1. Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

2. 2025年1月9日 UTC 14:02investigating

   As of 14:00 UTC, Learnosity is working on identifying the cause of degraded performance in our Analytics APIs, Reports and Data. Authoring and Assessment stacks remain unaffected Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.

3. 2025年1月9日 UTC 14:13monitoring

   As of 14:10 UTC, Learnosity has restored full service for all new requests to all Analytics APIs. We are monitoring stability and analyzing all queued requests. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.

4. 2025年1月9日 UTC 14:50resolved

   As of 14:50 UTC, after a further 60 minutes of issue-free monitoring, we are resolving this issue affecting availability of session information via our Analytics stack. Reports API and Data API are fully operational. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalized any next steps or preventative measures required. Please reach out if you have any questions or concerns.

5. 2025年1月14日 UTC 17:25postmortem

   **Affected Systems and Regions** On 2025-01-09, Learnosity experienced a brief partial outage affecting our analytics stacks, specifically the Reports API and Data API in the AMER region. The issue began at 13:30 UTC and was resolved at 14:06 UTC, lasting 36 minutes. All other API's were unaffected and there was no loss of data. **Investigation** We discovered that a large number of atypical, inefficient Data API queries requested by customers were taking too long to complete. This prevented other queries from running in a timely manner, creating a backlog. It was determined that an additional database index would significantly improve the response times of these types of queries. **Resolution** Immediately upon discovering this issue, the impacted database instances were successfully scaled up to ease the backlog of requests. The additional index was implemented and all remaining queries were processed quickly. Affected APIs returned to normal operations, and further monitoring ensured the issue was fully resolved. **Prevention** Following further testing, the new index is working well and has become a permanent part of the system. We are also adding new automated monitoring and regression testing to ensure similar requests perform as expected.

1. 2024年10月1日 UTC 15:09investigating

   As of 15:00 UTC, we are currently investigating an issue affecting the event delivery that powers the live progress report (live-activitystatus-by-user) in VA (us-east-1). Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

2. 2024年10月1日 UTC 15:25identified

   As of 15:20 UTC, the Systems Engineering team has identified the cause for the slower than normal delivery of events and is proceeding with mitigation steps. Learnosity Support and Systems Engineering teams are continuing to actively investigate the issue, and will follow on with an update and resolution as soon as possible.

3. 2024年10月1日 UTC 15:40monitoring

   As of 15:35 UTC, the Systems Engineering team has deployed a fix, the errors have stopped, and currently all systems are functional. We will continue to monitor for a time before resolving the incident but we are currently seeing no errors. We will follow on with an update and resolution as soon as possible.

4. 2024年10月1日 UTC 16:07resolved

   As of 16:05 UTC, We are have resolved the issue affecting live progress reports in VA (us-east-1). As the recovery remains stable, we are now resolving the incident that was affecting live progress reports. Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalised any next steps or preventative measures required. Please reach out if you have any questions or concerns.

1. 2024年9月13日 UTC 14:35investigating

   As of 14:30 UTC, We are investigating a potential issue affecting the delivery of some Events API events (used in the live proctoring / live-activitystatus-by-user report) in us-east-1 (VA). Assessments are otherwise proceeding normally and can be saved and submitted without any issue. This issue if confirmed would only affect the delivery of Events API events. Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

2. 2024年9月13日 UTC 15:09identified

   As of 15:00 UTC, the Learnosity Systems Engineering team has identified the issue preventing the assessment events from being delivered and are working on the implementation of remediation steps. Learnosity Support and Systems Engineering teams are continuing to work on the issue, and will follow on with an update and resolution as soon as possible.

3. 2024年9月13日 UTC 15:34monitoring

   As of 15:30 UTC, events are flowing normally once again and all systems are operational. We will continue to monitor for some time before we resolve the issue fully to ensure we have a stable recovery.

4. 2024年9月13日 UTC 16:03resolved

   As of 16:00 UTC, after monitoring and confirming the stable state of the fixes put into place, we have resolved the Events API events delivery issue affecting proctored assessments in us-east-1 (VA). Learnosity Support and Systems Engineering teams will follow up with a post mortem once we have completed root cause analysis and finalised any next steps or preventative measures required. Please reach out if you have any questions or concerns.

1. 2024年5月22日 UTC 18:27investigating

   As of 18:10 UTC, we are trying to determine if an issue exists causing a very small number of adaptive sessions to not trigger a Firehose event after scoring in US-East-1. This is just an internal investigation at present, without confirmation, and we have no evidence that any impact outside the described implementation is affected. Learnosity Support and Systems Engineering teams are actively investigating the issue, and will follow on with an update and resolution as soon as possible.

2. 2024年5月22日 UTC 18:33investigating

   As of 18:30 UTC, Learnosity Support and Systems Engineering teams are continuing to investigate this unconfirmed issue and will follow on with an update as soon as possible.

3. 2024年5月22日 UTC 18:47investigating

   As of 18:45 UTC, Learnosity Support and Systems Engineering teams are continuing to investigate an unconfirmed issue, examining less than 500 potentially affected sessions out of 4.93 million submitted over the same period. We will follow on with an update as soon as possible.

4. 2024年5月22日 UTC 19:01monitoring

   As of 19:00 UTC, we have been unable to find any systemic problems, while the number of potentially impacted sessions remains at one hundredth of one percent ( 0.01%) of total submissions over the period investigated, likely due to a content issue. Learnosity Support and Systems Engineering teams will continue to monitor for 30 minutes and expect to close this investigation at that time.

5. 2024年5月22日 UTC 19:42resolved

   As of 19:40 UTC, we believe we've identified content to be the root cause of this tiny number of sessions not passing through scoring and triggering a Firehose event. We've reached out to customer proactively and will help address this issue directly. We are now calling this issue resolved and apologize for any distractions. Please reach out if you have any questions or concerns.