Loanmaps 장애 기록

We're investigating an issue with saving in the loan application interface

This issue appears to be isolated to co-borrowers and page four data. We are working on a correction now.

A fix has been implemented and we are preparing to deploy the change

Deployment with the update is in progress

A fix has been deployed and we are monitoring to ensure there are no further issues.

This incident has been resolved.

We have identified an issue with icons not loading and confirmed it is an issue with our icon provider's service. They are currently investigating.

We've disabled loading icons from our icon provider and routed requests internally. Once our icon provider's service has been restored, we will switch back over to them.

Resolving, as we have routed all icon loading to our internal servers.

We experienced an issue today where an expired SSL certificate was being used on one of our load balanced web servers. This was causing intermittent certificate expiration messages for end users. We monitor all load balanced web servers for expiring certificates, however we found today that one of the servers had an unused configuration section that was somehow being referenced when requesting the certificate for that server. By removing this unused configuration, the server started referencing the correct certificate from the correct configuration. Because we removed the unused configuration, this is not an issue we expect to happen again at the next certificate renewal date (in 2022).

We have identified an issue with icons not loading and confirmed it is an issue with our icon provider's service. They are currently investigating.

We've disabled loading icons from our icon provider and routed requests internally. Once our icon provider's service has been restored, we will switch back over to them.

We are aware of an issue causing our database server to be under high load. This is impacting the general performance of the system. This issue is being actively investigated.

We identified an issue with one of our database queries that loads the "Not Assigned" section of the "Pre-Close" tab on the dashboard. We are working to improve the performance of this query now.

Clarification to the previous update: The query in question is causing overall high load on the database server, which is impacting other areas of the system.

We are currently working on testing a possible fix for this issue and will have more information to share shortly.

We have finished updating the query and are deploying it into the production environment now.

This issue was fixed in production at 3:00 PM. Monitoring for the past hour shows that it has been resolved.

One of the production servers we run on is currently experiencing issues. At this time, we have that server offline to prevent any further issues for our users. This may cause a slightly slower experience. We will be performing maintenance tonight to restore full functionality.

This incident has been resolved.

The infrastructure that we run on experienced an outage, taking down our production and monitoring systems. The infrastructure was down for approximately 30 minutes, and it took about 15 minutes to restore our production environment functionality, as all servers were started from a cold state.

We're experiencing an elevated level of memory usage on the primary web application. Our team is investigating this and updates will be provided as information is available. This elevated level of memory may cause intermittent instability within the application, and our team is closely monitoring the situation while a solution is being crafted.

After experiencing some issues earlier in the day, the system has been stable this afternoon. We currently have a scheduled maintenance event tonight to address some of these issues, and will update this incident tomorrow based on the results of that maintenance.

The update has been deployed, and we are now closely monitoring the application.

Memory has been holding stable this morning without any new incidents. Will continue to monitor for the rest of the day. Another update will be provided in the afternoon.

The elevated memory level issue has been resolved. We have not seen an elevated memory level today.

This incident has been resolved.

We had an issue with elevated memory that caused some system instability. The issue was resolved quickly and stability has returned back to normal.

There was an elevated level of errors this afternoon caused by a database configuration issue. This issue was identified and solved, and scheduled maintenance has been scheduled to perform additional work related to this incident.

This morning, reports of users experiencing an elevated level of errors were received. The underlying cause of this was a disk capacity issue on our database server. This was quickly corrected, and we will be performing maintenance later tonight to improve the capacity issue.