Ashby 장애 기록

For approximately 11 minutes from 10:19 AM PDT to 10:30 AM PDT, Ashby users experienced long-running loading screens and, in some cases, errors. This was the result of requests to our servers taking too long to respond. Candidates applying for jobs were unaffected. Our maintenance this weekend will address the root cause of this issue by increasing our database capacity.

Due to an issue with a downstream provider, newly uploaded resumes are not being automatically parsed into structured data. Resume files remain available in the app, but contact info, work history, and plain-text content are not currently being extracted from newly-uploaded resumes. Application form autofill, AI job criteria evaluation, and resume-based search may be affected for new resumes. Email and Chrome extension resume imports may also fail until parsing is restored.

Newly-uploaded resumes are now being parsed successfully. Resumes that failed to parse during the incident will be backfilled.

This incident has been resolved. Resumes that failed to parse during the incident have been backfilled.

From 11:30am until 1:40pm PDT, candidate PDF exports failed with a generic message. The cause of this issue has already been identified and fixed.

We are currently investigating this issue.

We are getting reports of Ashby being unavailable in Sydney. We're also getting limited reports of Ashby being unavailable in other regions, like Portland and San Francisco. We're working with an upstream provider to determine the root cause.

Our upstream provider, Cloudflare, has confirmed a connectivity issue and is working on a fix.

Our upstream provider, Cloudflare, has implemented a fix and our tests indicate they have resolved the issue.

We are currently investigating this issue.

We've confirmed Ashby is available again in the Sydney region. We will work with our upstream provider to understand what happened.

Starting at 4:30PM (UTC), Ashby stopped being able to process inbound data from Dropbox Sign. E-Signature requests were sent, but status updates were not recorded.

A fix has been implemented, and data is in the process of being recovered.

This incident has been resolved.

We're experiencing an issue sending Slack and email notifications for interview reminders between 4:00PM and 5:00PM (UTC)

We're experiencing an issue sending Slack and email notifications for interview reminders between 4:00PM and 7:00PM (UTC)

Conditional offer fields are not correctly respecting conditions, preventing some offers from being sent out. We have identified the root cause and are remediating.

We have remediated the root cause and are monitoring.

This incident has been resolved.

Integrations degraded due to an infrastructure issue. Affected services include Google Workspace integration (including sending via Gmail); ATS syncs; autocomplete for candidate resumes/locations; and some AI features.

We are aware of some customers being unable to load saved dashboards. We are investigating this issue.

This issue has been diagnosed and a fix has been deployed. We're monitoring for further issues but expect this to be working for all customers now.

This incident has been resolved.

We've identified an issue that's causing embedded job pages to not load in Safari

We've identified that this issue affects any job page (embedded or not) in Safari.

We have implemented a fix and are monitoring. Affected candidates or any user will need to refresh any links from jobs.ashbyhq.com or you.ashbyhq.com.

This incident has been resolved.

## **Summary** On Monday, June 15, from approximately 4:00 AM PDT to 10:26 AM PDT, Safari and iOS users encountered an error with embedded and Ashby-hosted job boards \([jobs.ashbyhq.com](http://jobs.ashbyhq.com)\) and could not view or apply to jobs.  We became aware of the problem at 9:43 AM PDT and recovered at 10:26 AM PDT. ## **Why did this happen?** Ashby's internal code build process processes the code that powers your hosted and embedded job boards. As part of our security practices, each file is published with verification data that browsers use to confirm the file they download is exactly what we intend to serve. We shipped a change to this build process that modified the files after their verification data had been generated, so the files no longer matched the verification data and failed to load in Safari and iOS mobile browsers. The same files did not fail in the same way in other browsers, such as Chrome and Firefox, which is why the failure appeared inconsistent amongst visitors. ## **How did we resolve this situation?** After confirming the failure was tied to a recent deploy, we rolled back to a known-good state to restore service. We confirmed that job boards loaded correctly in Safari, on mobile devices, and in other browsers before resuming normal deployments. ## **How did we get to this point?** The change altered our shipped code in a way we did not anticipate, and two gaps let it reach production undetected:  1. Our automated job board tests ran only against Chrome. 2. We didn’t have a check verifying file integrity at the point of publishing, where this particular failure surfaced. ## **What have we put in place to prevent this from happening in the future?** 1. We added an automated check that verifies file integrity immediately before publishing; any mismatch now blocks the deployment of updates to Ashby. We confirmed it catches this exact failure. 2. We corrected how integrity verification is applied to our job board code, so that all browsers enforce it consistently. 3. We expanded our automated job board tests to run against Safari and Firefox, not just Chrome. 4. We have added additional monitoring, including on mobile browsers, to alert us directly when a job board fails to load in this way.

We are investigating an issue with the AI Notetaker which may affect the Meeting Bot's ability to attend a meeting

All upstream provider issues have been resolved. We are investigating the affected AI Notetaker instances.

Our upstream provider has indicated that their service is operating normally, and we consider the incident resolved. Please reach out to support@ashbyhq.com in case you experience any delays or issues with AI notetaker calls going forward.

This incident has been resolved.

We are currently investigating this issue.

We are currently investigating the issue

We are continuing to investigate this issue.

We are seeing signs of recovery, though there may be residual instability. AI features will be unavailable as we mitigate the remaining issues.

The application has recovered. AI features are still affected, and we are actively working to mitigate the issues.

AI features are enabled again, and we're processing the backlog of accumulated work.

All systems are now operational, we are continuing to monitor

This incident has been resolved.

## **Summary** On Sunday, May 31st, at 10pm PST, Ashby was slow and had a high error rate for approximately 1 hour, from 10pm to 11:10pm \(PST\). During this time, the Ashby main application would have been slow, and approximately 50% of user actions would have resulted in Ashby displaying an error. Some candidates may have noticed a slower response to their job applications, but all applications were successful and none were lost. ## **Why did this happen?** On Sunday, May 31st, at 4pm PST, an account with one of our AI providers became unavailable. As a result, calls to that provider started to error. These calls are configured to automatically retry periodically until they succeed, but the retries caused a steady increase in the number of such calls per minute. In turn, this drained the capacity of a shared subsystem that manages many different processes at Ashby. In particular, that shared subsystem is used for storing user session data. As the shared subsystem became overwhelmed, user sessions became slower to respond. This caused requests to our web servers to take significantly longer to process. Which then caused our web servers to begin running out of capacity. At 6am, capacity reached a critical low. At this point, our system began rejecting user actions. This is done to prevent complete system failure. ## **How did we resolve the situation?** At 10:04pm PST, our on-call engineers were notified. By 10:16pm PST, our on-call engineers were beginning an investigation. At 11:03pm PST, we identified the subsystem that had reached capacity. At 11:07pm PST, we stopped processing certain types of automated actions. At 11:09pm PST, we identified the issue with the external provider and rectified it immediately. At 11:10pm PST, error rates were down to 10%. At 11:20pm PST, error rates were down to normal levels \(~0%\). At 11:54pm PST, normal service was resumed, and no data was lost. ## **What have we put in place to prevent it from happening in the future?** We’ve done an internal postmortem on this incident and have implemented or plan to implement a variety of changes that achieve three things: 1. Reduced the likelihood of future failure 2. Faster incident response times 3. Lower impact on customers in the event of failure Specifically, we have done the following: * We were already in the process of replacing the subsystem at the center of the outage. We have accelerated that work. The replacement is not vulnerable to this particular kind of failure. * We’ve added additional monitoring around our AI providers to alert us to failures like this sooner. * We are prioritizing a fix for the retry mechanism that led to this incident. * We are migrating our user session storage to a new subsystem that is not vulnerable to this particular kind of failure.

We are investigating intermittent elevated error rates on the job board API.

We have identified the issue and are working on a fix.

We have rolled out the fix and are monitoring performance.

This incident has been resolved.

We have identified an issue with external/agency recruiters being unable to add notes to candidates' profiles. A fix is being deployed.

We have deployed the fix and are monitoring the situation.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We are currently investigating reports of columns not showing up in Application Review

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

This incident has been resolved.

We've identified an issue that's causing some resumes to upload slowly.

We've implemented a fix. New resume uploads should no longer be slow, and existing uploads in progress should complete within the next hour. We are monitoring and will post an update in 30m.

The issue has been resolved.

We are investigating an issue with the AI Notetaker - the Meeting Bot fails to show up to the interview.

This incident has been resolved.