История сбоев Shiphawk

1. 2 сентября 2026 г. в 16:24 UTCmonitoring

   Веб-сервисы FedEx в настоящее время испытывают ухудшение производительности, и некоторые клиенты сообщили о проблемах с бронированием поставок FedEx. Мы продолжим следить за ситуацией, пока FedEx работает над решением вопроса. Текущее время ответа FedEx доступно здесь: https://www.shippingapimonitor.com/history.html?api=fedex

2. 2 сентября 2026 г. в 21:59 UTCresolved

   Решение принято на стороне FedEx.

Автоматический перевод официального обновления инцидента.

1. 25 августа 2026 г. в 14:10 UTCinvestigating

   Некоторые клиенты испытывают медлительность при доступе к WMS. Наша команда активно изучает этот вопрос.

2. 25 августа 2026 г. в 16:01 UTCmonitoring

   Реализовано исправление, и мы отслеживаем результаты.

3. 25 августа 2026 г. в 17:40 UTCresolved

   Этот инцидент был урегулирован. Мы поделимся более подробной информацией, как только они будут доступны.

4. 27 августа 2026 г. в 22:21 UTCpostmortem

   ## Post-Incident Report: WMS Slowness and Errors for Warehouses on One Database Instance - August 25, 2026 **Status:** Resolved **Incident window:** August 25, 2026, ~04:30 - 08:45 PDT \(07:30 - 11:45 ET\) **Affected:** Customers whose databases are hosted on one WMS database instance in our US-East warehouse group: page load times of up to 20x normal, and intermittent errors on scanner and admin screens during the worst period. **Not affected:** All other WMS database instances and warehouse groups, the TMS platform, data integrity \(no transaction was lost, duplicated, or partially applied\), and all completed work - every transaction that was accepted was processed correctly. ### Were you affected? Impact was limited to customers whose WMS databases are hosted on one specific database instance in our US-East warehouse group, and only during the morning of August 25 \(approximately 04:30 - 08:45 PDT / 07:30 - 11:45 ET\). If you did not see slow page loads in the WMS during that window, your environment was not involved. All other WMS database instances and warehouse groups, and the entire TMS platform, operated normally throughout. ### Summary On the evening of August 24, a third-party ETL service that copies ShipHawk data into our data warehouse restarted a large batch of its sync jobs at once against one of our production databases. Each restarted job began re-reading a backlog of historical change data at full speed, in parallel and without any rate limiting. Read volume climbed to roughly five times the expected peak and consumed most of the disk bandwidth available to that database instance. This began overnight, when warehouse activity is light, so it had no effect on operations at the time. When US-East morning shifts started on August 25, normal WMS activity was added on top of the already-saturated channel and the instance reached its bandwidth limit. From the application's point of view this appeared as slow database responses: queries that normally return in milliseconds took far longer, work queued up behind them, and pages loaded slowly. Where a database response exceeded the application's timeout, the page returned an error instead of loading. The service remained operational throughout. Across the affected instance, customers processed approximately 70% of the volume normally handled in that window \(picks, packs and inventory moves\), though the experience was slow and at times difficult to work with, and the degree of impact varied between customers. The situation resolved by revoking the ETL service database credentials entirely at 08:37 PDT. The database recovered within eight minutes. Delayed work flushed through over the following two hours, and all affected warehouses were back to normal pace. No customer action was or is required, and no data was affected. No transaction was lost, duplicated, or partially applied - we verified this against integration logs covering the full incident window. Work that was submitted either completed correctly or failed cleanly before making any change. This is covered in more detail below. ### What was affected Impact was limited to customers whose databases are hosted on the affected WMS database instance in the US-East warehouse group. Throughout the incident window the system was slow across the board - scanner and admin pages that normally load in well under a second took many times longer - and where a database response exceeded the application timeout, the page returned an error rather than loading. What this looked like in practice: **Warehouse floor:** scanner pages \(picking, moving, adjusting inventory\) loaded very slowly; a worker who retried while a page was stuck could receive an error page and have to go back and repeat the action. **Errors were concentrated in a single burst rather than spread across the incident.** Most of them fell within one 15-minute window at the peak of the congestion \(06:15 - 06:30 PDT\). Counting confirmed error pages in our web-server and application logs, the most affected warehouse saw 71 in that window. **The system remained up throughout.** Every submitted transaction either completed correctly or failed cleanly before making any change. During the deepest slowdown window we can show hundreds of transactions completing successfully for the users who continued working. ### What was NOT affected **Data integrity.** The errors occurred at the very start of request processing, before any change was made. No transaction was lost, duplicated, or partially applied. Every fulfillment, inventory move, and shipment posting that completed did so correctly - we verified the integration logs for the incident window. **Order and shipment synchronization to ERPs and marketplaces** completed correctly throughout; postings that queued up during the slowdown were delivered in full during the catch-up \(verified in integration logs - no failed postings\). **All other environments.** Warehouses on our other database instances, and the entire TMS platform, operated normally. **Security and tenancy.** No security boundary was involved at any point. The third-party service in question is a data-integration vendor operating under credentials we issued; the issue was the volume of its reads, not any unauthorized access. ### Timeline \(all times PDT; add 3 hours for ET\) | Time | Event | | --- | --- | | Aug 24, 22:54 - 22:57 | The ETL service restarts ~22 sync jobs against the database within a three-minute window. Each begins re-reading historical change data at full speed. | | Aug 24, 22:54 - 23:50 | Read volume climbs to roughly five times the expected peak, consuming most of the disk bandwidth available to the instance. Overnight warehouse traffic is light, so there is no customer-visible effect yet. | | Aug 25, ~04:30 | US-East warehouse morning shifts begin. Combined demand exceeds the capped network speed; queues start building and the first pages begin rendering slower than normal. | | 05:30 | Automated response-time monitoring alerts as warehouse activity ramps up; customer reports of slowness arrive in the same period. Investigation begins. | | 06:00 - 07:00 | Peak congestion: database connections spike to ~15x normal as requests pile up; the wave of scanner-screen errors occurs \(06:15-06:30\). | | 06:50 | Root cause identified: disk bandwidth over the instance limit; the vendor's replication streams identified as the driver. | | 07:00 | Heaviest internal report queries disabled to free capacity - partial relief. | | 07:20 - 08:30 | ETL service sync jobs are paused in waves in its console and its database sessions terminated; the service automatically reconnects within seconds each time and continues reading. During this period it starts additional jobs. | | 08:35 | The ETL service database credentials are locked and its sessions terminated a final time. | | 08:38 - 08:45 | Database queues drain; page response times return to normal. Customer impact ends. | ### Why resolution took ~3 hours from first reports Three factors extended the timeline. First, the trigger occurred seven hours before symptoms. The ETL service's re-read ran overnight and had already consumed the available bandwidth, but with warehouse activity light at that hour the constraint produced only a slight change in system response times - below our alerting thresholds - so it went undetected. Our automated monitoring did alert once warehouse activity ramped up in the morning, but by then the underlying change was seven hours old and there was no recent deployment or configuration change to point to. Second, the ETL service's replication reads are invisible to standard database query logs - they use a replication protocol rather than queries - so identifying them as the consumer required correlating disk, network, and connection-level evidence. Third, the ETL service is built to survive interruptions: pausing its jobs and terminating its connections both failed as mitigations because it reconnects automatically within seconds, and it restarted additional jobs while we were pausing others. Only revoking its credentials stopped it. ### What we are changing **Tuning WMS response-time alert thresholds.** The condition behind this incident was present for seven hours overnight, but under light load it moved response times too little to cross our alert thresholds - so the first alert came only once warehouse activity ramped up and customers were already affected. We are tuning those thresholds to be sensitive to smaller shifts in WMS response time, including at low load, so events like this are caught and acted on before they reach customers. This includes alerting on the specific leading indicators of this incident - disk bandwidth consumption and disk queue depth. **The database has been migrated to an instance type with substantially more disk bandwidth**, giving significant headroom above peak demand to absorb spikes of this kind. **We are continuing our investigation with the ETL vendor.** We have an open case with them seeking an explanation for the simultaneous job restart, and requiring rate limiting and concurrency caps for re-reads against customer sources. That work is ongoing.

Автоматический перевод официального обновления инцидента.

1. 20 августа 2026 г. в 14:06 UTCinvestigating

   Мы получили сообщения о том, что TMS WebPortal возвращает ошибки или не загружается для некоторых клиентов. Мы активно изучаем проблему и будем предоставлять обновления по мере поступления дополнительной информации.

2. 20 августа 2026 г. в 15:02 UTCidentified

   Этот вопрос был определен, и в настоящее время осуществляется исправление.

3. 20 августа 2026 г. в 15:16 UTCmonitoring

   Реализовано исправление, и мы отслеживаем результаты.

4. 20 августа 2026 г. в 18:06 UTCresolved

   Этот инцидент был урегулирован.

5. 20 августа 2026 г. в 19:35 UTCpostmortem

   # Post-Incident Report: Elevated API and Login Errors - August 20, 2026 **Status:** Resolved **Incident window:** August 20, 2026, 06:31 - 08:11 PDT \(13:31 - 15:11 UTC\) **Affected:** ShipHawk API and dashboard requests in shared production environments, plus the login service. Impact was partial rather than a complete outage: approximately 25% of overall API traffic on [shiphawk.com](http://shiphawk.com) failed during its affected window; failure rates within affected environments ranged from approximately 37% to 48%, and approximately 41% of login-service requests failed. **Not affected:** In-Cart rating \(and all /api/v4/rates requests\), background processing \(all scheduled jobs, write backs, webhooks, async label generation, tracking and carrier communications ran normally\), data integrity. ## Summary On the morning of August 20, an operating-system critical security update published by Ubuntu - and applied automatically by our standard patching process - contained a defect in the web server component \(nginx\) that sits in front of the ShipHawk application. The affected package was published on August 19 as [USN-8563-3](https://ubuntu.com/security/notices/USN-8563-3). Ubuntu confirmed that this update introduced a regression and published [USN-8563-4](https://ubuntu.com/security/notices/USN-8563-4) the same day, reverting the problematic change pending further investigation. While the faulty version was running, the proxy layer corrupted the URL of many incoming requests before handing them to the application. The application could not match the corrupted URLs to any known endpoint and answered **404 Not Found**. The failures were immediate, clean rejections: no request was partially processed, routed to the wrong account, or lost after acceptance. Our servers do not all download and install operating-system security updates at the same moment; update checks and installation windows are staggered across hosts. As a result, some servers downloaded the faulty nginx build before Ubuntu published the corrected package, while others checked later and downloaded the corrected build directly. Only the servers that had already downloaded the faulty package became affected when their scheduled installation ran. This is why the issue appeared intermittent: otherwise-identical requests could fail or succeed depending on which server handled them. Even on servers running the faulty nginx package, only a subset of requests failed. The regression affected specific nginx routing rules rather than the entire proxy configuration, so many URL patterns continued to work normally on an affected server. The incident was fully resolved by 08:11 PDT after every affected server was upgraded to the corrected package and verified healthy. No customer action was or is required. ## What was affected The numbers below count **only failures caused by this incident**. Ordinary 404 responses \(lookups of records that genuinely don't exist, invalid URLs, bot traffic\) were identified by their distinct response signature and excluded. | Environment | Scope | Impacted window \(PDT\) | Failed requests | | --- | --- | --- | --- | | sh-p-1 environment | 2 of 3 web servers | 06:34 - 08:08 | ≈37% of requests on affected servers; ≈25% of overall API traffic | | Login service | Both servers | 06:33 - 08:08 | ≈41% of login-service requests | | sh-p-2 environment | 2 of 3 web servers | 06:31 - 08:08 | ≈47% of requests on affected servers | | sh-p-3 environment | 2 of 3 web servers | 06:31 - 08:11 | ≈48% of requests on affected servers | ‌ What this looked like in practice: * **API integrations** received HTTP 404 responses for valid requests. Because failures were immediate and stateless, client retries could succeed when they landed on an unaffected server. * **Dashboard and login** pages failed to load or sign in intermittently. * Failures depended on the exact URL: some request types passed through unaffected even on faulty servers, adding to the intermittent appearance. ## What was NOT affected * **In-cart rating requests.** All rating requests from the web portal, e-commerce platforms, ERP platforms and regular API requests to `/api/v4/rates` were working as usual. * **Background jobs were not affected at all.** All asynchronous processing - scheduled jobs, write backs, inventory sync, webhook deliveries, document and label generation, carrier and ERP communications - runs behind the proxy layer and continued normally throughout the incident. No queued work was lost or delayed. * **Data integrity.** No data was lost, altered, or corrupted. Requests either completed normally or were rejected outright. * **Security and tenancy.** No request was routed to another account, and no security boundary was crossed. The corruption occurred after all access controls were applied. The underlying Ubuntu update was a preventive security patch; the vulnerability it addressed was not exploited on our systems. ## Timeline \(all times PDT, August 20, 2026\) * **Aug 19 \(daytime\)** - Ubuntu publishes a security update for nginx; a defect is reported, and Ubuntu publishes a corrected package the same day. The corrected version propagates to public update mirrors overnight. * **Aug 19, 18:24 - 23:09** - The nightly update checks on the later-affected servers download the day's nginx update. At these moments, the faulty build is still the newest available on the mirrors. This step only downloads the package; installation happens during the next morning's patch window. * **Aug 20, 04:12 - 05:05** - Another group of servers runs its nightly update check after the corrected build has reached the mirrors. These servers download the fixed version and remain healthy throughout the incident. * **~06:00** - A routine, unrelated application configuration update is applied for upcoming releases. It has no effect on currently released functionality and **plays no role in the incident**, but because it is the only known change that morning, it becomes the first suspect once errors appear. * **06:00** - The nightly automated patching window begins rolling nginx updates across environments. Some servers already have the corrected package downloaded, while others have the faulty package. * **06:31 - 06:34 - INCIDENT START.** As the rolling automated patch window progresses, the previously downloaded faulty nginx package is installed on multiple web and login servers across shared production environments. Because patch schedules are staggered, not all servers update at once, and some servers remain healthy. The first customer-facing failed requests begin at **06:31**. * **06:35** - Automated external monitoring alerts on elevated errors. **Investigation begins immediately.** * **06:36 - 07:15** - Engineers first investigate the ~06:00 configuration update, the only known application-level change with closely matching timing. It is ruled out, and attention turns to the web/proxy layer. * **06:52** - The rolling patch window continues and the faulty package is activated on additional servers. Impact increases as more affected servers restart onto the faulty nginx version, while servers that downloaded Ubuntu's corrected package remain healthy. * **06:55** - A remaining web server updates using Ubuntu's corrected package and stays healthy throughout, continuing to serve its share of traffic correctly. * **07:18 - 07:19** - Affected web servers are restarted as a mitigation attempt. This has no effect because the faulty nginx package remains installed. * **07:20 - 07:55** - Suspect servers are removed from load-balancer rotation. Symptoms persist because the login service and application environments are independently affected, which materially widens the search. * **07:41** - The URL-corruption pattern is identified in application logs. * **07:45 - 08:00** - Per-server testing isolates the faulty servers. The only difference from healthy servers is the nginx package version. The faulty build is matched to Ubuntu's published regression notice and corrected package. * **08:02 - 08:11** - The corrected package is installed across all affected servers. Error rates return to normal immediately on each server as it restarts onto the fixed version. The final affected environment returns to normal at **08:11 - INCIDENT FULLY RESOLVED**. * **08:11\+** - Full verification is completed: every server is individually tested, and API, dashboard, login, and production environments are confirmed healthy. ## Why resolution took ~95 minutes from alert Detection was fast, but three factors slowed diagnosis. First, a routine configuration change earlier that morning was the only known change in the environment and had to be ruled out - automated OS patching does not appear in any application-level change log. Second, the failure was intermittent by nature: unaffected servers continued serving normally, and even affected servers successfully handled request types whose routing rules were not impacted. Third, removing the suspect servers from rotation did not stop the errors - because other tiers were independently affected - which initially pointed the investigation away from those servers. ## What we are changing **1. Stage operating-system security patches before production.** Automated OS- and nginx-level security updates, including critical patches, will first be installed on non-production servers. Automated application-level validation will exercise representative API, dashboard, and login paths against the updated servers before the same package versions are allowed to roll into production. Production rollout will begin only after those checks pass. **2. Faster version-level diagnosis.** Our incident runbooks now include immediate comparison of package versions and restart history across servers whenever identically-configured servers behave differently.

Автоматический перевод официального обновления инцидента.

1. 26 июня 2026 г. в 15:47 UTCmonitoring

   We are seeing a degraded performance with FedEx API, some customers reported issues with booking FedEx shipments. We are actively monitoring the situation and will provide updates as soon as more information becomes available or the issue is resolved.

2. 26 июня 2026 г. в 20:49 UTCresolved

   Resolved by FedEx

1. 19 июня 2026 г. в 15:38 UTCmonitoring

   We are seeing a degraded performance with USPS Endicia web services. We will continue to monitor as USPS Endicia works to resolve this. https://status.endicia.com

2. 20 июня 2026 г. в 02:17 UTCresolved

   This incident has been resolved.

1. 18 мая 2026 г. в 18:21 UTCidentified

   PrintNode is currently experiencing issues. Some regions may be experiencing degraded performance when attempting to print. See https://www.printnode.com/en/status for details.

2. 18 мая 2026 г. в 21:43 UTCresolved

   Resolved by PrintNode.

1. 23 февраля 2026 г. в 20:12 UTCidentified

   We are seeing a degraded performance with FedEx web services, some customers reported issues with booking shipments. FedEx has acknowledged an issue on their end. We are actively monitoring the situation and will provide updates as soon as more information becomes available or the issue is resolved.

2. 24 февраля 2026 г. в 22:52 UTCresolved

   Resolved by FedEx.

1. 20 октября 2025 г. в 20:37 UTCinvestigating

   Amazon Web Services had a major outage today that effected a large number software services across the internet, including ShipHawk. While ShipHawk's software did not go down, the outage has led to intermittent reliability for many parcel and LTL carriers regarding rating and label generation. We are working with AWS to resolve these issues as soon as possible. https://health.aws.amazon.com/health/status

2. 20 октября 2025 г. в 20:56 UTCmonitoring

   Amazon Web Services had a major outage today that effected a large number software services across the internet, including ShipHawk. While ShipHawk's software did not go down, the outage has intermittently affected some carriers regarding rating and label generation. Everything is working currently and we are working with AWS to ensure system responsiveness. https://health.aws.amazon.com/health/status

3. 20 октября 2025 г. в 23:28 UTCresolved

   The AWS issue affecting carriers has been resolved.

1. 10 октября 2025 г. в 17:51 UTCinvestigating

   We are seeing a degraded performance with USPS Endicia web services. We will continue to monitor as USPS Endicia works to resolve this. https://status.endicia.com

2. 10 октября 2025 г. в 18:18 UTCresolved

   Resolved by USPS Endicia

1. 29 сентября 2025 г. в 18:05 UTCidentified

   Customers using WWEX / SpeedShip integrations may experience issues with rating and booking shipments due to an outage of the WWEX / SpeedShip API. The provider has notified us that they are actively working on a resolution. We will continue monitoring and provide updates as they become available.

2. 29 сентября 2025 г. в 22:40 UTCresolved

   Confirmed as resolved by WWEX.

1. 20 июня 2025 г. в 13:30 UTCidentified

   PrintNode is currently experiencing connectivity issues. Some regions may be experiencing degraded performance when attempting to print. See https://www.printnode.com/en/status for details.

2. 20 июня 2025 г. в 13:42 UTCidentified

   Connectivity was restored to normal approximately 7 minutes ago; we are monitoring the situation and will post further information if and when it becomes available.

3. 20 июня 2025 г. в 17:21 UTCmonitoring

   A fix has been implemented and we are monitoring the results.

4. 21 июня 2025 г. в 04:07 UTCresolved

   This incident has been resolved.

1. 5 мая 2025 г. в 17:27 UTCidentified

   Pitney Bowes has notified us they are experiencing technical issues with PB Expedited Delivery Services. Customers who use Pitney Bowes may experience slowness or unavailability of Pitney Bowes shipping rates or printing labels. We will continue to monitor their progress as they resolve this issue. For additional information use Pitney Bowes status page - https://status.pitneybowes.com

2. 6 мая 2025 г. в 23:10 UTCresolved

   This incident has been resolved.

1. 14 апреля 2025 г. в 20:24 UTCinvestigating

   We are currently investigating an issue affecting one of our WMS environments. A subset of customers may be unable to access the system. Our team is actively working to identify the root cause and restore full access as soon as possible. We will continue to provide updates as we make progress. Thank you for your understanding and patience.

2. 14 апреля 2025 г. в 20:34 UTCmonitoring

   The affected WMS environment is now back online, and system access has been restored for impacted customers. We are currently monitoring the environment to ensure stability.

3. 14 апреля 2025 г. в 20:47 UTCresolved

   This incident has been resolved.

1. 11 марта 2025 г. в 11:52 UTCinvestigating

   Some customers are experiencing slowness with the rating API. Our team is actively investigating the issue. We will provide updates as we learn more.

2. 11 марта 2025 г. в 12:00 UTCmonitoring

   The issue causing slowness in the rating API has been identified and resolved.

3. 11 марта 2025 г. в 22:34 UTCresolved

   This incident has been resolved.

1. 30 января 2025 г. в 13:42 UTCidentified

   Pitney Bowes has notified us they are experiencing technical issues with PB Expedited Delivery Services. Customers who use Pitney Bowes may experience slowness or unavailability of Pitney Bowes shipping rates or printing labels. We will continue to monitor their progress as they resolve this issue. For additional information use Pitney Bowes status page - https://status.pitneybowes.com

2. 30 января 2025 г. в 15:59 UTCresolved

   Resolved by Pitney Bowes

1. 25 июля 2024 г. в 17:53 UTCidentified

   We are seeing a degraded performance with UPS web services. We will continue to monitor as UPS works to resolve this. To see current response times from UPS you can check: - https://downdetector.com/status/ups/ - https://www.shippingapimonitor.com/history.html?api=ups

2. 25 июля 2024 г. в 20:00 UTCmonitoring

   UPS resolved the issue on their side, we will continue to monitor it.

3. 26 июля 2024 г. в 00:14 UTCresolved

   This incident has been resolved.

1. 18 июня 2024 г. в 17:03 UTCidentified

   USPS Endicia has noted that they have fixed the postage printing issue. See USPS Endicia status page for more details: https://status.endicia.com/

2. 18 июня 2024 г. в 23:19 UTCresolved

   Resolved by Endicia.

1. 17 июня 2024 г. в 16:01 UTCidentified

   We are seeing a degraded performance with FedEx web services. We will continue to monitor as FedEx works to resolve this. To see current response times from FedEx you can check: https://www.shippingapimonitor.com/history.html?api=fedex

2. 18 июня 2024 г. в 17:01 UTCresolved

   Resolved by FedEx.

1. 22 мая 2024 г. в 17:46 UTCidentified

   We are seeing a degraded performance with FedEx web services. We will continue to monitor as FedEx works to resolve this. To see current response times from FedEx you can check: https://www.shippingapimonitor.com/history.html?api=fedex

2. 22 мая 2024 г. в 20:20 UTCresolved

   Resolved by FedEx team.

1. 17 мая 2024 г. в 19:18 UTCidentified

   We are seeing a degraded performance with UPS web services. We will continue to monitor as UPS works to resolve this. To see current response times from UPS you can check: - https://downdetector.com/status/ups/ - https://www.shippingapimonitor.com/history.html?api=ups

2. 17 мая 2024 г. в 22:41 UTCresolved

   Resolved by UPS.