Снижение производительности Query API влияет на американские проекты
Начало 26 августа 2026 г. в 21:39 UTC · 1h 7m
OutageСерьёзный инцидент
Затронутые компоненты
Application Availability (US)
investigating
Mixpanel испытывает ухудшение производительности API-интерфейса запросов, что приводит к ответам HTTP 500 при запросе отчетов для проектов с резиденцией данных США. Проекты с ЕС и IN Data Residency остаются без изменений.
Мы ценим ваше терпение, пока наши инженеры работают над восстановлением нормальной функциональности. Мы будем публиковать обновления прогресса на нашей странице статуса.
Если у вас есть какие-либо вопросы, пожалуйста, обратитесь в службу поддержки.
identified
Этот вопрос был определен, и в настоящее время осуществляется исправление.
monitoring
Было реализовано исправление, и мы видим улучшенные показатели успеха для API запросов. Мы продолжим мониторинг для обеспечения стабильности.
resolved
Этот инцидент был урегулирован.
Автоматический перевод официального обновления инцидента.
Проблемы заполнения свойств событий в выпадающих меню
В настоящее время мы испытываем проблемы с заполнением свойств событий в выпадающих меню. Мы ценим ваше терпение, пока наши инженеры работают над восстановлением функциональности. Если у вас есть вопросы, пожалуйста, свяжитесь с [email protected]
identified
Мы продолжаем работать над решением этой проблемы.
monitoring
Реализовано исправление, и мы отслеживаем результаты.
resolved
Этот инцидент был урегулирован.
Автоматический перевод официального обновления инцидента.
We are aware that the Get-Report tool in the Mixpanel MCP server is currently failing when called with skip_results: false. Report metadata is unaffected. We are investigating the issue and will provide updates as we have them.
identified
The issue has been identified and fix is being implemented.
monitoring
A fix has been deployed and we're now monitoring the results. Report queries in the Mixpanel MCP server should be functioning normally. We'll continue to watch closely and provide a final update once we've confirmed full resolution.
resolved
This incident has been resolved.
Вопросы, обслуживающие ответы агента Mixpanel
Начало 22 июля 2026 г. в 08:03 UTC · 6h 44m
Pending
investigating
В настоящее время мы испытываем проблемы с агентом Mixpanel в веб-приложении. Наша инженерная команда была предупреждена и изучает проблему и работает над восстановлением ее функциональности. Мы ценим ваше терпение, поскольку мы работаем над решением этой проблемы.
Если у вас есть какие-либо вопросы, пожалуйста, обратитесь в службу поддержки.
investigating
команда продолжает исследовать проблему и дальнейшие шаги. Спасибо за ваше терпение.
monitoring
Реализовано исправление, и мы отслеживаем результаты.
resolved
Этот инцидент был урегулирован.
Автоматический перевод официального обновления инцидента.
Временная задержка приема данных для проектов в США
Начало 11 июля 2026 г. в 07:53 UTC · 8h 1m
IssuesНезначительный инцидент
Затронутые компоненты
Ingestion API Availability (US)
investigating
В настоящее время мы испытываем задержки в приеме данных, что влияет на данные в реальном времени для проектов, зарегистрированных в проектах США, что приводит к задержкам для подмножества проектов. Несмотря на то, что данные не теряются, наша команда инженеров активно изучает этот вопрос и работает над восстановлением функциональности в режиме реального времени. Мы ценим ваше терпение в это время.
Если у вас есть какие-либо вопросы, пожалуйста, обратитесь в службу поддержки.
investigating
Мы продолжаем расследование этого вопроса.
identified
Этот вопрос был определен, и в настоящее время осуществляется исправление.
identified
Мы продолжаем работать над решением этой проблемы.
monitoring
Реализовано исправление, и мы отслеживаем результаты.
resolved
Этот инцидент был урегулирован.
postmortem
# Summary
Between approximately **11:35 PM PT on July 10 and 7:19 AM PT on July 11, 2026**, data ingestion for Mixpanel projects in the US region ran behind by up to ~2 hours. During this window, reports and dashboards temporarily showed incomplete data — recent time ranges could appear as sharp, artificial drops in metrics such as active users or revenue. **No data was lost.** All events were queued durably and processed in full once the backlog cleared; metrics returned to accurate values on their own, with no customer action required.
The cause originated in our ingestion controls: for a small number of very high-volume projects, the ingestion rates we allowed had drifted out of alignment with the capacity provisioned for those projects. Large historical imports — entirely legitimate use of the platform — were therefore admitted faster than their infrastructure could absorb, and two properties of our pipeline turned that localized overload into a region-wide delay. The fixes below realign those controls so that any import, of any size, is automatically kept within safe bounds.
# What happened
Mixpanel's ingestion pipeline is sharded and multi-tenant: each project's data is distributed across a set of partitions sized for its expected volume, and for throughput, events from many customers are processed together in batches. This design delivers high efficiency, but it depends on one invariant: the rate at which we admit a project's traffic must match the capacity provisioned for it. When that invariant holds, even very large imports are absorbed smoothly.
Here, it did not hold. A small number of very large historical data imports ran on projects whose allowed ingestion rates had, over time, grown well beyond their provisioned partition capacity. The excess volume concentrated onto specific partitions as **hot spots**, saturating the portion of the streaming fleet serving them. Two factors then widened the impact:
* **Batched, multi-tenant processing amplified the hot spots.** Because events from many customers travel together in batches, slowness and failures on the overloaded partitions delayed unrelated customers' events sharing those batches.
* **Automatic scale-up was ineffective.** Adding capacity cannot dissolve a hot spot of this kind, because the overloaded partitions stay pinned to the same infrastructure, and a capacity limit in a component of our streaming infrastructure prevented the added capacity from taking effect.
Together, these turned what should have been a brief, self-healing slowdown into a multi-hour delay requiring manual intervention.
# Timeline \(Pacific Time, July 10–11\)
* **11:35 PM** — Automated alerting detected the ingestion backlog; on-call engineer engaged immediately.
* **12:49 AM** — Status page incident posted; impact scoped to the US region only.
* **1:23 AM** — The largest contributing import was paused.
* **1:55–6:48 AM** — Progressive mitigations: additional traffic sources throttled, certain backlogged data deferred with the owning customer's agreement, failure isolation enabled in the pipeline, and affected infrastructure nodes replaced.
* **7:19 AM** — Backlog fully processed; all projects current. Status page moved to monitoring, then resolved after a stable observation period.
# Root cause
1. **Ingestion rate limits misaligned with provisioned capacity.** For the driving projects, the rates our platform admitted had grown out of step with the infrastructure provisioned for them, so legitimate high-volume imports were let in faster than their partitions could absorb. This is the systemic root cause. The imports themselves were a legitimate use of the platform.
2. **Batched, multi-tenant processing amplified the overload.** Failures on the overloaded partitions delayed unrelated customers' events sharing the same processing batches, spreading a localized problem across the platform.
3. **Traffic to the overloaded partitions could not be redistributed.** Partition-to-server assignment in the streaming layer was not load-aware, so the hot spots stayed pinned to the same servers regardless of fleet size — total capacity was sufficient, but it could not be brought to bear. A scaling limit in a component of our streaming infrastructure compounded this by preventing scale-up from adding capacity, which delayed diagnosis until engineers intervened manually.
# What we're changing
The end state we are building toward: **every project's ingestion limits automatically match its provisioned capacity, so that imports of any size, including full historical backfills and data warehouse syncs, can run without advance coordination, and one project's volume is prevented from affecting another's data freshness.**.
Already deployed — improving our ingestion handling:
* **Improved hot-spot handling.** Traffic distribution in the streaming layer is now load-aware, spreading concentrated load more evenly across the fleet, and individual problem items are now retried separately instead of holding up the rest of their batch — reducing, though not eliminating, the impact a localized overload can have on unrelated traffic.
* **Re-provisioned the highest-volume workloads** onto appropriately sized infrastructure, prioritized by risk.
Already deployed — changing how we operate the third-party streaming service:
* **Audited and re-tuned the fleet's capacity profile** so individual servers have substantially more headroom for concentrated load, and worked with the provider to resolve the scaling limitation encountered during the incident.
In progress:
* **Capacity-aware rate limiting** — closing the gaps between individually granted project rate limits and each project's real provisioned capacity, extending rate limiting to ingestion paths that previously lacked it, and coupling any future limit increase to a capacity increase, which is designed to prevent this class of misalignment from recurring..
* **Finer-grained volume monitoring and alerting** so capacity misalignment is detected and corrected before it can affect any customer.
* Evaluating stronger workload isolation/backlog recovery prioritization for bulk/backfill traffic paths, so historical imports have reduced impact on live traffic
# Common questions
* **Was any data lost?** No. Events were durably queued throughout the incident and were fully processed once the backlog cleared. Any metric drops seen during the window were a display artifact of the delay and self-corrected.
* **Do I need to coordinate large imports or backfills with Mixpanel?** No — Our goal is for the platform to keep any import within safe rates automatically, so that backfills and warehouse syncs can run without scheduling or notice. We are still rolling out the capacity-aware controls that deliver this. In the meantime, if you are planning an unusually large import or backfill, we recommend coordinating with your account team so we can confirm capacity in advance.
* **How is this prevented going forward?** The systemic fix is tightening the gaps between individually granted project rate limits and each project's real provisioned capacity, and extending rate limiting to the ingestion paths that previously lacked it — so overload of this kind is stopped at admission. In addition, the scaling limitation that prolonged the incident is fixed, the pipeline now retries individual problem items separately so a localized overload has far less impact on unrelated traffic, and bulk traffic is being further isolated from live traffic.
* **Can individual projects be prioritized during recovery?** This capability did not exist during the incident — all projects recovered at the same rate. We are evaluating prioritization mechanisms for backlog recovery as part of our follow-up work.
We apologize for the disruption and for the concern the temporarily depressed metrics caused. Please reach out through your account team or support with any questions.
Автоматический перевод официального обновления инцидента.
Mixpanel испытывает ухудшение производительности с помощью нашего API запросов, включая повышенную задержку. Вы можете увидеть отчеты о медленной загрузке или ошибки запросов. Мы ценим ваше терпение, пока наши инженеры работают над восстановлением нормальной функциональности. Мы будем публиковать обновления прогресса на нашей странице статуса.
Если у вас есть какие-либо вопросы, пожалуйста, обратитесь в службу поддержки.
identified
Этот вопрос был определен, и в настоящее время осуществляется исправление.
identified
Задержка запросов стабилизировалась, хотя наша команда продолжает исследовать основную причину ухудшения производительности. Мы будем продолжать публиковать обновления по мере продвижения нашего расследования. Спасибо за ваше терпение.
monitoring
Реализовано исправление, и мы отслеживаем результаты.
resolved
Этот инцидент был урегулирован.
Автоматический перевод официального обновления инцидента.
Issue with Inviting and Deleting Users
Начало 3 июня 2026 г. в 20:05 UTC · 1h 22m
Pending
investigating
Mixpanel is currently experiencing a disruption in the ability to invite and delete internal users within organizations. We appreciate your patience while our engineers work to restore this functionality. If you have any questions, please contact [email protected]
identified
The issue has been identified and a fix is being implemented.
resolved
This incident has been resolved.
Temporary Data Ingestion Delay for US, India and EU projects
Начало 2 июня 2026 г. в 11:02 UTC · 5h 51m
Pending
investigating
We are experiencing delays with our data ingestion and shuffling pipeline to projects with all projects. No data is being lost but as a result, real-time data is delayed. We appreciate your patience while our engineers work to restore real-time functionality. If you have any questions, please contact support
identified
The issue has been identified and a fix is being implemented.
monitoring
A fix has been implemented and we are monitoring the results.
resolved
This incident has been resolved.
Temporary Data Ingestion Delay for US, India and EU projects
We are experiencing delays with our data ingestion and shuffling pipeline to projects with US & India data residency. No data is being lost but as a result, real-time data is delayed. We appreciate your patience while our engineers work to restore real-time functionality. If you have any questions, please contact support
investigating
We have now identified an ingestion delay for EU projects, and are continuing to investigate the issue. Thank you for your patience.
identified
The issue has been identified, and a fix is being implemented.
monitoring
A fix has been implemented and we are monitoring the results.
resolved
This incident has been resolved.
Board Access Issues
Начало 19 мая 2026 г. в 19:03 UTC · 2h 15m
Pending
investigating
A subset of users are currently facing issues accessing boards that they previously had access to view. Our Engineering team is actively investigating and will update shortly.
identified
The issues has been identified and a fix is being implemented. As a workaround boards can be explicitly shared with users who are having issues viewing them currently.
monitoring
A fix has been implemented and we are monitoring the results.
resolved
This incident has been resolved.
Query API degraded performance
Начало 14 мая 2026 г. в 23:17 UTC · 1d 3h
IssuesНезначительный инцидент
Затронутые компоненты
Application Availability (US)
investigating
We are experiencing degraded performance with our Query API, including increased latency. You may see slow-loading reports, incomplete query results, query errors, or data discrepancies. We appreciate your patience while our engineers work to restore normal functionality. We will post progress updates on our status page. If you have any questions, please contact support.
identified
The issue has been identified and a fix is being implemented.
identified
We are continuing to work on a fix for this issue.
identified
We've continued to make progress on the issue affecting some US projects. Query success rates have returned to normal levels, and the related processing delays have also recovered.
A subset of affected projects may still see incomplete data in query results while our recovery process runs. We're actively working on this and will share another update in 2 hours.
Impact remains limited to our US region. We do not currently believe any data has been permanently lost.
identified
Recovery is progressing well. Query success rates remain at normal levels, and missing data has now been restored for a portion of affected projects.
We're continuing the recovery process for the remaining affected projects and currently estimate full recovery within approximately 3–5 hours.
Some customers may still see incomplete data in query results until this work is complete.
identified
We are continuing to work on resolving this issue. Our current estimate for full recovery is approximately 3–5 hours. Some customers may still see incomplete data in reports until recovery is complete. We will continue to provide updates.
identified
Recovery is continuing to progress and query latency has returned to normal levels. Our current estimate for full recovery is approximately 3–4 hours. Some customers may still see incomplete data in reports until recovery is complete. We will continue to provide updates.
identified
We are investigating an increase in query latency that appears to be unrelated to the ongoing recovery. You may experience slower-loading reports. Our team is actively looking into the cause and we will provide an update shortly.
Recovery is continuing to progress. Our current estimate for full recovery is approximately 3–4 hours. Some customers may still see incomplete data in reports until recovery is complete. We will continue to provide updates.
identified
Query latency has returned to normal levels. We are continuing to monitor.
identified
Query latency has returned to normal levels and has been resolved.
Recovery is continuing to progress, and our current estimate for full recovery is approximately 4–5 hours. Some customers may still see incomplete data in reports until recovery is complete. We will continue to provide updates.
identified
Recovery is continuing to progress. Our current estimate for full recovery is approximately 1-2 hours. Some customers may still see incomplete data in reports until recovery is complete. We will continue to provide updates.
monitoring
A fix has been implemented and we are monitoring the results.
resolved
This incident has been resolved.
postmortem
# Mixpanel RCA: Transient Data Access Issue, May 14, 2026
## Summary
On Thursday, May 14, 2026 at approximately 2:30 PM PT, a routine but infrequent cleanup operation in Mixpanel's storage system mistakenly removed a portion of production data files in addition to the unused files it was intended to remove. Some customers experienced query errors during the hours that followed. We detected the issue within minutes, deployed mitigations the same evening that returned query success rates and latency to normal, and restored the affected files from backup by 5:15 PM PT on Friday, May 15. Mixpanel's ingestion pipeline was not affected and no event data was lost in transit.
## What happened
This incident was triggered by a storage cleanup procedure that runs periodically to remove files no longer referenced by Mixpanel's metadata. The procedure was more involved than usual: it followed a recent enhancement to our file storage strategy that left a set of unused files behind in our storage backend, and addressing them required extending our standard cleanup approach to cover a new code path.
As part of executing this extended cleanup, an engineer generated the list of files to delete using a SQL query whose date filter was not strictly earlier than the reference snapshot it was being compared against. As a result, a small set of legitimate production files that had been written in the gap window between the snapshot and the filter date were incorrectly classified as unused and removed.
The deletion ran for roughly half an hour before internal alerting caught the resulting query failures and the operation was stopped. The trigger was operator error against an ambiguous runbook, not a defect in the live serving path or in our ingestion pipeline.
## Customer impact
Impact unfolded in two phases.
The first phase ran from Thursday at approximately 2:30 PM PT until 8:11 PM PT — roughly five and a half hours. During this window, customers across the platform may have seen slower or failed queries when their requests touched files that had been deleted. The breadth and severity varied by project depending on which data each query touched. By 8:11 PM PT, mitigations had fully rolled out — queries automatically retried against an alternate availability zone, and a fallback path was put in place to serve missing files from a backup datastore. After this point, query success rate and latency returned to normal.
The second phase lasted from 8:11 PM PT Thursday through approximately 5:15 PM PT Friday, May 15. During this window, fewer than 2% of customers were still affected — specifically, those whose deleted files had not yet been fully restored from backup. The vast majority of these files were recovered by Friday afternoon. A small number of projects \(under 30\) had files that could not be fully recovered from backup, and we are following up with those accounts directly.
## Timeline \(Pacific Time\)
* May 14, 2:30 PM — Cleanup operation begins
* May 14, 3:11 PM — Internal alerting flags query failures; the cleanup operation is stopped within minutes
* May 14, 4:07 PM — Status page banner posted
* May 14, 4:45 PM — Mitigation deployed: queries automatically retry against an alternate availability zone
* May 14, 7:12 PM — Mitigation deployed: queries fall back to a backup datastore for missing files
* May 14, 8:11 PM — Query success rate and latency fully restored to normal levels
* May 15, 5:15 PM — File restore from backup complete; status page banner resolved
## Why this happened
Several contributing factors lined up.
The runbook for this cleanup procedure had ambiguous wording around the ordering and timing of its inputs. It had been recently authored to handle the new file-storage code path and had not gone through a formal review before being used.
Our cleanup tooling did not programmatically enforce the safety invariant that the date filter must be strictly before the reference snapshot. That invariant lived only in operator-authored SQL.
The extended cleanup was being executed in parallel across two storage layers by two different engineers, which increased the room for error.
## What we're doing to prevent recurrence
We have already made or have actively in flight the following changes.
We are adding programmatic safeguards to our cleanup tooling so that an input set whose date filter is not safely before the reference snapshot is rejected before any deletion occurs, along with a reconciliation step that flags any production-referenced file before deletion proceeds.
Destructive cleanup operations will now run in phased stages, starting with internal projects and pausing for a holding period before any broader execution.
Destructive storage operations now require a second engineer to sign off on the exact deletion set and to be present during execution, matching the practice we already follow for database migrations.
We have updated the cleanup runbook with explicit guidance on input timing, required safety buffers, and an enforced review process for any runbook covering a destructive operation.
Longer term, we are working to eliminate the manual portion of this cleanup procedure entirely and route it through our existing automated cleanup infrastructure, so the class of failure that produced this incident is no longer reachable through human input.
## Closing
Reliability and data integrity are foundational to the trust our customers place in Mixpanel, and we recognize the impact this incident had on the teams who rely on us. We are sorry for the disruption. If you have questions about how this incident may have affected a specific project, please reach out to your account team or Mixpanel Support.
Data Volume Monitoring degraded performance
Начало 13 мая 2026 г. в 16:34 UTC · 21h 1m
Pending
identified
Data Volume Monitoring is again experiencing degraded performance due to a recurring upstream provider disruption. We're mitigating now.
identified
We are continuing to work on a fix for this issue.
resolved
This incident has been resolved.
Data Volume Monitoring degraded performance
Начало 12 мая 2026 г. в 17:17 UTC · 3h 57m
Pending
Затронутые компоненты
Application Availability (US)
investigating
We are currently investigating an issue that leads to degraded performance of Data Volume Monitoring in a subset of US-based projects. We truly appreciate your patience and apologize for the inconvenience. If you have any questions, please contact support (https://mixpanel.com/get-support).
identified
An upstream service provider outage is causing this issue. We are deploying a fix to mitigate the disruption.
monitoring
A fix has been implemented, and we are monitoring the result.
resolved
This incident has been resolved.
Snowflake pipeline exports degraded
Начало 6 мая 2026 г. в 00:00 UTC · 20h 4m
Pending
investigating
A subset of projects are experiencing issues exporting data to the Snowflake warehouse via Mixpanel pipelines. We are currently investigating this issue.
identified
We have identified the issue affecting Snowflake pipeline exports for a subset of projects, and our engineering team is working on a resolution.
resolved
This incident has been resolved.
Credit Card Processing Interruption
Начало 9 апреля 2026 г. в 20:03 UTC · 1h 41m
Pending
identified
We are currently experiencing an interruption with our credit card processing. Our engineers have identified the issue and are working on a fix.
monitoring
A fix has been deployed, and all impacted accounts have had their billing re-run. If you continue to receive errors, please ensure the card on file is up to date. If you are still experiencing issues, please submit a support ticket.
resolved
This incident has been resolved.
Degraded Query API Performance impacting EU Projects
Начало 16 марта 2026 г. в 16:50 UTC · 4h 56m
IssuesНезначительный инцидент
Затронутые компоненты
Application Availability (EU)
identified
Mixpanel is experiencing increased query latency for projects with EU residency, which may result in HTTP 500 responses when querying or saving reports. Projects with US and IN residency remain unaffected. Our engineering team is working on a resolution.
monitoring
A fix has been implemented and we are monitoring the results.
resolved
This incident has been resolved.
Error loading Mixpanel Webapp
Начало 2 марта 2026 г. в 21:00 UTC · 53m
Pending
investigating
We are currently experiencing issues with loading Mixpanel.com and are working to restore service as quickly as possible. Our team is investigating the issue and will provide updates as soon as we have more information. Thank you for your patience.
identified
The issue has been identified and a fix is being implemented.
monitoring
A fix has been implemented and we are monitoring the results.
resolved
This incident has been resolved. If Mixpanel is still not loading, please clear your cache and cookies.
Degraded MCP Availability
Начало 2 марта 2026 г. в 19:37 UTC · 6h 5m
Pending
investigating
A subset of users are experiencing OAuth issues when connecting to our MCP server. We are currently investigating this issue.
identified
The issue has been identified and a fix is being implemented.
monitoring
A fix has been implemented and we are monitoring the results.
monitoring
We have partially mitigated the issue and are continuing to monitor for any further impact. We are also actively working to address the root cause.
resolved
This incident has been resolved.
Degraded MCP Availability
Начало 2 марта 2026 г. в 18:02 UTC · 37m
Pending
investigating
A subset of users are experiencing OAuth issues when connecting to our MCP server. We are currently investigating this issue.
identified
The issue has been identified and a fix is being implemented.
monitoring
A fix has been implemented and we are monitoring the results.
resolved
This incident is resolved.
Delays with Integrations Syncs (Cohort Exports)
Начало 30 января 2026 г. в 18:39 UTC · 8h 2m
Pending
Затронутые компоненты
Data Export
investigating
We are currently experiencing delays with Cohort Syncs to external destinations, including custom webhooks and engagement platforms. Our team is investigating the issue and actively working on a resolution. We sincerely apologize for the inconvenience and thank you for your kind understanding.
If you have any questions, please contact support
monitoring
A fix has been implemented, and we are monitoring the results. Recurring cohort syncs should resume on the next scheduled run with no additional action required.
monitoring
We are continuing to monitor for further issues. We have identified that a subset of Cohort Syncs are still experiencing delays and are working to resolve these remaining cases.
monitoring
Additional fixes have been implemented. We are continuing to monitor for any further issues.