Clevertap 장애 기록

Our MEC region customers are experiencing downtime in some of the components. This is due to the disruption in AWS services in the region. We are working with AWS team to bring systems back up. We will provide another update in a hour, or sooner if we have additional information to share.

We have implemented some infra changes and all systems are operational. We are monitoring the systems now.

Systems are down right now. The disruption of AWS services continues in META region and is impacting CleverTap components. Alternate ways to bring the systems backup are being explored. Next update will be shared in one hour.

Disruption in AWS services in the META region is still on-going. Due to this, CleverTap systems are down. The situation is being monitored and further updates will be shared as soon as we have more information.

We have made necessary infra changes to get the systems up and running. Your Success managers have been communicating with you on this. This will continue to happen to address specific queries, relay information about what followed the AWS outage.

There was an outage of the segment refresh module in the context of journeys impacting journey movement across segments for a day. It was fixed within a few hours, resuming from the following day.

We are investigating reports of Dashboard query failures impacting some accounts within the EU region. Data ingestion from both API and SDK is processing with delays for these accounts.

All our systems are back to normalcy. We are currently monitoring this for some time before closing the incident.

This incident has been resolved and all our systems are working as expected.

We are investigating reports of Dashboard Login issues and query failures impacting our accounts within the EU region. Data ingestion from both API and SDK is processing with delays for these accounts. Please expect an update within 30 - 45 minutes.

We have identified the issue and are working on the fix. Login issue has been resolved. At the moment, impacted components are campaign delivery and Data exports.

The issue is still being investigated and we optimising systems. Campaign Delivery and Data exports are still impacted. Please expect the next update in 45 minutes.

We have implemented a fix and all the systems are back to normal now. We are monitoring the systems currently.

The incident has been resolved and all the systems are back to normal.

We are investigating reports of Dashboard query failures impacting some accounts within the SG region. Data ingestion from both API and SDK is processing with delays for these accounts. Please expect an update within 30 minutes.

Root cause has been identified and the fix has been deployed. Systems are back to normal and we are currently monitoring closely.

The partial outage for some clients in the SG region has been resolved and the system is back to normalcy.

We are investigating reports of Dashboard query failures impacting some accounts within the EU region. Data ingestion from both API and SDK is processing with delays for these accounts. Please expect an update within 30 minutes. We apologize for the inconvenience and will update this status page as soon as we have determined the root cause.

We have identified the cause and are working on bringing the system back to normal level of operation for the affected accounts. ETA: 15 minutes.

The partial outage for some clients in the EU region has been resolved and the system is back to normal operation.

Cloudflare is facing issues with its European endpoints. Some of our customers were not able to login to the dashboard. Campaigns/exports/ingestion is working as normal. We have bypassed Cloudflare and directly routed the traffic to our endpoints. The dashboard is working now. We will update this incident shortly.

We have bypassed Cloudflare for now and dashboard logins are working fine. Systems are back to normalcy. We are keeping a watch on Cloudflare status before moving back the traffic via Cloudflare.

We are upgrading the affected cluster in the EU1 region. Please allow us 10-20 minutes to upgrade. Affected users will not be able to use dashboard during this period.

The cluster has been upgraded and systems are back to normal. Users can login and perform the dashboard actions normally. We are observing this for the next few minutes before closing the incident.

This incident is resolved and all systems are back to normal.

One of our clusters in EU1 was unresponsive for 20 minutes. Users on that cluster could not login to the platform for 20 minutes between IST 10:30 AM to 10.50 AM (GMT 5:00 AM to GMT 5:20 AM)

We are experiencing degraded performance on some of our IN (instances). Data ingestion will be delayed and some of the campaigns will be running slow/delayed. Dashboard queries will be slow/not working for a few customers. We are currently investigating the incident and will update here in the next 30 minutes.

The degraded performance issue is resolved. And ingestion of data is getting cleared for the affected instances. We are monitoring the situation and will update here when all delay is cleared.

This incident has been resolved and all the services are back to normal and the delay is cleared for the affected hosts completely.

We are experiencing degraded performance on some of our IN (instances). Data ingestion will be delayed and some of the campaigns will be running slow/delayed. Dashboard queries will be slow. We are currently investigating the incident and will update here in the next 30 minutes.

We are upgrading the issue to impact: Major as this is affecting the dashboard logins, trigger campaigns and delayed ingestion.

The degraded performance issue is resolved. And ingestion of data is getting cleared for the affected instances. We are monitoring the situation and will update here when all delay is cleared.

This incident has been resolved and all the services are back to normal and the delay is cleared for the affected hosts completely.

We are currently facing a hardware issue with one the the underlying nodes of one of our IN clusters. The customers on this cluster are facing delayed ingestion, slow/delayed campaign deliveries, and delays in import/exports/reports. We have identified the issue and working with the infrastructure team to bring the node back. We are expecting approximately 2 hours for the systems to get back to normal. We will keep updating the status here.

We are continuing to work with the infrastructure team to fix the issue. We are expecting a slight delay in getting this node back online. We will keep updating the status regularly here.

The affected node is back to normal. All our systems are back to normal. We are currently monitoring the services. We will close the incident in a few minutes.

This incident has been resolved and all our systems are stable and normal. We are closing the incident.

We needed unplanned maintenance on one of the data stores. This has resulted in a partial outage for a few of our customers. The campaign deliveries/data exports are partially unavailable for these customers. Currently, we have identified the issue and fixing it. We are bringing up the data store and backup systems and monitoring the performance. The expected time is 1 hour to resolve.

The data store is back up now and we are verifying the instance(s) before moving into monitoring status.

The affected system is back to normalcy. We are currently monitoring this for some time before closing the incident.

This incident has been resolved and all our systems are working as expected.

We are currently investigating an issue that is causing outage on our Europe SDK Endpoint. Please note that our SDK has a built-in retry mechanism and the traffic will come back to normal when the issue is resolved. We will provide further updates in 30 minutes.

We have deployed the changes which resolved the issue with our Europe SDK endpoint. The ingestion is back to normal and we are currently monitoring the system. We will observe this for some time before closing the incident.

This incident has been resolved and SDK ingestion has resumed to normal.

We are currently facing a user information sync issue. A few accounts/users are not able to login to our EU1 dashboard. We are currently fixing the sync issue. The users will be able to login as soon as their information is synced up. We will update here with the progress. As an alternative measure, users/accounts who are facing issues can access the following page to perform their activities: https://eu1.dashboard.clevertap.com/old-login.html All other components of the platform are working fine: 1) Ingestion (SDK/API) is working fine 2) Scheduled campaigns/live campaigns are running fine 3) Import/Export functionality is working fine

Update: We have completed the data sync for 90% of the accounts and users. We will be completing the remaining set of users in the next 30 minutes. We will post the update here.

All user information and account information is synced. All users can login and conduct their operations normally. This incident is resolved and all components are working as expected.

We are currently investigating an issue that is causing slowness on the EU1 dashboard. We are investigating this issue and would update the status soon.

We are upgrading the issue to critical as the following things are affected: 1) Ingestion of data is delayed (SDK/API/sFTP import) 2) Dashboard operations are affected/degraded 3) Campaign delivery is affected We are currently working on bringing the services up. The estimated time is approximately 120 minutes or lesser. We will update the status in the next 30 minutes. We apologize for the inconvenience caused.

We have identified the issue and fixed it. The systems are back to their normal status. We are currently monitoring the platform and would update here in some time.

This incident is marked as resolved and all systems are back to normal.

We are currently investigating an issue that is causing outage on our Europe SDK Endpoint. Please note that our SDK has a built-in retry mechanism and the traffic will come back to normal when the issue is resolved. We will provide further updates in 30 minutes.

We have deployed the changes which resolved the issue with our Europe sdk endpoint. The ingestion is coming back to normal and we are currently monitoring the system. We will observe this for some time before closing the incident.

This incident has been resolved and SDK ingestion has resumed to normal.

We are currently investigating an issue that is causing degraded performance/outage on our Europe API Endpoints. Our customers will see an increased rate of errors during this period. Please allow us some time to diagnose and fix the issue. We appreciate your patience and we will continue to update the status here.

We are continuing to work on this issue. We have identified some of the possible causes and working on addressing those. We will update this space in the next 30 minutes with further updates.

A fix has been implemented and all our Europe API endpoints are up and running. We are monitoring the situation.

This incident has been resolved. Our Europe API endpoint is back to normal.

We are currently investigating an issue that is causing degraded performance/partial outage on our Europe and India SDK and API Endpoints. Please note that our SDK has a built-in retry mechanism and the traffic will come back to normal when the issue is resolved. This will also affect the action-based (trigger) campaigns/journeys. We appreciate your patience and we will continue to update the status here.

The issue has been identified and a fix is being implemented. We will update the progress in the next few minutes.

Europe endpoints are back to normal and the data is getting processed. We are currently working on India Endpoints and will update the status here soon.

We are taking more time to identify the exact issue with India endpoints that is causing the downtime. Thank you for your patience and we will continue to update the progress here.

India SDK endpoints are back to normal and the data is getting processed. We are currently working on India API endpoints and the service is partially back to normal and currently exhibiting degraded performance. We are working on this and will update the status here soon

We are continuing to work on a fix for this issue.

Our India API endpoints are back to normal and we are monitoring the system for some time. Status update: All our endpoints (SDK as well as API) are up for India and Europe. We will observe this for some time before closing the incident.

This incident has been resolved and all endpoints are back to normal. Data ingestion has resumed to normal.

We are currently investigating an issue that is causing degraded performance/partial outage on our Europe SDK Endpoint. Please note that our SDK has a built-in retry mechanism and the traffic will come back to normal when the issue is resolved. We will provide further updates in 30 minutes.

We are continuing to work with the Amazon AWS team to get the system back to normal. Currently, this issue has affected the ingestion in the Europe region for SDK traffic. This will also affect the action-based (trigger) campaigns/journeys. We appreciate your patience and we will continue to update the status here.

The issue has been identified and a fix is being implemented. We will update the progress in the next few minutes.

We have deployed the changes and currently monitoring the system. We will observe this for some time before closing the incident.

This incident has been resolved and SDK ingestion has resumed to normal.