52 Tokenex incidents · oktober 2022 — official updates, affected components, duration and resolution details.
We are currently investigating intermittent issues with the TokenEx APIv2 endpoint. Teams are engaged, and updates will be posted here as they become available.
This incident has been resolved. Users might have observed intermittent latency or timeouts when calling the TokenEx APIv2 endpoint between 20:02 and 20:09 UTC. Since 20:09 UTC, all systems have operating within normal parameters.
Vi undersøger et potentielt problem, der påvirker 3D Secure Engine, der synes at have oprindelse uden for vores platform og primært påvirker Mastercard transaktioner. Vores team er aktivt at undersøge situationen og vil levere opdateringer, efterhånden som flere oplysninger bliver tilgængelige.
Vores undersøgelse viser, at problemet stammer fra Mastercard side og forårsager Mastercard 3D Secure godkendelser til at mislykkes. Vi har kontaktet Mastercard og følger aktivt situationen med dem. Kun Mastercard 3D Sikker godkendelse er påvirket. Vi vil give yderligere opdateringer, efterhånden som flere oplysninger bliver tilgængelige. For de seneste oplysninger, se den relaterede status side: status.3dsecure.io.
Vi observerer ikke længere Mastercard godkendelsesfejl og kan bekræfte, at normale operationer er genoptaget. Mastercard har endnu ikke officielt bekræftet hændelsen som løst. Vi vil fortsat overvåge situationen nøje og levere yderligere opdateringer, efterhånden som de bliver tilgængelige.
Vi kan bekræfte, at spørgsmålet er løst. Tjenesten fungerer normalt, og vi har ikke observeret nogen yderligere effekt. Vi vil fortsat overvåge situationen for at sikre stabilitet.
Automatisk oversat fra den officielle hændelsesopdatering.
We are currently experiencing a service disruption affecting Payment Orchestration. Teams are engaged and actively investigating. Further updates will follow as more information becomes available.
Our teams are continuing to work on resolving the issue. We will provide further updates as they become available.
Transaction processing have recovered to normal levels but reporting is still impacted. Our team is investigating the network issues with our provider.
Teams are continuing to work toward full service restoration. Transactions are processing successfully, however, they are not yet visible in the admin UI, dashboards, search, reporting, or export mechanisms. This data will become visible once services are fully restored.
Transaction processing is fully operational and payments continue to be processed normally. The impact is limited to visibility, which means recent transactions may not currently appear in the Admin UI or Dashboards even though they are being processed successfully. Our teams are continuing to work on restoring the affected services and further updates will follow.
Search/reporting functionality has been restored, and data is currently synchronizing. Users should begin to see transaction data in the portal and dashboards. Teams are continuing to monitor while synchronization completes and to ensure ongoing stability.
Data synchronization has completed, and users should now see current transaction data in the portal and dashboards. All systems are operating within normal parameters.
We are observing intermittent service availability issues affecting the TokenEx APIs in the US region. Teams are currently engaged and investigating.
The issue has been identified, and teams are addressing. Most services, with the exception of APIv2, are once again within normal operational parameters.
Services have stabilized as of 10:40 am Central Time. Teams are continuing to monitor.
This incident has been resolved.
The Universal Tokenization Production Environment (US) will undergo scheduled Production Patching on Thursday, June 18th, from 5am - 7am CST (11:00 UTC - 13:00 UTC). Impact: All production services are expected to remain fully operational throughout this maintenance window. If you have any questions or concerns about this scheduled maintenance, please contact our support team.
We are continuing to investigate this issue.
Created Incident instead of Maintenance Activity.
We identified an issue resulting in a brief service interruption impacting a subset of customers on our platform. Normal operations have now resumed, and all affected systems have recovered. Our team remains engaged and is actively investigating the situation to gather pertinent details and determine the underlying root cause. We will continue to monitor system health closely. We are currently not expecting any further interruption. Please make sure that the service is fully functional on your side as well. If you experience any ongoing irregularities, please contact our support team.
We are investigating a service interruption that impacted the TokenEx APIv2 endpoints in our US region. Impact was observed from 4:40 pm to 4:54 pm US Central Time. Services are currently stable, and technical teams are engaged.
This incident is resolved. Teams have confirmed system stability, and no further impacts are expected.
We are currently investigating elevated errors and intermittent connectivity issues for IXOPAY services. Our team is engaged and actively analyzing the impact on our transaction processing and overall service availability.
We have identified the underlying cause of the elevated transaction processing errors and intermittent connectivity issues affecting IXOPAY services. Our team is actively working on implementing a fix and restoring full service stability. We continue to monitor the environment closely and will provide further updates as progress is made.
Mitigation measures have been implemented for the elevated transaction processing errors and intermittent connectivity issues affecting IXOPAY services. Transaction processing and service availability have stabilized, and our team is currently monitoring the environment closely to ensure continued normal operations.
We are currently investigating elevated errors and intermittent connectivity issues for IXOPAY services. Our team is engaged and actively analyzing the impact on our transaction processing and overall service availability.
The issue causing elevated transaction processing errors and intermittent connectivity issues affecting IXOPAY services has been resolved. Services are operating normally again, and our team is continuing to monitor the environment closely to ensure ongoing stability.
The incident affecting IXOPAY services has been fully resolved. Our monitoring has confirmed stable operations, and all services are functioning normally at this time. We apologize for the inconvenience and appreciate your patience during this time.
This morning at 6:11 AM CST(12:11 PM UTC), we experienced an issue with the active firewall VM in our US datacenter, which triggered an automatic failover event. The resulting service interruption lasted less than one minute. Systems have been operating normally since the failover completed. We continue to monitor the environment to ensure stability.
We are investigating intermittent response failures on the TokenEx APIv1 endpoints in the US region, initially observed at 8:38 am Central Time. We will provide updates here as investigation continues.
Teams have identified the potential cause of the issue and are actively remediating.
A fix has been implemented and the team is monitoring the results. We believe the issue to be resolved as of 9:28 am CT.
The issue has been resolved and the TokenEx APIv1 endpoints are operating normally. The cause was identified to be a planned configuration change, which was reverted. Monitoring since the fix was applied has not shown any additional failures. A post-mortem review will be conducted internally to avoid reoccurrence in the future.
We are currently experiencing issues with our EU support phone line (+43 1 353 05 12-247). We have contacted our telephone provider, and they are investigating. In the meantime, please use the support portal to submit tickets.
The issue has been identified, and we are currently working on a solution. In the meantime, if you require urgent assistance, please use our preliminary EU support phone line at +43 1 353 05 05-4567.
We are pleased to confirm that the functionality of the EU Support Line has been restored. Our team has verified the fix, and the incident is resolved.
We are currently experiencing issues with our US support phone line (1-877-217-5718). We have contacted our telephony provider, and they are investigating. In the meantime, please use the support portal to submit tickets.
Our US support phone line (1-877-217-5718) is now fully operational. Thank you for your patience while we worked with our provider to restore service. We’re sorry for any inconvenience this may have caused and appreciate your understanding
We observed degraded performance on APIv1 starting at approximately 11:21 UTC. After investigation, we identified the issue as being related to a single node. APIv1 performance has returned to normal at 12:33 UTC.
Between 10:59 PM and 11:09 PM UTC, the APIv2 service experienced a brief issue impacting a small percentage of production traffic. Service availability was fully restored at 11:09 PM UTC. The system's automatic recovery process for this specific node failed to initiate as expected, resulting in a temporary loss of availability for customers routed to that node. The cause of the failure has been identified and corrected. We are implementing additional monitoring checks and proactive remediation steps to specifically detect and instantly isolate similar scenarios in the future.
We are currently investigating elevated errors and intermittent connectivity issues for IXOPAY services that rely on external network infrastructure. We have identified this issue to be linked to a widespread service disruption reported by a key content delivery network provider (Cloudflare Incident: https://www.cloudflarestatus.com/incidents/8gmgl950y3h7). Our team is engaged and actively analyzing the impact on our transaction processing and overall service availability. We are closely monitoring the situation as the external provider implements mitigation measures.
We are continuing to investigate this issue.
We are observing a significant reduction in elevated error rates and are seeing a strong trend toward the restoration of normal transaction processing levels. Our teams are continuing to monitor the stability of the connections closely and remain engaged to confirm full service recovery following the mitigation efforts initiated by the external provider.
Our team has successfully completed routing traffic off the Cloudflare network. Service is now being served directly from our data centers. We are currently monitoring all critical transaction paths and key performance indicators to confirm full stability and normalized service performance.
All systems remain fully operational after routing traffic off Cloudflare’s network. To prevent any further impact on transaction processing today, we will apply a change tomorrow during business hours to route traffic back through Cloudflare. A brief follow-up update will be posted once the change has been completed.
Our team is about to begin routing traffic back through Cloudflare’s network.
Routing back onto Cloudflare’s network has been successfully completed. All systems remain fully operational, and no impact to transaction processing was observed during the transition.
Between 13:54 and 13:57 UTC some customers experienced an elevated number of intermittent api error responses during transaction processing. This issue has been resolved, and services have resumed normal operation. We have identified the root cause of this issue and will continue to monitor transaction volume statistics closely to ensure stability.
Due to a one-time maintenance script run, the database experienced degraded performance between 12:03–12:06 UTC and again between 12:20–12:30 UTC. The issue has been resolved and services are operating normally. Our teams continue to monitor transaction processing closely.
We are currently experiencing an issue that is preventing users from logging in to our Customer Portal ( Salesforce ). Our technical team is actively investigating and working on a solution. We will provide updates as soon as more information becomes available. If a matter requires urgent attention, please contact us directly via our 24/7 Hotline at +4313530512-247.
Our team is currently working closely with Salesforce to implement a fix. We appreciate your patience and will continue to provide updates as progress is made. If a matter requires urgent attention, please contact us directly via our 24/7 Hotline at +4313530512-247.
During scheduled Service Cloud maintenance, an unforeseen issue arose that temporarily prevented user login. This disruption occurred between 8:20 UTC and 14:50 UTC on April 10. The issue has since been resolved, and users are now able to successfully log in to the portal. We appreciate your patience during this brief interruption.
We have noticed intermittent timeouts in incoming traffic impacting a small subset of transactions. We are investigating the issue.
We have identified the issue to be a routing issue. We are in contact with our upstream network provider to reroute the affected traffic.
Our ISP has further escalated the issue to isolated the bad network hop. The error rate remains stable but impact remains low.
The error rate has returned to regular levels. We are no longer observing timeouts. We will continue to monitor the health of transactions and stay in contact with our upstream provider.
We have identified the issue to be related to PSPs running under AWS. As the error rate has remained at healthy levels for the last 30 minutes we consider this incident resolved.
Between 23:01 and 23:14 UTC, we've observed intermittent timeouts when connecting to Visa Directory Server. The timeout rate has subsided without IXOPAY's intervention. The issue was caused by a network routing problem outside our sphere. We are in contact with our data center provider for further investigation.
This incident has been resolved.