20 Vinsolutions incidents · July 2018 — official updates, affected components, duration and resolution details.
We are monitoring, and taking proactive measures where possible, in response to intermittent DNS provider issues which may be impacting our integrator community.
A fix has been implemented and we are monitoring the results.
Service is restored. From 1:40 P.M. to 4:15 P.M. EDT, Multiple CAI services and product lines experienced DNS related failures.
Mashery's service is currently degraded, which may affect Cox Automotive / Connect CRM API requests. You can obtain direct updates from Tibco here: https://status.cloud.tibco.com/incidents/bjzqyq4rvdmf.
Update from Tibco/Mashery (https://status.cloud.tibco.com/incidents/bjzqyq4rvdmf) "Based on the available information at this time, we believe that the degradation has been addressed and we are seeing recovery. All Mashery service components are in good health. Mashery team is currently monitoring the situation. We are committed to providing you follow-up updates through Status page as soon as we have more information." Based on the above information, as well as verification that API requests are being fulfilled properly, we are closing this incident.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
We have scheduled a maintenance window for this evening to work on the performance issues with the Call Details API. Until then, the system will be operational, but in a degraded state.
Maintenance and verification is complete.
Our Call Details API response times have degraded and we are investigating the matter. The issue results in excessive HTTP 504 (Gateway Timeout) responses. We are investigating.
The Call Details API is now responding as expected.
The Call Details API is being affected by an incident with our network infrastructure provider beginning at approx. 5pm UTC. Multiple teams have been engaged and are working on the issue. Errors have begun to clear but we continue to monitor the situation.
We are continuing to monitor for any further issues.
This incident has been resolved.
We are investigating reports of errors in response to some vehicle inventory requests using the Digital Showroom API.
The issue has been identified and a fix will be applied momentarily.
The incident affecting Vehicle Inventory queries has been resolved.
We are currently investigating reports of intermittent HTTP 500 / 504 responses affecting primarily the Contact endpoints. Multiple teams are engaged as this is a system-wide issue affecting the CRM.
We are also investigating timeouts associated with requests to the Lead endpoints.
We have placed the Contact API in maintenance mode while teams troubleshoot it. Requests to this endpoint will be responded to with HTTP 503 - Service Unavailable, with an additional header "Scheduled Maintenance."
CRM performance has improved significantly for clients; however, we will leave the Contact endpoint in maintenance mode while engineering and I/T bring online other processes which were disabled during troubleshooting. We will remove maintenance mode once we receive the all clear that supporting processes are again online.
After confirming CRM stability, we are now taking the Contact API out of maintenance mode. We continue to monitor the CRM's performance for slowness and will take corrective actions as necessary.
CRM and API performance have returned to normal levels.
We are currently investigating an issue affecting our Leads APIs.
The issue has been resolved.
We identified the issue causing HTTP 401 responses when requests are made to our Organizations and Users endpoints.
The incident has been resolved and we have confirmed that requests for Organizations and Users are processing as expected.
We are currently investigating an issue affecting our API infrastructure which results in API request timeouts.
A fix has been implemented and we are monitoring the results. We are seeing fewer HTTP 5xx errors.
This incident has been resolved.
We are currently investigating an issue with our Identity services which impacts the CRM and APIs. The issue began at approximately 10:45am CDT.
A fix has been implemented and we will monitor throughout the day.
This incident has been resolved.
We are currently investigating an issue affecting authentication.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating an issue which affects the Call Details API.
The Call Details API incident has been resolved.
Call details API is experiencing an issue. No other APIs are affected.
The Call Details API is operational and requests are successfully processing.
We are working on a fix to an issue affecting the creation, and updating, of leads via the API.
We are gathering more information.
We are working with our API Gateway provider to understand why Leads requests are returning HTTP 500 responses intermittently.
This incident has been resolved.
The Leads API is not able to associate a vehicle of interest in some cases. The root cause has been identified and a fix is being developed.
The fix is planned to be deployed to production tomorrow morning. We will send an update after successful deployment.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating timeouts which result in HTTP 500 responses to Leads API requests.
A fix is being developed for issues affecting the Lead PUT method.
The fix for the PUT Lead errors has passed QA and will roll to production soon.
The fix has been deployed and verified in Production. When notes are included with POST / PUT Customer and Lead requests, HTTP 500 responses do not appear and the operation completes successfully.
This incident has been resolved.
We are currently investigating reports of unexpected HTTP 500 responses to Lead PUT requests. The same request body, submitted subsequently, results in HTTP 200 (success).
We've seen a reduction in the percentage of HTTP 500 responses; but, we are also developing a fix to further reduce the frequency of these responses to the PUT Leads requests.
We've implemented changes to reduce the frequency of the HTTP 500 errors. We will monitor for 1 week to gather data about the success of our changes before "resolving" this incident.
This incident has been resolved based on feedback from our Partners and a measurable reduction in HTTP 500 responses.
We are currently investigating issues reported with the Leads API.
After researching the issue, we found that, when an invalid AutoID is used in the body of the associateVehicleOfInterest operation, the server responds with HTTP 500. To prevent the HTTP 500 response, please be sure to use valid AutoIDs provided by the getVehicleInventory operation. In the future, we will provide better messaging to handle this scenario.
Dealer hierarchy services were impaired between 16:47 and 17:13 UTC, causing API requests for Dealership data to fail with HTTP 500 responses.