Chargeover Status

Connectivity issues have been resolved. Some payments may need to be re-attempted. Some syncs to external platforms (QuickBooks, Xero, Salesforce, etc.) may need re-sync.

This has been resolved. We are investigating what occurred here still. A postmortem will follow.

This incident has been resolved.

This incident has been resolved. A postmortem will follow.

Syncing for all affected integrations is complete now. A postmortem will follow.

ChargeOver was inaccessible for most users from approximately 9:44am CT to 9:58am CT, due to a database connection issue. No data loss occurred. Scheduled subscription invoice creation and payment processing were not affected - all scheduled payments processed normally. We are still investigating root cause. A postmortem will follow.

# Incident details We received messages from internal & external users that they were not able to log into their instances. After investigation, we confirmed that the login sequence was taking an abnormal amount of time. This behavior was present on July 18th from `14:00 CST` to `15:14 CST`. The impact of this incident was limited to only the login process for ChargeOver. This did not affect payment processing, hosted pages, or other automated processes. # Root cause ChargeOver uses a third party metrics service, PostHog, to track user input to help improve the platform. Requests to PostHog were determined to be the cause of the slow login sequence. We found a URL endpoint was no longer being serviced, updated the pointing of our load balancer, and rebooted any systems in order for the change to take affect. # Incident timeline * 14:00 CDT - Received notice of logins not working. * 14:11 CDT - Assembled a team to start investigating root causes and update our status page. * 14:44 CDT - Identified the issue with the third party metrics service. Determined that logins were working, but they were taking significantly longer than expected. * 15:07 CDT - Implemented fix for third party metrics service and deployed to user instances. Began monitoring for any further issues. * 15:14 CDT - Verified that logins were working as expected and began drawing up plans for future remediation. # Remediation plan We understand that it is frustrating to lose access to your ChargeOver instance and we are very sorry that this has happened. In the future, we will handle these requests so that they do not get in the way of normal use by separating the logic independently of the rest of ChargeOver. We will also be implementing an SOP to make sure that any services that may cause disruptions won't affect the app in this way.

# Incident details On July 5th, a firewall-related issue caused ChargeOver to be inaccessible for approximately 90 minutes. Customers were unable to log in or access the ChargeOver application during this time frame. # Root cause At approximately 3:27pm CT, one of the high-availability firewalls experienced an error which caused it to stop passing traffic into/out-of ChargeOver’s internal network. Although the firewall was configured in a way which should trigger automatic fail-over \(we use industry standard Netgate firewalls, which use CARP to share IP and network status information and monitor for fail-over\) to a secondary firewall, the automatic fail-over did not occur. ChargeOver staff were immediately and automatically notified, and after troubleshooting were able to resolve the outage by power-cycling the affected firewall. # Incident timeline * 3:27pm CT - Firewall experiences an error, which stops some inbound/outbound traffic. ChargeOver staff are immediately alerted. * 3:28pm CT - ChargeOver staff respond and begin troubleshooting. * 4:57pm CT - Problem resolved after power-cycling the affected firewall. # Remediation plan We are still investigating what caused the error which took the firewall offline, as well as why the firewall did not fail over correctly to a secondary firewall when the error occurred. As we investigate further, we’ll have a better idea of whether this requires replacement or reconfiguration of the firewall. We are improving documentation, to enable our engineering team to troubleshoot quicker/isolate root cause quicker for future incidents. We are also working with data center staff to establish some better monitoring tools to help us troubleshoot quicker/isolate root cause quicker.

We have posted a postmortem for this incident here: * [https://status.chargeover.com/incidents/3zfsdz4msf3s](https://status.chargeover.com/incidents/3zfsdz4msf3s)

## Incident details ChargeOver experienced an outage on June 7th, and an outage again on June 8th, as a result of a planned data center power outage, and an unplanned networking stack -related failure. On Saturday, June 7th, ChargeOver was inaccessible from approximately 13:12 CT to 16:53 CT. On Sunday, June 8th, ChargeOver was inaccessible from approximately 13:32 CT to 17:05 CT. ## Root cause ChargeOver’s primary data center \(Flexential in Chaska, MN\) performed a planned maintenance task which de-energized the four uninterruptible power supply \(UPS\) systems serving entire data center, one UPS system at a time. ChargeOver’s network infrastructure is designed to be redundant, and so ChargeOver has two separate power feeds coming from the four data center UPS systems. ChargeOver’s stacked network switches are plugged into either the first power feed, or the second power feed, to provide redundancy. When one of ChargeOver’s redundant network switches lost power due to the data center UPS maintenance, the expectation was that the switch would automatically fail over to the second redundant switch. Either the switches did not fail over correctly, or the switch lost it’s configuration on fail-over, which resulted in ChargeOver’s systems becoming unavailable to the outside world. ## Incident timeline * May 29 - Flexential informed ChargeOver of anticipated maintenance. Due to a miscommunication, it was unclear to ChargeOver staff that the maintenance would impact our environment. ‌ * June 7 - 13:12 CT - Flexential de-energizes and then re-energizes the first power feed, causing a network switch to go offline. The network switch either fails to fail-over correctly, or loses it’s configuration as the power feed comes back on. ChargeOver staff are immediately notified. * June 7 - 13:19 CT - Attempts to reset the network switch fail to restore service. * June 7 - 16:51 CT - One network switch is removed, and the configuration on the second switch is re-loaded. * June 7 - 16:51 CT - Network availability is restored. ‌ * June 7 - 13:32 CT - Flexential de-energizes and then re-energizes the second power feed, causing a network switch to go offline. The network switch either fails to properly boot, or re-loses it’s configuration as the power feed comes back on. ChargeOver staff are immediately notified. * June 7 - 14:41 CT - Attempts to reset the network switch fail to restore service. * June 7 - 17:03 CT - The configuration on the second switch is re-loaded. * June 7 - 17:05 CT - Network availability is restored. ## Remediation plan We deeply apologize for the downtime, and are already working towards many improvements towards our infrastructure. * We have replaced a suspected-faulty network switch. * We are working internally to establish some better procedures for coordinating planned maintenance intervals in the future. * We are investigating network switch options and fail-over configuration, to identify possible improvements we may be able to make to our network stack to avoid future service interruptions of this type. * We are investigating possible UPS upgrades or replacements we may be able to make to provide extra power redundancy beyond what is already provided by the Flexential data center.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.

The scheduled maintenance has been completed.