Cronologia delle interruzioni di Kickserve

The issue has been identified and a fix is being implemented.

AWS issues seem to have cleared up. We will monitor and post an update if conditions change. Thank you.

This incident has been resolved.

We're seeing some cases of the Kickserv web application not loading properly. We're looking into it now.

We've tracked down the issue to problems at Amazon Web Services (our hosting provider). We've disabled some components of the Kickserv web application, which has restored most functionality. Bear with us as we continue to investigate. Thank you.

This incident has been resolved.

Kickserv's mobile and web applications are currently unavailable. We're looking into the problem and will keep you posted as soon as we have an update. Thanks for bearing with us.

We're working on a fix for the service interruption now.

Web and mobile apps are once again functioning normally. Thank you for bearing with us.

Early on the morning of January 12, some Kickserv customers experienced slow loading and errors when using the web and mobile apps. This was caused by unusually high system activity tied to New Year–related reporting and background processing, which temporarily overwhelmed part of our infrastructure. As a result, many application pages (especially dashboards and settings) were sluggish or timed out for about an hour. We identified the issue, increased system capacity, and fully restored normal performance by 10:21 AM US Eastern time. We know interruptions like this are frustrating, and we’re taking steps to better handle seasonal usage spikes in the future, and improve early detection so this particular issue doesn’t happen again. Thanks for your patience.

We've received reports that the dispatch view is not loading properly. We're looking into it now. Until this is resolved, please use a different view (Schedule, Calendar, Resources, or Map) to view your jobs for the day. We'll update this post as soon as we have more information.

This incident has been resolved.

In our efforts to modernize our mobile app, we updated our date-formatting code. We missed one place where the old method is still called. Users will find that saving new or existing tasks on any job record will generate an error with a 'Try Again' page. This error is purely visual and the task is successfully saved to the job record. When users encounter this, it's recommended to simply tap the 'Try Again' button a few times and the error will resolve and take you back to the job record. We have identified the issue, we are working on the fix and we'll have a new build submitted for approval to both major app stores shortly. Depending on approval times, the update should be publicly available this coming week. We're sorry for any inconvenience, and we thank you for your patience.

Update: The issue has been identified and tested. We've pushed new versions with the fix to both major app stores. Please keep the app updated to make sure you have all the latest stability enhancements and bug fixes. Thank you for your patience and have a good week.

Both iOS and Android should be working now, once users download the latest update.

This incident has been resolved.

Navigating and searching the Kickserv web application is resulting in an issue where our users are getting directed to our main sign in page or showing an invalid customer with an error referencing degraded state. Our team is investigating the cause.

All systems are operational.

Some emails sent from Kickserv free accounts to Yahoo or AOL email addresses are being sent to customers' junk mail folders. This is a result of spammers attempting to use Kickserv to send unsolicited email, raising our spam score at certain email providers. We have added measures to eliminate spam, and are working with Yahoo and AOL to resolve this problem more quickly, but spam scores take time to correct themselves. Please be patient and let your Yahoo and AOL customers know to check their junk mail folders for emails from Kickserv. You can also send notifications via text message instead of (or in addition to) email. This gets your customers notified more quickly without the need to check email at all. Check your Messaging settings in Kickserv to enable text messages.

This incident has been resolved.

Normally, links in our text messages are shortened by a third-party service (Rebrandly) to fit and look similar to the following: https://kck.sv/mrz2 However, Rebrandly's API is currently reporting an error, and our links are going unshortened: https://app.kickserv.com/servicechampions/self_service/jobs/11240?ksrc=sms&token=xH-4tCHXbcx39KBZPvkEAoSlh56LHnNQua1Q2lHsdmU This does not impact the behavior of text messages or prevent customers from visiting links in text messages. In rare cases (mostly on older devices), the text message may be split into multiple parts if it is too long. We are working with Rebrandly to resolve the issue and get our link shortening working properly again as soon as possible.

This incident has been resolved.

Our latest iOS app release (3.10.13) is incorrectly using test data. If you use an iOS device, please do not update your app just yet. We’ll get that fixed today.

While we work to get a fix approved and released on the iOS App Store, we've made a configuration change on our servers to allow the mobile app to use production data. You'll still see the message "MOBILE_STAGING_SERVER_URL" at the bottom left of your screen, but your Kickserv account's jobs, customers, and employees should all be correct now. We recommend force-closing and relaunching the app for good measure. We'll update this space when a new version with the permanent fix is available for download. Thanks for bearing with us.

Version 3.10.14 is now available on the iOS App Store. This version contains the fix. If you have version 3.10.13 with the data error, please update your app now. Once you've updated, you will no longer see the "MOBILE_STAGING_SERVER_URL" message.

With the latest mobile app release (version 3.10.10), we discovered an issue when adding files to a job or customer. If you check the "Share in Customer Center" option, and do not enter a description, the app may crash. We are investigating this now, but there's an easy workaround. Just make sure you enter a description. You can even enter some text and then delete it before uploading. We'll let you know as soon as we have a fix available.

The issue has been identified and a fix is being implemented.

We have released the fix for this issue in version 3.10.11. It is available on the Google Play Store for Android devices and in review in the App Store for Apple devices. We'll let you know as soon as Apple releases to the App Store.

This incident has been resolved.

We've received reports that the New Opportunities tab is not showing opportunities. We're looking into the issue now.

We are continuing to investigate this issue.

We identified the issue as a bug introduced with some recent changes to our opportunities filter. We rolled back those changes and you should once again see your new opportunities back where they belong. We're sorry for the inconvenience.

This incident has been resolved.

On Monday afternoon, we deployed a new option to the filters you see at the top of the Jobs lists \(Opportunities, Jobs, and Invoices\). The new option allows you to choose a custom date range \(from any date, to any date\) rather than being limited to a specific timeframe \(this month, last year, etc\). Unfortunately, there was a bug in the code for the new option that went undiscovered during QA testing. Our product and success team members test every fix and new feature before approving it for deployment. This means our software is tested by the people who are most familiar with the ways that real-world users use it. However, it also means our QA process is less formalized and there is room for improvement. We’re sorry this happened and we know you depend on our software to run your business. Reliability and trust are of prime importance to us. We will guard against this in the future by adding specific steps and hints to testers before releasing code changes to QA, and making sure that the code review process includes verifying that QA documentation is in place.

We've received reports that email notifications are not being delivered in some cases. We're looking into it now.

We have identified this as an issue with our email sending queue. We updated some settings on the queue and emails are sending normally. Please let us know if you experience any further delivery issues. Thanks!

We are continuing to monitor for any further issues.

This incident has been resolved.

Towards the end of last week, the success team received a few reports of undelivered tasks and agendas. The engineering team investigated and found that the part of our system that queues up background tasks—like emails and texts—occasionally backed up to the extent that a task timed out after a number of retries. There are several queues in the system that share a finite set of resources, and things like QuickBooks syncs and search indexing sometimes generate a large number of background tasks. The problem went away on its own with no obvious cause, and troubleshooting became more challenging without the ability to see the system under load. Engineers focused on other pressing support and maintenance issues. Unfortunately, the condition degraded over the long weekend until it affected many accounts on Monday, May 29th. This was probably exacerbated by higher-than-normal, post-Memorial Day weekend activity. For example: everyone arriving at the office on Tuesday and running a big QuickBooks sync at the same time. We’re addressing this specific issue, and guarding against future occurrences, by doing the following: * Adding more monitoring for task notifications and other essential background tasks in the system. In the event that the notification queue starts to enter a retry situation, we will alert engineering and reprioritize the notification queue higher so that tasks do not get dropped. * Prioritizing updates to infrastructure software. The part of our system that runs background tasks is starting to get out of date, and newer versions offer improved performance and stability. We are sorry this happened and are grateful to you for bearing with us.

Over the weekend, we deployed a minor update to our credit card processing. The update passed QA, but once in production, the new code triggered an unexpected error condition at Stripe (Kickserv's credit card processor). This caused some credit card payments to fail with an error message. We rolled back the update Monday morning and all payments since have processed without issues. If you or your customers processed a credit card payment over the weekend (from May 19 to the morning of May 22), check to make sure those payments went through, and run them again if necessary. We're sorry for the inconvenience.

SMS text messages from Kickserv are currently not being received by T-mobile subscribers. We're investigating with our SMS provider now.

We are continuing to work with Twilio (our text message provider) to identify and resolve this. We'll keep you posted. In the meantime, you can always use Kickserv's built-in messaging to communicate with T-Mobile customers.

Text message deliveries to T-Mobile subscribers are back to normal this morning. We are continuing to work with Twilio to identify the root cause of the previous outage.

We are continuing to monitor for any further issues.

This incident has been resolved.

We've received reports that some accounts are unable to log manual (non-credit card) payments in Kickserv. We're currently looking into the issue and will update this page as soon as we have a resolution.

We've identified the issue with payment logging and are pushing a fix now.

The fix for logging non-credit-card payments has proven more tricky than we anticipated. In the meantime, a page refresh or two should clear up the issue and allow you to log those payments.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We're receiving reports of problems adding notes and attachments to a job. Both web and mobile applications are affected. We're investigating now and will update this page as soon as we have identified the issue.

We've traced the problem to unexpected latency issues with Amazon Web Services' Simple Storage Service (S3). We've added additional compute resources to compensate and are monitoring the situation in case we need to take further action.

We have adjusted our autoscaling to take into account periods of higher S3 latency. If you continue to experience issues when adding notes or files, please let our support team know.

We are investigating an issue which is preventing email replies from being received by the Kickserv web application. We hope to have it resolved shortly. Until we do, if you are awaiting a reply from a customer, you may want to follow up with them by phone or separate email. We'll update you when we have more information.

We are once again receiving inbound email replies. It appears that Postmark (our email provider) had a misconfigured setting on their end. Thanks for bearing with us.

Postmark has let us know that many of the missing email replies can still be processed on a delayed basis. They're running those now. Keep an eye on your message inbox.

This incident has been resolved.

We are seeing DNS problems resolving web URLs created by our link shortening service. These URLs start with kck.sv and are used by our text messaging service and a few other areas of our web application. We are investigating now. In the meantime, if your customers are having problems viewing payment requests, you can send them a link to the Customer Center by visiting the customer's screen in Kickserv, selecting the Overview tab, and scrolling to the Customer Center section. Thanks for bearing with us while we sort this out.

Due to severe technical limitations at the registrar for our kck.sv domain, we will switching to kck.im for our text message links. We'll update this incident when that's done. Any payment requests will need to be resent, as the old kck.sv links will no longer work. Note that you can always send your customer a link to their Customer Center--where they can view and pay invoices and requests--by visiting the customer's screen in Kickserv, selecting the Overview tab, and scrolling to the Customer Center section.

The issue has been identified and a fix is being implemented.

Service has been restored for the domain kck.sv, so existing short links for payment requests and to the Customer Center are once again working. As previously mentioned, though, we will be switching to kck.im very soon to guard against reoccurrence of the issue. We hope to switch back to kck.sv at some point in the future, if and when it becomes possible.

This incident has been resolved.

Amazon Web Services, the hosting provider for Kickserv and much of the rest of the Internet, is having issues with one of their server availability zones. The Kickserv application itself is hosted in multiple zones and is unaffected, but Elastic Cloud (our search engine service) has been affected by the outage. We are working with them to try and resolve the situation. In the meantime, instead of searching for jobs or customers, we recommend using the Customers list, finding your customer by name, and selecting the job from the customer’s Jobs tab. We’ll post updates as we have them.

Elastic has restored our search service. It will take some time for our servers to get the search index caught up with recent data changes, and this may affect certain aspects of Kickserv's performance--for instance, some event reminders may be slightly delayed. We are working to get everything completely back to normal, and we'll update this page as soon as we're there. Thanks for bearing with us.

The search index is now up to date and everything should once again be functioning normally. Please let our support team know if you encounter more issues.

Amazon Web Services (AWS), the hosting provider for Kickserv and much of the Internet, is reporting various issues today in its us-east-1 region. Kickserv is hosted in multiple regions, so our performance has so far seen little impact, but this could change at any point. Many services we use also rely on AWS, so you may encounter issues that are out of our control. We'll update this page as events warrant. Thanks for bearing with us. For more details, watch the AWS status page at https://status.aws.amazon.com

AWS has resolved their issues, so Kickserv should be operating normally once again.