51 Phoneburner incidents · december 2018 — official updates, affected components, duration and resolution details.
Vi undersøger webstedsproblemer. Ingeniører har identificeret problemet og arbejder på en løsning.
Problemet er løst. Vi overvåger
Site er tilbage til normal drift.
Automatisk oversat fra den officielle hændelsesopdatering.
We are investigating a fix.
This incident has been resolved.
We are investigating performance issues with the website.
Site performance is improving. We're monitoring.
Site performance is back to normal operation.
We have identified an intermittent issue with the dialer where the buttons don't work as expected. We have identified it as an issue with one of our vendors. We're working closely with them to resolve the issue.
The vendor is still researching the issues on their end.
We have started the process of moving all the usage away from the problematic vendor. We are monitoring. We recommend any users experiencing the issues log out of your account and login again to ensure you pickup the new configurations.
Traffic is migrated away from the problematic vendor and we'll continue monitoring.
We are investigating issues with the site
We've identified the source of the issue and we're remediating. We're starting to see performance improvements.
We are seeing full site performance recovery. We are monitoring.
Site performance has been restored. We've determined that an upstream provider was having performance issues that degraded some aspects of our site. We have mitigated those issue and the site is performing well. We'll continue monitoring.
We are investigating degraded website performance
Engineers have identified the issue and we're seeing performance recover. We're monitoring.
Site performance has been restored.
We have identified the issue and are working on a resolution.
Engineering team has identified a fix and an update and/or fix should be deployed shortly.
Calls are completing successfully. We are monitoring the make sure the system is stable.
The incident is resolved.
We are investigating site performance issues
We are observing improved performance. The team is continuing to monitor.
Site performance has been fully restored.
We are investigating the issue.
The team continues to investigate the issue. A new update will come shortly.
We are continuing to work on a fix for the issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are investing issues with the site loading slowly.
We have identified the source of the issue and the team is working on resolution.
We're observing site performance improvements. We're monitoring.
The site performance has been restored.
We have identified an issue with some outbound calls through the dialer. This includes call transfers as well. Engineers are actively working through the issues.
Issue has been identified and resolved. We are monitoring.
Systems are fully operational.
We are currently researching issues related to slow site performance.
We have identified the source of the issue and are deploying a fix.
A fix has been implemented and performance is recovering. We are monitoring.
Issues are not resolved. We are triaging a deployment issue.
Fixes are confirmed in good working order. We are monitoring performance.
Site performance is fully restored and operating normally.
We are currently investigating an issue with the dialer.
We have identified the issue and are working on a fix.
We are actively resolving the issue and we're starting to see recovery.
Dialer has been restored and fully functional.
We are currently investigating reports of a slow and/or unresponsive site.
The team is still researching. We'll update again shortly.
The team believes there is a performance problem with some in-browser integrations. We have disabled that integration and are monitoring performance. We also believe this does not impact everyone.
The team has confirmed that the 3rd party library in the browser is in fact driving high CPU and making the browser window unresponsive. The integration has been disabled, and the site is in working as expected. We are closing the incident and the team will continue to monitor.
We are investigating issues with site performance
Site access is fully restored.
We are experiencing issues the site. The team is researching.
We have identified the db instances in the cluster and are fixing.
Site is recovering as instances are restored.
Site is recovered with some indications of degraded performance. The team is still researching and monitoring.
Site is fully restored. We will continue to monitor.
We have noticed degraded performance on the site. The team is researching.
The team has identified some problematic instances in out the database cluster. A fix is being worked on now. There is no impact on data - just on site performance.
Problematic instances are fixed and site is fully operational.
We are investigating issues with accessing the site.
The team has identified a potential source cause. We are investigating further.
The team is still researching. Nothing new to report at the moment.
The team has resolved the root cause and is monitoring.
This incident is resolved.
We are currently experiencing site issues. We're investgiating
We have a identified a resource issue and are working on a fix.
Sites are showing improved performance and we are monitoring.
Systems are fully operational.
We are currently experiencing a service outage with SMS, Inbound (vPhone), Number purchasing, and CallerID verification services being impacted. Our team is working with our telecommunications providers to restore the service. Outbound Dial Sessions are not impacted by this outage. We apologize for any inconvenience. We will send you an update at 10 am Pacific Time on Wednesday, January 25th.
10 am PT Update: We are still experiencing issues with SMS, Inbound (vPhone), Number purchasing, and CallerID verification services and continue to work with our carrier partners to resolve the issue. Our next update will be 2 pm PT or sooner.
Here is a quick update to this existing outage - all outbound calls are fully down. All calls from the PhoneBurner platform are no longer working. We expect that this outage will last for the rest of the day. We will send an update at 2pm Pacific Time.
Outbound dialing, SMS, Inbound (vPhone), Number purchasing, and CallerID verification remain unavailable. The team is working around the clock to restore service. We'll update again at 5pm Pacific.
Outbound dialing, SMS, Inbound (vPhone), Number purchasing, and CallerID verification remain unavailable. Our teams continue to work with our telecommunications providers to restore all services. An update will be provided at 8am Eastern on Thursday, January 26th.
Outbound dialing, SMS, Inbound (vPhone), Number purchasing, and CallerID verification remain unavailable. Our teams continue to work with our telecommunications providers to restore all services. An update will be provided at 11am Eastern.
We apologize for this late update. Outbound dialing, SMS, Inbound (vPhone), Number purchasing, and CallerID verification remain unavailable. Our teams continue to work with our telecommunications providers to restore all services. An update will be provided at 3pm Eastern.
Our system is currently down due to upstream telephone carrier issues. As you may be aware, the FCC issued a Public Notice on Tuesday, January 24th where PhoneBurner was named as a service provider for an enterprise who has allegedly made non-compliant calls. Our attorneys are meeting with the FCC and our carrier partners to resolve this issue. Once we have resolved this issue, we will restore service to you as soon as possible. We understand the importance of staying connected and apologize for any disruption this may have caused. We will continue to communicate with you via this page as soon as new information becomes available. We appreciate your patience and understanding. Sincerely, Team PhoneBurner
On Tuesday afternoon, our company was made aware of a Public Notice posted by the FCC to all US-Based voice providers. As part of that notice, PhoneBurner was listed as one of the service providers for a company placing alleged illegal calls. The notice was clear that calls originating from our platform, by the customer named in the Notice, were to be suspended within a prescribed timeline. We had no advance notice of this Public Notice and, to our knowledge, neither did our carrier partners. As you have experienced, some of our carrier partners have interpreted the Public Notice to include all PhoneBurner traffic. In our opinion, this clearly was not the intent of the Public Notice by the FCC. We are working aggressively to get our services restored, and there are a few material updates to provide you. First, we have retained legal counsel that specializes in working with the FCC. Second, our legal counsel had meetings this morning with the FCC on our behalf. We believe that the interpretation of that meeting was consistent with the Public Notice. Simply put: carriers were asked to remove one company’s traffic and not all PhoneBurner call traffic. Third, our legal counsel, in partnership with the FCC and PhoneBurner, is identifying a series of modest and near-term steps that can be taken to help us reactivate our outbound dialing services. This is the most important step. Fourth, to help appropriately set expectations, service could be restored in the next few hours to the next few days depending on some of the factors described above. We will share more information regarding timing of service restoration with you as soon as we have it. Our teams are working around the clock on this, and service restoration is our top priority. Thank you for your trust in PhoneBurner. Please know we are doing all that we can to reactivate our outbound dialing services. Sincerely, Team PhoneBurner
Our systems remain down as of 11:30 PT Thursday. Ours teams are working with telecom partners around the clock to restore service as soon as possible. As soon as we have an accurate estimate for restoration we will publish immediately.
SERVICE RESTORATION (partial): As of 12:33 AM PT (Friday 1/27) we have resumed our outbound dialing service. We continue to work on SMS, Inbound (vPhone) and Number Purchasing. We'll update status as we make progress.
Thank you for your continued patience. We have identified one call performance issue with our newly restored outbound service that is being actively researched by our operations team. We will provide updates as they become available.
We believe the earlier identified issue as of 8:01 AM PT has been resolved. Our teams continue to monitor our systems, and we believe you should be experiencing normal dial sessions. Thank you for your patience. We will continue to provide updates as needed.
A quick Saturday evening update - outbound dialing was fully restored and stabilized Friday. Whisper/Barge was restored late Friday evening. We anticipate click-to-call will be restored before end of day Monday, if not sooner. SMS, Inbound, and number purchasing is expected to be restored by end of day Wednesday, if not sooner. We'd like to take a moment to thank all of our customers who have reached out to express their support and encouragement recently.
Monday morning update: Outbound dialing continues to be stable and fully operational. Click-to-call was restored as of Sunday evening. SMS, Inbound, and number purchasing are expected to be restored by end of day Wednesday, if not sooner. We truly appreciate your continued support and patience.
Unfortunately, shortly after posting our last update, we identified degraded service with our outbound dialing that is being actively researched by our operations team. We will provide updates as they become available.
Outbound dialing is partially restored. Our team is monitoring closely, and we will continue to provide updates.
As of 9:30am Pacific, outbound dialing appears to be fully restored.
We want to begin by telling you how truly sorry we are for the service challenges we have faced since last week. This has been an unprecedented time for all of us, and our 80-person team has been working around the clock to fully restore the exceptional services and quality that PhoneBurner is known for. While our outbound functionality has been restored, we've experienced some speed bumps along the way. We are aware that some of our customers had connection issues today, and we are cautiously optimistic that all outbound dialing issues have been resolved. Here's where everything currently stands: Fully Restored: - Outbound - fully restored with the acknowledgement things may not be perfect yet Will be restored tomorrow by close of business: - Number purchasing - Inbound calls - Inbound SMS Will be restored gradually over the next several days: - Existing Local ID & vPhone numbers - Outbound SMS We will follow up tomorrow with additional updates. We continue to be thankful to all of our loyal customers. Your notes of support and gratitude have given us strength and motivation. Thank you.
Premium features and numbers are in the process of being restored. Updates are available on the following: -Inbound -Number Purchasing -Local IDs -SMS (available starting Friday) For additional instructions on activating these features, please check your email or login to PhoneBurner for in-app notifications that will guide you to a short video.
Quick update on the newly restored number purchasing. We've successfully provisioned hundreds of numbers this morning. However, larger orders are experiencing delays. The orders have all been submitted for processing, and we're working with our number provider to fulfill all outstanding number purchases. As the numbers are fulfilled, they will appear on your vPhone or Local Id numbers page. Please do not submit a duplicate order.
Number purchasing has been fully restored and purchased numbers will appear on your vPhone or Numbers page. A small number of customers are still experiencing dropped audio while using the browser connection to initiate dial sessions. If you are experiencing any issues, we recommend you use your phone to dial in. We are working to resolve this issue and will keep you updated as things progress. Thanks for your continued patience.
For customers who have previously purchased numbers and/or have outbound SMS capability, we have two important updates: Previously Purchased Numbers: Numbers (Local ID and vPhone) purchased on or before January 25 are in the process of being restored. Each account is being individually reviewed before numbers will become available. Once an account is approved, the numbers in your account will gradually activate as they are restored. Note this process is ongoing and some of your numbers may become available before others. Some numbers will begin to be available starting as early as Monday, February 6. Please note this process normally takes much longer, but we are working with our partners to turn this around as quickly as possible. Outbound SMS: A process is actively in place to restore SMS functionality and comply with regulatory compliance for A2P (Application-to-Person) SMS. If you are using SMS in your account, make sure you have completed your Business Profile and provided opt-in details. You should have received an email today with more detailed instructions to start the restoration process. We thank you in advance for your patience while we do proper due diligence to comply with carrier regulations and provide you with the best experience possible.
Update - Our team has been working nonstop to improve services and monitor accounts. Read on for updates to Local ID, vPhone and outbound SMS. Numbers - Local ID and vPhone numbers purchased on or before January 25 are in the process of being restored. Please note this process is ongoing and some of your numbers may become available before others. You can check the status of your numbers in your PhoneBurner account. Here are definitions of what those statuses mean: - Active (done & working) - Restore in Progress (in progress) - Pending Restore (in the queue to be restored) - In Review (being vetted by compliance) SMS - SMS now requires you to complete your Business Profile and provide SMS opt-in details. Log into your account, go to settings, and you can find your Business Profile under My Account - https://www.phoneburner.com/myaccount/businessprofile/index Once the profile is complete it will be submitted for compliance review. Once approved, we'll enable your SMS messaging and notify you. We don’t have a timeline for how long approval will take. Thanks for your patience.
We are aware of an issue causing some calls to not connect. We have just implemented a change that should correct for this.
The connection issue from this morning appears to be resolved.
We are currently experiencing some instances of lower call quality. We are actively working to resolve this.
For roughly one hour today from 12:30pm - 1:45pm PT some of our customers experienced lower call quality when making outbound calls. Our team resolved this issue at roughly 1:45pm. We will continue to monitor the situation. We apologize for these issues as we continue to fully restore all of our systems.
We are continuing to investigate this issue.
Update - We have important updates regarding vPhone/Local ID numbers and SMS. Numbers Local ID and vPhone numbers purchased on or before January 25 are in the process of being restored. At this time, some numbers have been restored so continue to check your numbers library for the status of yours. Here are definitions of what those statuses mean: - Active (done & working) - Restore in Progress (in progress) - Pending Restore (in the queue to be restored) - In Review (being vetted by compliance) SMS For those still awaiting SMS approval, we need some additional info! Please fill out this form and we will submit your request for SMS on your behalf. http://phoneburner.com/sms-form This is in addition to having your business profile completed. If you still haven’t taken that step, you can log into your account and find your Business Profile page here: https://www.phoneburner.com/myaccount/businessprofile/index We know this is a lot, but as the industry and compliance evolves, we are doing what we can to help protect your business.
Now that the dialer is fully operational and premium services are restored, this will be the last update we provide using this communication path. While our platform was in the process of coming back to full operations, we used these Statuspage updates to deliver information to you as the situation was quickly evolving. Now that our primary services have been restored, we are going to transition back to our typical communication channels and use the Statuspage only in the event of future service outages. We are aware of some uncommon call quality issues that are improving each day. We are also aware that there are still some customers who are waiting for their numbers to be restored and for compliance approval for SMS. We will continue to communicate regularly with you via email and in-app messaging. We recommend adding phoneburner.com as a safe sender in your email settings to ensure you continue to receive any important messages from us. We thank you for all of your patience and look forward to a bright future together.