51 Binary Lane incidents · июнь 2018 г. — official updates, affected components, duration and resolution details.
В настоящее время мы расследуем этот вопрос.
Этот вопрос был определен, и в настоящее время осуществляется исправление.
Реализовано исправление, и мы отслеживаем результаты.
Этот инцидент был урегулирован.
Автоматический перевод официального обновления инцидента.
We are currently investigating this issue.
BinaryLane is responding to a large-scale denial-of-service attack targeting our entire autonomous system. The attack has been significant enough to affect our primary upstream provider, who has had to drop our ports several times to protect their wider network — most recently because the attack resumed each time we were re-added. We are currently announcing traffic via our secondary upstream, which is keeping the platform reachable but with reduced throughput and higher latency for traffic outside Sydney, where the secondary provider's footprint is concentrated. We have a meeting at 4:00 PM AEST with our primary upstream to agree a sustainable path forward, and in parallel we are working with another Australian carrier (with whom we have direct connectivity in all our datacentres) to bring additional capacity online for mitigation. Customer data is not at risk — this is a network-level attack, not an intrusion. We'll post a further update immediately after the 4:00 PM meeting. Thank you for your patience.
We've now met with our primary upstream and agreed a plan to return stability to the network, and that plan is being implemented now. In parallel, over the past hour we've continued to work with additional providers to bring online further capacity for ongoing DDoS mitigation. We will post the next update at 6:30 PM AEST. Thank you for your continued patience.
Update — services generally restored, ongoing monitoring We are continuing to mitigate what remains the largest DDoS attack we have observed. Our primary upstream has implemented local policies that are materially reducing the impact, and we've engaged several additional partners to absorb traffic — some have been unable to handle the volume, but others are now contributing capacity successfully. Services are now generally available, and we will continue close monitoring over the next 12 hours. We'll post a further update if the picture changes materially, or once the monitoring window is complete. Thank you for your patience through today.
We have continued to make steady progress restoring direct connectivity at each of our city POPs over the weekend, and the majority of our locations have now been returned to normal direct service. Perth and Adelaide are the two remaining sites still being carried via Sydney while the upstream paths there are brought back online. Customers in those two cities will continue to see slightly elevated latency on international destinations while the backhaul is in place, but services remain available. We expect to return Perth and Adelaide to direct service later today. Across the weekend the network remained steady — once we had reached the stable configuration on Friday evening, we observed no further customer-affecting events, and overall capacity has held throughout. We have been monitoring closely the whole time and have not needed to take any further action beyond the recovery work already in progress. We are leaving this incident open until every POP has been returned to its normal direct-service configuration, rather than closing it once services are usable, because we want our public record of this event to reflect a complete restoration rather than a partial one. We will continue to post updates as the remaining sites come back online, and a final close-out summary once everything is fully restored. Thank you again for your patience through Friday's disruption and across the weekend. We sincerely apologise for the impact, and we appreciate the understanding shown by customers while we worked through the incident.
Adelaide and Perth were returned to direct-service connectivity at 6:24pm AEST last night (Monday 18 May), completing the restoration of all city POPs to their normal upstream paths. The network is now stable and operating in its normal configuration across all locations. Our work now shifts from active restoration to investigation and remediation. We are continuing to work with our upstream providers to understand the full picture of Friday's event and to identify structural changes that will reduce our exposure to a similar attack pattern in future. A portion of the technical investigation is being handled in cooperation with the Australian Signals Directorate under an open case. A full post-incident review will be published once the investigation phase is complete and the ASD engagement is at a point where we can share the technical picture without compromising it. We will notify status-page subscribers when that is available, on a timeframe of weeks rather than days. To reiterate from earlier updates: this was a network-level denial-of-service event affecting the reachability of our public IP space. It was not a security breach or intrusion, no customer systems were accessed, and no customer data was involved in any way. Customer workloads continued to operate normally throughout. We will leave this incident open in a monitoring state until the post-incident review is published. Thank you for your patience throughout, and for the constructive feedback received from customers over the weekend.
This update formally closes the incident of Friday 15 May. To summarise the event: from 12:30 PM AEST our network was targeted by a large-scale volumetric denial-of-service attack directed evenly across our entire public IP space — the largest we have observed. The volume exceeded the international transit capacity of our primary upstream, who dropped our connectivity several times to protect their wider network. Services were kept reachable via alternate paths through the afternoon, with general availability restored by 7:12 PM that evening and all city POPs returned to their normal direct-service configuration by Monday 18 May. As stated throughout: this was a network-level attack on reachability. It was not a breach or intrusion, no customer systems were accessed, and no customer data was involved. Our engagement with the Australian Signals Directorate on this incident has concluded from our side, with all requested material provided or accounted for. We again thank our customers for their patience and the constructive feedback we received. This incident is now resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
This host node has encountered an irrecoverable hardware fault. We will begin restoring affected servers from latest backup (if purchased) or fresh OS installation shortly.
Restore/reinstall of affected servers is underway.
This incident has been resolved.
We are currently investigating this issue.
We are continuing to investigate this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
A host outage occurred on sydcompute78. VPS are being migrated to other hosts and rebooted as necessary. Investigations are ongoing.
This incident has been resolved.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
A host outage occurred on sydcompute56 and sydcompute57. VPS are being migrated to other hosts and rebooted as necessary. Investigations are ongoing.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
We are continuing to investigate this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Cloud servers on host node sydcompute55 lost network connectivity due to a network card failure. Service was restored at approx 21:10 UTC
