Juicefs outage history and incident timeline

Outrage in Asia Regions

Started May 12, 2019 at 6:26 PM UTC 0m

IssuesMinor incident

May 12, 2019 at 6:26 PM UTCresolved
May 13, 2019 02:15 +08 Investigating - Recieved alarm on disrupted service in for Asia regions (Singapore). May 13, 2019 02:26 +08 Resolved - Restarted those instanses and add more memory, the service was fully recovered.

Started May 15, 2018 at 1:38 PM UTC 0m

OutageMajor incident

May 15, 2018 at 1:38 PM UTCresolved
May 15, 2018 21:08 +08 Investigating - Two of the three meta service for UCloud North failed, which caused service outage. May 15, 2018 21:38 +08 Resolved - - Apply for new available zone E and distribute meta services in zone B, D, E to reduce the outage chance. - Lower the alarm threshold to detect failure sooner

Started February 18, 2018 at 4:55 AM UTC 0m

OutageMajor incident

February 18, 2018 at 4:55 AM UTCresolved
February 18, 2018 12:08 +08 Investigating - Two of the three meta services failed to accept new connections. No syslog for past 20 hours. February 18, 2018 12:55 +08 Resolved - - Reboot node to recover - Rollback recent changes in syslog configuration

Started January 22, 2018 at 12:54 PM UTC 0m

OutageMajor incident

January 22, 2018 at 12:54 PM UTCresolved
January 22, 2018 20:42 +08 Investigating - Out of memory when rotating changelog to disk. It causes slow response in election and finally losing leader. January 22, 2018 20:54 +08 Resolved - - Increase memory - Adjust alarm threshold for memory

Started November 16, 2017 at 8:26 PM UTC 0m

IssuesMinor incident

November 16, 2017 at 8:26 PM UTCresolved
November 17, 2017 04:16 +08 Investigating - 12:18 Received alarm about disruption of JuiceFS metadata service in UCloud beijing region. Investigation started. 12:20 Saw that two follower nodes can't replay change log from leader, kept restarting. 12:23 Restarted the previous leader. 12:26 Service is fully recovered, back to normal. 13:30 Find the bug that cause the failure of applying. 20:30 Deployed the bugfix. November 17, 2017 04:26 +08 Resolved - This is caused by a bug introduced in 4.2.2, which have a slightly different logic to apply change log, that introduce different internal result and fail the checksum. The bug was fixed and deployed.

Started July 3, 2017 at 6:50 AM UTC 0m

IssuesMinor incident

July 3, 2017 at 6:50 AM UTCresolved
July 3, 2017 14:30 +08 Scheduled - Deploy a new release with huge performance improvements. July 3, 2017 14:30 +08 Active - 6:30 Deployed to Aliyun HK region, 6:40 Start to deploy to all regions. July 3, 2017 14:50 +08 Completed - 6:50 Deployed to all regions successfully.

Started May 16, 2017 at 6:30 AM UTC 0m

IssuesMinor incident

May 16, 2017 at 6:30 AM UTCresolved
May 16, 2017 14:12 +08 Investigating - 6:14 Got a report that JuiceFS in AWS Beijing region is slow, timeout to connect to metadata service. 6:16 Start investigating. We realized that a node from UCloud Beijing region is elected as active to serve requests from AWS Beijing. The egress of AWS Beijing region is slow and dropping packets. 6:30 The egress of AWS Beijing is back to normal. May 16, 2017 14:30 +08 Resolved - 13:34 All the node for AWS Beijing are moved to AWS Beijing to make sure that slowness of egress will not degrade the performance of JuiceFS. We also double check all other regions to make sure that the metadata service is access within the same region for better performance.