It’s been over two years since I built my home NAS box. A couple days ago I logged in and noticed some file operation was taking a while, so I did a “zpool status” and was shocked to see one of the disks had failed at some point. I really have no idea when it happened, but the “last scan” date said something about December, so it could have been months. I was a bad Sysadmin and didn’t have alerting for this case – silent HA failover is a risk most people don’t think of, though coincidentally it’s one of the projects I’m currently working on at work.
Anyway, here’s what the failed zpool looked like:
root@lunix:~# zpool status
pool: lunix1
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: scrub repaired 0 in 12h54m with 0 errors on Sun Dec 2 14:18:15 2018
config:
NAME STATE READ WRITE CKSUM
lunix1 DEGRADED 0 0 0
mirror-0 ONLINE 0 0 0
sdb ONLINE 0 0 0
sdc ONLINE 0 0 0
mirror-1 DEGRADED 0 0 0
sdd ONLINE 0 0 0
11769402787959493007 UNAVAIL 0 0 0 was /dev/sde1
errors: No known data errors
Fortunately, back when I built this thing, I bought a spare disk which has been sitting on the shelf in the box from Newegg for 2 years. I replaced it a little while ago and ran “zpool import” to bring the zpool back in, and then did the “zpool replace”:
root@lunix:~# zpool replace -f lunix1 11769402787959493007 sdd
root@lunix:~# zpool status
pool: lunix1
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Apr 26 15:47:41 2019
21.9M scanned out of 3.71T at 3.12M/s, 345h50m to go
10.5M resilvered, 0.00% done
config:
NAME STATE READ WRITE CKSUM
lunix1 DEGRADED 0 0 0
mirror-0 ONLINE 0 0 0
sdb ONLINE 0 0 0
sdc ONLINE 0 0 0
mirror-1 DEGRADED 0 0 0
sde ONLINE 0 0 0
replacing-1 UNAVAIL 0 0 0
11769402787959493007 FAULTED 0 0 0 was /dev/sde1
sdd ONLINE 0 0 0 (resilvering)
errors: No known data errors
345 hours to go?! I ran zpool status about 20 minutes later and got a much better number:
root@lunix:~# zpool status
pool: lunix1
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Apr 26 15:47:41 2019
125G scanned out of 3.71T at 101M/s, 10h18m to go
62.3G resilvered, 3.28% done
config:
NAME STATE READ WRITE CKSUM
lunix1 DEGRADED 0 0 0
mirror-0 ONLINE 0 0 0
sdb ONLINE 0 0 0
sdc ONLINE 0 0 0
mirror-1 DEGRADED 0 0 0
sde ONLINE 0 0 0
replacing-1 UNAVAIL 0 0 0
11769402787959493007 FAULTED 0 0 0 was /dev/sde1
sdd ONLINE 0 0 0 (resilvering)
errors: No known data errors
Anyway, this was my first experience replacing a failed disk under a ZFS raid, so it was pretty exciting. The hardest part of the process was figuring out which disk was the failed one. The bad one didn’t even show up as a device in /dev, so what I ended up doing was “hdparm -i /dev/sd?” for each disk that was there, noting the serial number, and then looking on the outside of each disk to see which one had a serial that wasn’t available to the OS. Overall it was a pretty good process.
Evan Hoffman's Blog 