Replacing a failed disk in a ZFS pool

Question

I've got a large ZFS disk pool; 3 nested RAIDZ2 vdevs.

I am documenting the process for replacing a failed disk for my colleagues and so simulated a disk failure by removing a disk from the host.

Sure enough, the vdev to which the disk belonged became degraded and the disk unavailable.

I offlined the disk like so...

zpool offline diskpool sdo

A quick 'zpool status' shows the disk as offline... so far so good.

I replaced the disk and confirmed on my SATA controller the new disk was detected, which it was. Then i tried to get linux to rescan the scsi bus to detect the disk. This is where my first problem occurs.

As far as I know, the following command is used to find the correct host bus to rescan...

grep mpt /sys/class/scsi_host/host?/proc_name

However on my Centos 7.2 system, this command has no output. It doesn't error, it just gives me null output and waits for my next command.

I'm using several specialist cards that allow me to connect many sata devices. I would normally rescan the bus with

echo "- - -" > /sys/class/scsi_host/hostX/scan

Where hostX is the correct host bus, but as I cannot find the host bus, I cannot complete this step.

Is there another way to get this info or has the command changed in Centos 7.2 or something?

Furthermore, I opted to reboot the machine to allow me to continue testing. Following a reboot, the ZFS pool was not attached. I had to manually import it with 'zpool import diskpool'. That worked fine, but strangely once its imported, if i do 'zpool status', I no longer see the device IDs like it showed me before...

 raidz2-2                            ONLINE       0     0     0
        /dev/sdd                     ONLINE       0     0     0
        /dev/sde                     ONLINE       0     0     0
        /dev/sdf                     ONLINE       0     0     0
        /dev/sdg                     ONLINE       0     0     0

Instead, it seems to have the drive serial numbers...

      raidz2-2                            ONLINE       0     0     0
        ata-ST8000AS0002-1NA17Z_Z840DG66  ONLINE       0     0     0
        ata-ST8000AS0002-1NA17Z_Z840DVE0  ONLINE       0     0     0
        ata-ST8000AS0002-1NA17Z_Z840CQFB  ONLINE       0     0     0
        ata-ST8000AS0002-1NA17Z_Z840DP2V  ONLINE       0     0     0

This will cause a problem in the future as if a further disk fails, I will struggle to identify the correct disk to replace.

Is there a way I can switch this back so I'm shown the device id again?

Thanks in advance!

Answer 1

ZFS detects disks not by their name in the filesystem, but by their UUID that is written onto the disk (or at least something similar -- not 100% sure that it's actually a UUID). When zpool import runs, the disks are enumerated, ZFS rebuilds all the pools, and then uses the device name (without actually including any directory IME, usually it's something like sda rather than /dev/sda) in the zpool status output. As such, if you move the drives around (or if the kernel moves the drives around, which can happen with modern kernels on modern hardware), zpool will still detect the disks in the same order as it did before; disks that appeared first in the output will again appear first in the output, even if the kernel doesn't enumerate them in that output anymore.

What happened to you here is probably that due to the fact that the original zpool import didn't work, the kernel could complete its boot, udev did a lot more work, and then by the time you did the manual zpool import, the default enumeration of all your disks turned out to have the serial number-based ones first, rather than the sdX-based ones. Most likely, the next time you reboot the machine, the used names will be back to the sdX scheme.

Luckily, resolving the names from one naming scheme to the other is fairly straightforward:

wouter@gangtai:/dev/disk/by-id$ ls -l
total 0
lrwxrwxrwx. 1 root root  9 Mar 31 18:15 ata-SAMSUNG_MZ7TE256HMHP-00004_S1RKNSAFC04685 -> ../../sda
lrwxrwxrwx. 1 root root 10 Mar 31 18:15 ata-SAMSUNG_MZ7TE256HMHP-00004_S1RKNSAFC04685-part1 -> ../../sda1
lrwxrwxrwx. 1 root root  9 Mar 31 18:15 wwn-0x50025388a089e89c -> ../../sda
lrwxrwxrwx. 1 root root 10 Mar 31 18:15 wwn-0x50025388a089e89c-part1 -> ../../sda1

There are multiple naming schemes (by-id, by-uuid, and by-path), all of which can be found under /dev/disk.

Having said all that, I must say I don't agree with your claim that it would be easier to figure out which disk is which by looking at the sdX names. Modern kernels no longer assign static device names to particular devices; this is why modern distributions use UUID-based fstab files, rather than sdX-based ones. The serial number, in fact, is a far more reliable way to figure out which is the broken disk; after all, it's written on the actual disk, in contrast to the sdX name, which may differ from boot to boot (I've actually encountered that on a ZFS box with sixteen hard disks). Any one of the other methods (by-uuid, by-id, and especially by-path in the enterprise-level multi-disk enclosures) is much more reliable than that.