[zfs-discuss] Upgrading vdev mirror from ashift=9 to ashift=12 : 512B to 4K transition

fermulator fermulator at sympatico.ca
Fri Jan 1 14:20:37 EST 2016


I'm in a situation where it seems I may have unknowingly locked myself 
into ashift=9 for some vdevs in a pool of mirrors.

Here's the basic premise:
  * initial adoption and research into ZFS, physical sector size, 
ashift, and block alignment
  * created a new zpool with ashift=12
  * added vdevs (some use newer 4K sector size, others consider of older 
512B sector size)

I had /assumed/ that ashift property per vdev would inherit from the 
zpool, but this appears to not be the case. In actuality, it seems like 
zfs automatically chose the ashift value according to the drives 
reported sector size:
  * vdev mirrors with ashift=12 for the vdevs that have newer drives (4K 
physical sector size)
  * vdev mirrors with ashift=9 for the vdevs using older drives (512B 
physical sector size)

Here's what I've got, and please look beyond the fact that two of my 
vdevs are not yet redundant :o

   pool: zfsmedia
  state: ONLINE
   scan: scrub in progress since Fri Jan  1 10:08:11 2016
     1.45T scanned out of 7.20T at 104M/s, 16h6m to go
     0 repaired, 20.06% done
config:

         NAME STATE     READ WRITE CKSUM
         zfsmedia ONLINE       0     0     0
           ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E0HHF92N ONLINE       0     
0     0
           mirror-1 ONLINE       0     0     0
             ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5LAPP8J ONLINE       0     
0     0
             ata-ST2000VN000-1H3164_W1H25K77 ONLINE       0     0     0
           mirror-2 ONLINE       0     0     0
             ata-ST2000DL003-9VT166_5YD0XWHR ONLINE       0     0     0
             ata-ST2000VN000-1H3164_W1H25JXM ONLINE       0     0     0
           mirror-3 ONLINE       0     0     0
             ata-ST2000DL003-9VT166_5YD18S0M ONLINE       0     0     0
             ata-ST2000DL003-9VT166_5YD18S73 ONLINE       0     0     0
           ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E0HHFYTV ONLINE       0     
0     0

$ sudo zdb | grep -e ashift -e vdev -e type | grep -v disk
     vdev_children: 5
     vdev_tree:
         type: 'root'
             ashift: 12
             type: 'mirror'
             ashift: 12
             type: 'mirror'
             ashift: 9
             type: 'mirror'
             ashift: 9
             ashift: 12

$ for disk in $(ls /dev/disk/by-id | egrep ata | grep -v part); do echo 
$disk; sudo smartctl -a /dev/disk/by-id/$disk | egrep Size; done
(snip)
ata-ST2000DL003-9VT166_5YD0XWHR
Sector Size:      512 bytes logical/physical
ata-ST2000DL003-9VT166_5YD15M4K
Sector Size:      512 bytes logical/physical
ata-ST2000DL003-9VT166_5YD18S0M
Sector Size:      512 bytes logical/physical
ata-ST2000DL003-9VT166_5YD18S73
Sector Size:      512 bytes logical/physical
ata-ST2000VN000-1H3164_W1H25JXM
Sector Sizes:     512 bytes logical, 4096 bytes physical
ata-ST2000VN000-1H3164_W1H25K77
Sector Sizes:     512 bytes logical, 4096 bytes physical
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5LAPP8J
Sector Sizes:     512 bytes logical, 4096 bytes physical
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5YK6CRE
Sector Sizes:     512 bytes logical, 4096 bytes physical
ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E0HHF92N
Sector Sizes:     512 bytes logical, 4096 bytes physical
ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E0HHFYTV
Sector Sizes:     512 bytes logical, 4096 bytes physical


Here's how things were created:
2015-08-31.22:43:14 zpool create -f -o ashift=12 -O compression=lz4 
zfspool /dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E0HHF92N
(snip)
2015-09-13.17:23:21 zpool add -f zfspool mirror 
/dev/disk/by-id/ata-ST2000DL003-9VT166_5YD15M4K 
/dev/disk/by-id/ata-ST2000VN000-1H3164_W1H25K77
2015-09-13.17:29:20 zpool add -f zfspool mirror 
/dev/disk/by-id/ata-ST2000DL003-9VT166_5YD0XWHR 
/dev/disk/by-id/ata-ST2000VN000-1H3164_W1H25JXM
2015-09-13.17:30:28 zpool add -f zfspool mirror 
/dev/disk/by-id/ata-ST2000DL003-9VT166_5YD18S0M 
/dev/disk/by-id/ata-ST2000DL003-9VT166_5YD18S73
2015-09-13.17:42:33 zpool add zfspool 
/dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E0HHFYTV
(snip)
etc.

Now, as the drives in the vdev with ashift=9 fail, we would presumably 
replace them with newer drives, and it's highly likely these drives will 
have 4K sector size.
But how can we "migrate" the vdev from ashift=9 to ashift=12?

I've done some searching and found that others have run into this 
scenario, but the solution is not clearly resolved in any of these posts 
thus far.

http://comments.gmane.org/gmane.os.omnios.general/5580
http://list.zfsonlinux.org/pipermail/zfs-discuss/2014-January/013501.html
https://github.com/zfsonlinux/zfs/issues/2024
https://github.com/zfsonlinux/zfs/commit/b28e57cb82c5d5a992b90c56f67dd7dbf9b6f296

Do we have documentation somewhere which explains how to transition? 
This scenario has got to be extremely common. If not already, then very 
soon for a vast majority of users.
Some documentation claims that it's impossible to even destroy a vdev, 
which seems to imply solving this problem requires a costly 
recreation/clone of the existing pool to a new one with proper 
future-proof ashift values set.

~fermulator
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.zfsonlinux.org/pipermail/zfs-discuss/attachments/20160101/14ccb68d/attachment.html>


More information about the zfs-discuss mailing list