[zfs-discuss] Currently sysctl recommended parameters to accelerate resilver?

Chris Siebenmann cks at cs.toronto.edu
Mon Dec 4 17:06:34 EST 2017

[belatedly and slowly:]
> So far so good, but the resilver (with the machine otherwise entirely
> idle) is taking forever to complete; "zpool status" in fact is showing
> just 28.5MB/s, while I know for a fact that these disks have at least
> 100MB/s sequential performance (and no, the pool is not fragmented as
> I just created it and restored its contents from a backup via "tar").

 I'm not sure this insures that metadata in the pool has been written
out contiguously (although it should ensure that data has been for large
enough files). I would expect that unpacking a tar archive would spread
out the creation of new files and directories over a fair amount of time,
which may naturally scatter their metadata around the disk as they're
written in different transaction groups.

 Unfortunately, my understanding is that scattered metadata is one
significant slowdown for resilvers (and scrubs), because ZFS has to
traverse through the metadata tree before it can start reading that
nicely contiguous data.

> To try and accelerate it, I adjusted  the old(?) sysctl "resilver"
> parameters, to wit:
>         echo 5000 > /sys/module/zfs/parameters/zfs_resilver_min_time_ms
>         echo 0 >/sys/module/zfs/parameters/zfs_resilver_delay
>         echo 512 > /sys/module/zfs/parameters/zfs_top_maxinflight
> But, to my great surprise, nothing changed: speed is still at 28.5MB/s
> after wating 15 minutes since the adjustment.
> Have these parameters, or their recommended values for fast resilvers,
> changed from 0.6.x to 0.7.x?

 As far as I know, these are still the best values for fast resilvers
that will issue as much IO as possible as fast as possible.

 Note that looking at the 'zpool status' resilver speed indicator is not
necessarily reliable, because it is the cumulative speed to date, not
the instantaneous speed. If a slow resilver has been running for a long
time and you speed it up, that speedup may take a significant amount
of time to be reflected in the MB/s numbers. What I do is look at the
difference in the amount scanned after, say, sixty seconds; that gives a
better idea of the instantaneous resilver rate.

 In general I would suggest looking at the IO stats of your drives.
This should give you an idea of the drive-level IO that's going on and
how your drives are performing; if you see the current one doing, say,
150 read IOs a second, you're probably seek-limited despite appearances.

(This is such a perennial issue that perhaps someday ZFS will expose
IO stats or kstats that provide insight on what the true current scrub
speed and scrub limits are. It would be nice.)

	- cks

More information about the zfs-discuss mailing list