[zfs-discuss] Slow read performance with Samba

Richard Elling richard.elling at richardelling.com
Tue Dec 4 15:27:51 EST 2018

> On Dec 3, 2018, at 7:24 PM, Chris Siebenmann via zfs-discuss <zfs-discuss at list.zfsonlinux.org> wrote:
>> "zfetchstats" showed 90% hit rate...
> I believe that this is not necessarily what you might think it is.
> The data printed out by Richard Elling's zfetchstats program and shown
> in /proc/spl/kstat/zfetchstats is whether or not ZFS's prefetcher
> thinks it can do anything. To simplify somewhat, ZFS maintains an
> idea of prefetch streams for files, which are IO patterns that it is
> (potentially) prefetching in. A zfetchstats 'hit' is whether an IO falls
> into such a prefetching stream and will trigger an attempt to prefetch
> more data; a 'miss' is an IO that does not fall into a prefetching
> stream and will not trigger any prefetching. A zfetchstats hit is a
> necessary prerequisite to usefully prefetching things, but it is not
> sufficient. If you have sequential IO and not too much of it, you
> should expect a very high zfetchstats hit rate.
> To see whether or not the prefetching actually *worked*, you need to
> look at the ARC stat demand_hit_predictive_prefetch. This is visible in
> /proc/spl/kstat/zfs/arcstats but may not be supported by the arcstat(s)
> program you may be using. The ARC stats have other things that talk
> about 'prefetch', but these again are not what you want; they are a
> count of whether attempts to prefetch data or metadata found what they
> were looking for in the ARC or whether they had to go to disk.

To follow on with Chris' suggestions, the best long-term approach is to monitor
these metrics with a monitoring system. Both telegraf and node_exporter do
collect all of the arcstats and zfetchstats metrics (plus others) and these can
then be stored in a variety of time-series databases. Using a CLI tool is so
1990s :-)

> The other way to look at how effective prefetching was is simply
> to start with an ARC that doesn't contain your data (in the extreme case,
> you can tell the Linux kernel to drop all its disk caches), read your
> data, and look at the ARC hit rate for 'demand data' or 'demand hit
> percentage'. A high hit rate means that prefetching and other methods
> of getting the data into the ARC actively helped things.

This is a bit extreme for most folks, though. A more extreme approach is
to use stap or bpf to monitor accesses. Perhaps some day, if there is enough
interest and spare time, we could document some of these techniques.
 -- richard

More information about the zfs-discuss mailing list