[zfs-discuss] Slow read performance with Samba
richard.elling at richardelling.com
Tue Dec 4 15:27:51 EST 2018
> On Dec 3, 2018, at 7:24 PM, Chris Siebenmann via zfs-discuss <zfs-discuss at list.zfsonlinux.org> wrote:
>> "zfetchstats" showed 90% hit rate...
> I believe that this is not necessarily what you might think it is.
> The data printed out by Richard Elling's zfetchstats program and shown
> in /proc/spl/kstat/zfetchstats is whether or not ZFS's prefetcher
> thinks it can do anything. To simplify somewhat, ZFS maintains an
> idea of prefetch streams for files, which are IO patterns that it is
> (potentially) prefetching in. A zfetchstats 'hit' is whether an IO falls
> into such a prefetching stream and will trigger an attempt to prefetch
> more data; a 'miss' is an IO that does not fall into a prefetching
> stream and will not trigger any prefetching. A zfetchstats hit is a
> necessary prerequisite to usefully prefetching things, but it is not
> sufficient. If you have sequential IO and not too much of it, you
> should expect a very high zfetchstats hit rate.
> To see whether or not the prefetching actually *worked*, you need to
> look at the ARC stat demand_hit_predictive_prefetch. This is visible in
> /proc/spl/kstat/zfs/arcstats but may not be supported by the arcstat(s)
> program you may be using. The ARC stats have other things that talk
> about 'prefetch', but these again are not what you want; they are a
> count of whether attempts to prefetch data or metadata found what they
> were looking for in the ARC or whether they had to go to disk.
To follow on with Chris' suggestions, the best long-term approach is to monitor
these metrics with a monitoring system. Both telegraf and node_exporter do
collect all of the arcstats and zfetchstats metrics (plus others) and these can
then be stored in a variety of time-series databases. Using a CLI tool is so
> The other way to look at how effective prefetching was is simply
> to start with an ARC that doesn't contain your data (in the extreme case,
> you can tell the Linux kernel to drop all its disk caches), read your
> data, and look at the ARC hit rate for 'demand data' or 'demand hit
> percentage'. A high hit rate means that prefetching and other methods
> of getting the data into the ARC actively helped things.
This is a bit extreme for most folks, though. A more extreme approach is
to use stap or bpf to monitor accesses. Perhaps some day, if there is enough
interest and spare time, we could document some of these techniques.
More information about the zfs-discuss