[zfs-discuss] Slow read performance

Alex Vodeyko alex.vodeyko at gmail.com
Sun Apr 1 01:23:47 EDT 2018


Re-created with ashift=12, xattr=sa, atime=off
"dd" writes = 2.7 GB/s, reads = 826 MB/s - so reads dropped ...

ashift=12: "iostat" dd writes
dm-61             0.00     0.00    0.40  142.20     0.00    47.10
676.45    17.73  120.13    0.00  120.47   5.07  72.28
dm-62             0.00     0.00    0.00  141.00     0.00    47.10
684.07    17.93  123.27    0.00  123.27   5.10  71.94

ashift=12: "iostat" dd reads
dm-61             0.00     0.00  113.80    0.00    14.22     0.00
256.00     0.07    0.62    0.62    0.00   0.47   5.36
dm-62             0.00     0.00  114.20    0.00    14.27     0.00
256.00     0.08    0.72    0.72    0.00   0.50   5.74

Re-created back with ashift=9, xattr=sa, atime=off
"dd" writes = 2.4 GB/s, reads = 1.7 GB/s

ashift=9: "iostat" dd writes
dm-61             0.00     0.00    0.00  152.60     0.00    41.86
561.79    12.59   80.76    0.00   80.76   4.77  72.80
dm-62             0.00     0.00    0.00  127.00     0.00    41.46
668.59    14.03  106.95    0.00  106.95   5.49  69.76

ashift=9: "iostat dd reads (showed 128k)
dm-61             0.00     0.00  209.40    0.00    26.15     0.00
255.76     0.07    0.31    0.31    0.00   0.30   6.20
dm-62             0.00     0.00  209.20    0.00    26.13     0.00
255.76     0.06    0.30    0.30    0.00   0.28   5.82

"fio" with 4 worker reads = 2.1 GB/s, "iostat" - reads are also showed = 128k
dm-61             0.00     0.00  260.40    0.00    32.50     0.00
255.62     0.47    1.80    1.80    0.00   0.99  25.68
dm-62             0.00     0.00  260.20    0.00    32.45     0.00
255.44     0.42    1.62    1.62    0.00   0.89  23.28

To remind - all of the above came from zpool of six 8+2 raidz2 (60 drives total)

For comparison I've created one zpool with single 8+2 raidz2 (10
drives) and rerun tests on it, so:

10 drives, 8+2 raidz2: ashift=12, xattr=sa, atime=off
"dd" writes = 1.4 GB/s, reads = 1 GB/s

ashift=12: "iostat" dd writes
dm-37             0.00     0.00    0.00  628.60     0.00   166.04
540.95    15.10   24.06    0.00   24.06   1.44  90.80
dm-41             0.00     0.00    0.00  739.80     0.00   167.50
463.68    12.08   16.49    0.00   16.49   1.21  89.28

ashift=12: "iostat" dd reads
dm-37             0.00     0.00  750.20    0.00    93.73     0.00
255.87     0.78    1.04    1.04    0.00   0.39  29.02
dm-41             0.00     0.00  751.40    0.00    93.90     0.00
255.93     0.33    0.44    0.44    0.00   0.30  22.66

10 drives, 8+2 raidz2: ashift=9, xattr=sa, atime=off
"dd" writes = 885 MB/s, reads = 1.2 GB/s

ashift=9: "iostat" dd writes
dm-37             0.00     0.00    0.00  461.60     0.00   104.54
463.81    16.49   34.82    0.00   34.82   1.92  88.52
dm-41             0.00     0.00    0.00  480.20     0.00   104.16
444.23    15.24   31.12    0.00   31.12   1.86  89.14

ashift=9: "iostat" dd reads
dm-37             0.00     0.00  782.40    0.00    97.65     0.00
255.61     0.32    0.40    0.40    0.00   0.27  21.00
dm-41             0.00     0.00  782.00    0.00    97.68     0.00
255.80     0.30    0.38    0.38    0.00   0.28  21.68

Something happens while using more than one 8+2 raidz2 in a pool.
Maybe setting some additional log/debug will help to understand?

Thanks,
Alex


2018-03-31 23:13 GMT+03:00 Gionatan Danti <g.danti at assyoma.it>:
> Il 31-03-2018 20:05 Alex Vodeyko ha scritto:
>>
>> Changed with ashift=9 (though I'm sure I've tested it w/o luck) and
>> xattr=sa and reads become much better (though writes dropped a bit)...
>> "dd" writes = 2.4 GB/s, reads = 1.7 GB/s
>
>
> Interesting. What happens with ashift=12 and xattr=sa ?
> What about iostat during the "fast" dd run? It showed 128K reads only?
>
> Thanks.
>
> --
> Danti Gionatan
> Supporto Tecnico
> Assyoma S.r.l. - www.assyoma.it
> email: g.danti at assyoma.it - info at assyoma.it
> GPG public key ID: FF5F32A8


More information about the zfs-discuss mailing list