[zfs-discuss] Slow read performance

Alex Vodeyko alex.vodeyko at gmail.com
Sun Apr 1 09:44:54 EDT 2018


Single dual-port LSI SAS 9300-8e HBA. port1 connected to JBOD's esmA,
and port2 - to esmB.
"multipath" manages 30 drives through port1 and other 30 through port2
(though I've also tested w/o multipath all 60 drives on one port).

It's not the best config, but anyway I would be happy to get ~2-3GB/s
from 30 drives (the best will be 4-5 GB/s from 60 drives)

Thanks,
Alex


2018-04-01 13:04 GMT+03:00 George Melikov <mail at gmelikov.ru>:
> How are your disks connected? Via how many controllers (of which model)?
>
> In my practice I remembered that raidz generates much more IOPS than mdraid
> and controller might be the bottleneck.
>
> ____________________________________
> Sincerely,
> George Melikov
>
>
>
>
> 01.04.2018, 08:24, "Alex Vodeyko via zfs-discuss"
> <zfs-discuss at list.zfsonlinux.org>:
>
> Re-created with ashift=12, xattr=sa, atime=off
> "dd" writes = 2.7 GB/s, reads = 826 MB/s - so reads dropped ...
>
> ashift=12: "iostat" dd writes
> dm-61 0.00 0.00 0.40 142.20 0.00 47.10
> 676.45 17.73 120.13 0.00 120.47 5.07 72.28
> dm-62 0.00 0.00 0.00 141.00 0.00 47.10
> 684.07 17.93 123.27 0.00 123.27 5.10 71.94
>
> ashift=12: "iostat" dd reads
> dm-61 0.00 0.00 113.80 0.00 14.22 0.00
> 256.00 0.07 0.62 0.62 0.00 0.47 5.36
> dm-62 0.00 0.00 114.20 0.00 14.27 0.00
> 256.00 0.08 0.72 0.72 0.00 0.50 5.74
>
> Re-created back with ashift=9, xattr=sa, atime=off
> "dd" writes = 2.4 GB/s, reads = 1.7 GB/s
>
> ashift=9: "iostat" dd writes
> dm-61 0.00 0.00 0.00 152.60 0.00 41.86
> 561.79 12.59 80.76 0.00 80.76 4.77 72.80
> dm-62 0.00 0.00 0.00 127.00 0.00 41.46
> 668.59 14.03 106.95 0.00 106.95 5.49 69.76
>
> ashift=9: "iostat dd reads (showed 128k)
> dm-61 0.00 0.00 209.40 0.00 26.15 0.00
> 255.76 0.07 0.31 0.31 0.00 0.30 6.20
> dm-62 0.00 0.00 209.20 0.00 26.13 0.00
> 255.76 0.06 0.30 0.30 0.00 0.28 5.82
>
> "fio" with 4 worker reads = 2.1 GB/s, "iostat" - reads are also showed =
> 128k
> dm-61 0.00 0.00 260.40 0.00 32.50 0.00
> 255.62 0.47 1.80 1.80 0.00 0.99 25.68
> dm-62 0.00 0.00 260.20 0.00 32.45 0.00
> 255.44 0.42 1.62 1.62 0.00 0.89 23.28
>
> To remind - all of the above came from zpool of six 8+2 raidz2 (60 drives
> total)
>
> For comparison I've created one zpool with single 8+2 raidz2 (10
> drives) and rerun tests on it, so:
>
> 10 drives, 8+2 raidz2: ashift=12, xattr=sa, atime=off
> "dd" writes = 1.4 GB/s, reads = 1 GB/s
>
> ashift=12: "iostat" dd writes
> dm-37 0.00 0.00 0.00 628.60 0.00 166.04
> 540.95 15.10 24.06 0.00 24.06 1.44 90.80
> dm-41 0.00 0.00 0.00 739.80 0.00 167.50
> 463.68 12.08 16.49 0.00 16.49 1.21 89.28
>
> ashift=12: "iostat" dd reads
> dm-37 0.00 0.00 750.20 0.00 93.73 0.00
> 255.87 0.78 1.04 1.04 0.00 0.39 29.02
> dm-41 0.00 0.00 751.40 0.00 93.90 0.00
> 255.93 0.33 0.44 0.44 0.00 0.30 22.66
>
> 10 drives, 8+2 raidz2: ashift=9, xattr=sa, atime=off
> "dd" writes = 885 MB/s, reads = 1.2 GB/s
>
> ashift=9: "iostat" dd writes
> dm-37 0.00 0.00 0.00 461.60 0.00 104.54
> 463.81 16.49 34.82 0.00 34.82 1.92 88.52
> dm-41 0.00 0.00 0.00 480.20 0.00 104.16
> 444.23 15.24 31.12 0.00 31.12 1.86 89.14
>
> ashift=9: "iostat" dd reads
> dm-37 0.00 0.00 782.40 0.00 97.65 0.00
> 255.61 0.32 0.40 0.40 0.00 0.27 21.00
> dm-41 0.00 0.00 782.00 0.00 97.68 0.00
> 255.80 0.30 0.38 0.38 0.00 0.28 21.68
>
> Something happens while using more than one 8+2 raidz2 in a pool.
> Maybe setting some additional log/debug will help to understand?
>
> Thanks,
> Alex
>
>
> 2018-03-31 23:13 GMT+03:00 Gionatan Danti <g.danti at assyoma.it>:
>
>  Il 31-03-2018 20:05 Alex Vodeyko ha scritto:
>
>
>  Changed with ashift=9 (though I'm sure I've tested it w/o luck) and
>  xattr=sa and reads become much better (though writes dropped a bit)...
>  "dd" writes = 2.4 GB/s, reads = 1.7 GB/s
>
>
>
>  Interesting. What happens with ashift=12 and xattr=sa ?
>  What about iostat during the "fast" dd run? It showed 128K reads only?
>
>  Thanks.
>
>  --
>  Danti Gionatan
>  Supporto Tecnico
>  Assyoma S.r.l. - www.assyoma.it
>  email: g.danti at assyoma.it - info at assyoma.it
>  GPG public key ID: FF5F32A8
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at list.zfsonlinux.org
> http://list.zfsonlinux.org/cgi-bin/mailman/listinfo/zfs-disc…


More information about the zfs-discuss mailing list