[zfs-discuss] Increase number of I/O workers?

jonny at jonny.eng.br jonny at jonny.eng.br
Thu Feb 11 15:57:57 EST 2016


On 02/11/2016 04:23 PM, Durval Menezes via zfs-discuss wrote:
>
> Hello Jonny,
>
> Sorry for the delay in getting back to you, was fixing a refrigerator 
> here (no joke).
>
> On Feb 11, 2016 3:24 PM, "Gordan Bobic via zfs-discuss" 
> <zfs-discuss at list.zfsonlinux.org 
> <mailto:zfs-discuss at list.zfsonlinux.org>> wrote:
> >
> > Both rsync and scrub are single-threaded. They will never generate 
> enough load to saturate the CPU. Try with 20 rsync instances running 
> simultaneously.
>
> I think Gordan nailed it: 1 rsync and one scrub will be a long way 
> from saturating anything, and are both blocking on I/O before coming 
> anywhere near saturating your cores.
>

I would agree, if the rates were minimally acceptable.  The same rsync, 
with ext3, raid1 and cbq, would do at least 10 times the performance 
within the same data and hardware.  We are talking about large 
sequential files.  4MB/s/disk is too slow even for an old pendrive!   :-)

> I would not try 20 rsyncs because then you'd be saturating your 
> network or even disk I/O (if you have lots of metadata). I'd use 20 
> instances fio, cp, dd or similar to locally generate your test I/O. 
> Use sequential I/O so you don't end up running out of IOPs and you you 
> should be saturating your cores with GZIP before long.
>
> Cheers,
> -- 
>    Durval.
> >
> > On Thu, Feb 11, 2016 at 5:14 PM, João Carlos Mendes Luís 
> <zfs-discuss at list.zfsonlinux.org 
> <mailto:zfs-discuss at list.zfsonlinux.org>> wrote:
> >>
> >> On 02/11/2016 01:19 PM, Durval Menezes wrote:
> >>>
> >>> Hello Jonny,
> >>>
> >>> On Thu, Feb 11, 2016 at 12:48 PM, João Carlos Mendes Luís 
> <zfs-discuss at list.zfsonlinux.org 
> <mailto:zfs-discuss at list.zfsonlinux.org>> wrote:
> >>>>
> >>>>    I am benchmarking ZoL for personal use, and got a case where I 
> think performance is limited by number of I/O threads (zd_rd_int, 
> z_wr_int, etc.), probably because I am using gzip compression.
> >>>
> >>>
> >>> Humrmmr... I've seen no such issues with either LZ4 or LZJB. I 
> would suggest you repeat your testing with LZ4.
> >>>
> >>>>
> >>>> The problem is that these threads are currently limited by number 
> of CPU cores, but I still have lots of idle CPU available. Even 
> scrubbing appears to be limited by this, drastically reducing overall 
> speed where block size is smaller (more compression achieved).
> >>>
> >>>
> >>> How many cores you have,
> >>
> >>
> >> 8 cores, 32GB RAM, AMD CPU
> >>
> >>> what benchmark did you run,
> >>
> >>
> >> zpool scrub
> >> zpool iostat -v 5
> >> iostat -dx 5
> >> top
> >> rsync
> >>
> >> For example, during scrub, while average request size is constantly 
> smaller than 128k, I/O is limited to 10MB/s, each z_rd_int thread is 
> using about 15% CPU, and scrub estimates finishing in 46 hours.  Near 
> the end, when average request size nears 128k, I/O reaches 180MB/s, 
> and z_rd_int CPU is negligible.  The whole scrub finishes within 20 hours.
> >>
> >> BTW: SATA disks, "home" setup.
> >>
> >>
> >>> and how many cores got 100% use?
> >>
> >>
> >> none!
> >>
> >> That's the problem.  I still have lots of CPU available, but scrub, 
> resilver, copy, etc. appear slower than it should be.
> >>
> >>
> >>>
> >>>>
> >>>>    AFAIK, it's not currently possible to increase the number of 
> these threads. Am I wrong? Already changed zio_taskq_batch_pct to 200, 
> but nothing changed.
> >>>
> >>>
> >>> I never had to change any parameters to saturate my systems with 
> either LZ4 or LZJB. But perhaps, as mine saturated with I/O,  your 
> case is different due to GZIP use (which I don't do exactly because it 
> uses too much CPU).
> >>
> >>
> >> It's the opposite.  It appears to be slow (due to compression?), 
> but I still have too much idle CPU available.
> >>
> >>>
> >>> Also, AFAIK ZFS doesn't use threads but full-blown kernel tasks to 
> do its thing. I could be wrong in that, but some fussing around /proc 
> here seems to indicate each of these tasks has exactly one thread (so 
> no multi-threading at all).
> >>>
> >>>>
> >>>>    Is there a reason to keep these limited?
> >>>
> >>>
> >>> To avoid starving the CPU not only for user process, but also for 
> other stuff going in the kernel at the same time?
> >>>
> >>>
> >>>>    Please, do not suggest to disable compression, as this is one 
> of the reasons I'm trying ZFS. Also, I'd be happy to help increase ZFS 
> performance for everybody use, not only datacenter folks.
> >>>
> >>>
> >>> I'd retest with LZ4; I don't think you have much sense of reaching 
> any kind of good performance with GZIP (in ZFS or otherwise) for any 
> kind of disk-intensive load.  If you do retest it, please let us know 
> the results.
> >>
> >>
> >> Last week I created a separate dataset with lz4 compression, for 
> performance reasons and some more testing.  It took about *6 hours* to 
> rsync 148G of data between datasets. Again, the same problem: slow to 
> read...
> >>
> >> I would expect gzip to be slow to write, never to read!
> >>
> >>
> >> Let me give you some real data from right now:
> >>
> >> - Killed most user apps, including browsers, etc.
> >> - Started some high speed reading: find /path -type f | xargs sum
> >> - The base dir is basically a vmware repo, with some virtual disks, 
> located in a lz4 compressed dataset:
> >>
> >> tank/vms  compressratio         1.59x -
> >> tank/vms  compression           lz4 local
> >> tank/vms  refcompressratio      1.59x -
> >>
> >> Latest output from zpool iostat -v 5 (the 4 way mirror is part of 
> the test, not a final config):
> >>
> >>                       capacity     operations bandwidth
> >> pool               alloc   free   read  write   read write
> >> -----------------  -----  -----  -----  -----  ----- -----
> >> tank               2.47T  1.16T    356      0 15.2M      0
> >>   mirror           2.47T  1.16T    356      0 15.2M      0
> >>     sdc                -      -     92      0 4.11M      0
> >>     sdd                -      -     91      0 4.06M      0
> >>     sde                -      -     86      0 3.88M      0
> >>     sdf                -      -     85      0 3.77M      0
> >>
> >>
> >> Latest output from iostat -dx 5:
> >>
> >> Device:         rrqm/s   wrqm/s     r/s     w/s rkB/s    wkB/s 
> avgrq-sz avgqu-sz   await r_await w_await  svctm %util
> >> sdc               0.00     0.00  108.12    0.00 5072.00     0.00    
> 93.82     0.04    0.38    0.38    0.00 0.37   4.05
> >> sdd               0.00     0.00  108.38    0.00 5038.50     0.00    
> 92.98     0.05    0.49    0.49    0.00 0.49   5.28
> >> sde               0.00     0.00   97.88    0.00 4572.50     0.00    
> 93.44     0.06    0.61    0.61    0.00 0.61   5.99
> >> sdf               0.00     0.00   95.62    0.00 4466.50     0.00    
> 93.42     0.05    0.55    0.55    0.00 0.55   5.26
> >>
> >> Latest output from top:
> >>
> >> Tasks: 492 total,   2 running, 490 sleeping,   0 stopped,   0 zombie
> >> %Cpu(s):  2.6 us,  8.7 sy,  0.0 ni, 78.7 id, 10.0 wa, 0.0 hi,  0.0 
> si,  0.0 st
> >> KiB Mem : 32914452 total,  8412620 free, 13802828 used, 10699004 
> buff/cache
> >> KiB Swap: 50331644 total, 49407140 free,   924504 used.  9627708 
> avail Mem
> >>
> >>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ 
> COMMAND
> >> 28646 root      20   0    8160   2116   1756 D  22.8 0.0   0:47.03 sum
> >>  1212 root       0 -20       0      0      0 S   7.9 0.0 238:35.75 
> z_rd_int_1
> >>  1214 root       0 -20       0      0      0 S   7.9 0.0 238:35.73 
> z_rd_int_3
> >>  1211 root       0 -20       0      0      0 S   7.6 0.0 238:41.44 
> z_rd_int_0
> >>  1216 root       0 -20       0      0      0 R   7.6 0.0 238:45.36 
> z_rd_int_5
> >>  1213 root       0 -20       0      0      0 S   7.3 0.0 238:51.33 
> z_rd_int_2
> >>  1217 root       0 -20       0      0      0 S   7.3 0.0 238:38.50 
> z_rd_int_6
> >>  1215 root       0 -20       0      0      0 S   6.9 0.0 238:36.28 
> z_rd_int_4
> >>  1218 root       0 -20       0      0      0 S   6.6 0.0 238:36.69 
> z_rd_int_7
> >>   641 root      20   0       0      0      0 S   2.0 0.0   0:45.44 
> dbu_evict
> >>   646 root      20   0       0      0      0 S   1.7 0.0   0:32.62 
> arc_user_evicts
> >>   645 root      20   0       0      0      0 S   1.3 0.0   2:26.67 
> arc_reclaim
> >>   628 root       0 -20       0      0      0 S   1.0 0.0   1:14.05 
> spl_kmem_cache
> >>
> >> Note that there's no limit in I/O, memory or CPU. There's no reason 
> to why the system is not performing better.
> >>
> >> I am copying this data to a new dataset, without compression, to 
> make these experiments again later.
> >>
> >>
> >> BTW:
> >>
> >> # cat /sys/module/zfs/version
> >> 0.6.5-149_g4b9ed69
> >> # cat /sys/module/spl/version
> >> 0.6.5-42_gd112232
> >> #
> >>
> >>
> >>
> >>
> >>>
> >>> Cheers, and good luck,
> >>> --
> >>>    Durval.
> >>>
> >>>
> >>>>
> >>>>
> >>>>     Thanks for any comment,
> >>>>
> >>>>                 Jonny
> >>>> _______________________________________________
> >>>> zfs-discuss mailing list
> >>>> zfs-discuss at list.zfsonlinux.org 
> <mailto:zfs-discuss at list.zfsonlinux.org>
> >>>> http://list.zfsonlinux.org/cgi-bin/mailman/listinfo/zfs-discuss
> >>>
> >>>
> >>
> >>
> >> _______________________________________________
> >> zfs-discuss mailing list
> >> zfs-discuss at list.zfsonlinux.org 
> <mailto:zfs-discuss at list.zfsonlinux.org>
> >> http://list.zfsonlinux.org/cgi-bin/mailman/listinfo/zfs-discuss
> >>
> >
> >
> > _______________________________________________
> > zfs-discuss mailing list
> > zfs-discuss at list.zfsonlinux.org <mailto:zfs-discuss at list.zfsonlinux.org>
> > http://list.zfsonlinux.org/cgi-bin/mailman/listinfo/zfs-discuss
> >
>
>
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at list.zfsonlinux.org
> http://list.zfsonlinux.org/cgi-bin/mailman/listinfo/zfs-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.zfsonlinux.org/pipermail/zfs-discuss/attachments/20160211/ae48d886/attachment-0001.html>


More information about the zfs-discuss mailing list