[zfs-discuss] Increase number of I/O workers?

Durval Menezes durval.menezes at gmail.com
Thu Feb 11 13:23:32 EST 2016


Hello Jonny,

Sorry for the delay in getting back to you, was fixing a refrigerator here
(no joke).

On Feb 11, 2016 3:24 PM, "Gordan Bobic via zfs-discuss" <
zfs-discuss at list.zfsonlinux.org> wrote:
>
> Both rsync and scrub are single-threaded. They will never generate enough
load to saturate the CPU. Try with 20 rsync instances running
simultaneously.

I think Gordan nailed it: 1 rsync and one scrub will be a long way from
saturating anything, and are both blocking on I/O before coming anywhere
near saturating your cores.

I would not try 20 rsyncs because then you'd be saturating your network or
even disk I/O (if you have lots of metadata). I'd use 20 instances fio, cp,
dd or similar to locally generate your test I/O. Use sequential I/O so you
don't end up running out of IOPs and you you should be saturating your
cores with GZIP before long.

Cheers,
-- 
   Durval.
>
> On Thu, Feb 11, 2016 at 5:14 PM, João Carlos Mendes Luís <
zfs-discuss at list.zfsonlinux.org> wrote:
>>
>> On 02/11/2016 01:19 PM, Durval Menezes wrote:
>>>
>>> Hello Jonny,
>>>
>>> On Thu, Feb 11, 2016 at 12:48 PM, João Carlos Mendes Luís <
zfs-discuss at list.zfsonlinux.org> wrote:
>>>>
>>>>    I am benchmarking ZoL for personal use, and got a case where I
think performance is limited by number of I/O threads (zd_rd_int, z_wr_int,
etc.), probably because I am using gzip compression.
>>>
>>>
>>> Humrmmr... I've seen no such issues with either LZ4 or LZJB. I would
suggest you repeat your testing with LZ4.
>>>
>>>>
>>>> The problem is that these threads are currently limited by number of
CPU cores, but I still have lots of idle CPU available. Even scrubbing
appears to be limited by this, drastically reducing overall speed where
block size is smaller (more compression achieved).
>>>
>>>
>>> How many cores you have,
>>
>>
>> 8 cores, 32GB RAM, AMD CPU
>>
>>> what benchmark did you run,
>>
>>
>> zpool scrub
>> zpool iostat -v 5
>> iostat -dx 5
>> top
>> rsync
>>
>> For example, during scrub, while average request size is constantly
smaller than 128k, I/O is limited to 10MB/s, each z_rd_int thread is using
about 15% CPU, and scrub estimates finishing in 46 hours.  Near the end,
when average request size nears 128k, I/O reaches 180MB/s, and z_rd_int CPU
is negligible.  The whole scrub finishes within 20 hours.
>>
>> BTW: SATA disks, "home" setup.
>>
>>
>>> and how many cores got 100% use?
>>
>>
>> none!
>>
>> That's the problem.  I still have lots of CPU available, but scrub,
resilver, copy, etc. appear slower than it should be.
>>
>>
>>>
>>>>
>>>>    AFAIK, it's not currently possible to increase the number of these
threads. Am I wrong? Already changed zio_taskq_batch_pct to 200, but
nothing changed.
>>>
>>>
>>> I never had to change any parameters to saturate my systems with either
LZ4 or LZJB. But perhaps, as mine saturated with I/O,  your case is
different due to GZIP use (which I don't do exactly because it uses too
much CPU).
>>
>>
>> It's the opposite.  It appears to be slow (due to compression?), but I
still have too much idle CPU available.
>>
>>>
>>> Also, AFAIK ZFS doesn't use threads but full-blown kernel tasks to do
its thing. I could be wrong in that, but some fussing around /proc here
seems to indicate each of these tasks has exactly one thread (so no
multi-threading at all).
>>>
>>>>
>>>>    Is there a reason to keep these limited?
>>>
>>>
>>> To avoid starving the CPU not only for user process, but also for other
stuff going in the kernel at the same time?
>>>
>>>
>>>>    Please, do not suggest to disable compression, as this is one of
the reasons I'm trying ZFS. Also, I'd be happy to help increase ZFS
performance for everybody use, not only datacenter folks.
>>>
>>>
>>> I'd retest with LZ4; I don't think you have much sense of reaching any
kind of good performance with GZIP (in ZFS or otherwise) for any kind of
disk-intensive load.  If you do retest it, please let us know the results.
>>
>>
>> Last week I created a separate dataset with lz4 compression, for
performance reasons and some more testing.  It took about *6 hours* to
rsync 148G of data between datasets.  Again, the same problem: slow to
read...
>>
>> I would expect gzip to be slow to write, never to read!
>>
>>
>> Let me give you some real data from right now:
>>
>> - Killed most user apps, including browsers, etc.
>> - Started some high speed reading: find /path -type f | xargs sum
>> - The base dir is basically a vmware repo, with some virtual disks,
located in a lz4 compressed dataset:
>>
>> tank/vms  compressratio         1.59x                  -
>> tank/vms  compression           lz4                    local
>> tank/vms  refcompressratio      1.59x                  -
>>
>> Latest output from zpool iostat -v 5 (the 4 way mirror is part of the
test, not a final config):
>>
>>                       capacity     operations    bandwidth
>> pool               alloc   free   read  write   read  write
>> -----------------  -----  -----  -----  -----  -----  -----
>> tank               2.47T  1.16T    356      0  15.2M      0
>>   mirror           2.47T  1.16T    356      0  15.2M      0
>>     sdc                -      -     92      0  4.11M      0
>>     sdd                -      -     91      0  4.06M      0
>>     sde                -      -     86      0  3.88M      0
>>     sdf                -      -     85      0  3.77M      0
>>
>>
>> Latest output from iostat -dx 5:
>>
>> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
>> sdc               0.00     0.00  108.12    0.00  5072.00     0.00
93.82     0.04    0.38    0.38    0.00   0.37   4.05
>> sdd               0.00     0.00  108.38    0.00  5038.50     0.00
92.98     0.05    0.49    0.49    0.00   0.49   5.28
>> sde               0.00     0.00   97.88    0.00  4572.50     0.00
93.44     0.06    0.61    0.61    0.00   0.61   5.99
>> sdf               0.00     0.00   95.62    0.00  4466.50     0.00
93.42     0.05    0.55    0.55    0.00   0.55   5.26
>>
>> Latest output from top:
>>
>> Tasks: 492 total,   2 running, 490 sleeping,   0 stopped,   0 zombie
>> %Cpu(s):  2.6 us,  8.7 sy,  0.0 ni, 78.7 id, 10.0 wa,  0.0 hi,  0.0 si,
0.0 st
>> KiB Mem : 32914452 total,  8412620 free, 13802828 used, 10699004
buff/cache
>> KiB Swap: 50331644 total, 49407140 free,   924504 used.  9627708 avail
Mem
>>
>>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+
COMMAND
>> 28646 root      20   0    8160   2116   1756 D  22.8  0.0   0:47.03 sum
>>  1212 root       0 -20       0      0      0 S   7.9  0.0 238:35.75
z_rd_int_1
>>  1214 root       0 -20       0      0      0 S   7.9  0.0 238:35.73
z_rd_int_3
>>  1211 root       0 -20       0      0      0 S   7.6  0.0 238:41.44
z_rd_int_0
>>  1216 root       0 -20       0      0      0 R   7.6  0.0 238:45.36
z_rd_int_5
>>  1213 root       0 -20       0      0      0 S   7.3  0.0 238:51.33
z_rd_int_2
>>  1217 root       0 -20       0      0      0 S   7.3  0.0 238:38.50
z_rd_int_6
>>  1215 root       0 -20       0      0      0 S   6.9  0.0 238:36.28
z_rd_int_4
>>  1218 root       0 -20       0      0      0 S   6.6  0.0 238:36.69
z_rd_int_7
>>   641 root      20   0       0      0      0 S   2.0  0.0   0:45.44
dbu_evict
>>   646 root      20   0       0      0      0 S   1.7  0.0   0:32.62
arc_user_evicts
>>   645 root      20   0       0      0      0 S   1.3  0.0   2:26.67
arc_reclaim
>>   628 root       0 -20       0      0      0 S   1.0  0.0   1:14.05
spl_kmem_cache
>>
>> Note that there's no limit in I/O, memory or CPU.  There's no reason to
why the system is not performing better.
>>
>> I am copying this data to a new dataset, without compression, to make
these experiments again later.
>>
>>
>> BTW:
>>
>> # cat /sys/module/zfs/version
>> 0.6.5-149_g4b9ed69
>> # cat /sys/module/spl/version
>> 0.6.5-42_gd112232
>> #
>>
>>
>>
>>
>>>
>>> Cheers, and good luck,
>>> --
>>>    Durval.
>>>
>>>
>>>>
>>>>
>>>>     Thanks for any comment,
>>>>
>>>>                 Jonny
>>>> _______________________________________________
>>>> zfs-discuss mailing list
>>>> zfs-discuss at list.zfsonlinux.org
>>>> http://list.zfsonlinux.org/cgi-bin/mailman/listinfo/zfs-discuss
>>>
>>>
>>
>>
>> _______________________________________________
>> zfs-discuss mailing list
>> zfs-discuss at list.zfsonlinux.org
>> http://list.zfsonlinux.org/cgi-bin/mailman/listinfo/zfs-discuss
>>
>
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at list.zfsonlinux.org
> http://list.zfsonlinux.org/cgi-bin/mailman/listinfo/zfs-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.zfsonlinux.org/pipermail/zfs-discuss/attachments/20160211/7a1e64d9/attachment.html>


More information about the zfs-discuss mailing list