[zfs-discuss] Re: Bottlenecking on CPU with SSDs, Tuning, Scheduling

Niels de Carpentier zfs at decarpentier.com
Wed Sep 11 17:06:23 EDT 2013

> On Wednesday, August 21, 2013 7:02:07 AM UTC-7, Gordan Bobic wrote:
>> Setup:
>> 2x SSD (Intel 520 + Kingston HyperX)
>> mirrored (using ZFS)
>> 2 deduped zvols, dd-ing 64GB from one to the other (bs=1G count=64).
>> 12x1GB zram L2ARCs.

How much RAM? What is your ARC size?
>> dedup=on
>> compression=zle
>> sync=disabled
>> iostat -x consistently shows %util=100 during the process on the zvols,
>> but the %util for the raw underlying disks is considerably lower (varies
>> 10%-95%).
>> That implies there is a major CPU bottleneck, and sure enough I see not
>> far from 50% of CPU overall being used on the machine (12-core (2x6)
>> 3.6GHz
>> Xeon!).
>> The biggest CPU hogs are:
>> 20-30% txg_sync
>> 10-20% spl_kmem_cache
>> 10-20% arc_adapt
>> 5-20% z_wr_iss (lots of these and the next 3!)
>> 5-20% z_wr_int
>> 5-20% z_rd_iss
>> 5-20% z_rd_int

The z_rd/z_wr processes might just be waiting on disk, spl_kmem_cache
suggests a memory issue.

My guess would be that your ARC is too small, and the reads and writes
from the same pool trashes each others caches. So likely a lot of extra
reads are done to get the dedup tables into memory again. A 4k block size
requires a lot of memory for the dedup tables. The L2ARC likely not filled
fast enough to get the evicted arc data in.

I did a test with a mirred pair of crucial M4's (120GB) (with a normal zfs
fs though) and got reasonable write speeds with compression=lze and
dedup=on. (148MB/s, compared to 188MB/s as a simple mirror, which is close
to the max write speeds of these I think).


To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe at zfsonlinux.org.

More information about the zfs-discuss mailing list