[zfs-discuss] CPU for ZFS compression

Andreas Dilger adilger at dilger.ca
Thu Feb 1 23:12:07 EST 2018

On Feb 1, 2018, at 7:18 PM, Edward Ned Harvey (zfsonlinux) via zfs-discuss <zfs-discuss at list.zfsonlinux.org> wrote:
>> From: zfs-discuss [mailto:zfs-discuss-bounces at list.zfsonlinux.org] On
>> Behalf Of Bogdan via zfs-discuss
>> I'm going to order entry level storage  server for virtual machines
>> images
>> backups:
>> • Storage: 12x3.5" 7200RPM
>> • Network 2x10Gbit
>> • Organization: RAIDZ2 with compression.
>> Maximum load: 2 input streams, ~300-350Mbyte/sec each.
>> What the CPU do I need to keep to fully utilize HDD IOPS? My typical
>> rate is
>> 1.75 for gzip-1 compression.
> If you use gzip (at any level), no matter what, your CPU will slow down your system. If you use lz4 (ideally) or lzjb (if for any reason you can't or don't want to use lz4) then no matter what, your CPU will not slow down your system, and for moderately compressible data, compression with lz4 or lzjb will actually *accelerate* your system. I say "no matter what" based on the assumption that you're considering Xeon processors and entry level storage, with relatively small pool. Some of these things will be different under different assumptions. It takes some *real* slow processors and real fast IO before the processor can become the bottleneck with lz4. There is a reason why lz4 is the default. Although you can sometimes get better compression with gzip, usually maximum compression is not what people care about.

It is worthwhile to mention that if gzip data compression is important to you,
you can get an Intel "Quick Assist Technology" (QAT) PCI hardware accelerator
for deflate de/compression and SSL network:


The ZFS QAT support was included in the ZoL 0.7.0 release, but does not yet
accelerate the ZFS crypto feature in 0.8 (though there is AES hardware support
in Xeon CPUs).

There is also optionally a faster QAT adapter in the C625-C628 chipsets:


Cheers, Andreas

(full disclosure, I work for Intel and worked with the QAT developers to add
support for QAT to ZFS)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 873 bytes
Desc: Message signed with OpenPGP
URL: <http://list.zfsonlinux.org/pipermail/zfs-discuss/attachments/20180201/74d3c8d6/attachment.sig>

More information about the zfs-discuss mailing list