[zfs-discuss] High performance SAN with ZoL

Andreas Bauer Andreas.Bauer at voltage.de
Sat Jan 2 10:35:40 EST 2016


Hello there,

I posted this summer here with the same subject, that is building a SAN
that caches most all of the day-to-day activities on flash while still
providing the big storage for data that is accessed once-in-a-moon.

Hardware:
Xeon E5v3 4-core 3 Ghz
128 GB DDR4
5x2 mirrored 2TB SATA Disks
4x SATA3 SSDs
10GbE

After discussion here, I set these parameters:

options zfs l2arc_noprefetch=0
options zfs l2arc_write_max=83886080
options zfs l2arc_write_boost=83886080

This solved one issue, namely that the L2ARC now gets filled to capacity.

After further googling, I set these parameters:

options zfs zfs_vdev_sync_write_min_active=64
options zfs zfs_vdev_sync_write_max_active=128
options zfs zfs_vdev_sync_read_min_active=64
options zfs zfs_vdev_sync_read_max_active=128
options zfs zfs_vdev_async_read_min_active=64
options zfs zfs_vdev_async_read_max_active=128
options zfs zfs_top_maxinflight=160

The system was running in production for a few months, and I always saw
the same thing: most of the reads are served from the spinning disks,
rather than from the SSD, even though most all of the data should be in
the L2ARC.

Performance is just not there as one would expect. With 4 SATA3 SSDs for
cache and a read-heavy workload, I would expect ZFS to easily saturate
the 10GbE even with small 8K and 16K random IO workloads, mostly from
either the ARC or the L2ARC.

Performance is actually way worse than a single local SSD with Ext4 :-(

To test it with a defined workload, over the holidays I destroyed the
pool and recreated the pool with an rsync from the backup server.

As expected, the L2ARC was filled just fine and has about 90% of the
data in it (one subvolume of 400G is configured with secondarycache=none).

To test it, I run a read-only workload (sequential) from the local
machine on the ZFS. This is the result:

                    capacity     operations    bandwidth
pool             alloc   free   read  write   read  write
----------------------  -----  -----  -----  -----  -----
tank      [...]  1,96T  7,10T    600      0  75,1M      0
  mirror  [...]   402G  1,42T    122      0  15,4M      0
    ata-WD[...]      -      -     63      0  7,99M      0
    ata-WD[...]      -      -     58      0  7,37M      0
  mirror  [...]   402G  1,42T    119      0  15,0M      0
    ata-WD[...]      -      -     64      0  8,12M      0
    ata-WD[...]      -      -     54      0  6,87M      0
  mirror  [...]   402G  1,42T    116      0  14,6M      0
    ata-WD[...]      -      -     66      0  8,37M      0
    ata-WD[...]      -      -     49      0  6,24M      0
  mirror  [...]   402G  1,42T    117      0  14,7M      0
    ata-WD[...]      -      -     50      0  6,37M      0
    ata-WD[...]      -      -     66      0  8,37M      0
  mirror  [...]   402G  1,42T    122      0  15,4M      0
    ata-WD[...]      -      -     76      0  9,62M      0
    ata-WD[...]      -      -     45      0  5,74M      0
cache     [...]      -      -      -      -      -      -
  ata-Sams[...]   346G  3,60G      0      0      0      0
  ata-Sams[...]   346G  3,58G      0      0      0      0
  ata-Sams[...]   346G  3,64G      0      0      0      0
  ata-Sams[...]   346G  3,61G      0      0      0      0

:-(

No usage of the L2ARC at all. In production, I see at least some usage.

What is going wrong here? The L2ARC has almost all of the data, but gets
no use. Instead, the data is taken from the hard disks.

ZFS 0.6.5.2-2 on Debian 8.2 with 4.3.3-2 kernel from testing. Same issue
was before with 3.16 Jessie Kernel and 4.2 Backports Kernel.

Thanks,

Andreas


More information about the zfs-discuss mailing list