[zfs-discuss] Slow reads with encrypted dataset

Luke Olson luke.s.olson at gmail.com
Tue Nov 27 23:30:55 EST 2018


This is a red flag to me: CPU is an Intel Xeon-D 1541. That's a low power
CPU. Try it on another system if you can, one with a more powerful CPU.

Luke


On Tue, Nov 27, 2018, 8:57 AM Reid Kelley via zfs-discuss <
zfs-discuss at list.zfsonlinux.org wrote:

> I am seeing extremely slow reads with an encrypted dataset. Read speeds on
> the encrypted dataset are approx 1/3 or less of the read speeds on the
> unencrypted dataset, write speeds are virtually identical. Initially this
> was discovered on an HDD pool with SSD as the new metadata/special devices.
> Testing with many pool configuration and disk types, this read speed issue
> is happening across the board. For simplicity sake this is now being tested
> on a small SSD only pool without special devices. In this test setup,
> unencrypted reads are ~ 400MB/s and encrypted reads are ~ 115MB/s. Bonnie++
> local synthetic benchmark was also run showing similar results in below.
> Has anyone else experienced/solved this issue?
> ### Describe how to reproduce the problem
> Create two datasets with same parameters except for name, mountpoint and
> encryption properties. ```zfs get all ssd/XXX``` output below is from the
> encrypted dataset. CPU is an Intel Xeon-D 1541 with 32GB ECC memory, disks
> are 4x Samsung old enterprise SSDs in a zpool of two mirrors, connected via
> 10 gigabit network to MacOS 10.14.1 using both SMB and AFP. Test data
> source is a ~20GB directory made of nine large video and zip files. Copy
> data source over SMB or AFP to the zfs dataset, then read it back to local
> machine.
>
> ##### Distro info
> Distribution Name | Ubuntu
> Distribution Version | 18.04.1 LTS
> Linux Kernel | 4.15.0-39-generic
> Architecture | x86_64
> ZFS Version | 0.8.0-rc2
> SPL Version | 0.8.0-rc2
>
> ##### zfs get-all on the encrypted dataset
> ```
> NAME          PROPERTY              VALUE                  SOURCE
> ssd/test-enc  type                  filesystem             -
> ssd/test-enc  creation              Mon Nov 26 18:43 2018  -
> ssd/test-enc  used                  21.0G                  -
> ssd/test-enc  available             818G                   -
> ssd/test-enc  referenced            21.0G                  -
> ssd/test-enc  compressratio         1.00x                  -
> ssd/test-enc  mounted               yes                    -
> ssd/test-enc  quota                 none                   default
> ssd/test-enc  reservation           none                   default
> ssd/test-enc  recordsize            128K                   default
> ssd/test-enc  mountpoint            /ssd/test-enc          default
> ssd/test-enc  sharenfs              off                    default
> ssd/test-enc  checksum              on                     default
> ssd/test-enc  compression           off                    local
> ssd/test-enc  atime                 off                    inherited from
> ssd
> ssd/test-enc  devices               on                     default
> ssd/test-enc  exec                  on                     default
> ssd/test-enc  setuid                on                     default
> ssd/test-enc  readonly              off                    default
> ssd/test-enc  zoned                 off                    default
> ssd/test-enc  snapdir               hidden                 default
> ssd/test-enc  aclinherit            restricted             default
> ssd/test-enc  createtxg             66660                  -
> ssd/test-enc  canmount              on                     default
> ssd/test-enc  xattr                 sa                     local
> ssd/test-enc  copies                1                      default
> ssd/test-enc  version               5                      -
> ssd/test-enc  utf8only              off                    -
> ssd/test-enc  normalization         none                   -
> ssd/test-enc  casesensitivity       sensitive              -
> ssd/test-enc  vscan                 off                    default
> ssd/test-enc  nbmand                off                    default
> ssd/test-enc  sharesmb              off                    default
> ssd/test-enc  refquota              none                   default
> ssd/test-enc  refreservation        none                   default
> ssd/test-enc  guid                  1881475390342226       -
> ssd/test-enc  primarycache          all                    default
> ssd/test-enc  secondarycache        all                    default
> ssd/test-enc  usedbysnapshots       0B                     -
> ssd/test-enc  usedbydataset         21.0G                  -
> ssd/test-enc  usedbychildren        0B                     -
> ssd/test-enc  usedbyrefreservation  0B                     -
> ssd/test-enc  logbias               latency                default
> ssd/test-enc  objsetid              1.02K                  -
> ssd/test-enc  dedup                 off                    default
> ssd/test-enc  mlslabel              none                   default
> ssd/test-enc  sync                  standard               default
> ssd/test-enc  dnodesize             auto                   local
> ssd/test-enc  refcompressratio      1.00x                  -
> ssd/test-enc  written               21.0G                  -
> ssd/test-enc  logicalused           21.0G                  -
> ssd/test-enc  logicalreferenced     21.0G                  -
> ssd/test-enc  volmode               default                default
> ssd/test-enc  filesystem_limit      none                   default
> ssd/test-enc  snapshot_limit        none                   default
> ssd/test-enc  filesystem_count      none                   default
> ssd/test-enc  snapshot_count        none                   default
> ssd/test-enc  snapdev               hidden                 default
> ssd/test-enc  acltype               off                    default
> ssd/test-enc  context               none                   default
> ssd/test-enc  fscontext             none                   default
> ssd/test-enc  defcontext            none                   default
> ssd/test-enc  rootcontext           none                   default
> ssd/test-enc  relatime              off                    default
> ssd/test-enc  redundant_metadata    all                    default
> ssd/test-enc  overlay               off                    default
> ssd/test-enc  encryption            aes-256-ccm            -
> ssd/test-enc  keylocation           prompt                 local
> ssd/test-enc  keyformat             passphrase             -
> ssd/test-enc  pbkdf2iters           342K                   -
> ssd/test-enc  encryptionroot        ssd/test-enc           -
> ssd/test-enc  keystatus             available              -
> ssd/test-enc  special_small_blocks  0                      default
> ```
> The properties on the unencrypted dataset are the same, just without
> encryption turned on.
>
> ##### SMB/AFP test
> Below is an example of ```iotop``` command output showing the read
> difference over AFP test runs.
> ```
> 25382 be/4 user1  106.22 M/s    0.00 B/s  0.00 % 88.06 % --- Encrypted
> 25382 be/4 user1  389.76 M/s    0.00 B/s  0.00 % 72.98 % --- No Encrypt
> ```
> ##### Bonnie++ test
> Test Parameters: ```bonnie++ -d /ssd/test-enc -r 32768 -u user1```
> Encrypted:
> ```
> Version  1.97       ------Sequential Output------ --Sequential Input-
> --Random-
> Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
> /sec %CP
> nas1            64G   184  99 711234  97 41019  18   462  98 60090  11
> 2165  90
> Latency             56495us     104ms   30115us   43211us    6288us
>  14473us
> Version  1.97       ------Sequential Create------ --------Random
> Create--------
> nas1                -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
> /sec %CP
>                  16 +++++ +++ +++++ +++ 16292  50 +++++ +++ +++++ +++
> +++++ +++
> Latency              2859us    1307us     479ms    2279us     255us
> 66us
> ```
> No Encrypt:
> ```
> Version  1.97       ------Sequential Output------ --Sequential Input-
> --Random-
> Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
> /sec %CP
> nas1            64G   169  99 676246  95 105841  38   465  99 152732  27
> 7864 446
> Latency             59580us     101ms     100ms   37700us    3000us
> 7597us
> Version  1.97       ------Sequential Create------ --------Random
> Create--------
> nas1                -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
> /sec %CP
>                  16 +++++ +++ +++++ +++ 18053  53 +++++ +++ +++++ +++
> +++++ +++
> Latency              2806us    1302us     421ms    2246us      29us
> 68us
> ```
> Dataset without encryption shows higher throughput/iops and much lower
> latency.
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at list.zfsonlinux.org
> http://list.zfsonlinux.org/cgi-bin/mailman/listinfo/zfs-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.zfsonlinux.org/pipermail/zfs-discuss/attachments/20181127/475ca03f/attachment.html>


More information about the zfs-discuss mailing list