Zfs slow down

sverre stoltenberg sverrest at opera.com
Mon May 23 05:18:16 EDT 2011


I have a equallogic iscsi share with zfs. Around 250 volumes, with about 
30-40 snapshots each, of the last 20 days, and the 1st each month. The 
host has 4 physical cpus, DELL R900, and 107GB memory, and is only used 
for rsync backups. The pool was created on opensolaris. It is running 
the latest git zfs/spl

root at dumbo:~# zpool list
NAME         SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
filebackup  14.6T  8.64T  5.98T    59%  1.49x  ONLINE  -

root at dumbo:~# zpool status
   pool: filebackup
  state: ONLINE
  scan: scrub canceled on Mon May  9 20:53:28 2011
config:

         NAME        STATE     READ WRITE CKSUM
         filebackup  ONLINE       0     0     0
           sdf       ONLINE       0     0     0
         logs
           sdd       ONLINE       0     0     0
         cache
           c4t2d0s0  UNAVAIL      0     0     0
           c4t3d0s0  UNAVAIL      0     0     0
           c4t4d0s0  UNAVAIL      0     0     0
           sdb       ONLINE       0     0     0
           sdc       ONLINE       0     0     0

root at dumbo:~/bin# zdb -D filebackup
DDT-sha256-zap-duplicate: 28326430 entries, size 318 on disk, 158 in core
DDT-sha256-zap-unique: 99674157 entries, size 408 on disk, 287 in core

dedup = 1.49, compress = 1.77, copies = 1.00, dedup * compress / copies 
= 2.64

Initialy it used dedup and compression, but recently turned both off to 
see if it could get the speed up, but only the load went down from 50-60 
to 15, the IO is horrible, between 0 and 2M read and write, but have 
spikes where it can read up to 6MB. When idle I can dd an iso image to 
/dev/zero with 75M/s.

Right now it is running 9 rsync processes:

root at dumbo:~/bin# iostat -mx 3
Linux 2.6.38.1 (dumbo)  05/23/2011      _x86_64_        (16 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
            0.04    0.00   10.84    0.02    0.00   89.10

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.41    23.82    0.49    1.09     0.01     0.03 44.00 
     0.01    8.61   0.83   0.13
sdb               0.01     1.23    0.00    1.23     0.00     0.20 326.74 
     0.01    4.92   3.23   0.40
sdd               0.01     0.00    0.01    0.00     0.00     0.00 29.94 
     0.00    1.03   0.62   0.00
sdc               0.01     1.24    0.00    1.24     0.00     0.20 326.93 
     0.01    4.97   3.31   0.41
dm-0              0.00     0.00    0.19   24.09     0.00     0.02 2.23 
    0.88   36.41   0.02   0.05
dm-1              0.00     0.00    0.00    0.00     0.00     0.00 8.00 
    0.00    3.32   0.40   0.00
dm-2              0.00     0.00    0.64    0.07     0.00     0.00 12.09 
     0.00    2.75   0.56   0.04
dm-3              0.00     0.00    0.04    0.47     0.00     0.00 10.96 
     0.00    7.46   0.81   0.04
dm-4              0.00     0.00    0.02    0.28     0.00     0.00 2.02 
    0.00    1.13   0.17   0.01
sdf               0.25     0.39   17.55    9.63     0.50     0.17 50.81 
     0.25    9.37   2.72   7.40

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
            0.02    0.00   19.64    0.00    0.00   80.34

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
sdb               0.00     1.33    0.00    0.67     0.00     0.06 176.00 
     0.00    4.00   4.00   0.27
sdd               0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
sdc               0.00     1.33    0.00    0.67     0.00     0.09 284.00 
     0.00    4.00   4.00   0.27
dm-0              0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
dm-3              0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
dm-4              0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
sdf               0.00     3.67    0.33   32.00     0.02     1.00 64.74 
     0.08    2.60   0.70   2.27

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
            0.00    0.00   17.16    0.00    0.00   82.84

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
sdb               0.00     1.00    0.00    0.67     0.00     0.01 39.00 
     0.00    4.00   4.00   0.27
sdd               0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
sdc               0.00     2.00    0.00    1.33     0.00     0.24 375.25 
     0.01    4.00   2.00   0.27
dm-0              0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
dm-3              0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
dm-4              0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
sdf               0.00     0.33    1.00   25.67     0.06     0.20 20.25 
     0.06    2.15   0.40   1.07

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
            0.03    0.00   13.19    0.00    0.00   86.78

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
sdb               0.00     0.33    0.00    0.33     0.00     0.08 512.00 
     0.00    4.00   4.00   0.13
sdd               0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
sdc               0.00     4.00    0.00    3.33     0.00     0.56 346.90 
     0.01    2.40   1.60   0.53
dm-0              0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
dm-3              0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
dm-4              0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
sdf               0.00     5.33    1.67  106.67     0.10     1.26 25.74 
     0.26    2.39   0.85   9.20

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
            0.00    0.00   18.70    0.00    0.00   81.30

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s 
avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
sdb               0.00     1.67    0.00    1.67     0.00     0.22 270.20 
     0.01    3.20   2.40   0.40
sdd               0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
sdc               0.00     0.00    0.00    0.67     0.00     0.05 140.00 
     0.00    2.00   2.00   0.13
dm-0              0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
dm-1              0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
dm-3              0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
dm-4              0.00     0.00    0.00    0.00     0.00     0.00 0.00 
    0.00    0.00   0.00   0.00
sdf               0.00     2.67    0.67   29.00     0.04     1.09 78.07 
     0.09    2.97   0.72   2.13

 From time to time iostat shows 0.00 all over.

Can it be a fragmentation problem? The san went full a year ago, but 
since then I have added more space, added SSD disks, more memory, and 
deleted a load of snapshots, and the speed was comfortable for a period. 
The slowness also appears if I replace the os disks with an illumos 
install, so I don't think it is linux specific.

I have a zfs pool on a lower specified machine, from the same san, and 
after turning off dedup and compression, it can write around 45-60M/s.

-- 
/S.



More information about the zfs-discuss mailing list