[zfs-discuss] Question around expected performance

Andreas Dilger adilger at dilger.ca
Thu Jun 30 18:33:05 EDT 2011


On 2011-06-30, at 2:25 PM, Selim wrote:
> This has been posed in various forms before, but I am trying to figure
> out where to find (if any) there are bottlenecks.
> 
> Celeron M440 1.86 GHz with 3GB RAM
> 7 WD WD10EACS-00D6B0 drives connected to a Marvell 88SE6440 SATA
> controller backplane
> Ubuntu 11.04 Server, 32-bit mode
> Gigabit ethernet
> Kernel boot up parameter "vmalloc=512M"
> 
> I am trying to do rsync large files (DVD images) over the Gig-E port
> into the this box that I built.
> 
> I get no more than ~16-18 MB/sec of transfer speed with the rsync.  I
> have heard of much higher speeds and wanted to know if there are
> things I can do to improve the transfer speed.

I assume any new CPU has multiple cores, so it is entirely possible
that one of the processes (e.g. rsync) is consuming 100% of that CPU.

> Below are snips from iostat:
> 
> http://pastebin.com/raw.php?i=LqX4vvh6
> 
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>          64.00    0.00   28.00    0.00    0.00    8.00

If you run "top" and press "1" it will show the per-core usage.
If CPU is the bottleneck, the next step is running oprofile to
see where the CPU being burned.  Also very useful would be to
run blocktrace and seekwatcher to visualize the IO on the disks.

> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00    0.00    0.00   0.00   0.00
> sdb               0.00    26.00    0.00   61.00     0.00  2739.00
> 89.80     0.22    3.57    0.00    3.57   1.70  10.40
> sdc               0.00    23.50    0.00   59.00     0.00  2736.50
> 92.76     0.20    3.36    0.00    3.36   1.83  10.80
> sdd               0.00    23.00    0.00   58.00     0.00  2737.75
> 94.41     0.25    4.24    0.00    4.24   2.03  11.80
> sde               0.00    26.00    0.00   55.50     0.00  2738.25
> 98.68     0.13    2.34    0.00    2.34   1.80  10.00
> sdh               0.00    27.50    0.00   56.50     0.00  2738.25
> 96.93     0.18    3.26    0.00    3.26   1.77  10.00
> sdf               0.00    26.50    0.00   56.50     0.00  2736.75
> 96.88     0.17    3.04    0.00    3.04   1.81  10.20
> sdg               0.00    27.50    0.00   59.50     0.00  2736.75
> 91.99     0.16    2.72    0.00    2.72   1.68  10.00
> 
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>          53.73    0.00   24.38    0.00    0.00   21.89
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00    0.00    0.00   0.00   0.00
> sdb               0.00    27.36    0.00   80.60     0.00  2795.27
> 69.36     0.29    3.65    0.00    3.65   1.63  13.13
> sdc               0.00    26.37    0.00   87.06     0.00  2793.78
> 64.18     0.23    2.67    0.00    2.67   1.37  11.94
> sdd               0.00    27.86    0.00   86.07     0.00  2795.52
> 64.96     0.25    2.89    0.00    2.89   1.43  12.34
> sde               0.00    27.36    0.00   86.07     0.00  2799.50
> 65.05     0.31    3.61    0.00    3.61   1.55  13.33
> sdh               0.00    28.86    0.00   80.60     0.00  2795.02
> 69.36     0.26    3.19    0.00    3.19   1.43  11.54
> sdf               0.00    30.35    0.00   88.06     0.00  2798.01
> 63.55     0.27    3.10    0.00    3.10   1.40  12.34
> sdg               0.00    29.35    0.50   87.56    31.84  2796.77
> 64.24     0.28    3.16   36.00    2.98   1.60  14.13
> 
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>          64.32    0.00   23.62    0.00    0.00   12.06
> 
> Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> sda               0.00     0.00    0.00    0.00     0.00     0.00
> 0.00     0.00    0.00    0.00    0.00   0.00   0.00
> sdb               0.00    24.62    0.00   61.31     0.00  2752.26
> 89.79     0.25    4.03    0.00    4.03   2.03  12.46
> sdc               0.00    24.12    0.00   68.84     0.00  2755.53
> 80.05     0.28    4.09    0.00    4.09   1.78  12.26
> sdd               0.00    24.12    0.00   67.84     0.00  2752.76
> 81.16     0.28    4.09    0.00    4.09   1.75  11.86
> sde               0.00    24.62    0.00   63.32     0.00  2749.75
> 86.86     0.36    5.68    0.00    5.68   2.13  13.47
> sdh               0.00    25.13    0.00   53.27     0.00  2749.75
> 103.25     0.23    4.23    0.00    4.23   2.30  12.26
> sdf               0.00    24.62    0.00   57.29     0.00  2751.01
> 96.04     0.27    4.77    0.00    4.77   2.28  13.07
> sdg               0.00    28.14    0.00   55.78     0.00  2753.02
> 98.71     0.39    6.92    0.00    6.92   2.74  15.28


Cheers, Andreas







More information about the zfs-discuss mailing list