[zfs-discuss] Re: write performance on VMware vsphere

dipe dipeit at gmail.com
Sat Jul 9 21:39:40 EDT 2011


I also tried these 2 options but it did not make a significant
difference

 zpool create -o ashift=9
 zpool create -o ashift=12


On Jul 9, 6:28 pm, dipe <dip... at gmail.com> wrote:
> Hi,
>
> I just checked out the latest stuff from github (rc5) and build it on
> a fresh install of openSUSE 11.4 x64 (Kernel 2.6.37) on our VMware
> vsphere 4.0 environment and am having some performance issues:
>
> installed these dependencies:
>
> zypper -n in kernel-source git zlib-devel libuuid-devel libblkid-devel
> libselinux-devel parted lsscsi e2fsprogs-devel
> zypper -n in --type pattern devel_rpm_build
>
> built spl and zfs, installed the rpms and created a zpool
>
> zpool create -f tank raidz pci-0000:00:10.0-scsi-0:0:1:0
> pci-0000:00:10.0-scsi-0:0:2:0 pci-0000:00:10.0-scsi-0:0:3:0
> pci-0000:00:10.0-scsi-0:0:4:0
>
> ox5:~ # zpool status tank
>   pool: tank
>  state: ONLINE
>  scan: none requested
> config:
>
>         NAME                               STATE     READ WRITE CKSUM
>         tank                               ONLINE       0     0     0
>           raidz1-0                         ONLINE       0     0     0
>             pci-0000:00:10.0-scsi-0:0:1:0  ONLINE       0     0     0
>             pci-0000:00:10.0-scsi-0:0:2:0  ONLINE       0     0     0
>             pci-0000:00:10.0-scsi-0:0:3:0  ONLINE       0     0     0
>             pci-0000:00:10.0-scsi-0:0:4:0  ONLINE       0     0     0
>
> errors: No known data errors
>
> ****** ran dd on the tank  ***************
>
> ox5:~ # dd if=/dev/zero of=/tank/dumpfile bs=4096 count=100000
> 100000+0 records in
> 100000+0 records out
> 409600000 bytes (410 MB) copied, 30.0652 s, 13.6 MB/s
> ox5:~ #
> ox5:~ # dd if=/dev/zero of=/tank/dumpfile bs=4096 count=100000
> 100000+0 records in
> 100000+0 records out
> 409600000 bytes (410 MB) copied, 16.5147 s, 24.8 MB/s
> ox5:~ #
> ox5:~ # dd if=/dev/zero of=/tank/dumpfile bs=4096 count=100000
> 100000+0 records in
> 100000+0 records out
> 409600000 bytes (410 MB) copied, 18.9506 s, 21.6 MB/s
> ox5:~ #
>
> ****** ran dd on tmp  ***************
>
> ox5:~ # dd if=/dev/zero of=/tmp/dumpfile bs=4096 count=100000
> 100000+0 records in
> 100000+0 records out
> 409600000 bytes (410 MB) copied, 5.23293 s, 78.3 MB/s
> ox5:~ #
> ox5:~ # dd if=/dev/zero of=/tmp/dumpfile bs=4096 count=100000
> 100000+0 records in
> 100000+0 records out
> 409600000 bytes (410 MB) copied, 6.67496 s, 61.4 MB/s
> ox5:~ #
> ox5:~ # dd if=/dev/zero of=/tmp/dumpfile bs=4096 count=100000
> 100000+0 records in
> 100000+0 records out
> 409600000 bytes (410 MB) copied, 5.39355 s, 75.9 MB/s
> ox5:~ #
> ox5:~ # dd if=/dev/zero of=/tmp/dumpfile bs=4096 count=100000
> 100000+0 records in
> 100000+0 records out
> 409600000 bytes (410 MB) copied, 6.63144 s, 61.8 MB/s
> ox5:~ #
> ox5:~ # dd if=/dev/zero of=/tmp/dumpfile bs=4096 count=100000
> 100000+0 records in
> 100000+0 records out
> 409600000 bytes (410 MB) copied, 7.27147 s, 56.3 MB/s
> ox5:~ #
> ox5:~ # dd if=/dev/zero of=/tmp/dumpfile bs=4096 count=100000
> 100000+0 records in
> 100000+0 records out
> 409600000 bytes (410 MB) copied, 6.69949 s, 61.1 MB/s
> ox5:~ #
>
> ****** and again on the zfs tank  ***************
>
> ox5:~ # dd if=/dev/zero of=/tank/dumpfile bs=4096 count=100000
> 100000+0 records in
> 100000+0 records out
> 409600000 bytes (410 MB) copied, 28.5647 s, 14.3 MB/s
> ox5:~ #
>
> Sure, there are some performance penalties because I use
> raidz .....but should the penalty be that high ?
>
> here is some additional output from dmesg:
>
>   4.535190] Fusion MPT base driver 3.04.17
> [    4.535196] Copyright (c) 1999-2008 LSI Corporation
> [    4.540186] Fusion MPT SPI Host driver 3.04.17
> [    4.540452] mptspi 0000:00:10.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
>
> [    4.541875] mptbase: ioc0: Initiating bringup
> [    4.612425] ioc0: LSI53C1030 B0: Capabilities={Initiator}
> [    4.773368] scsi2 : ioc0: LSI53C1030 B0, FwRev=01032920h, Ports=1,
> MaxQ=128, IRQ=17
> [    4.884717] scsi 2:0:0:0: Direct-Access     VMware   Virtual
> disk     1.0  PQ: 0 ANSI: 2
> [    4.884745] scsi target2:0:0: Beginning Domain Validation
> [    4.885368] scsi target2:0:0: Domain Validation skipping write
> tests
> [    4.885370] scsi target2:0:0: Ending Domain Validation
> [    4.885411] scsi target2:0:0: FAST-40 WIDE SCSI 80.0 MB/s ST (25
> ns, offset 127)
> [    4.886866] scsi 2:0:1:0: Direct-Access     VMware   Virtual
> disk     1.0  PQ: 0 ANSI: 2
> [    4.886890] scsi target2:0:1: Beginning Domain Validation
> [    4.887544] scsi target2:0:1: Domain Validation skipping write
> tests
> [    4.887560] scsi target2:0:1: Ending Domain Validation
> [    4.887604] scsi target2:0:1: FAST-40 WIDE SCSI 80.0 MB/s ST (25
> ns, offset 127)
> [    4.888145] scsi 2:0:2:0: Direct-Access     VMware   Virtual
> disk     1.0  PQ: 0 ANSI: 2
> [    4.888152] scsi target2:0:2: Beginning Domain Validation
> [    4.888788] scsi target2:0:2: Domain Validation skipping write
> tests
> [    4.888791] scsi target2:0:2: Ending Domain Validation
> [    4.888832] scsi target2:0:2: FAST-40 WIDE SCSI 80.0 MB/s ST (25
> ns, offset 127)
> [    4.889230] scsi 2:0:3:0: Direct-Access     VMware   Virtual
> disk     1.0  PQ: 0 ANSI: 2
> [    4.889235] scsi target2:0:3: Beginning Domain Validation
> [    4.889846] scsi target2:0:3: Domain Validation skipping write
> tests
> [    4.889848] scsi target2:0:3: Ending Domain Validation
> [    4.889893] scsi target2:0:3: FAST-40 WIDE SCSI 80.0 MB/s ST (25
> ns, offset 127)
> [    4.890305] scsi 2:0:4:0: Direct-Access     VMware   Virtual
> disk     1.0  PQ: 0 ANSI: 2
> [    4.890310] scsi target2:0:4: Beginning Domain Validation
> [    4.890926] scsi target2:0:4: Domain Validation skipping write
> tests
> [    4.890928] scsi target2:0:4: Ending Domain Validation
> [    4.890971] scsi target2:0:4: FAST-40 WIDE SCSI 80.0 MB/s ST (25
> ns, offset 127)
> [    4.891356] scsi 2:0:5:0: Direct-Access     VMware   Virtual
> disk     1.0  PQ: 0 ANSI: 2
> [    4.891362] scsi target2:0:5: Beginning Domain Validation
> [    4.891970] scsi target2:0:5: Domain Validation skipping write
> tests
> [    4.891972] scsi target2:0:5: Ending Domain Validation
> [    4.892066] scsi target2:0:5: FAST-40 WIDE SCSI 80.0 MB/s ST (25
> ns, offset 127)
> [    4.892812] sd 2:0:1:0: [sdb] 16777216 512-byte logical blocks:
> (8.58 GB/8.00 GiB)
> [    4.892956] sd 2:0:0:0: [sda] 209715200 512-byte logical blocks:
> (107 GB/100 GiB)
> [    4.893191] sd 2:0:1:0: [sdb] Test WP failed, assume Write Enabled
> [    4.893275] sd 2:0:1:0: [sdb] Cache data unavailable
> [    4.893278] sd 2:0:1:0: [sdb] Assuming drive cache: write through
> [    4.893295] sd 2:0:0:0: [sda] Test WP failed, assume Write Enabled
> [    4.893332] sd 2:0:0:0: [sda] Cache data unavailable
> [    4.893337] sd 2:0:0:0: [sda] Assuming drive cache: write through
> [    4.893881] sd 2:0:2:0: [sdc] 16777216 512-byte logical blocks:
> (8.58 GB/8.00 GiB)
> [    4.894090] sd 2:0:0:0: [sda] Test WP failed, assume Write Enabled
> [    4.894134] sd 2:0:2:0: [sdc] Test WP failed, assume Write Enabled
> [    4.894172] sd 2:0:0:0: [sda] Cache data unavailable
> [    4.894174] sd 2:0:0:0: [sda] Assuming drive cache: write through
> [    4.894195] sd 2:0:2:0: [sdc] Cache data unavailable
> [    4.894197] sd 2:0:2:0: [sdc] Assuming drive cache: write through
> [    4.894432] sd 2:0:2:0: [sdc] Test WP failed, assume Write Enabled
> [    4.894463] sd 2:0:2:0: [sdc] Cache data unavailable
> [    4.894465] sd 2:0:2:0: [sdc] Assuming drive cache: write through
> [    4.894942] sd 2:0:1:0: [sdb] Test WP failed, assume Write Enabled
> [    4.894973] sd 2:0:1:0: [sdb] Cache data unavailable
> [    4.894975] sd 2:0:1:0: [sdb] Assuming drive cache: write through
> [    4.895660] sd 2:0:5:0: [sdf] 16777216 512-byte logical blocks:
> (8.58 GB/8.00 GiB)
> [    4.895748] sd 2:0:4:0: [sde] 16777216 512-byte logical blocks:
> (8.58 GB/8.00 GiB)
> [    4.895996] sd 2:0:5:0: [sdf] Test WP failed, assume Write Enabled
> [    4.896077] sd 2:0:4:0: [sde] Test WP failed, assume Write Enabled
> [    4.896328] sd 2:0:4:0: [sde] Cache data unavailable
> [    4.896331] sd 2:0:4:0: [sde] Assuming drive cache: write through
> [    4.896643] sd 2:0:3:0: [sdd] 16777216 512-byte logical blocks:
> (8.58 GB/8.00 GiB)
> [    4.896764] sd 2:0:4:0: [sde] Test WP failed, assume Write Enabled
> [    4.896822] sd 2:0:4:0: [sde] Cache data unavailable
> [    4.896824] sd 2:0:4:0: [sde] Assuming drive cache: write through
> [    4.896837] sd 2:0:3:0: [sdd] Test WP failed, assume Write Enabled
> [    4.896866] sd 2:0:3:0: [sdd] Cache data unavailable
> [    4.896867] sd 2:0:3:0: [sdd] Assuming drive cache: write through
> [    4.897125] sd 2:0:5:0: [sdf] Cache data unavailable
> [    4.897127] sd 2:0:5:0: [sdf] Assuming drive cache: write through
> [    4.897245] sd 2:0:3:0: [sdd] Test WP failed, assume Write Enabled
> [    4.897297] sd 2:0:3:0: [sdd] Cache data unavailable
> [    4.897299] sd 2:0:3:0: [sdd] Assuming drive cache: write through
> [    4.897672] sd 2:0:5:0: [sdf] Test WP failed, assume Write Enabled
> [    4.897704] sd 2:0:5:0: [sdf] Cache data unavailable
> [    4.897705] sd 2:0:5:0: [sdf] Assuming drive cache: write through
> [    4.901441]  sdf: unknown partition table
> [    4.901741] sd 2:0:5:0: [sdf] Test WP failed, assume Write Enabled
> [    4.901770] sd 2:0:5:0: [sdf] Cache data unavailable
> [    4.901772] sd 2:0:5:0: [sdf] Assuming drive cache: write through
> [    4.901864] sd 2:0:5:0: [sdf] Attached SCSI disk
> [    4.910506]  sda: sda1 sda2 sda3
> [    4.911260] sd 2:0:0:0: [sda] Test WP failed, assume Write Enabled
> [    4.911298] sd 2:0:0:0: [sda] Cache data unavailable
> [    4.911300] sd 2:0:0:0: [sda] Assuming drive cache: write through
> [    4.911448] sd 2:0:0:0: [sda] Attached SCSI disk
> [    4.930460]  sdc: sdc1 sdc9
> [    4.931078] sd 2:0:2:0: [sdc] Test WP failed, assume Write Enabled
> [    4.931108] sd 2:0:2:0: [sdc] Cache data unavailable
> [    4.931110] sd 2:0:2:0: [sdc] Assuming drive cache: write through
> [    4.931244] sd 2:0:2:0: [sdc] Attached SCSI disk
> [    4.931322]  sdb: sdb1 sdb9
> [    4.931715] sd 2:0:1:0: [sdb] Test WP failed, assume Write Enabled
> [    4.931742] sd 2:0:1:0: [sdb] Cache data unavailable
> [    4.931744] sd 2:0:1:0: [sdb] Assuming drive cache: write through
> [    4.931824] sd 2:0:1:0: [sdb] Attached SCSI disk
> [    4.934119]  sde: sde1 sde9
> [    4.934685] sd 2:0:4:0: [sde] Test WP failed, assume Write Enabled
> [    4.934715] sd 2:0:4:0: [sde] Cache data unavailable
> [    4.934716] sd 2:0:4:0: [sde] Assuming drive cache: write through
> [    4.934823] sd 2:0:4:0: [sde] Attached SCSI disk
> [    4.950626]  sdd: sdd1 sdd9
> [    4.951219] sd 2:0:3:0: [sdd] Test WP failed, assume Write Enabled
> [    4.951248] sd 2:0:3:0: [sdd] Cache data unavailable
> [    4.951250] sd 2:0:3:0: [sdd] Assuming drive cache: write through
> [    4.951400] sd 2:0:3:0: [sdd] Attached SCSI disk
> [    4.974172] ACPI: acpi_idle registered with cpuidle
> [    5.002034] udev[83]: starting version 166
> [    6.088717] PM: Marking nosave pages: 000000000009f000 -
> 0000000000100000
> [    6.088728] PM: Marking nosave pages: 000000002fef0000 -
> 000000002ff00000
> [    6.088735] PM: Basic memory bitmaps created
> [    6.092215] PM: Basic memory bitmaps freed
> [    6.229951] PM: Starting manual resume from disk
> [    6.229955] PM: Hibernation image partition 8:1 present
> [    6.229956] PM: Looking for hibernation image.
> [    6.230675] PM: Image not found (code -22)
> [    6.230677] PM: Hibernation image not present or could not be
> loaded.
> [    6.395003] EXT4-fs (sda2): mounted filesystem with ordered data
> mode. Opts: acl,user_xattr
> [    6.614834] EXT4-fs (sda2): re-mounted. Opts: acl,user_xattr
> [    9.405914] preloadtrace: systemtap: 1.4/0.149, base:
> ffffffffa0113000, memory: 43data/40text/26ctx/13net/392alloc kb,
> probes: 44
> [   10.259273] udev[405]: starting version 166
> [   10.769078] [433]: VMCI: shared components initialized.
> [   10.769158] Probing for vmci/PCI.
> [   10.769288] vmci 0000:00:07.7: PCI INT A -> GSI 16 (level, low) ->
> IRQ 16
> [   10.769330] Found vmci/PCI at 0x1080, irq 16.
> [   10.769377] VMCI: using capabilities 0x4.
> [   10.769437] [433]: VMCI: Host capability check: PASSED.
> [   10.769522] Registered vmci device.
> [   10.769592] [433]: VMCI: Using guest personality
> [   10.769595] [433]: VMCI: host components initialized.
> [   10.769708] [433]: VMCI: Module registered
> (name=vmci,major=10,minor=59).
> [   10.769710] [433]: VMCI: Using host personality
> [   10.769711] [433]: VMCI: Module (name=vmci) is initialized
> [   10.892764] ACPI: AC Adapter [ACAD] (on-line)
> [   11.006353] input: Sleep Button as /devices/LNXSYSTM:00/device:00/
> PNP0C0E:00/input/input2
> [   11.006455] ACPI: Sleep Button [SLPB]
> [   11.006520] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/
> input/input3
> [   11.006550] ACPI: Power Button [PWRF]
> [   11.049817] ACPI: resource piix4_smbus [io  0x1040-0x1047]
> conflicts with ACPI region SMB_ [io  0x1040-0x104b 64bit pref]
> [   11.049823] ACPI: If an ACPI driver is available for this device,
> you should use it instead of the native driver
> [   11.115822] Fusion MPT misc device (ioctl) driver 3.04.17
> [   11.116325] mptctl: Registered with Fusion MPT base driver
> [   11.116328] mptctl: /dev/mptctl @ (major,minor=10,220)
> [   11.195147] Floppy drive(s): fd0 is 1.44M
> [   11.213232] FDC 0 is a post-1991 82077
> [   11.217516] input: PC Speaker as /devices/platform/pcspkr/input/
> input4
> [   11.227436] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
> [   11.265954] shpchp: Standard Hot Plug PCI Controller Driver
> version: 0.4
> [   11.312628] parport_pc 00:08: reported by Plug and Play ACPI
> [   11.312735] parport0: PC-style at 0x378, irq 7 [PCSPP,TRISTATE]
> [   11.474291] scsi 1:0:0:0: Attached scsi generic sg0 type 5
> [   11.474351] sd 2:0:0:0: Attached scsi generic sg1 type 0
> [   11.474433] sd 2:0:1:0: Attached scsi generic sg2 type 0
> [   11.474491] sd 2:0:2:0: Attached scsi generic sg3 type 0
> [   11.474561] sd 2:0:3:0: Attached scsi generic sg4 type 0
> [   11.474640] sd 2:0:4:0: Attached scsi generic sg5 type 0
> [   11.474710] sd 2:0:5:0: Attached scsi generic sg6 type 0
> [   11.589624] sr0: scsi3-mmc drive: 1x/1x xa/form2 cdda tray
> [   11.589644] cdrom: Uniform CD-ROM driver Revision: 3.20
> [   11.590449] sr 1:0:0:0: Attached scsi CD-ROM sr0
> [   11.668479] ppdev: user-space parallel port driver
> [   11.737002] e1000: Intel(R) PRO/1000 Network Driver - version
> 7.3.21-k8-NAPI
> [   11.737005] e1000: Copyright (c) 1999-2006 Intel Corporation.
> [   11.737319] e1000 0000:02:00.0: PCI INT A -> GSI 18 (level, low) ->
> IRQ 18
> [   11.739335] e1000 0000:02:00.0: setting latency timer to 64
> [   12.147597] e1000 0000:02:00.0: eth0: (PCI:66MHz:32-bit)
> 00:50:56:9c:34:5a
> [   12.147611] e1000 0000:02:00.0: eth0: Intel(R) PRO/1000 Network
> Connection
> [   12.784517] Adding 2103292k swap on /dev/sda1.  Priority:-1 extents:
> 1 across:2103292k
> [   13.658956] device-mapper: uevent: version 1.0.3
> [   13.659643] device-mapper: ioctl: 4.18.0-ioctl (2010-06-29)
> initialised: dm-de... at redhat.com
> [   14.044671] Btrfs loaded
> [   14.048057] device fsid 8d4cc104e90eaa0c-f06bb9541de01f86 devid 1
> transid 70 /dev/sda3
> [   15.903905] type=1400 audit(1310258862.979:2): apparmor="STATUS"
> operation="profile_load" name="/bin/ping" pid=901
> comm="apparmor_parser"
> [   15.989041] type=1400 audit(1310258863.067:3): apparmor="STATUS"
> operation="profile_load" name="/sbin/klogd" pid=906
> comm="apparmor_parser"
> [   16.105961] type=1400 audit(1310258863.183:4): apparmor="STATUS"
> operation="profile_load" name="/sbin/syslog-ng" pid=910
> comm="apparmor_parser"
> [   16.213272] type=1400 audit(1310258863.291:5): apparmor="STATUS"
> operation="profile_load" name="/sbin/syslogd" pid=914
> comm="apparmor_parser"
> [   16.690809] type=1400 audit(1310258863.767:6): apparmor="STATUS"
> operation="profile_load" name="/usr/lib/apache2/mpm-prefork/apache2"
> pid=919 comm="apparmor_parser"
> [   16.690929] type=1400 audit(1310258863.767:7): apparmor="STATUS"
> operation="profile_load" name="/usr/lib/apache2/mpm-prefork/apache2//
> DEFAULT_URI" pid=919 comm="apparmor_parser"
> [   16.691037] type=1400 audit(1310258863.767:8): apparmor="STATUS"
> operation="profile_load" name="/usr/lib/apache2/mpm-prefork/apache2//
> HANDLING_UNTRUSTED_INPUT" pid=919 comm="apparmor_parser"
> [   16.691204] type=1400 audit(1310258863.767:9): apparmor="STATUS"
> operation="profile_load" name="/usr/lib/apache2/mpm-prefork/apache2//
> phpsysinfo" pid=919 comm="apparmor_parser"
> [   16.816951] type=1400 audit(1310258863.895:10): apparmor="STATUS"
> operation="profile_load" name="/usr/lib/dovecot/deliver" pid=923
> comm="apparmor_parser"
> [   16.961797] type=1400 audit(1310258864.039:11): apparmor="STATUS"
> operation="profile_load" name="/usr/lib/dovecot/dovecot-auth" pid=927
> comm="apparmor_parser"
> [   20.382850] SPL: Loaded module v0.6.0, using hostid 0x6b8ce058
> [   20.390811] microcode: CPU0 sig=0x6f1, pf=0x1, revision=0xb3
> [   20.434098] zunicode: module license 'CDDL' taints kernel.
> [   20.434104] Disabling lock debugging due to kernel taint
> [   20.456776] microcode: CPU1 sig=0x6f1, pf=0x1, revision=0xb3
> [   20.464778] microcode: Microcode Update Driver: v2.00
> <tig... at aivazian.fsnet.co.uk>, Peter Oruba
> [   21.759529] ZFS: Loaded module v0.6.0, ZFS pool version 28, ZFS
> filesystem version 5
> [   27.521181] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow
> Control: None
> [   27.535982] ADDRCONF(NETDEV_UP): eth0: link is not ready
> [   27.536966] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [   38.240148] eth0: no IPv6 routers present
> [   43.268129] BIOS EDD facility v0.16 2004-Jun-25, 6 devices found



More information about the zfs-discuss mailing list