[zfs-discuss] extreme load average for ZFS-intensive tasks

Ben Kurtz bkurtz at ucsd.edu
Fri Apr 8 17:06:32 EDT 2016


Hi,

I'm having trouble with ZFS causing very high load levels (see a couple of example 'top' outputs below) on our new disk server.  Specifically, the system has 4 cores, and I'm seeing load averages as high as 45 and more when ZFS is working hard (seems to be worse when there are multiple independent operations going on, such as a user trying to access files at the same time as a pool scrub).  The result is of course that the server is effectively unresponsive because threads can't get CPU time.  I'm wondering if there's some configuration option that I need to set to make this behave better.

Some configuration details:
* Ubuntu 14.04 with Linux 3.19 kernel
* 8GB RAM
* no deduplication
* 18 6TB disks in two RAIDZ1 vdevs (9 disks each)
* Attached via HighPoint r750 HBA
* pool is around 50% full
* a month ago, (albeit with only one user actively using the system at that point), thing seemed to be working normally; scrub speeds ~500 MB/sec, load averages reasonable.

I asked about this on stackexchange, and got one answer suggesting that I should change the zfs_vdev_scrub_min_active and zfs_vdev_scrub_max_active properties, though I wasn't clear whether the person intended to suggest increasing or decreasing the values.  In any case, the specific values they suggested represented significant increases over the defaults I was already seeing.  I tested out a more moderate increase in the _max_active parameter, and so far it seems to have only locked up the server even worse during the scrub (though I did briefly see speeds around 500 MB/sec before it locked up).  Increasing these parameters seems counterintuitive to me, as they are supposed to be max concurrent operations, and it seems like I already have too many of those, but maybe I don't understand how this works.

Anyway, any help would be much appreciated. 

Thanks much!
--Ben K


top - 10:48:53 up 3 days, 12:27,  2 users,  load average: 17.24, 8.80, 5.52
Tasks: 572 total,  18 running, 554 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us, 83.4 sy,  0.0 ni,  3.9 id, 11.4 wa,  0.0 hi,  1.3 si,  0.0 st
KiB Mem:   8017104 total,  7163288 used,   853816 free,      232 buffers
KiB Swap: 97656240 total,    11860 used, 97644384 free.     5708 cached Mem

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                               
3926 root       0 -20       0      0      0 S 100.0  0.0   1:51.74 z_rd_iss                                                                                             
3920 root       0 -20       0      0      0 R 100.0  0.0   1:50.92 z_rd_iss                                                                                             
3921 root       0 -20       0      0      0 R 100.0  0.0   1:49.71 z_rd_iss                                                                                             
3925 root       0 -20       0      0      0 S 100.0  0.0   1:50.45 z_rd_iss                                                                                             
3923 root       0 -20       0      0      0 S 100.0  0.0   1:48.97 z_rd_iss                                                                                             
3919 root       0 -20       0      0      0 S 100.0  0.0   1:49.91 z_rd_iss                                                                                             
3922 root       0 -20       0      0      0 S 100.0  0.0   1:50.56 z_rd_iss                                                                                             
3924 root       0 -20       0      0      0 S 100.0  0.0   1:50.15 z_rd_iss                                                                                             
4429 root      20   0       0      0      0 R 100.0  0.0  52:24.07 txg_sync                                                                                             
4024 root       1 -19       0      0      0 S  41.9  0.0 154:39.17 z_wr_iss                                                                                             
4023 root       1 -19       0      0      0 S  41.3  0.0 154:39.59 z_wr_iss                                                                                             
4025 root       1 -19       0      0      0 S  41.3  0.0 155:02.86 z_wr_iss                                                                                             
4795 root      20   0       0      0      0 S  12.7  0.0 565:37.57 nfsd                                                                                                 
3987 root       0 -20       0      0      0 S   9.0  0.0   1:44.08 z_rd_int_5                                                                                           
3940 root       0 -20       0      0      0 R   8.6  0.0   1:43.94 z_rd_int_1                                                                                           
4003 root       0 -20       0      0      0 S   8.6  0.0   1:43.88 z_rd_int_6                                                                                           
3974 root       0 -20       0      0      0 S   8.5  0.0   1:44.14 z_rd_int_3                                                                                           
3939 root       0 -20       0      0      0 S   8.4  0.0   1:43.70 z_rd_int_1                                                                                           
3948 root       0 -20       0      0      0 S   8.4  0.0   1:43.33 z_rd_int_1                                                                                           
3966 root       0 -20       0      0      0 S   8.4  0.0   1:44.76 z_rd_int_3                                                                                           
3928 root       0 -20       0      0      0 S   8.3  0.0   1:44.04 z_rd_int_0                                                                                           
3999 root       0 -20       0      0      0 S   8.3  0.0   1:44.05 z_rd_int_6                                                                                           
3973 root       0 -20       0      0      0 S   8.2  0.0   1:44.14 z_rd_int_3                                                                                           
3980 root       0 -20       0      0      0 S   8.2  0.0   1:44.34 z_rd_int_4                                                                                           
3988 root       0 -20       0      0      0 S   8.2  0.0   1:43.68 z_rd_int_5                                                                                           
4022 root       0 -20       0      0      0 S   8.2  0.0   1:43.82 z_rd_int_7                                                                                           
3934 root       0 -20       0      0      0 S   8.1  0.0   1:44.03 z_rd_int_0                                                                                           
4016 root       0 -20       0      0      0 S   8.1  0.0   1:44.20 z_rd_int_7                                                                                           
3938 root       0 -20       0      0      0 S   8.0  0.0   1:43.72 z_rd_int_0                                                                                           
3941 root       0 -20       0      0      0 S   8.0  0.0   1:43.67 z_rd_int_1                                                                                           
3953 root       0 -20       0      0      0 S   8.0  0.0   1:43.72 z_rd_int_2                                                                                           
3969 root       0 -20       0      0      0 S   8.0  0.0   1:43.72 z_rd_int_3                                                                                           
3985 root       0 -20       0      0      0 R   8.0  0.0   1:43.84 z_rd_int_4                                                                                           
4004 root       0 -20       0      0      0 S   8.0  0.0   1:43.56 z_rd_int_6                                                                                           
4009 root       0 -20       0      0      0 S   8.0  0.0   1:44.02 z_rd_int_6                                                                                           
4010 root       0 -20       0      0      0 S   8.0  0.0   1:43.39 z_rd_int_6                                                                                           
4012 root       0 -20       0      0      0 S   8.0  0.0   1:43.78 z_rd_int_7                                                                                           
3931 root       0 -20       0      0      0 R   7.9  0.0   1:43.70 z_rd_int_0                                                                                           
3957 root       0 -20       0      0      0 S   7.9  0.0   1:43.98 z_rd_int_2                                                                                           
3995 root       0 -20       0      0      0 S   7.9  0.0   1:43.99 z_rd_int_5                                                                                           
4002 root       0 -20       0      0      0 S   7.9  0.0   1:44.20 z_rd_int_6                                

top - 11:03:45 up 3 days, 12:42,  2 users,  load average: 47.49, 38.33, 23.91
Tasks: 573 total,   5 running, 568 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us, 87.1 sy,  0.0 ni,  0.4 id, 11.2 wa,  0.0 hi,  1.2 si,  0.0 st
KiB Mem:   8017104 total,  7261788 used,   755316 free,      232 buffers
KiB Swap: 97656240 total,    11860 used, 97644384 free.     6668 cached Mem

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                               
3922 root       0 -20       0      0      0 S  73.2  0.0   5:56.51 z_rd_iss                                                                                             
3926 root       0 -20       0      0      0 S  72.9  0.0   5:57.70 z_rd_iss                                                                                             
3924 root       0 -20       0      0      0 R  72.8  0.0   5:56.17 z_rd_iss                                                                                             
3923 root       0 -20       0      0      0 R  72.8  0.0   5:53.78 z_rd_iss                                                                                             
3920 root       0 -20       0      0      0 S  72.4  0.0   5:55.24 z_rd_iss                                                                                             
3919 root       0 -20       0      0      0 S  72.4  0.0   5:52.71 z_rd_iss                                                                                             
3921 root       0 -20       0      0      0 S  72.0  0.0   5:52.40 z_rd_iss                                                                                             
3925 root       0 -20       0      0      0 R  72.0  0.0   5:52.54 z_rd_iss                                                                                             
4429 root      20   0       0      0      0 R  56.7  0.0  55:36.92 txg_sync                                                                                             
4023 root       1 -19       0      0      0 S   7.3  0.0 155:11.13 z_wr_iss                                                                                             
4024 root       1 -19       0      0      0 S   7.0  0.0 155:10.34 z_wr_iss                                                                                             
4025 root       1 -19       0      0      0 S   6.9  0.0 155:33.40 z_wr_iss                                                                                             
3963 root       0 -20       0      0      0 S   2.9  0.0   1:53.28 z_rd_int_3                                                                                           
3941 root       0 -20       0      0      0 S   2.9  0.0   1:52.88 z_rd_int_1                                                                                           
3947 root       0 -20       0      0      0 S   2.9  0.0   1:53.27 z_rd_int_1                                                                                           
3942 root       0 -20       0      0      0 S   2.9  0.0   1:53.07 z_rd_int_1                                                                                           
3929 root       0 -20       0      0      0 S   2.8  0.0   1:52.85 z_rd_int_0                                                                                           
3988 root       0 -20       0      0      0 S   2.8  0.0   1:52.92 z_rd_int_5                                                                                           
4003 root       0 -20       0      0      0 S   2.8  0.0   1:53.06 z_rd_int_6                                                                                           
3978 root       0 -20       0      0      0 S   2.8  0.0   1:52.42 z_rd_int_4                                                                                           
3992 root       0 -20       0      0      0 S   2.8  0.0   1:52.59 z_rd_int_5                                                                                           
3944 root       0 -20       0      0      0 S   2.8  0.0   1:53.25 z_rd_int_1                                                                                           
3952 root       0 -20       0      0      0 S   2.8  0.0   1:52.48 z_rd_int_2                                                                                           
3989 root       0 -20       0      0      0 S   2.8  0.0   1:52.88 z_rd_int_5                                                                                           



More information about the zfs-discuss mailing list