[zfs-discuss] ARC_RECLAIM99% CPU UTILIZATION

Frank Xenal xn9733 at gmail.com
Fri Jan 8 00:37:50 EST 2016


Hello,

Recently upgraded from 0.6.3.1 to  zfs-0.6.5.3-1.el6.x86_64

Note: Reason for upgrading was that I was getting very high CPU usage with
"arc_adapt" and the NFS mounts would be non-responsive and would have no
choice but to reboot the system.

Since the upgrade, now I'm seeing  "arc_reclaim" at a constant 99% cpu
utilization after very high NFS traffic -- mainly during nightly backups.
However, this time the nfs mounts are still responsive. I also see lots of
"arc_prune" processing running.

System information:

OS: Centos 6.7
       Linux 2.6.32-573.12.1.el6.x86_64 #1 SMP Tue Dec 15 21:19:08 UTC 2015
x86_64 x86_64 x86_64 GNU/Linux
ZFS pools:   4 very active pools, each with 10 disks in raidz2
RAM: 256G
Note: this server is strictly an NFS server


Looking for some help with determining what is causing this. I'm concerned
that the system NFS mounts will eventually become unresponsive and will
have to reboot.

Thanks!


Following are some zfs config and debug information:

SLAB information:
==============
--------------------- cache
-------------------------------------------------------  ----- slab ------
 ---- object -----  --- emergency ---
name                                    flags      size     alloc slabsize
 objsize  total alloc   max  total alloc   max  dlock alloc   max
spl_vn_cache                          0x00020     98304     48048     8192
     104     12    11    11    504   462   462      0     0     0
spl_vn_file_cache                     0x00020     65536     32928     8192
     112      8     7     7    336   294   294      0     0     0
spl_zlib_workspace_cache              0x00240         0         0  2144960
  268072      0     0     0      0     0     0      0     0     0
ddt_cache                             0x00040   2390784   2187328   199232
   24856     12    11    11     96    88    88      0     0     0
zio_buf_20480                         0x00042  27095040  19660800   200704
   20480    135   135   159   1080   960  1272      0     0     0
zio_data_buf_20480                    0x00042  45760512  25231360   200704
   20480    228   228   366   1824  1232  2928      0     0     0
zio_buf_24576                         0x00042  26849280  21037056   233472
   24576    115   115   142    920   856  1136      0     0     0
zio_data_buf_24576                    0x00042  47861760  29097984   233472
   24576    205   205   296   1640  1184  2368      0     0     0
zio_buf_28672                         0x00042  35676160  28901376   266240
   28672    134   134   134   1072  1008  1072      0     0     0
zio_data_buf_28672                    0x00042  43663360  31653888   266240
   28672    164   164   166   1312  1104  1328      0     0     0
zio_buf_32768                         0x00042  34086912  23330816   299008
   32768    114   114   134    912   712  1072      0     0     0
zio_data_buf_32768                    0x00042  53821440  37486592   299008
   32768    180   180   199   1440  1144  1592      0     0     0
zio_buf_40960                         0x00042  36454400  32768000   364544
   40960    100   100   145    800   800  1160      0     0     0
zio_data_buf_40960                    0x00042  80564224  53411840   364544
   40960    221   221   287   1768  1304  2296      0     0     0
zio_buf_49152                         0x00042  52469760  46399488   430080
   49152    122   122   133    976   944  1064      0     0     0


zfs.conf
=========
options zfs zfs_arc_min=10737418240
options zfs zfs_arc_max=68719476736


arcstat:
=========
6 1 0x01 91 4368 12850802983 183440709950683
name                            type data
hits                            4    137887521
misses                          4    61047442
demand_data_hits                4    4931870
demand_data_misses              4    11851169
demand_metadata_hits            4    117889449
demand_metadata_misses          4    39705706
prefetch_data_hits              4    1086655
prefetch_data_misses            4    1184105
prefetch_metadata_hits          4    13979547
prefetch_metadata_misses        4    8306462
mru_hits                        4    71392731
mru_ghost_hits                  4    6220387
mfu_hits                        4    51428588
mfu_ghost_hits                  4    2599822
deleted                         4    57251112
mutex_miss                      4    276401
evict_skip                      4    11019879240
evict_not_enough                4    320113180
evict_l2_cached                 4    0
evict_l2_eligible               4    444038314496
evict_l2_ineligible             4    121226022912
evict_l2_skip                   4    0
hash_elements                   4    1319696
hash_elements_max               4    17601910
hash_collisions                 4    15187078
hash_chains                     4    25670
hash_chain_max                  4    7
p                               4    64424509440
c                               4    68719476736
c_min                           4    10737418240
c_max                           4    68719476736
size                            4    68966084976
hdr_size                        4    538457296
data_size                       4    918016
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.zfsonlinux.org/pipermail/zfs-discuss/attachments/20160108/b47157a5/attachment-0001.html>


More information about the zfs-discuss mailing list