[zfs-discuss] ARC_RECLAIM99% CPU UTILIZATION

Gordan Bobic gordan.bobic at gmail.com
Fri Jan 8 05:33:59 EST 2016


Interesting, I have seen arc_reclaim get stuck at 100% CPU usage and
effectively permanently hang the machine, but only on a 32-bit ARM. I just
assumed it was something that only happens on 32-bit deployments (which are
known to still be problematic). I haven't seen it happen on x86-64.


On Fri, Jan 8, 2016 at 5:37 AM, Frank Xenal via zfs-discuss <
zfs-discuss at list.zfsonlinux.org> wrote:

> Hello,
>
> Recently upgraded from 0.6.3.1 to  zfs-0.6.5.3-1.el6.x86_64
>
> Note: Reason for upgrading was that I was getting very high CPU usage with
> "arc_adapt" and the NFS mounts would be non-responsive and would have no
> choice but to reboot the system.
>
> Since the upgrade, now I'm seeing  "arc_reclaim" at a constant 99% cpu
> utilization after very high NFS traffic -- mainly during nightly backups.
> However, this time the nfs mounts are still responsive. I also see lots of
> "arc_prune" processing running.
>
> System information:
>
> OS: Centos 6.7
>        Linux 2.6.32-573.12.1.el6.x86_64 #1 SMP Tue Dec 15 21:19:08 UTC
> 2015 x86_64 x86_64 x86_64 GNU/Linux
> ZFS pools:   4 very active pools, each with 10 disks in raidz2
> RAM: 256G
> Note: this server is strictly an NFS server
>
>
> Looking for some help with determining what is causing this. I'm concerned
> that the system NFS mounts will eventually become unresponsive and will
> have to reboot.
>
> Thanks!
>
>
> Following are some zfs config and debug information:
>
> SLAB information:
> ==============
> --------------------- cache
> -------------------------------------------------------  ----- slab ------
>  ---- object -----  --- emergency ---
> name                                    flags      size     alloc slabsize
>  objsize  total alloc   max  total alloc   max  dlock alloc   max
> spl_vn_cache                          0x00020     98304     48048     8192
>      104     12    11    11    504   462   462      0     0     0
> spl_vn_file_cache                     0x00020     65536     32928     8192
>      112      8     7     7    336   294   294      0     0     0
> spl_zlib_workspace_cache              0x00240         0         0  2144960
>   268072      0     0     0      0     0     0      0     0     0
> ddt_cache                             0x00040   2390784   2187328   199232
>    24856     12    11    11     96    88    88      0     0     0
> zio_buf_20480                         0x00042  27095040  19660800   200704
>    20480    135   135   159   1080   960  1272      0     0     0
> zio_data_buf_20480                    0x00042  45760512  25231360   200704
>    20480    228   228   366   1824  1232  2928      0     0     0
> zio_buf_24576                         0x00042  26849280  21037056   233472
>    24576    115   115   142    920   856  1136      0     0     0
> zio_data_buf_24576                    0x00042  47861760  29097984   233472
>    24576    205   205   296   1640  1184  2368      0     0     0
> zio_buf_28672                         0x00042  35676160  28901376   266240
>    28672    134   134   134   1072  1008  1072      0     0     0
> zio_data_buf_28672                    0x00042  43663360  31653888   266240
>    28672    164   164   166   1312  1104  1328      0     0     0
> zio_buf_32768                         0x00042  34086912  23330816   299008
>    32768    114   114   134    912   712  1072      0     0     0
> zio_data_buf_32768                    0x00042  53821440  37486592   299008
>    32768    180   180   199   1440  1144  1592      0     0     0
> zio_buf_40960                         0x00042  36454400  32768000   364544
>    40960    100   100   145    800   800  1160      0     0     0
> zio_data_buf_40960                    0x00042  80564224  53411840   364544
>    40960    221   221   287   1768  1304  2296      0     0     0
> zio_buf_49152                         0x00042  52469760  46399488   430080
>    49152    122   122   133    976   944  1064      0     0     0
>
>
> zfs.conf
> =========
> options zfs zfs_arc_min=10737418240
> options zfs zfs_arc_max=68719476736
>
>
> arcstat:
> =========
> 6 1 0x01 91 4368 12850802983 183440709950683
> name                            type data
> hits                            4    137887521
> misses                          4    61047442
> demand_data_hits                4    4931870
> demand_data_misses              4    11851169
> demand_metadata_hits            4    117889449
> demand_metadata_misses          4    39705706
> prefetch_data_hits              4    1086655
> prefetch_data_misses            4    1184105
> prefetch_metadata_hits          4    13979547
> prefetch_metadata_misses        4    8306462
> mru_hits                        4    71392731
> mru_ghost_hits                  4    6220387
> mfu_hits                        4    51428588
> mfu_ghost_hits                  4    2599822
> deleted                         4    57251112
> mutex_miss                      4    276401
> evict_skip                      4    11019879240
> evict_not_enough                4    320113180
> evict_l2_cached                 4    0
> evict_l2_eligible               4    444038314496
> evict_l2_ineligible             4    121226022912
> evict_l2_skip                   4    0
> hash_elements                   4    1319696
> hash_elements_max               4    17601910
> hash_collisions                 4    15187078
> hash_chains                     4    25670
> hash_chain_max                  4    7
> p                               4    64424509440
> c                               4    68719476736
> c_min                           4    10737418240
> c_max                           4    68719476736
> size                            4    68966084976
> hdr_size                        4    538457296
> data_size                       4    918016
>
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at list.zfsonlinux.org
> http://list.zfsonlinux.org/cgi-bin/mailman/listinfo/zfs-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.zfsonlinux.org/pipermail/zfs-discuss/attachments/20160108/52ce22dc/attachment-0001.html>


More information about the zfs-discuss mailing list