[zfs-discuss] ARC_RECLAIM99% CPU UTILIZATION

Frank Xenal xn9733 at gmail.com
Fri Jan 8 07:44:51 EST 2016


update:  Finally, the system's nfs mounts became unresponsive and was
forced to begin a reboot. However, in the middle of preparing to reboot,
the system rebooted automatically.

@Gordan Bobic,  in your case did you have to ultimately reboot? Also, what
steps have you taken to fix the issue, if any?


Thanks!

On Fri, Jan 8, 2016 at 5:33 AM, Gordan Bobic <gordan.bobic at gmail.com> wrote:

> Interesting, I have seen arc_reclaim get stuck at 100% CPU usage and
> effectively permanently hang the machine, but only on a 32-bit ARM. I just
> assumed it was something that only happens on 32-bit deployments (which are
> known to still be problematic). I haven't seen it happen on x86-64.
>
>
> On Fri, Jan 8, 2016 at 5:37 AM, Frank Xenal via zfs-discuss <
> zfs-discuss at list.zfsonlinux.org> wrote:
>
>> Hello,
>>
>> Recently upgraded from 0.6.3.1 to  zfs-0.6.5.3-1.el6.x86_64
>>
>> Note: Reason for upgrading was that I was getting very high CPU usage
>> with "arc_adapt" and the NFS mounts would be non-responsive and would have
>> no choice but to reboot the system.
>>
>> Since the upgrade, now I'm seeing  "arc_reclaim" at a constant 99% cpu
>> utilization after very high NFS traffic -- mainly during nightly backups.
>> However, this time the nfs mounts are still responsive. I also see lots of
>> "arc_prune" processing running.
>>
>> System information:
>>
>> OS: Centos 6.7
>>        Linux 2.6.32-573.12.1.el6.x86_64 #1 SMP Tue Dec 15 21:19:08 UTC
>> 2015 x86_64 x86_64 x86_64 GNU/Linux
>> ZFS pools:   4 very active pools, each with 10 disks in raidz2
>> RAM: 256G
>> Note: this server is strictly an NFS server
>>
>>
>> Looking for some help with determining what is causing this. I'm
>> concerned that the system NFS mounts will eventually become unresponsive
>> and will have to reboot.
>>
>> Thanks!
>>
>>
>> Following are some zfs config and debug information:
>>
>> SLAB information:
>> ==============
>> --------------------- cache
>> -------------------------------------------------------  ----- slab ------
>>  ---- object -----  --- emergency ---
>> name                                    flags      size     alloc
>> slabsize  objsize  total alloc   max  total alloc   max  dlock alloc   max
>> spl_vn_cache                          0x00020     98304     48048
>> 8192      104     12    11    11    504   462   462      0     0     0
>> spl_vn_file_cache                     0x00020     65536     32928
>> 8192      112      8     7     7    336   294   294      0     0     0
>> spl_zlib_workspace_cache              0x00240         0         0
>>  2144960   268072      0     0     0      0     0     0      0     0     0
>> ddt_cache                             0x00040   2390784   2187328
>> 199232    24856     12    11    11     96    88    88      0     0     0
>> zio_buf_20480                         0x00042  27095040  19660800
>> 200704    20480    135   135   159   1080   960  1272      0     0     0
>> zio_data_buf_20480                    0x00042  45760512  25231360
>> 200704    20480    228   228   366   1824  1232  2928      0     0     0
>> zio_buf_24576                         0x00042  26849280  21037056
>> 233472    24576    115   115   142    920   856  1136      0     0     0
>> zio_data_buf_24576                    0x00042  47861760  29097984
>> 233472    24576    205   205   296   1640  1184  2368      0     0     0
>> zio_buf_28672                         0x00042  35676160  28901376
>> 266240    28672    134   134   134   1072  1008  1072      0     0     0
>> zio_data_buf_28672                    0x00042  43663360  31653888
>> 266240    28672    164   164   166   1312  1104  1328      0     0     0
>> zio_buf_32768                         0x00042  34086912  23330816
>> 299008    32768    114   114   134    912   712  1072      0     0     0
>> zio_data_buf_32768                    0x00042  53821440  37486592
>> 299008    32768    180   180   199   1440  1144  1592      0     0     0
>> zio_buf_40960                         0x00042  36454400  32768000
>> 364544    40960    100   100   145    800   800  1160      0     0     0
>> zio_data_buf_40960                    0x00042  80564224  53411840
>> 364544    40960    221   221   287   1768  1304  2296      0     0     0
>> zio_buf_49152                         0x00042  52469760  46399488
>> 430080    49152    122   122   133    976   944  1064      0     0     0
>>
>>
>> zfs.conf
>> =========
>> options zfs zfs_arc_min=10737418240
>> options zfs zfs_arc_max=68719476736
>>
>>
>> arcstat:
>> =========
>> 6 1 0x01 91 4368 12850802983 183440709950683
>> name                            type data
>> hits                            4    137887521
>> misses                          4    61047442
>> demand_data_hits                4    4931870
>> demand_data_misses              4    11851169
>> demand_metadata_hits            4    117889449
>> demand_metadata_misses          4    39705706
>> prefetch_data_hits              4    1086655
>> prefetch_data_misses            4    1184105
>> prefetch_metadata_hits          4    13979547
>> prefetch_metadata_misses        4    8306462
>> mru_hits                        4    71392731
>> mru_ghost_hits                  4    6220387
>> mfu_hits                        4    51428588
>> mfu_ghost_hits                  4    2599822
>> deleted                         4    57251112
>> mutex_miss                      4    276401
>> evict_skip                      4    11019879240
>> evict_not_enough                4    320113180
>> evict_l2_cached                 4    0
>> evict_l2_eligible               4    444038314496
>> evict_l2_ineligible             4    121226022912
>> evict_l2_skip                   4    0
>> hash_elements                   4    1319696
>> hash_elements_max               4    17601910
>> hash_collisions                 4    15187078
>> hash_chains                     4    25670
>> hash_chain_max                  4    7
>> p                               4    64424509440
>> c                               4    68719476736
>> c_min                           4    10737418240
>> c_max                           4    68719476736
>> size                            4    68966084976
>> hdr_size                        4    538457296
>> data_size                       4    918016
>>
>>
>> _______________________________________________
>> zfs-discuss mailing list
>> zfs-discuss at list.zfsonlinux.org
>> http://list.zfsonlinux.org/cgi-bin/mailman/listinfo/zfs-discuss
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.zfsonlinux.org/pipermail/zfs-discuss/attachments/20160108/850bec7d/attachment-0001.html>


More information about the zfs-discuss mailing list