[zfs-discuss] system hangs running find on .zfs subdirectory
wingfeathera at gmail.com
Mon Apr 30 05:20:59 EDT 2012
On 29 Apr 2012, at 17:07, Andy wrote:
> Hello all.
> Anyone else found that a system hang can be provoked easily by running a simple 'find /tank1/data1/.zfs/snapshot -name somefile' command a few times, looking in the new .zfs subdirectory, where all my snapshots are mounted? The filesystem currently has ~2300 files/subdirectories, I have 110 snapshots, neither of which I'd consider to be 'big data'. There's nothing else happening on the VM.
> I'm running the current rc8, on CentOS 6.2 x86_64 2.6.32-220.7.1.el6.x86_64 (yes this does, or should, support the use of .zfs). The system (a VM) has 1400MB RAM. However, I read about a possible need to tune settings for stability, so I tried changing 'vm.min_free_kbytes', 'zfs_arc_max' and 'zfs_arc_meta_limit' (will look out the values I used if anyone's interested), but couldn't stop the system freeze, even though I could tell that the arc was not taking so much memory after tuning zfs_arc_max. I don't think this is a case of running out of memory anyway, because I changed the VM to give it 8GB of RAM, and the problem seemed to happen just as easily. Output from 'top' and 'free' did not show the system running anywhere near out of memory.
> Is there anything I can do to help diagnose this? I don't get anything on the VM's console when this happens. I'm not sure where else to look to get diagnostics. Is there something else I need to enable, to see what's happening?
> Thanks in advance Andy
I don't know if this will be any help, but I was experiencing similar hangs from running rsyncs. It doesn't seem to matter how much memory you have, simply because ZFS's memory usage grows to fill your memory as it caches things. So adding memory doesn't help. The major thing that worked for me was changing the value of min_free_kbytes. I've set mine to 256M now (262144) and I haven't had a single hang since. I think 128M would also be fine, but that does still crash for me if compression is enabled. 256M has been stable for me even with compression so far.
According to the latest wisdom, zfs_arc_max should be set to 1/2 your system ram at the very most. I've set mine to 1/4 of my RAM for now and have no problems. Of course, this will impact performance somewhat.
It seems that the pending VM/ARC improvement patches should improve (fix?) all crash behaviour relating to memory stress.
More information about the zfs-discuss