[zfs-discuss] Re: High IO cripples system

Christ Schlacta aarcane at aarcane.org
Fri Dec 16 16:22:52 EST 2011

As I posted below, I have 4TB and have ZFS ARC clamped between 128MB 
minimum and 2GB total for a 5TB Raidz array with dedup formerly enabled, 
but turned off, compression formerly enabled, but turned off.

Neither have ever been used in this particular dataset.  I clamp my ARC 
at half because I DO have a lot of procees running, and am often using 
nearly all of that memory.  I'd rather always have a little extra free 
than ever run out though, because I run without swap.  When swap on ZFS 
is viable, I plan to migrate.

I ordered some ECC ram to replace the non-ECC ram that's in there now, 
and I'll have 8GB instead of 4.  However, I'm expecting the problem to 
stay the same.  I'm going to double how much memory I have for ARC 
(256-4GB), and I expect to encounter the same problem in twice the 
time.  I'll definitely report back once the memory comes in.

One thing is:  I wish I could figure out how to gather some sort of 
debugging information so I can provide some feedback on what's going 
wrong, where, or how...  but I don't know how.  If anyone knows how to 
enable debugging mode, verbose output to a console, or anything else 
useful, PLEASE post to this thread or a new one and make the information 
more available!

On 12/15/2011 23:30, Cori wrote:
> How much memory do you have?
> ZFS normally uses all of a servers memory less 1GB for for disk cache. 
> So, it's recommended that you have at LEAST 1GB of memory + 1GB of 
> memory per Terabyte of storage used.
> If you enable dedupe, then you can expect to use dramatically more. 
> 4-32GB/TB.  Otherwise dedupe will be slowed down.
> Cori
> On Sat, Dec 10, 2011 at 8:57 PM, Marcel J.F. Knopper <marcel at ckxs.com 
> <mailto:marcel at ckxs.com>> wrote:
>     Hi, can't help you with getting more information but I've been
>     experimenting with dedup and compression in various scenarios and
>     seen complete hangs too. And I to did not get any usefull
>     information regarding the exact cause of the hang.
>     My conclusion after all the tests is that dedup needs more memory
>     than I have to keep some performance. So performance is nil if the
>     dataset is more than a couple of GB's. And compression seem to
>     cause system hangs when under load.
>     Regards,
>     MarcelK.
>     ________________________________________
>     Van: Christ Schlacta [aarcane at aarcane.org
>     <mailto:aarcane at aarcane.org>]
>     Verzonden: zaterdag 10 december 2011 8:21
>     Aan: zfs-discuss at zfsonlinux.org <mailto:zfs-discuss at zfsonlinux.org>
>     Onderwerp: [zfs-discuss] Re: High IO cripples system
>     So this time it crashed, and I got no useful information out of
>     dmesg or
>     anything else.  I have, however, encountered the error twice
>     since, but
>     again no useful messages.  I get the same kernel messages about hung
>     tasks that I have posted previously, however, nothing new.
>     For the time being I'm trying to bring some virtualization online.
>     However, if high IO cripples ZFS, and vicariously, my system, then
>     I may
>     not be able to accomplish my goal.  The system does generally recover
>     from high IO related hangs, it just takes a while.  I think it may be
>     related to the drop_caches issues that people keep reporting, but
>     if so,
>     constantly dropping caches is not a viable long-term alternative.
>     Is there anything I can do to trick ZFS into providing more useful
>     information?  I can do radical stuff only on thursdays, but anything
>     not-too-invasive I can do pretty much anywhen.
>     On 12/9/2011 19:32, Christ Schlacta wrote:
>     > High IO cripples my ZFS system.  I was SSH'd in, and running a
>     command
>     > on a block device /dev/zvol/tank/images/ubuntu-natty-base (compress,
>     > dedup off) and the system has frozen up so bad I can't connect via
>     > SSH.  I've had similar freezes under other high loads.  Not really
>     > sure what's causing it, or where the weak point is, but it's been
>     > persistent.  WHen the system un-freezes, I'll post a reply with some
>     > dmesg output and anything interesting I can find.  I've got a cache
>     > device on SSD, and and an ARC limited to between 128MB and 2GB, on a
>     > 4GB system.  I've also got a per device cache (I don't recall the
>     > module parameter, but it's per physical disk) of either 8 or 6 MB.
>     > Everything else should be default.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.zfsonlinux.org/pipermail/zfs-discuss/attachments/20111216/f72eb5cc/attachment.html>

More information about the zfs-discuss mailing list