[zfs-discuss] Re: High IO cripples system
aarcane at aarcane.org
Fri Dec 16 16:22:52 EST 2011
As I posted below, I have 4TB and have ZFS ARC clamped between 128MB
minimum and 2GB total for a 5TB Raidz array with dedup formerly enabled,
but turned off, compression formerly enabled, but turned off.
Neither have ever been used in this particular dataset. I clamp my ARC
at half because I DO have a lot of procees running, and am often using
nearly all of that memory. I'd rather always have a little extra free
than ever run out though, because I run without swap. When swap on ZFS
is viable, I plan to migrate.
I ordered some ECC ram to replace the non-ECC ram that's in there now,
and I'll have 8GB instead of 4. However, I'm expecting the problem to
stay the same. I'm going to double how much memory I have for ARC
(256-4GB), and I expect to encounter the same problem in twice the
time. I'll definitely report back once the memory comes in.
One thing is: I wish I could figure out how to gather some sort of
debugging information so I can provide some feedback on what's going
wrong, where, or how... but I don't know how. If anyone knows how to
enable debugging mode, verbose output to a console, or anything else
useful, PLEASE post to this thread or a new one and make the information
On 12/15/2011 23:30, Cori wrote:
> How much memory do you have?
> ZFS normally uses all of a servers memory less 1GB for for disk cache.
> So, it's recommended that you have at LEAST 1GB of memory + 1GB of
> memory per Terabyte of storage used.
> If you enable dedupe, then you can expect to use dramatically more.
> 4-32GB/TB. Otherwise dedupe will be slowed down.
> On Sat, Dec 10, 2011 at 8:57 PM, Marcel J.F. Knopper <marcel at ckxs.com
> <mailto:marcel at ckxs.com>> wrote:
> Hi, can't help you with getting more information but I've been
> experimenting with dedup and compression in various scenarios and
> seen complete hangs too. And I to did not get any usefull
> information regarding the exact cause of the hang.
> My conclusion after all the tests is that dedup needs more memory
> than I have to keep some performance. So performance is nil if the
> dataset is more than a couple of GB's. And compression seem to
> cause system hangs when under load.
> Van: Christ Schlacta [aarcane at aarcane.org
> <mailto:aarcane at aarcane.org>]
> Verzonden: zaterdag 10 december 2011 8:21
> Aan: zfs-discuss at zfsonlinux.org <mailto:zfs-discuss at zfsonlinux.org>
> Onderwerp: [zfs-discuss] Re: High IO cripples system
> So this time it crashed, and I got no useful information out of
> dmesg or
> anything else. I have, however, encountered the error twice
> since, but
> again no useful messages. I get the same kernel messages about hung
> tasks that I have posted previously, however, nothing new.
> For the time being I'm trying to bring some virtualization online.
> However, if high IO cripples ZFS, and vicariously, my system, then
> I may
> not be able to accomplish my goal. The system does generally recover
> from high IO related hangs, it just takes a while. I think it may be
> related to the drop_caches issues that people keep reporting, but
> if so,
> constantly dropping caches is not a viable long-term alternative.
> Is there anything I can do to trick ZFS into providing more useful
> information? I can do radical stuff only on thursdays, but anything
> not-too-invasive I can do pretty much anywhen.
> On 12/9/2011 19:32, Christ Schlacta wrote:
> > High IO cripples my ZFS system. I was SSH'd in, and running a
> > on a block device /dev/zvol/tank/images/ubuntu-natty-base (compress,
> > dedup off) and the system has frozen up so bad I can't connect via
> > SSH. I've had similar freezes under other high loads. Not really
> > sure what's causing it, or where the weak point is, but it's been
> > persistent. WHen the system un-freezes, I'll post a reply with some
> > dmesg output and anything interesting I can find. I've got a cache
> > device on SSD, and and an ARC limited to between 128MB and 2GB, on a
> > 4GB system. I've also got a per device cache (I don't recall the
> > module parameter, but it's per physical disk) of either 8 or 6 MB.
> > Everything else should be default.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the zfs-discuss