[zfs-discuss] Re: Re: no activity, uses all memory then crashes

Brian Behlendorf behlendorf1 at llnl.gov
Mon Jun 4 14:29:36 EDT 2012


On Mon, 2012-06-04 at 11:06 -0700, Gregor Kopka wrote:
> 
> Am 04.06.2012 18:44, schrieb Niels de Carpentier:
> >> On Mon, 2012-06-04 at 18:14 +0200, Niels de Carpentier wrote:
> >>> It mainly slab inefficiency. A 262144 byte block can only hold 31 4096
> >>> objects max, so more than half the memory is always wasted.
> >> I don't quite follow here. How big are "4096 objects"? (Obviously,
> >> they're not 4096 bytes if a 262144 byte block can only hold 31 of them.)
> >> Shouldn't ZFS take that size into account? In other words, why is ZFS
> >> trying to allocate blocks in inefficient sizes? If more than half of a
> >> block will *always* be wasted, the block is *at least* twice as big as
> >> it should be.
> > It's not ZFS, it's the slab, which is the linux specific spl layer.
> > The problem is the blocks need to be aligned on 4096 boundaries as well.
> > Since the slab uses a small header before each object, they have to be
> > spaced 4096 bytes apart. I'm working on a patch that stores the headers
> > separately from the objects, which allows tight packing of the objects. I
> > just have been so busy with work that I haven't had time to work on it the
> > last months.
> Does ZFS has its own SLAB allocator, or is it using the one in the kernel?

It has its own because it requires particularly large objects which
isn't handled well by the kernel.

> Asking since SLUB is available in the kernel which dosn't suffer from 
> the small headers:
> http://lwn.net/Articles/229984/ and http://lwn.net/Articles/229096/ for 
> details

The long term fix here is to back the ARC directly with the Linux page
cache instead of a slab.  Then we can shift the rest of the ZFS slab
consumers to the standard Linux slabs.

However, that work is still a ways off so I'm all for further optimizing
the current SPL implementation.  The big issue as Niels correctly
described it is that the required memory alignment for ARC buffers can
waste a significant amounts of memory.  Packing the objects more tightly
can be done but it's a bit tricky.

Niels if you have some thoughts on this, or better yet a preliminary
patch, I'd love to take a look.

-- 
Thanks,
Brian



More information about the zfs-discuss mailing list