[zfs-discuss] ZFS consuming twice ARC's size and crashing system

Sander Smeenk ssmeenk at freshdot.net
Tue Aug 20 03:23:57 EDT 2013


Quoting Gordan Bobic (gordan.bobic at gmail.com):

> What I find works quite well is setting the arc_max quite small (up
> to about 1GB per TB of usable disk space). Then I set up one ZRAM
> per CPU core on the machine so that their total size equals the
> amount I want to use for ZFS caching, and I set up those ZRAM
> devices as L2ARC. Enabling ARC-ing prefetch data can help achieve
> better hit ratios in most cases. If your disk:RAM ratio is very
> high, set L2ARC to metadata-only.

I am now running with 32 ZRAM-caches in my pool and arc_max set to 25%
of memory. The remaining memory, minus 8GB for 'OS stuff', is used by
the ZRAMs.

First tests are promising, though i also had good runs with the stock
config so this could still go anywhere... ;-))

I adapted a script to setup ZRAMs from the 'zram-config' package in
Ubuntu (which just puts ZRAMs as swap) which scales nicely when i
change arcsize and/or number of CPUs:

| #!/bin/sh
| # $Id: zramsetup 284 2013-08-19 18:31:02Z sanders $
| 
| nrdevices=$(grep -c ^processor /proc/cpuinfo | sed 's/^0$/1/')
| if modinfo zram | grep -q ' zram_num_devices:' 2>/dev/null; then
|     modprobeargs="zram_num_devices=${nrdevices}"
| elif modinfo zram | grep -q ' num_devices:' 2>/dev/null; then
|     modprobeargs="num_devices=${nrdevices}"
| else
|     echo "Unknown / unsupported 'zram' module?"
|     exit 1
| fi
| 
| modprobe zram $modprobeargs
| 
| # Calculate memory to use for zram [ in bytes! ]
| totalmem=$(free -b | awk '/^Mem: */ {print $2}')
| arcsize=$(cat /sys/module/zfs/parameters/zfs_arc_max)
| ossize=$((8 * 1024 * 1024 * 1024))
| availmem=$((totalmem-$((arcsize+ossize))))
| memperdev=$((availmem / 2 / nrdevices))
| 
| echo "Total memory : $totalmem"
| echo "Max ARC size : $arcsize"
| echo "OS memory    : $ossize"
| echo ""
| echo "Unused memory: $availmem"
| echo "Per device   : $memperdev (x$nrdevices)"
| echo ""
| 
| # initialize the devices
| for i in $(seq ${nrdevices}); do
|     echo $memperdev > /sys/block/zram$((i - 1))/disksize
| done


> >So far setting the arc_max to 25-30% of RAM and vm.min.free_kbytes to
> >512MB has kept my systems from crashing.

I also set:
| # grep -v "^#" /etc/sysctl.d/zfs.conf | grep -v "^$"
| vm.min_free_kbytes=4026532
| vm.dirty_background_ratio=5
| vm.swappiness=10
| vm.vfs_cache_pressure=10000

But by now i have tweaked so much i don't really know what helps and
what doesn't. All these tips/tweaks were gathered from various zfs
mailinglists / websites / howto's etc..

Oh, and i 'echo 3 > /proc/sys/vm/drop_caches' every 30 minutes.


Thanks for all the good tips and help so far!!
-Sndr.
-- 
| The world is so full of these wonderful things,
| i'm sure we should all be as happy as kings.
| 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7  FBD6 F3A9 9442 20CC 6CD2

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe at zfsonlinux.org.



More information about the zfs-discuss mailing list