[zfs-discuss] memory congestion and high txg_sync txg_quiesce CPU utilization and very bad performance

Tren Blackburn iam at theendoftime.net
Thu Dec 18 17:21:14 EST 2014

On December 18, 2014 at 2:13:49 PM, Gordan Bobic (gordan.bobic at gmail.com) wrote:
On 12/18/2014 09:01 PM, Tren Blackburn wrote: 
> On December 18, 2014 at 12:02:41 PM, Gordan Bobic 
> (gordan.bobic at gmail.com <mailto:gordan.bobic at gmail.com>) wrote: 
>> On 12/18/2014 06:58 PM, Tren Blackburn wrote: 
>> >> rsync does not use ram directly, only indirectly through Linux's vm 
>> >> system, and that is what I would like to tune. Its also worth noting 
>> >> that the filesystem being backed up is GPFS, which like ZFS does its 
>> >> own caching strategies as well. 
>>>> > I'll have to refute that. rsync most definitely does use ram. This is 
>> > from a ZFS box that is pretty much an analog for yours. A disaster 
>> > recovery point that can be flipped to prod: 
>>>> > [0] root at IMbackup.office:~# ps wwaux | grep rsync 
>>>> > root 15221 1.7 9.5 3312232 3146772 pts/0 D+ Dec02 403:33 rsync 
>> > -av --progress --exclude .snapshot /fas2040a/ . 
>>>> > root 15222 0.3 7.5 2667976 2500612 pts/0 S+ Dec02 75:34 rsync 
>> > -av --progress --exclude .snapshot /fas2040a/ . 
>>>> > root 15223 1.7 8.0 2822564 2655532 pts/0 S+ Dec02 405:09 rsync 
>> > -av --progress --exclude .snapshot /fas2040a/ . 
>>>> > As you can see, rsync is using about 8ish GB of RAM. This is actual RAM 
>> > usage, not page cache. 
>> That doesn't sound entirely sane. Are you sure you are running the 
>> latest version of rsync and not an old buggy version? 
> rsync with centos 6.4. 

That's reasonably recent, but not fully up to date. It may be worth 
checking the changelog for bug fixes that could be in play. 

>> If that really is a legitimate use of RAM, I can only think of rsync 
>> using it as metadata. Do the inodes in that subtree add up to anywhere 
>> near 8GB? 
> There's about 75 million inodes from this source I'm syncing from 
> currently. This is the initial replication step, so it needs to trawl 
> through everything. Memory hasn't been an issue, nor has performance. 
> ARC is limited to 10gb and set to metadata. About 18GB in the page cache 
> (due to rsyncing from a NFS source), and there's about 1.5GB of ram free 
> at all times. This is on a 32GB box. 

You may want to consider invoking separate rsyncs for each subdirectory 
in that tree. Divide and conquer, so to speak. 
That's the plan for the next rsync. But luckily things haven't been an issue the way they're running. Speed has been consistent, and we're not in a huge rush. 

>> You may want to look into using something like lsyncd that will copy 
>> each file upon modification, rather than repeatedly doing a full 
>> recursive trawl (that only happens once at startup so it can establish a 
>> base line from which to track changes from). 
> Possibly, but that's not how we're looking to use this. But thanks for 
> the suggestions. 

If you aren't looking at async near-real-time replication, you should 
almost certainly be using zfs send. That will be much faster and much 
less resource intensive. 
This isn't ZFS to ZFS. This is a NetApp to ZFS. 

Anyhow, I feel bad for this off topic chat taking over the OP's thread. Let's split this off. :)


To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe at zfsonlinux.org. 

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe at zfsonlinux.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.zfsonlinux.org/pipermail/zfs-discuss/attachments/20141218/e5b71f00/attachment.html>

More information about the zfs-discuss mailing list