[zfs-discuss] sending stream too slow on busy system

Sam Van den Eynde svde.tech at gmail.com
Tue May 24 05:52:19 EDT 2016


Wrt my previous mail: if the test without load on the send side works
perfectly fine with identical receiver configuration, this is of course not
applicable to you. Perhaps I got carried away too much here :-)

​
> This interests me a lot for your use case, as I am investigating perhaps
> something similar.



> My key question is simple: how can one make sure that 100% of all ZFS
> received streams go to the SLOG device before being written to HDD?



> It seems that, even with sync=always, it is not a trivial thing. In my
> particular use case, the SLOG device will easily hold the complete
> incremental stream it is receiving. However, since the stream does not seem
> to go through the SLOG device, ZFS receive causes a lot of immediate seek
> overhead to the HDD, and that causes the system IO to slow down. ZFS
> receive jobs can take quite some time, where they would complete in a few
> minutes on an idle system with just one stream being received. And that
> makes sense, as the HDD config on this system is very bad for IOPS (single
> HDD). The system only holds temporary data that is redundant.



> The idea behind this could work perfectly fine, but in practice I haven't
> gotten it to work the way I like it yet (disclaimer: I just started on it).
> Even if the SLOG device would not be useful to optimize the final IO stream
> going to the HDD (so HDD seek time needed would still be the same), it
> would at least allow me to keep the send/receive window low. That is my
> main concern for now. I could of course make a "mini-pool" out of the
> current SLOG device and send the data through from there, but for now I
> would like to see if I can figure this out first.



> I'm investigating things like this blog post
> http://nex7.blogspot.be/2013/04/zfs-intent-log.html
>  but need to do further testing wrt. the parameters mentioned.


Perhaps something to look into as well.​
>
>
On Tue, May 24, 2016 at 11:48 AM, Sam Van den Eynde <svde.tech at gmail.com>
wrote:

> with SSD cache/zil.
>>
>
>
> This interests me a lot for your use case, as I am investigating perhaps
> something similar.
>
>
> My key question is simple: how can one make sure that 100% of all ZFS
> received streams go to the SLOG device before being written to HDD?
>
>
> It seems that, even with sync=always, it is not a trivial thing. In my
> particular use case, the SLOG device will easily hold the complete
> incremental stream it is receiving. However, since the stream does not seem
> to go through the SLOG device, ZFS receive causes a lot of immediate seek
> overhead to the HDD, and that causes the system IO to slow down. ZFS
> receive jobs can take quite some time, where they would complete in a few
> minutes on an idle system with just one stream being received. And that
> makes sense, as the HDD config on this system is very bad for IOPS (single
> HDD). The system only holds temporary data that is redundant.
>
> The idea behind this could work perfectly fine, but in practice I haven't
> gotten it to work the way I like it yet (disclaimer: I just started on it).
> Even if the SLOG device would not be useful to optimize the final IO stream
> going to the HDD (so HDD seek time needed would still be the same), it
> would at least allow me to keep the send/receive window low. That is my
> main concern for now. I could of course make a "mini-pool" out of the
> current SLOG device and send the data through from there, but for now I
> would like to see if I can figure this out first.
>
> I'm investigating things like this blog post
> http://nex7.blogspot.be/2013/04/zfs-intent-log.html but need to do
> further testing wrt. the parameters mentioned.
>
> Perhaps something to look into as well.
>
>
> On Tue, May 24, 2016 at 12:25 AM, David Bruzos via zfs-discuss <
> zfs-discuss at list.zfsonlinux.org> wrote:
>
>> Hi again,
>> The system has six SATA rotating rust drives with SSD cache/zil.  The
>> SATA drives are configured in three mirrored vdevs.  It is significantly
>> busy with small and random'ish virtual machine work.  At his point, it
>> looks like it may make it in about five days (asuming nothing goes wrong).
>>
>> It seems like zfs send performance drops quiet rapidly when the system
>> gets even a little busy, is that typical and/or expected behavior.
>>
>> Oh, btw, resumable send/recv would be a good thing.
>>
>> David
>>
>> On Mon, May 23, 2016 at 03:13:17PM -0700, Paul Dagnelie wrote:
>> > Apparently, ZoL doesn't have resumable receive yet... I thought they'd
>> > pulled that in by now.  In that case, zfs_pd_bytes_max is the only
>> tunable
>> > you can mess with that will help you much.  Sending 1 TB in 1 week is
>> very
>> > slow; what kind of storage is your pool using? Is the system very busy
>> with
>> > other IO?
>> >
>> > On Mon, May 23, 2016 at 3:06 PM, David Bruzos <
>> dbruzos at succinctsystems.com>
>> > wrote:
>> >
>> > > It does not appear that additional buffering would help much in my
>> > > particular situation, but I will check it out anyway.
>> > >
>> > > Thanks.
>> > >
>> > >
>> > > On Mon, May 23, 2016 at 03:38:32PM -0400, ZFS Discuss ML wrote:
>> > > > Have a look at mbuffer. I've used it and it seems to speed up
>> replication
>> > > > considerably
>> > > >
>> > > >
>> > > > On Mon, May 23, 2016 at 2:54 PM, Paul Dagnelie via zfs-discuss <
>> > > > zfs-discuss at list.zfsonlinux.org> wrote:
>> > > >
>> > > > > First, what version are you running?  zfs send has been going
>> through
>> > > some
>> > > > > changes lately, and the tunables are changing with it.  Second,
>> you
>> > > > > probably want to look into resumable send/recv; if your send is
>> going
>> > > to
>> > > > > take a week, you don't want to have to restart on day 6 because
>> your
>> > > > > network dropped some packets on the floor.
>> > > > >
>> > > > > On Sun, May 22, 2016 at 5:29 PM, David Bruzos via zfs-discuss <
>> > > > > zfs-discuss at list.zfsonlinux.org> wrote:
>> > > > >
>> > > > >> Hello all:
>> > > > >> I have a reasonably busy system that I need to stream about a TB
>> of
>> > > data
>> > > > >> from in order to migrate some virtual machines to a remote site,
>> but
>> > > the
>> > > > >> zfs send is really slow.  A similar machine with no VMs running
>> seems
>> > > to
>> > > > >> send about 10-15 times faster than the busy box.
>> > > > >> My plan is to do an initial send and then a final incremental
>> send, in
>> > > > >> order to cut the down time to a minimum, but at this rate, it
>> will
>> > > take me
>> > > > >> a week to complete the initial send.  Of course, something is
>> likely
>> > > to go
>> > > > >> wrong during this long time, so I'll have to start over.
>> > > > >>
>> > > > >> What are some tunable parameters that can increase ZFS streaming
>> > > > >> performance that I can play with while the system is up and
>> running
>> > > and are
>> > > > >> not likely to cause some kind a nasty crash and/or sideaffect?
>> > > > >>
>> > > > >> Thanks!
>> > > > >>
>> > > > >> David
>> > > > >>
>> > > > >> _______________________________________________
>> > > > >> zfs-discuss mailing list
>> > > > >> zfs-discuss at list.zfsonlinux.org
>> > > > >> http://list.zfsonlinux.org/cgi-bin/mailman/listinfo/zfs-discuss
>> > > > >>
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Paul Dagnelie
>> > > > >
>> > > > > _______________________________________________
>> > > > > zfs-discuss mailing list
>> > > > > zfs-discuss at list.zfsonlinux.org
>> > > > > http://list.zfsonlinux.org/cgi-bin/mailman/listinfo/zfs-discuss
>> > > > >
>> > > > >
>> > >
>> > > > _______________________________________________
>> > > > zfs-discuss mailing list
>> > > > zfs-discuss at list.zfsonlinux.org
>> > > > http://list.zfsonlinux.org/cgi-bin/mailman/listinfo/zfs-discuss
>> > >
>> > >
>> >
>> >
>> > --
>> > Paul Dagnelie
>> _______________________________________________
>> zfs-discuss mailing list
>> zfs-discuss at list.zfsonlinux.org
>> http://list.zfsonlinux.org/cgi-bin/mailman/listinfo/zfs-discuss
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.zfsonlinux.org/pipermail/zfs-discuss/attachments/20160524/4c87251a/attachment.html>


More information about the zfs-discuss mailing list