[zfs-devel] Support for external data transformation in ZFS

Matthew Ahrens mahrens at delphix.com
Mon Oct 24 12:27:31 EDT 2016

Niklas, this idea sounds very similar to what Dan implemented for
compressed send/receive.  This hasn't yet been integrated to OpenZFS but
you could use it as a foundation for your project.  See the new code in
receive_read_record(), arc_loan_compressed_buf(), etc.  This also supports
compressed ("raw") reads from the compressed ARC.



On Mon, Oct 24, 2016 at 7:01 AM, Niklas Behrmann via zfs-devel <
zfs-devel at list.zfsonlinux.org> wrote:

> Hello,
> I am about to start my thesis at the group Scientific Computing at the
> University of Hamburg. The thesis involves changes to ZFS which we would
> like to discuss with you.
> The IPCC for Lustre project is aiming for compression support in Lustre at
> multiple levels. To increase throughput performance a prior thesis started
> the implementation of online compression with lz4 in Lustre. The data is
> compressed on client-side and send as sparse data stripes to the server.
> This leads to insufficient readaheads due to the gaps in the incomplete
> written stripes.
> In order to solve this problem the idea is to integrate the compressed
> stripes in ZFS as if they were compressed by ZFS. To achieve this
> additional information about the uncompressed size of the data is needed.
> First it was considered to create a new function *zpl_write* with a added
> parameter for the uncompressed size but this would require a change of the
> VFS Layer. So instead the plan is to create a new *dmu_write* function
> since Lustre communicates with ZFS through the dmu Layer. This may result
> in a function *dmu_write_transformed(objset_t *os, uint64_t object,
> uint64_t offset, uint64_t orig_size, uint64_t size, const void *buf,
> dmu_tx_t *tx)*. Then instead of compressing the blocks in
> *zio_write_bp_init* the passed data size could be used to set the *lsize*.
> It would be great if you could tell us what you think about this project
> or provide some ideas and suggestions.
> Best regards,
> Niklas
> _______________________________________________
> zfs-devel mailing list
> zfs-devel at list.zfsonlinux.org
> http://list.zfsonlinux.org/cgi-bin/mailman/listinfo/zfs-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.zfsonlinux.org/pipermail/zfs-devel/attachments/20161024/585196df/attachment.html>

More information about the zfs-devel mailing list