[zfs-devel] Support for external data transformation in ZFS

Niklas Behrmann 1behrman at informatik.uni-hamburg.de
Mon Oct 24 10:01:45 EDT 2016


I am about to start my thesis at the group Scientific Computing at the 
University of Hamburg. The thesis involves changes to ZFS which we would 
like to discuss with you.
The IPCC for Lustre project is aiming for compression support in Lustre 
at multiple levels. To increase throughput performance a prior thesis 
started the implementation of online compression with lz4 in Lustre. The 
data is compressed on client-side and send as sparse data stripes to the 
server. This leads to insufficient readaheads due to the gaps in the 
incomplete written stripes.
In order to solve this problem the idea is to integrate the compressed 
stripes in ZFS as if they were compressed by ZFS. To achieve this 
additional information about the uncompressed size of the data is needed.
First it was considered to create a new function /zpl_write/ with a 
added parameter for the uncompressed size but this would require a 
change of the VFS Layer. So instead the plan is to create a new 
/dmu_write/ function since Lustre communicates with ZFS through the dmu 
Layer. This may result in a function /dmu_write_transformed(objset_t 
*os, uint64_t object, uint64_t offset, uint64_t orig_size, uint64_t 
size, const void *buf, dmu_tx_t *tx)/. Then instead of compressing the 
blocks in /zio_write_bp_init/ the passed data size could be used to set 
the /lsize/.
It would be great if you could tell us what you think about this project 
or provide some ideas and suggestions.

Best regards,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.zfsonlinux.org/pipermail/zfs-devel/attachments/20161024/e53d4194/attachment.html>

More information about the zfs-devel mailing list