[zfs-devel] Support for external data transformation in ZFS
1behrman at informatik.uni-hamburg.de
Mon Oct 24 10:01:45 EDT 2016
I am about to start my thesis at the group Scientific Computing at the
University of Hamburg. The thesis involves changes to ZFS which we would
like to discuss with you.
The IPCC for Lustre project is aiming for compression support in Lustre
at multiple levels. To increase throughput performance a prior thesis
started the implementation of online compression with lz4 in Lustre. The
data is compressed on client-side and send as sparse data stripes to the
server. This leads to insufficient readaheads due to the gaps in the
incomplete written stripes.
In order to solve this problem the idea is to integrate the compressed
stripes in ZFS as if they were compressed by ZFS. To achieve this
additional information about the uncompressed size of the data is needed.
First it was considered to create a new function /zpl_write/ with a
added parameter for the uncompressed size but this would require a
change of the VFS Layer. So instead the plan is to create a new
/dmu_write/ function since Lustre communicates with ZFS through the dmu
Layer. This may result in a function /dmu_write_transformed(objset_t
*os, uint64_t object, uint64_t offset, uint64_t orig_size, uint64_t
size, const void *buf, dmu_tx_t *tx)/. Then instead of compressing the
blocks in /zio_write_bp_init/ the passed data size could be used to set
It would be great if you could tell us what you think about this project
or provide some ideas and suggestions.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the zfs-devel