[zfs-discuss] Compressratio vs du

Richard Elling richard.elling at richardelling.com
Tue Feb 20 14:46:16 EST 2018



> On Feb 16, 2018, at 3:35 AM, Audun via zfs-discuss <zfs-discuss at list.zfsonlinux.org> wrote:
> 
> This seems intriguing, so I tried exploring this a bit:
> 
> So I created a test file by concatenating /etc/hosts a bunch of times
> to make sure it's easily compressible, and not all zeroes:
> # ls -lh /tank/test/file3
> -rw-r--r-- 1 root root 183K Feb 15 15:24 /tank/test/file3

FWIW,
ls can show the size in addition to the length of a file. Try:
  ls -lsh /tank/test/file3

stat can also show more allocation details.

Know that if copies > 1, then the size is expected to be larger than the length.
 -- richard

> 
> # ls -l /tank/test/file3
> -rw-r--r-- 1 root root 186840 Feb 15 15:24 /tank/test/file3
> 
> du shows the file to take 17k:
> 
> # du -h /tank/test/file3
> 17K     /tank/test/file3
> # du --block-size=1 /tank/test/file3
> 16896   /tank/test/file3
> 
> Side note, 16896 is 4*4096 + 512
> 
> # du -h --apparent-size /tank/test/file3
> 183K    /tank/test/file3
> 
> # du  --block-size=1 --apparent-size /tank/test/file3
> 186840  /tank/test/file3
> 
> The file compresses nicely.
> 
> Here is output from zdb:
> 
> # zdb -vvv -O tank test/file3
> 
>    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
>    156175    2   128K   128K    16K     512   256K  100.00  ZFS plain
> file (K=inherit) (Z=inherit)
>                                               168   bonus  System attributes
>        dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED
>        dnode maxblkid: 1
>        uid     1
>        gid     1
>        atime   Thu Feb 15 15:25:34 2018
>        mtime   Thu Feb 15 15:24:56 2018
>        ctime   Thu Feb 15 15:24:56 2018
>        crtime  Thu Feb 15 15:23:13 2018
>        gen     18796876
>        mode    100644
>        size    186840
>        parent  222678
>        links   1
>        pflags  40800000004
> Indirect blocks:
>               0 L1  0:c399652000:1000 20000L/1000P F=2 B=18796896/18796896
>               0  L0 0:c399632000:1000 20000L/1000P F=1 B=18796895/18796895
>           20000  L0 0:c399650000:1000 20000L/1000P F=1 B=18796896/18796896
> 
>                segment [0000000000000000, 0000000000040000) size  256K
> 
> 
> I'm no expert, but I read this as the file taking 12.5k (12800 bytes) on disk:
> dnsize (512) + 3x indirect blocks of 0x1000P (4k/4096) each.
> 
> Where does du get 17k from? There seems to be exactly 4096 bytes too
> much from what I see in zdb.
> 
> The pool does not have ashift set explicitly, but disks are 4k and zdb
> -C shows ashift=12 on the single mirrored vdev in this pool.
> 
> I'm no good at reading zdb output, it would be really great if someone
> would point to some documentation.
> 
> 
> --
> Best Regards,
> 
> Audun Gangsto
> 
> 
> On 15 February 2018 at 09:31, Stefan Ring via zfs-discuss
> <zfs-discuss at list.zfsonlinux.org> wrote:
>> On Wed, Feb 14, 2018 at 10:27 PM, Joel Krauska via zfs-discuss
>> <zfs-discuss at list.zfsonlinux.org> wrote:
>>> Hello,
>>> 
>>> I've been trying to better understand how to evaluate effective compression
>>> ratios.
>>> 
>>> I've come across many examples that compare the output from
>>> 
>>> du -s . and   du -s --apparent-size
>>> as a technique to validate/confirm compression.
>>> 
>>> eg.
>>> # dd if=/dev/zero of=zeros bs=1M count=100
>>> 
>>> # du -s zeros
>>> 1 zeros
>>> 
>>> vs
>>> 
>>> # du -s --apparent-size zeros
>>> 102400 zeros
>>> 
>>> Using this technique I've been attempting to establish which
>>> directories/datasets compress well and which do not.
>>> 
>>> However on the whole I'm seeing a pretty big disparity between what du
>>> reports (typically 3x-4x) and what 'zfs get compressratio' reports
>>> (typically 7-8x).
>> 
>> If you write only zeroes, that will be converted to a sparse file, and
>> I guess that this does not even count as compression, as there is
>> nothing to compress.
>> _______________________________________________
>> zfs-discuss mailing list
>> zfs-discuss at list.zfsonlinux.org
>> http://list.zfsonlinux.org/cgi-bin/mailman/listinfo/zfs-discuss
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at list.zfsonlinux.org
> http://list.zfsonlinux.org/cgi-bin/mailman/listinfo/zfs-discuss



More information about the zfs-discuss mailing list