[zfs-discuss] LZ4 compression not living up to hype

Chris Peden chris at cptechconsulting.com
Sat Aug 3 01:35:53 EDT 2013


without compression:
peedypool   18.1G  7.23T      0    187      0  20.6M
peedypool   18.2G  7.23T      0    490      0  52.2M
peedypool   18.3G  7.23T      0    712      0  75.2M
peedypool   18.6G  7.23T      0    628      0  64.0M
peedypool   18.7G  7.23T      0    731      0  81.0M
peedypool   19.0G  7.23T      0    537      0  58.2M
peedypool   19.1G  7.23T      0    739      0  84.7M
peedypool   19.4G  7.23T      0    595      0  62.3M
peedypool   19.5G  7.23T      0    599      0  67.9M
peedypool   19.6G  7.23T      0    712      0  81.3M
peedypool   19.8G  7.23T      0    606      0  64.0M
peedypool   20.1G  7.23T      0    587      0  65.1M
peedypool   20.2G  7.23T      0    747      0  84.9M
peedypool   20.3G  7.23T      0    601    511  64.9M
peedypool   20.6G  7.23T      2    625  12.5K  66.0M
peedypool   20.8G  7.23T      1    704  2.00K  81.0M
peedypool   20.9G  7.23T      4    647  4.50K  69.0M
peedypool   21.2G  7.23T      1    542  2.00K  59.4M
peedypool   21.3G  7.23T      2    726  2.25K  83.0M
peedypool   21.5G  7.23T      4    669  4.25K  73.8M
peedypool   21.7G  7.23T      3    551  3.75K  55.2M
peedypool   21.9G  7.23T      1    728  10.5K  78.9M
peedypool   22.0G  7.23T      3    748  14.2K  83.3M
peedypool   22.3G  7.23T      2    512  1.75K  54.2M
peedypool   22.4G  7.23T      2    669  2.00K  76.1M
peedypool   22.6G  7.23T      1    758  1.75K  85.4M
peedypool   22.9G  7.23T      4    554  12.5K  54.0M
peedypool   23.0G  7.23T      2    648  2.25K  73.7M
peedypool   23.1G  7.23T      1    743  2.00K  79.4M
peedypool   23.3G  7.23T      4    597  14.5K  62.8M
peedypool   23.6G  7.23T      2    590  2.25K  66.6M
peedypool   23.7G  7.23T      2    720  2.25K  81.4M
peedypool   23.9G  7.23T      4    629  4.25K  67.5M
peedypool   24.1G  7.23T      1    600  10.5K  62.7M
peedypool   24.3G  7.23T      2    703  2.50K  79.9M
peedypool   24.4G  7.23T      2    670  2.75K  73.5M
peedypool   24.7G  7.23T      3    513  4.50K  54.4M
peedypool   24.8G  7.23T      1    608  2.25K  68.3M
peedypool   25.0G  7.23T      2    705  2.50K  80.5M
peedypool   25.1G  7.23T      3    639  4.25K  68.0M
peedypool   25.4G  7.23T      2    559  2.50K  57.6M
peedypool   25.5G  7.23T      1    773  8.50K  87.0M
peedypool   25.7G  7.22T      3    640  3.75K  70.8M
peedypool   26.0G  7.22T      2    547  2.25K  58.9M
peedypool   26.1G  7.22T      2    712  2.25K  81.5M
peedypool   26.2G  7.22T      3    682  2.75K  74.7M
peedypool   26.5G  7.22T      2    546  12.0K  54.1M
peedypool   26.7G  7.22T      2    685  2.25K  73.1M
peedypool   26.8G  7.22T      1    701  10.5K  80.0M
peedypool   26.9G  7.22T      1    561  2.00K  60.4M
peedypool   27.2G  7.22T      0    748      0  85.5M
peedypool   27.5G  7.22T      0    658      0  71.8M
peedypool   27.6G  7.22T      0    802      0  92.2M
peedypool   27.9G  7.22T      0    646    255  70.2M
peedypool   28.0G  7.22T      0    744      0  83.2M
peedypool   28.3G  7.22T      0    684      0  76.3M
peedypool   28.4G  7.22T      0    723      0  78.2M
peedypool   28.7G  7.22T      0    662      0  72.3M
peedypool   28.8G  7.22T      0    742      0  82.8M
peedypool   29.1G  7.22T      0    710      0  80.0M
peedypool   29.3G  7.22T      0    690      0  75.4M
peedypool   29.5G  7.22T      0    785      0  91.5M
peedypool   29.6G  7.22T      0    629      0  64.2M
peedypool   29.9G  7.22T      0    739      0  84.0M
peedypool   30.0G  7.22T      0    663      0  71.6M
peedypool   30.3G  7.22T      0    607      0  61.3M
peedypool   30.4G  7.22T      0    792      0  91.2M
peedypool   30.7G  7.22T      0    670      0  73.1M
peedypool   30.8G  7.22T      0    754      0  85.5M
peedypool   31.1G  7.22T      0    714      0  80.4M
peedypool   31.4G  7.22T      0    705      0  75.9M
peedypool   31.5G  7.22T      0    787      0  91.7M
peedypool   31.6G  7.22T      0    628      0  65.3M
peedypool   31.9G  7.22T      0    646      0  68.0M
peedypool   32.0G  7.22T      0    735      0  80.0M
peedypool   32.2G  7.22T      0    619      0  61.7M
peedypool   32.5G  7.22T      0    746      0  86.2M
peedypool   32.7G  7.22T      0    661      0  72.0M
peedypool   32.9G  7.22T      0    762      0  88.7M
peedypool   33.1G  7.22T      0    668      0  72.3M
peedypool   33.3G  7.22T      0    797      0  92.3M
peedypool   33.4G  7.22T      0    550      0  57.7M
peedypool   33.7G  7.22T      0    519      0  50.5M
peedypool   33.8G  7.22T      0    754      0  87.4M
peedypool   34.1G  7.22T      0    603      0  63.9M
peedypool   34.2G  7.22T      0    740      0  85.3M
peedypool   34.3G  7.22T      0    550      0  58.6M
peedypool   34.6G  7.22T      0    642      0  73.4M
peedypool   34.7G  7.22T      0    660      0  72.0M
peedypool   35.0G  7.22T      0    545      0  56.2M
peedypool   35.1G  7.22T      0    696      0  80.3M
peedypool   35.2G  7.22T      0    547      0  59.9M
peedypool   35.4G  7.22T      0    480      0  47.2M
peedypool   35.6G  7.22T      0    625      0  71.3M
peedypool   35.7G  7.22T      0    606      0  65.4M
peedypool   35.8G  7.22T      0    547      0  58.6M
peedypool   36.1G  7.21T      0    459      0  48.7M


with compression (lz4)

peedypool   36.1G  7.21T      0    101      0  11.7M
peedypool   36.2G  7.21T      0    646      0  70.5M
peedypool   36.5G  7.21T      0    497      0  49.4M
peedypool   36.6G  7.21T      0    696      0  78.7M
peedypool   36.7G  7.21T      0    678      0  72.8M
peedypool   37.0G  7.21T      0    507      0  51.8M
peedypool   37.1G  7.21T      0    647      0  72.3M
peedypool   37.3G  7.21T      0    745      0  78.6M
peedypool   37.4G  7.21T      0    590      0  60.2M
peedypool   37.7G  7.21T      0    494      0  53.8M
peedypool   37.8G  7.21T      0    678      0  75.9M
peedypool   38.0G  7.21T      0    696      0  77.1M
peedypool   38.1G  7.21T      3    599  2.75K  62.4M
peedypool   38.4G  7.21T      2    505  2.50K  53.9M
peedypool   38.5G  7.21T      1    676  2.25K  74.9M
peedypool   38.7G  7.21T      2    701  2.50K  79.1M
peedypool   38.8G  7.21T      4    635  13.0K  61.8M
peedypool   39.1G  7.21T      2    509  2.50K  54.1M
peedypool   39.2G  7.21T      1    496  2.25K  54.0M
peedypool   39.4G  7.21T      2    679  2.50K  75.1M
peedypool   39.5G  7.21T      2    602  2.50K  63.9M
peedypool   39.7G  7.21T      1    305  10.5K  30.8M
peedypool   39.8G  7.21T      2    428  2.75K  46.5M
peedypool   39.9G  7.21T      1    688  2.50K  77.0M
peedypool   40.1G  7.21T      2    687  3.75K  76.3M
peedypool   40.2G  7.21T      4    589  2.75K  60.8M
peedypool   40.5G  7.21T      2    509  1.50K  54.0M
peedypool   40.6G  7.21T      1    611  1.25K  67.1M
peedypool   40.8G  7.21T      2    687  1.50K  76.7M
peedypool   40.9G  7.21T      2    700  1.50K  71.3M
peedypool   41.2G  7.21T      3    532  11.7K  54.1M
peedypool   41.3G  7.21T      1    518  1.50K  54.7M
peedypool   41.5G  7.21T      2    702  1.75K  78.2M
peedypool   41.6G  7.21T      1    699  1.50K  76.7M
peedypool   41.8G  7.21T      4    584  3.50K  59.9M
peedypool   42.0G  7.21T      1    508  1.50K  53.8M
peedypool   42.2G  7.21T      2    612  1.75K  66.9M
peedypool   42.3G  7.21T      2    718  1.75K  74.2M
peedypool   42.5G  7.21T      2    695  11.0K  75.4M
peedypool   42.8G  7.21T      3    537  3.25K  54.6M
peedypool   42.9G  7.21T      2    506  2.00K  54.5M
peedypool   43.0G  7.21T      1    602  1.25K  65.7M
peedypool   43.2G  7.21T      1    681  2.25K  74.8M
peedypool   43.3G  7.21T      1    676  2.00K  74.2M
peedypool   43.5G  7.21T      3    570  4.25K  57.2M
peedypool   43.7G  7.21T      1    578  4.50K  54.7M
peedypool   43.9G  7.21T      1    687  10.5K  71.3M
peedypool   44.0G  7.21T      2    681  12.5K  74.0M
peedypool   44.2G  7.21T      2    677  3.00K  72.2M
peedypool   44.5G  7.21T      3    536  2.25K  54.5M
peedypool   44.6G  7.21T      1    516   1023  55.2M
peedypool   44.7G  7.21T      2    797  1.50K  89.7M
peedypool   45.0G  7.21T      1    686    767  72.2M
peedypool   45.2G  7.21T      0    823      0  95.1M
peedypool   45.4G  7.21T      0    723      0  76.9M
peedypool   45.6G  7.21T      0    851      0  97.2M
peedypool   45.9G  7.21T      0    704    255  75.9M
peedypool   46.0G  7.21T      0    783      0  85.6M
peedypool   46.3G  7.20T      0    793      0  89.9M
peedypool   46.4G  7.20T      0    693      0  71.7M
peedypool   46.7G  7.20T      0    789      0  88.3M
peedypool   46.8G  7.20T      0    705      0  73.5M
peedypool   47.1G  7.20T      0    770      0  85.4M
peedypool   47.3G  7.20T      0    717      0  76.9M
peedypool   47.5G  7.20T      0    820      0  95.1M
peedypool   47.7G  7.20T      0    676      0  67.2M
peedypool   48.0G  7.20T      0    806      0  92.4M
peedypool   48.2G  7.20T      0    705      0  74.5M
peedypool   48.4G  7.20T      0    837      0  95.8M
peedypool   48.7G  7.20T      0    689      0  73.0M
peedypool   48.8G  7.20T      0    825      0  93.0M
peedypool   49.1G  7.20T      0    699      0  75.9M
peedypool   49.2G  7.20T      0    734      0  79.2M
peedypool   49.5G  7.20T      0    702      0  74.1M
peedypool   49.6G  7.20T      0    803      0  90.0M
peedypool   49.9G  7.20T      0    705      0  76.3M
peedypool   50.1G  7.20T      0    750      0  82.6M
peedypool   50.3G  7.20T      0    709      0  77.9M
peedypool   50.5G  7.20T      0    721      0  80.0M
peedypool   50.7G  7.20T      0    608      0  65.3M
peedypool   50.9G  7.20T      0    627      0  66.8M
peedypool   51.0G  7.20T      0    667      0  71.0M
peedypool   51.2G  7.20T      0    591      0  60.2M
peedypool   51.4G  7.20T      0    694      0  79.2M
peedypool   51.6G  7.20T      0    676      0  72.2M
peedypool   51.8G  7.20T      0    703      0  79.2M
peedypool   51.9G  7.20T      0    650      0  70.8M
peedypool   52.2G  7.20T      0    498      0  51.4M
peedypool   52.3G  7.20T      0    571      0  59.5M
peedypool   52.5G  7.20T      0    652      0  73.5M
peedypool   52.6G  7.20T      0    525      0  53.8M
peedypool   52.7G  7.20T      0    586      0  60.1M
peedypool   53.0G  7.20T      0    662      0  75.1M
peedypool   53.1G  7.20T      0    646      0  71.0M
peedypool   53.4G  7.20T      0    542      0  57.1M
peedypool   53.5G  7.20T      0    770      0  87.3M
peedypool   53.7G  7.20T      0    575      0  59.3M
peedypool   53.9G  7.20T      0    359      0  38.2M


I noticed the load during the compression copy peaked at about 2.86.  So could it be my CPU that is unable to do the compression quick enough and thus the speed drop?

Also, to answer your other questions.  This is 64bit ubuntu.  No dedup, nothing special set, other than case sensitivty=mixed.  Files being copies are camera raw files and quicktime movies.  Win7 box is physical box, not a vm.  Measured the speed with windows resource monitor.

Based on your results I think its the CPU, as an i7 is way faster than my AMD.

Chris



More information about the zfs-discuss mailing list