The new tool is called pzlib, it is open source, it decompresses zlib streams in files and makes them bigger so we can compress them later with better compressors. In the end, it restores the original file. It has 3 versions, the newer ones were buggy but this one turned out to be stable and creates reproducible results. Similar tools were usually very slow. When restoring files, pzlib supports multi-threading to speed up the operation. By default it uses 50% of the available cores.
Source code and binaries of pzlib are available here:
http://fileforums.com/showthread.php?t=97888
You need to be logged in to download, you can use these:
username: bugmenot_
password: bugmenot
Let’s create the same whonix.tar file
touch -d “2000-01-01 00:00:00 UTC” gateway.ova
touch -d “2000-01-01 00:00:00 UTC” workstation.ova
7za a -ttar whonix.tar gateway.ova workstation.ova
pzlib should create a 8,69 GB whonix.pz file out of 3,89 GB
pzlib e -v whonix.tar -o whonix.pz
srep will turn this into a 2,62 GB whonix.srep file
srep -m4f -hash- whonix.pz whonix.srep
let’s compress this file with xz using 7za to get to 1,20 GB
7za a -txz whonix.xz whonix.srep
Note: It is possible to lower the final size about 50 MB more but that requires a 64 bit 7za executable when compressing due to memory requirements, which means some people wouldn’t be able to reproduce the archive (with 32 bit OS/VM or low RAM) but it doesn’t change decompression requirements much.
These are my SHA256 hashes:
whonix.tar - 3,89 GB
F2265763F18717328E10FB8FA7FBC589B6E4D8C84F3DFEBDFE69213C51108557
whonix.pz - 8,69 GB
2977DF5AF4FF54DE27EAA54B4E82BB3246F9D4C2DD0EF9C63C3C6D63B2182C29
whonix.srep - 2,62 GB
F5B787CD063F876DB2FAE7D6764A0273CD80615DABF9102249D01C9639F31DCF
whonix.xz - 1,20 GB
656C3765AF80D6EF9AF892528811DCF39E5A37ED89073009D6788E2DB3C8E58A
single step decompression without temporary files:
7za x whonix.xz -so | srep -d -hash- - - | pzlib d - -o - | 7za x -ttar -si
on my PC, it takes 8 minutes to extract the gateway.ova and workstation.ova files with this command