[HOME] [DOWNLOAD] [DOCS] [NEWS] [SUPPORT] [TIPS] [ISSUES] [CONTRIBUTE] [DONATE]

KVM (qcow) Compression Solution

Development
#1

I just started to test out KVM version; both the downloaded one and one I compiled from source (the one I compiled from source was missing a kernel though, I will post that in new topic).

My host system I downloaded the ‘Whonix-Gateway-8.tar.gz’ file is Ubuntu Server 14.04 running btrfs file system. When I extract the file it is 100G. Also, when I built the file from source, the final qcow2 image in ‘whonix_binary’ was also 100G.

In order to be able to untar the qcow2 file at the expected size of 2.6G and not 100G I would suggest you add one more step to the build process as shown in the example below.

Running qemu-img convert on the freshly created qcow2 image will compress it and allow it to be extracted from a tar at the 2.6G file size instead of the 100G file size. The qcow2 image itself is reduced from 100G to 2.6G as also shown and the tar.gz image is still 1.2G. As you will note we are converting a qcow2 to a qcow2 :slight_smile:

qemu-img convert -O qcow2 Whonix-Gateway-8.qcow2 Whonix-Gateway-8-downloaded.qcow2

Sample output:

# ls -l
> -rw-r--r--  1 root root 107390828544 Feb 20 18:48 Whonix-Gateway-8.qcow2
> -rw-r--r--  1 root root   1211742737 Feb 26 07:55 Whonix-Gateway-8.tar.gz

# qemu-img convert -O qcow2 Whonix-Gateway-8.qcow2 Whonix-Gateway-8-downloaded.qcow2

> -rw-r--r--  1 root root   2233729024 Mar 23 17:19 Whonix-Gateway-8-downloaded.qcow2
> -rw-r--r--  1 root root 107390828544 Feb 20 18:48 Whonix-Gateway-8.qcow2
> -rw-r--r--  1 root root   1211742737 Feb 26 07:55 Whonix-Gateway-8.tar.gz
#2

For reference, my answer is here:
https://www.whonix.org/forum/index.php/topic,203.msg1269.html#msg1269

My host system I downloaded the 'Whonix-Gateway-8.tar.gz' file is Ubuntu Server 14.04 running btrfs file system. When I extract the file it is 100G. Also, when I built the file from source, the final qcow2 image in 'whonix_binary' was also 100G.
Are you sure about this? Please run the following commands (adjust file names):

du -h --apparent-size file.img du -h file.img
And share the results here.

In order to be able to untar the qcow2 file at the expected size of 2.6G and not 100G I would suggest you add one more step to the build process as shown in the example below.

Running qemu-img convert on the freshly created qcow2 image will compress it and allow it to be extracted from a tar at the 2.6G file size instead of the 100G file size. The qcow2 image itself is reduced from 100G to 2.6G as also shown and the tar.gz image is still 1.2G. As you will note we are converting a qcow2 to a qcow2 :slight_smile:

qemu-img convert -O qcow2 Whonix-Gateway-8.qcow2 Whonix-Gateway-8-downloaded.qcow2

Sample output:

[code]

ls -l

-rw-r–r-- 1 root root 107390828544 Feb 20 18:48 Whonix-Gateway-8.qcow2
-rw-r–r-- 1 root root 1211742737 Feb 26 07:55 Whonix-Gateway-8.tar.gz

qemu-img convert -O qcow2 Whonix-Gateway-8.qcow2 Whonix-Gateway-8-downloaded.qcow2

-rw-r–r-- 1 root root 2233729024 Mar 23 17:19 Whonix-Gateway-8-downloaded.qcow2
-rw-r–r-- 1 root root 107390828544 Feb 20 18:48 Whonix-Gateway-8.qcow2
-rw-r–r-- 1 root root 1211742737 Feb 26 07:55 Whonix-Gateway-8.tar.gz
[/code]

Converting from qcow2 to qcow2 during the build process sounds like a hack. This “compression” should be possible with one command?

By the way, latest implementation can be found here:
https://github.com/Whonix/Whonix/blob/master/build-steps.d/2400_convert-img-to-qcow2
https://github.com/Whonix/whonix-developer-meta-files/blob/master/release/compress_qcow2

I am happy to entertain suggestions.

When “converting qcow2 to qcow2” I guess what we are doing is loosing these two options.

-o cluster_size=2M -o preallocation=metadata

I guess when you add these options, no size will be saved? Can you try please? Please share your “du -h --apparent-size file.img” and “du -h file.img” results.

If you’re interested why we are using these settings in the first place, see:

#3

I will provide all my results as request, most likely, tomorrow since my raid array is currently rebuilding (switching from raid 6 to raid 5 on-the-fly). I already deleted the images since I was running out of disk space but can copy them back when its complete.

#4

As promised, here are the results.

Without needing to study the results closely my conclusion is that it appears preallocation of the metadata has file system thinking its fully allocated. Since the container is only using the expected 2.6G everything seems fine. Its just a bit confusing.

Thank you for taking the time to respond to this and I hope this thread will find useful for someone else in the future as well.

## Grab Whonix Gateway from Sourceforge
# wget http://downloads.sourceforge.net/project/whonix/current/8/Whonix-Gateway-8.tar.gz

## File size of download
# ls -lh
-rw-r--r-- 1 root         root 1.2G Feb 26 07:55 Whonix-Gateway-8.tar.gz

## Un-tar; some unknown keyword?
# tar xvfz Whonix-Gateway-8.tar.gz 
tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.security.selinux'
Whonix-Gateway-8.qcow2

## Uncompressed file look really big, but is it?
# ls -lh
-rw-r--r-- 1 1000 1000 101G Feb 20 18:48 Whonix-Gateway-8.qcow2
-rw-r--r-- 1 root root 1.2G Feb 26 07:55 Whonix-Gateway-8.tar.gz

## Apparently it just looks big since du is showing sparse size
# du -h --apparent-size Whonix-Gateway-8.qcow2 && du -h --apparent-size Whonix-Gateway-8.qcow2 
101G    Whonix-Gateway-8.qcow2
2.5G    Whonix-Gateway-8.qcow2

## Ok, lets convert it as requested to see result.  
# qemu-img convert -o cluster_size=2M -o preallocation=metadata -O qcow2 Whonix-Gateway-8.qcow2 Whonix-Gateway-8-downloaded.qcow2
# ls -lh
-rw-r--r-- 1 root root 101G Mar 24 04:51 Whonix-Gateway-8-downloaded.qcow2
-rw-r--r-- 1 1000 1000 101G Feb 20 18:48 Whonix-Gateway-8.qcow2
-rw-r--r-- 1 root root 1.2G Feb 26 07:55 Whonix-Gateway-8.tar.gz

## Result is the same as extracted downloaded file; looks big but only taking up 2.5G disk space
# du -h --apparent-size Whonix-Gateway-8-downloaded.qcow2 && du -h Whonix-Gateway-8-downloaded.qcow2 
101G    Whonix-Gateway-8-downloaded.qcow2
2.5G    Whonix-Gateway-8-downloaded.qcow2

## So lets convert it without the -o cluster_size=2M -o preallocation=metadata options
# qemu-img convert -O qcow2 Whonix-Gateway-8.qcow2 Whonix-Gateway-8-downloaded-no-special-options.qcow2
-rw-r--r-- 1 root root 2.6G Mar 24 04:54 Whonix-Gateway-8-downloaded-no-special-options.qcow2
-rw-r--r-- 1 root root 101G Mar 24 04:51 Whonix-Gateway-8-downloaded.qcow2
-rw-r--r-- 1 1000 1000 101G Feb 20 18:48 Whonix-Gateway-8.qcow2
-rw-r--r-- 1 root root 1.2G Feb 26 07:55 Whonix-Gateway-8.tar.gz

# Well, 'df -h' now shows actual size used on disk
# du -h --apparent-size Whonix-Gateway-8-downloaded-no-special-options.qcow2 && du -h Whonix-Gateway-8-downloaded-no-special-options.qcow2 
2.6G   Whonix-Gateway-8-downloaded-no-special-options.qcow2
2.6G   Whonix-Gateway-8-downloaded-no-special-options.qcow2
[Imprint] [Privacy Policy] [Cookie Policy] [Terms of Use] [E-Sign Consent] [DMCA] [Contributors] [Investors] [Priority Support] [Professional Support]