Reducing size of ova images

EDIT by Patrick:

For users, if you want to reclaim space, reduce disk size, see user documetation:


Original post:

It is possible to seriously reduce the size of the vanilla ova images of both Whonix-Gateway and Whonix-Workstation VM. In VirtualBox, the easiest way is to import the ova as usual, fire them up with a live-boot ISO of any Linux distributions (without networking for the paranoids), and then to run

zerofree /dev/sda1

as root in the console (/dev/sda1 being the Whonix partition).

Back in the host, convert the vmdk file to a vdi file and compact the disk:

vboxmanage clonehd <whonix*.vmdk> <whonix*.vdi> --format VDI
vboxmanage modifyhd <whonix*.vdi> --compact

And then in VirtualBox, replace the vmdk disks by the new vdi disks and reexport the image appliances.

Current sizes after this (and with both images up-to-date!*):
Whonix-Gateway > 1.1 GB instead of 1.7 GB
Whonix-Workstation > 1.3 GB instead of 2.0 GB

Is there any security issue that I am not aware of? Otherwise I think it might be a good idea to reduce the size of the ova files that are available for download. Especially considering that many are probably downloading them over Tor.

*After running apt-get clean in both VM

2 Likes

Please create a ticket against Whonix_15 on phabricator.whonix.org.

Can you patch Whonix build process to add that step?

We however must not boot the image and then redistribute. (Very unclean.
Creates logs, entropy seeds and non-determinism [related to reproducible
builds].)

Instead I guess it should be possible to run zerofree has to be run
against the image directly (against the block device). Not sure against
the raw image is enough or if against the vdi image would be required.
(In the latter case I don’t know there is a tool available to get a
blockdevice.)

Hi Patrick,
I’ve just created an account on phabricator (Onion_Knight). Waiting for your approval.
Not sure what you mean by “patching Whonix build process”, but I am currently building Whonix with a few more insructions to zerofree the raw images as block devices. I’ll write the results here.

1 Like

Your account has been approved! If you have any problems logging in you can ask for help in this thread.

https://forums.whonix.org/t/phabricator-account-sign-ups-now-needs-manual-confirmation/

1 Like

So I did a fresh build with the --target raw option in a virtual debian 8 i386 machine.

Once the raw files were ready, I manually ran the following commands (against both .raw images):

sudo losetup -f -P <whonix-*.raw>
sudo zerofree $(sudo losetup -j <whonix-*.raw> | awk '{print $1}' | cut -d: -f 1)p1
sudo losetup -D

After that, I manually launched the 2500_convert-raw-to-vdi, 2600_create-vbox-vm and 2700_export-vm scripts against the previously zeroed raw images.

Note that I had to comment the line
sudo -u "$user_name" VBoxManage modifyvm "$VMNAME" --synthcpu on
in the 2600 script, as I ran into a weird fatal error:
VBoxManage: error: Unknown option: --synthcpu
But I assume it is unrelated to this topic (maybe related to the 32 bits VM?).
_EDIT: OK, this is normal as the feature is not supported anymore in VBox 5 . âš“ T408 --synthcpu was removed from VirtualBox, use --cpuid-portability-level or --cpuidremoveall?

The end result is quite impressive:
Whonix-Gateway-13.0.0.1.4.ova: now 862.8MB instead of 1.8GB!
Whonix-Workstation-13.0.0.1.4.ova: now 990MB instead of 2.1GB!

Moreover, the KVM files seem to also benefit from the zeroing of the raw images. The Whonix-Gateway qcow2 file is reduced to a 826MB tar.gz file (instead of the 1.3GB distributed .xz) . But I don’t know how actual distributed Whonix KVM images are being produced (they are .xz files, I don’t know how to make that?).

For the peace of mind I imported these .ova files into VirtualBox and tested them. Everything worked as usual, only VBoxGuest Additions didn’t seem to work. Again, probably unrelated bug to this topic, but did I miss something during the build process? It’s my first time building Whonix.

I was less successful when trying to add these manual instructions into the 2500_convert-raw-to-vdi script:

   sudo -u "$user_name" \
 losetup -f -P "$binary_image_raw" 
   sudo -u "$user_name" \
       zerofree $(losetup -j "$binary_image_raw"  | awk '{print $1}' | cut -d: -f 1)p1 
   sudo -u "$user_name" \
       losetup -D 
   sudo -u "$user_name" \
      VBoxManage convertfromraw "$binary_image_raw" "$HOMEVAR/VirtualBox VMs/$VMNAME/$VMNAME.vdi"

It always resulted in a failure (permission denied). I don’t know why I ran into a permission denied error as the script is run with sudo rights. I tried using kpartx as in the mount-raw script but without success.

Unfortunately, I am afraid my limited bash script skills do not allow me to further patch the build process by myself without great pain. But I am sure someone more advanced will figure it out quickly, and I’ll be happy to learn how!

2 Likes

Thanks, added a ticket:

https://phabricator.whonix.org/T790

It’s my first ticket, hope it’s OK.

I posted it against Whonix_15, but couldn’t it be also applied to Whonix 13 and 14? I think the solution is quite straightforward, and the benefits very tangible.

2 Likes

onion_knight:

Thanks, added a ticket:

âš“ T790 Reducing the size of raw files

It’s my first ticket, hope it’s OK.

Thanks!

I posted it against Whonix_15, but couldn’t it be also applied to Whonix 13 and 14?

Whonix 13 is already released. No grave changes after release.

Whonix 14 is overdue. Whonix development is understaffed. I want to
avoid introducing new possibly breaking bugs (even if just build
process) and building new downloadable Whonix images just for this.

The right time for this is during Whonix 15 development.

onion_knight:

Not sure what you mean by “patching Whonix build process”

Contributing to Whonix development and changing the source files as
required to add this feature.

onion_knight:

So I did a fresh build with the --target raw option in a virtual debian 8 i386 machine.

Once the raw files were ready, I manually ran the following commands (against both .raw images):

sudo losetup -f -P <whonix-*.raw>
sudo zerofree $(sudo losetup -j <whonix-*.raw> | awk '{print $1}' | cut -d: -f 1)p1
sudo losetup -D

After that, I manually launched the 2500_convert-raw-to-vdi, 2600_create-vbox-vm and 2700_export-vm scripts against the previously zeroed raw images. Note that I had to comment the line
sudo -u "$user_name" VBoxManage modifyvm "$VMNAME" --synthcpu on
in the 2600 script, as I ran into a weird fatal error:
VBoxManage: error: Unknown option: --synthcpu
But I assume it is unrelated to this topic (maybe related to the 32 bits VM?).

Your Whonix build script sources are probably outdated. Not using that
option anymore.

   ## Hide hosts CPU info. This does not have a GUI option.
   ## Was removed from VirtualBox.
   ## https://phabricator.whonix.org/T408
   #sudo $SUDO_OPTS VBoxManage modifyvm "$VMNAME" --synthcpu on

The end result is quite impressive:
Whonix-Gateway-13.0.0.1.4.ova: now 862.8MB instead of 1.8GB!
Whonix-Workstation-13.0.0.1.4.ova: now 990MB instead of 2.1GB!

Wow, excellent!

Moreover, the KVM files seem to also benefit from the zeroing of the raw images. The Whonix-Gateway qcow2 file is reduced to a 826MB tar.gz file (instead of the 1.3GB distributed .xz) . But I don’t know how actual distributed Whonix KVM images are being produced

(they are .xz files, I don’t know how to make that?).

prepare_release

https://github.com/Whonix/whonix-developer-meta-files/blob/master/release/prepare_release in function libvirt_compress.

./packages/whonix-developer-meta-files/release/prepare_release

–flavor whonix-gateway --target qcow2 --build

For the peace of mind I imported these .ova files into VirtualBox and tested them. Everything worked as usual, only VBoxGuest Additions didn’t seem to work. Again, probably unrelated bug to this topic, but did I miss something during the build process? It’s my first time building Whonix.

Yes, probably unrelated. Probably fixed in current source code version.

I was less successful when trying to add these manual instructions into the 2500_convert-raw-to-vdi script:

Better a separate build step.

OK

Don’t know, I used the following link (current version):

https://www.whonix.org/wiki/Dev/Build_Documentation/13_full

OK, I’ll try this way.

Agreed.

Also, could you confirm the build process runs an apt-get clean in both images before ending the chroot? I think I have seen it somewhere but can’t remember where exactly. This of course also significantly reduces the images size.

Imho the best way to zero the disk space would be before running the scripts to convert raw to qcow2 or vdi. So either in 2300_run-chroot-scripts-post-d or make an extra step like 2310_zero or something like that.
In the 2300 script one of the last steps is to unmount the image. If you zero the space before unmounting it should result in lower sizes for both the qcow2 and the vdi image.
For testing you could just use the zerofree command or something like

dd if=/dev/zero of=/path/to/chroot/zero.img
rm of=/path/to/chroot/zero.img

The virt-sparsify command should also do the same directly on the raw image after unmounting or as part of the 2400 or 2500 script.
In the 1700 script is an apt-clean step.

For development (for next version, not released version), one should use
the master branch, not Whonix 13 branch.

onion_knight:

Also, could you confirm the build process runs an apt-get clean in both images before ending the chroot? I think I have seen it somewhere but can’t remember where exactly. This of course also significantly reduces the images size.

here (and a lot more cleanup):

https://github.com/Whonix/whonix-initializer/blob/master/usr/lib/anon-dist/chroot-scripts-post.d/80_cleanup#L328

https://github.com/Whonix/Whonix/blob/master/build-steps.d/2300_run-chroot-scripts-post-d
runs these.

A good new build step name would be:

2350_shrink-raw

Then it would run at the perfect time - so far the theory!

  • 2300_run-chroot-scripts-post-d
  • 2350_shrink-raw
  • 2400_convert-raw-to-qcow2
  • 2500_convert-raw-to-vdi

zerofree alternatives that don’t require mounting beforehand - if these
are (almost) equally efficient - are strongly preferred. Much simpler.
Less risk of too many mount/umount and running in some upstream bug
because of this.

If virt-sparsify works, that would be great. It’s part of libguestfs-tools.

There is an outdated comment in Whonix source code on libguestfs-tools
which I will remove now. (Whonix build script is no longer used to build
Qubes-Whonix.)

  ## XXX: No longer installing libguestfs-tools.
  ##      Breaks Non-Qubes-Whonix builds within Qubes-Whonix.
  ##      https://github.com/QubesOS/qubes-issues/issues/1974

So if there is a tool providing same shrinking functionality in another
package fewer dependencies that would be even better. But
libguestfs-tools would be acceptable too.

Master branch = Whonix 14 as in here?
https://www.whonix.org/wiki/Dev/Build_Documentation/14_full

Regarding zeroing, this is my understanding so far:

  • dd command needs a lot of time and space. It will claim all the available space until the image reaches its limits (i.e. 100 GB here). The underlying host needs to be able to allocate this space.

  • zerofree is faster and doesn’t involve image growing but needs mounting the image (as block device or read only)

  • virt-sparsify seems by far the best method. It does not need mounting as it can be run directly against the image (virt-sparsify --in-place <file.raw>) and is very fast (like less than a minute).

I did a try with virt-sparsify, manually, on a raw Whonix-Gateway 13 image:

virt-sparsify --in-place Whonix-Gateway-13.0.0.1.4.raw

I then run manually the 2500, 2600 and 2700 scripts against the zeroed raw image.
Result: 862.8 MB for Whonix-Gateway-13.0.0.1.4.ova, which is exactly the same result as with the zerofree method! So I would say we should go for it!

Of course, needs more testing, integration in scripts, and running against the master branch images (both for virtualbox and kvm).

@Patrick
libguestfs-tools does not need to be installed in Whonix. Only in the building machine.

EDIT:
Result for Whonix-Workstation 13 (ova):
990MB (again same result as with zerofree!)

virt-sparsify execution time for Whonix-Gateway:

real 0m20.425s
user 0m10.052s
sys 0m1.332s

virt-sparsify execution time for Whonix-Workstation (don’t know why it went faster although the image is bigger):

real 0m11.290s
user 0m8.244s
sys 0m1.180s

on a debian8 i386 VirtualBox VM, 4096 RAM, 1 CPU

1 Like

virt-sparsify then. :slight_smile:

onion_knight:

@Patrick
libguestfs-tools does not need to be installed in Whonix. Only in the building machine.

Yes, for sure.

Always the tip of the development. So at the moment yes.

I did a test with Whonix 14 and virt-sparsify:

  • Whonix-Gateway.ova: 934MB instead of 1.8GB
  • Whonix-Workstation.ova: 1.1GB instead of 2.2GB

Question: the build scripts produce .ova and .raw files with long seemingly random names, like Whonix-Gateway-14.0.0.6.9-13-g58ed9c2b63c8baddc3ebb0b65fbdd9c50d679cf7.ova. Is it normal?

This is a first proposal for 2350_shrink-raw. Needs reviewing, editing, probably adding debugging/error handling capacities? So far it’s working:

#!/bin/bash

## Copyright (C) 2012 - 2018 ENCRYPTED SUPPORT LP <adrelanos@riseup.net>
## See the file COPYING for copying conditions.

set -x
set -e

true "INFO: Currently running script: $BASH_SOURCE $@"

MYDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

cd "$MYDIR"
cd ..
cd help-steps

source pre
source colors
source variables

shrink_raw() {

    virt-sparsify --in-place "$binary_image_raw"
}

main() {

    if [ "$WHONIX_BUILD_FLAVOR" = "whonix-gateway" ]; then
      shrink_raw
    elif [ "$WHONIX_BUILD_FLAVOR" = "whonix-workstation" ]; then
      shrink_raw
    elif [ "$WHONIX_BUILD_FLAVOR" = "whonix-custom-workstation" ]; then
      shrink_raw
    else
       error "ERROR: Invalid WHONIX_BUILD_FLAVOR $WHONIX_BUILD_FLAVOR. Please report this bug!"
    fi
}   

main "$@"

Note: the whonix-build script often fails during the umount-raw part of 1700_install-packages (umount: /home/user/whonix_binary/Whonix-Workstation_image: target is busy). For the Whonix-Workstation I had to do all the other steps manually…

EDIT:
libvirt.xz (KVM) image for Workstation is now 746.3 MB, Gateway is 605.2 MB

onion_knight:

I did a test with Whonix 14 and virt-sparsify:

  • Whonix-Gateway.ova: 934MB instead of 1.8GB
  • Whonix-Workstation.ova: 1.1GB instead of 2.2GB

Terrific!

Question: the build scripts produce .ova and .raw files with long seemingly random names, like Whonix-Gateway-14.0.0.6.9-13-g58ed9c2b63c8baddc3ebb0b65fbdd9c50d679cf7.ova. Is it normal?

Yes.

The trailing 13 in Whonix-Gateway-14.0.0.6.9-13 means “13 commits
after git tag 14.0.0.6.9” with git head at git commit has
g58ed9c2b63c8baddc3ebb0b65fbdd9c50d679cf7.

It’s to make sure no one builds a tag and claims it’s version so and so
while it’s not a tagged version.

The issue goes away once a newer git tag gets created. Really unrelated.
No worries. Really normal. (The code for setting version and file names
is in help-steps/variables.)

onion_knight:

This is a first proposal for 2350_shrink-raw. Needs reviewing,

Looks really good! Without having tested it, I don’t see anything wrong
with it at first look.

probably adding debugging

I guess no need or not possible.

I would expect the virt-sparsify command to work every time and when it
fails (such as if file not exists…?), it exits non-zero, right? Should
be enough for now. If it makes trouble, which I doubt, we can think
about this more.

error handling capacities?

No need. This is generally sorted out already. (As a fallback:)

set -e

Primarily:

source pre

Note: the whonix-build script often fails during the umount-raw part of 1700_install-packages (umount: /home/user/whonix_binary/Whonix-Workstation_image: target is busy). For the Whonix-Workstation I had to do all the other steps manually…

Probably unrelated? Please create a new thread for this.

Store the output of ps aux (or similar) before running Whonix build
script. Run Whonix build script. When this happens, run ps aux again.
Compare. See if there is some process still running inside the chroot?

If fear not… I fear Whonix build script does nothing wrong.

Try rebooting after building a VM. Sometimes it happens when mounting /
unmounting “too often”.

There is some obscure bug in kpartx and I doubt it was ever really fixed:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=734794

Nice! What is the procedure now? Do I suggest this script on phabricator? Or do we continue the discussion here?

It exits non-zero if file doesn’t exist, yes. The command can be run in verbose mode (-v option) which might be helpful. Note that virt-sparsify can be run against any kind of file, it doesn’t seem to check if the file is a virtual disk image. Not sure of the implications if the file is not a virtual disk, probably none (it runs normally and then quits with “Sparsify in-place operation completed with no errors”, although the shrinking is not performed.

Yes, but I was thinking more of the checks performed in the main() function ( if [ "$WHONIX_BUILD_FLAVOR" = "whonix-gateway" ]; then ), do we need to add more checking?

Regarding the unmounting bug, I will try to give a few more tries and open a new thread. But maybe worth taking into account the following:

  • All the build is done in a virtual machine (virtualbox)
  • The VM vdi is located on an external hdd which is formatted in NTFS (maybe kpartx doesnt like that?)
  • the VM already has a user “user” with password “changeme”

For now, the only solution I found is to reboot the machine… It only happened with the Workstation, not t he Gateway. With and without the added 2350 script.