Thanks, added a ticket:
https://phabricator.whonix.org/T790
It’s my first ticket, hope it’s OK.
I posted it against Whonix_15, but couldn’t it be also applied to Whonix 13 and 14? I think the solution is quite straightforward, and the benefits very tangible.
onion_knight:
Thanks, added a ticket:
⚓ T790 Reducing the size of raw files
It’s my first ticket, hope it’s OK.
Thanks!
I posted it against Whonix_15, but couldn’t it be also applied to Whonix 13 and 14?
Whonix 13 is already released. No grave changes after release.
Whonix 14 is overdue. Whonix development is understaffed. I want to
avoid introducing new possibly breaking bugs (even if just build
process) and building new downloadable Whonix images just for this.
The right time for this is during Whonix 15 development.
onion_knight:
Not sure what you mean by “patching Whonix build process”
Contributing to Whonix development and changing the source files as
required to add this feature.
onion_knight:
So I did a fresh build with the
--target rawoption in a virtual debian 8 i386 machine.Once the raw files were ready, I manually ran the following commands (against both .raw images):
sudo losetup -f -P <whonix-*.raw> sudo zerofree $(sudo losetup -j <whonix-*.raw> | awk '{print $1}' | cut -d: -f 1)p1 sudo losetup -DAfter that, I manually launched the
2500_convert-raw-to-vdi, 2600_create-vbox-vmand2700_export-vmscripts against the previously zeroed raw images. Note that I had to comment the line
sudo -u "$user_name" VBoxManage modifyvm "$VMNAME" --synthcpu on
in the 2600 script, as I ran into a weird fatal error:
VBoxManage: error: Unknown option: --synthcpu
But I assume it is unrelated to this topic (maybe related to the 32 bits VM?).
Your Whonix build script sources are probably outdated. Not using that
option anymore.
## Hide hosts CPU info. This does not have a GUI option.
## Was removed from VirtualBox.
## https://phabricator.whonix.org/T408
#sudo $SUDO_OPTS VBoxManage modifyvm "$VMNAME" --synthcpu on
The end result is quite impressive:
Whonix-Gateway-13.0.0.1.4.ova: now 862.8MB instead of 1.8GB!
Whonix-Workstation-13.0.0.1.4.ova: now 990MB instead of 2.1GB!
Wow, excellent!
Moreover, the KVM files seem to also benefit from the zeroing of the raw images. The Whonix-Gateway qcow2 file is reduced to a 826MB tar.gz file (instead of the 1.3GB distributed .xz) . But I don’t know how actual distributed Whonix KVM images are being produced
(they are .xz files, I don’t know how to make that?).
prepare_release
https://github.com/Whonix/whonix-developer-meta-files/blob/master/release/prepare_release in function libvirt_compress.
./packages/whonix-developer-meta-files/release/prepare_release
–flavor whonix-gateway --target qcow2 --build
For the peace of mind I imported these .ova files into VirtualBox and tested them. Everything worked as usual, only VBoxGuest Additions didn’t seem to work. Again, probably unrelated bug to this topic, but did I miss something during the build process? It’s my first time building Whonix.
Yes, probably unrelated. Probably fixed in current source code version.
I was less successful when trying to add these manual instructions into the
2500_convert-raw-to-vdi script:
Better a separate build step.
OK
Don’t know, I used the following link (current version):
https://www.whonix.org/wiki/Dev/Build_Documentation/13_full
OK, I’ll try this way.
Agreed.
Also, could you confirm the build process runs an apt-get clean in both images before ending the chroot? I think I have seen it somewhere but can’t remember where exactly. This of course also significantly reduces the images size.
Imho the best way to zero the disk space would be before running the scripts to convert raw to qcow2 or vdi. So either in 2300_run-chroot-scripts-post-d or make an extra step like 2310_zero or something like that.
In the 2300 script one of the last steps is to unmount the image. If you zero the space before unmounting it should result in lower sizes for both the qcow2 and the vdi image.
For testing you could just use the zerofree command or something like
dd if=/dev/zero of=/path/to/chroot/zero.img
rm of=/path/to/chroot/zero.img
The virt-sparsify command should also do the same directly on the raw image after unmounting or as part of the 2400 or 2500 script.
In the 1700 script is an apt-clean step.
For development (for next version, not released version), one should use
the master branch, not Whonix 13 branch.
onion_knight:
Also, could you confirm the build process runs an
apt-get cleanin both images before ending the chroot? I think I have seen it somewhere but can’t remember where exactly. This of course also significantly reduces the images size.
here (and a lot more cleanup):
https://github.com/Whonix/Whonix/blob/master/build-steps.d/2300_run-chroot-scripts-post-d
runs these.
A good new build step name would be:
2350_shrink-raw
Then it would run at the perfect time - so far the theory!
zerofree alternatives that don’t require mounting beforehand - if these
are (almost) equally efficient - are strongly preferred. Much simpler.
Less risk of too many mount/umount and running in some upstream bug
because of this.
If virt-sparsify works, that would be great. It’s part of libguestfs-tools.
There is an outdated comment in Whonix source code on libguestfs-tools
which I will remove now. (Whonix build script is no longer used to build
Qubes-Whonix.)
## XXX: No longer installing libguestfs-tools.
## Breaks Non-Qubes-Whonix builds within Qubes-Whonix.
## https://github.com/QubesOS/qubes-issues/issues/1974
So if there is a tool providing same shrinking functionality in another
package fewer dependencies that would be even better. But
libguestfs-tools would be acceptable too.
Master branch = Whonix 14 as in here?
https://www.whonix.org/wiki/Dev/Build_Documentation/14_full
Regarding zeroing, this is my understanding so far:
dd command needs a lot of time and space. It will claim all the available space until the image reaches its limits (i.e. 100 GB here). The underlying host needs to be able to allocate this space.
zerofree is faster and doesn’t involve image growing but needs mounting the image (as block device or read only)
virt-sparsify seems by far the best method. It does not need mounting as it can be run directly against the image (virt-sparsify --in-place <file.raw>) and is very fast (like less than a minute).
I did a try with virt-sparsify, manually, on a raw Whonix-Gateway 13 image:
virt-sparsify --in-place Whonix-Gateway-13.0.0.1.4.raw
I then run manually the 2500, 2600 and 2700 scripts against the zeroed raw image.
Result: 862.8 MB for Whonix-Gateway-13.0.0.1.4.ova, which is exactly the same result as with the zerofree method! So I would say we should go for it!
Of course, needs more testing, integration in scripts, and running against the master branch images (both for virtualbox and kvm).
@Patrick
libguestfs-tools does not need to be installed in Whonix. Only in the building machine.
EDIT:
Result for Whonix-Workstation 13 (ova):
990MB (again same result as with zerofree!)
virt-sparsify execution time for Whonix-Gateway:
| real | 0m20.425s |
|---|---|
| user | 0m10.052s |
| sys | 0m1.332s |
virt-sparsify execution time for Whonix-Workstation (don’t know why it went faster although the image is bigger):
| real | 0m11.290s |
|---|---|
| user | 0m8.244s |
| sys | 0m1.180s |
on a debian8 i386 VirtualBox VM, 4096 RAM, 1 CPU
virt-sparsify then. 
onion_knight:
@Patrick
libguestfs-tools does not need to be installed in Whonix. Only in the building machine.
Yes, for sure.
Always the tip of the development. So at the moment yes.
I did a test with Whonix 14 and virt-sparsify:
Question: the build scripts produce .ova and .raw files with long seemingly random names, like Whonix-Gateway-14.0.0.6.9-13-g58ed9c2b63c8baddc3ebb0b65fbdd9c50d679cf7.ova. Is it normal?
This is a first proposal for 2350_shrink-raw. Needs reviewing, editing, probably adding debugging/error handling capacities? So far it’s working:
#!/bin/bash
## Copyright (C) 2012 - 2018 ENCRYPTED SUPPORT LP <adrelanos@riseup.net>
## See the file COPYING for copying conditions.
set -x
set -e
true "INFO: Currently running script: $BASH_SOURCE $@"
MYDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
cd "$MYDIR"
cd ..
cd help-steps
source pre
source colors
source variables
shrink_raw() {
virt-sparsify --in-place "$binary_image_raw"
}
main() {
if [ "$WHONIX_BUILD_FLAVOR" = "whonix-gateway" ]; then
shrink_raw
elif [ "$WHONIX_BUILD_FLAVOR" = "whonix-workstation" ]; then
shrink_raw
elif [ "$WHONIX_BUILD_FLAVOR" = "whonix-custom-workstation" ]; then
shrink_raw
else
error "ERROR: Invalid WHONIX_BUILD_FLAVOR $WHONIX_BUILD_FLAVOR. Please report this bug!"
fi
}
main "$@"
Note: the whonix-build script often fails during the umount-raw part of 1700_install-packages (umount: /home/user/whonix_binary/Whonix-Workstation_image: target is busy). For the Whonix-Workstation I had to do all the other steps manually…
EDIT:
libvirt.xz (KVM) image for Workstation is now 746.3 MB, Gateway is 605.2 MB
onion_knight:
I did a test with Whonix 14 and virt-sparsify:
- Whonix-Gateway.ova: 934MB instead of 1.8GB
- Whonix-Workstation.ova: 1.1GB instead of 2.2GB
Terrific!
Question: the build scripts produce .ova and .raw files with long seemingly random names, like Whonix-Gateway-14.0.0.6.9-13-g58ed9c2b63c8baddc3ebb0b65fbdd9c50d679cf7.ova. Is it normal?
Yes.
The trailing 13 in Whonix-Gateway-14.0.0.6.9-13 means “13 commits
after git tag 14.0.0.6.9” with git head at git commit has
g58ed9c2b63c8baddc3ebb0b65fbdd9c50d679cf7.
It’s to make sure no one builds a tag and claims it’s version so and so
while it’s not a tagged version.
The issue goes away once a newer git tag gets created. Really unrelated.
No worries. Really normal. (The code for setting version and file names
is in help-steps/variables.)
onion_knight:
This is a first proposal for
2350_shrink-raw. Needs reviewing,
Looks really good! Without having tested it, I don’t see anything wrong
with it at first look.
probably adding debugging
I guess no need or not possible.
I would expect the virt-sparsify command to work every time and when it
fails (such as if file not exists…?), it exits non-zero, right? Should
be enough for now. If it makes trouble, which I doubt, we can think
about this more.
error handling capacities?
No need. This is generally sorted out already. (As a fallback:)
set -e
Primarily:
source pre
Note: the whonix-build script often fails during the umount-raw part of 1700_install-packages (
umount: /home/user/whonix_binary/Whonix-Workstation_image: target is busy). For the Whonix-Workstation I had to do all the other steps manually…
Probably unrelated? Please create a new thread for this.
Store the output of ps aux (or similar) before running Whonix build
script. Run Whonix build script. When this happens, run ps aux again.
Compare. See if there is some process still running inside the chroot?
If fear not… I fear Whonix build script does nothing wrong.
Try rebooting after building a VM. Sometimes it happens when mounting /
unmounting “too often”.
There is some obscure bug in kpartx and I doubt it was ever really fixed:
Nice! What is the procedure now? Do I suggest this script on phabricator? Or do we continue the discussion here?
It exits non-zero if file doesn’t exist, yes. The command can be run in verbose mode (-v option) which might be helpful. Note that virt-sparsify can be run against any kind of file, it doesn’t seem to check if the file is a virtual disk image. Not sure of the implications if the file is not a virtual disk, probably none (it runs normally and then quits with “Sparsify in-place operation completed with no errors”, although the shrinking is not performed.
Yes, but I was thinking more of the checks performed in the main() function ( if [ "$WHONIX_BUILD_FLAVOR" = "whonix-gateway" ]; then ), do we need to add more checking?
Regarding the unmounting bug, I will try to give a few more tries and open a new thread. But maybe worth taking into account the following:
For now, the only solution I found is to reboot the machine… It only happened with the Workstation, not t he Gateway. With and without the added 2350 script.
If you want more output and sanity checking:
pv to invent a progresspv to see other use examples of pv)error. (Might happen inBut all of that is very optional. (Meaning: can merge either way even as
is.) Seems pretty good. Perhaps I’ll include even in Whonix 14.
Maybe even decreases build times (libvirt compression)? (Unimportant
question.)
Decreases upload time due to smaller sizes, and download times for
everyone, which is amazing!
onion_knight:
Nice! What is the procedure now? Do I suggest this script on phabricator? Or do we continue the discussion here?
A phabricator ticket would be good as reminder (used as todo list).
Description can be minimal. With link to this forum discussion.
Script on phabricator: no need.
Script:
Could you post a github pull request (or git push a git branch somewhere
(github if you don’t mind)) please?
Discussion can continue here.
It exits non-zero if file doesn’t exist, yes. The command can be run in verbose mode (-v option) which might be helpful.
-v sounds good in theory. How does the output look like?
Users love some sort of progress indicator. Otherwise they have a
nervous ctrl+c finger.
Note that virt-sparsify can be run against any kind of file, it doesn’t seem to check if the file is a virtual disk image. Not sure of the implications if the file is not a virtual disk, probably none (it runs normally and then quits with “
Sparsify in-place operation completed with no errors”, although the shrinking is not performed.
Ok. Can live with that.
Yes, but I was thinking more of the checks performed in the main() function (
if [ "$WHONIX_BUILD_FLAVOR" = "whonix-gateway" ]; then), do we need to add more checking?
Ah. I guess no. Should be quite similar to
2300_run-chroot-scripts-post-d. So if it is as good as that one,
should be alright.
Regarding the unmounting bug, I will try to give a few more tries and open a new thread. But maybe worth taking into account the following:
- All the build is done in a virtual machine (virtualbox)
- The VM vdi is located on an external hdd which is formatted in NTFS (maybe kpartx doesnt like that?)
This might make a difference indeed even if it should not.
- the VM already has a user “user” with password “changeme”
For now, the only solution I found is to reboot the machine… It only happened with the Workstation, not t he Gateway.
With and without the added 2350 script.
Yes. I doubt the new script is fault.
Ok, then I’ll let it as it is.
It seems to decrease the .ova build time, probably also libvirt. But it needs more thorough benchmarking. From the end user side, ova importation is however much faster, as it is twice lighter.
I don’t know how to do that yet. I’ll look into the documentation.
virt-sparsify does have a built-in progress indicator:
Regarding the verbose output. It is indeed very verbose. Please tell me where I can upload the output file, it will be more convenient for you to read than copy-pasting all the lines here. For the peace of mind, I think it ought to be reviewed by you and/or other Whonix devs.
Output log shows that the raw image seems to be booted with qemu-system-x86_64 during the shrinking operation. Is it bad news? Does it leave logs? Is there anything dangerous taking place? Does it change anything inside the virtual disk apart from removing unused space? How can one verify that?
Here is the qemu-system command used by virt-sparsify --in-place:
/usr/bin/qemu-system-x86_64 \
-global virtio-blk-pci.scsi=off \
-nodefconfig \
-enable-fips \
-nodefaults \
-display none \
-machine accel=kvm:tcg \
-m 500 \
-no-reboot \
-rtc driftfix=slew \
-no-hpet \
-global kvm-pit.lost_tick_policy=discard \
-kernel /var/tmp/.guestfs-1000/appliance.d/kernel \
-initrd /var/tmp/.guestfs-1000/appliance.d/initrd \
-object rng-random,filename=/dev/urandom,id=rng0 \
-device virtio-rng-pci,rng=rng0 \
-device virtio-scsi-pci,id=scsi \
-drive file=/home/user/whonix_binary/Whonix-Workstation-14.0.0.6.9-13-g58ed9c2b63c8baddc3ebb0b65fbdd9c50d679cf7.raw,cache=writeback,discard=unmap,format=raw,id=hd0,if=none \
-device scsi-hd,drive=hd0 \
-drive file=/var/tmp/.guestfs-1000/appliance.d/root,snapshot=on,id=appliance,cache=unsafe,if=none,format=raw \
-device scsi-hd,drive=appliance \
-device virtio-serial-pci \
-serial stdio \
-device sga \
-chardev socket,path=/run/user/1000/libguestfsgb8EHA/guestfsd.sock,id=channel0 \
-device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
-append 'panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=xterm-256color'
NOTE: I have imported in VirtualBox the Whonix-Workstation ova file produced after virt-sparsify and inspected the content of /var/log. I see no logs other than the ones produced on first boot with VirtualBox. wtmp log only shows the first boot with VirtualBox, nothing seems to have been left by the qemu-system command run by virt-sparsify. Where else should I look?
Shrinking (without booting during shrinking) could even help with Verifiable Builds - Whonix (if it is deterministic).
Will for sure be reviewed / tested by me after git branch or pull request. (Even without can’t let this drop on the floor but would really help the “usual” way.)
Very bad news. - If it boots the actual Whonix raw image and not a separate helper image or so. Maybe it boots it in a way not modifying anything.
Yes.
Quite possibly. Could result in starting services such as the systemd entropy that creates a seed file, which should not be published and/or shared among all Whonix users. In worst case starts Tor service and prepopulates /var/lib/tor. Booting the image and making changes really ought to be avoided - very unprofessional.
We can look inside"the images. Compare files and contents. Build script with --report true to have it analyzed. Build twice. Compare the reports.
Or manually build twice and then use help-steps/analyze_image.
Context:
Where could I upload the output log so you can have a look on it?
phabricator. But no need. Since the image gets mounted we ought to analyze the image anyhow.
Any tool that doesn’t boot the image to shrink it?