[HOME] [DOWNLOAD] [DOCS] [BLOG] [SUPPORT] [TIPS] [ISSUES] [Priority Support]

Reducing size of ova images


#42

Yes this is what I am testing now.


#43

I took a closer look at the build log. virt-sparsify actually seems to use direct kernel boot with the kernel from the currently running machine + some custom initrd where the magic happens. The generated machine id is initramfs internal, at least I could not find any different or new files between the raw image before and after running virt-sparsify.
The real root is mounted but not booted. The relevant stuff for shrinking occurs during the end of the script i.e. mounting with -o discard and using fstrim. The latter one is quite similar to zerofree. See here https://ext4.wiki.kernel.org/index.php/Ext4_VM_Images.
Since in practice there is no difference in the final size between those commands it probably does not matter which one to use.


#44

Let’s stay still away from virt-sparsify. At the moment it may be ok, but we don’t know how the implementation of virt-sparsify changes over the years.

This is because we don’t have safeguards in place for sanity testing. That is automated checking if modifications to the image are made, https://www.whonix.org/wiki/Verifiable_Builds, deterministic builds.


#45

Just a little update: the new functions/scripts worked fine with the Whonix-Gateway, I am trying with the Workstation (building from scratch). It takes time. I’ll keep you updated.


#46

OK, seems to work now, except that my build attempts ALWAYS fail during the unmount-raw function of script 1700 (I have to reboot the machine and launch the remaining scripts manually)… But this is most probably unrelated to this topic.

Should I do a few more tries (maybe on a ext4 partition, not NTFS), or I can already commit the new scripts?


#47

I did a few more tries, both on NTFS and ext4 partitions.

I confirm zeroing works now! .ova files are shrunk as expected. I will update the code on git later today. Please test and review!

As usual, the 1700 script fails during unmounting (device busy)… Since it happens before the new 2350 script, and nothing else was changed, I am assuming it is not related to the zeroing function. Last time it happened I did a check with lsof and it showed that a single process, gpg-agent had actually some files still open in the Workstation image (see screenshot below). Has anyone already seen that?


#48

I put everything on GitHub. Please check and review the code.


#50

mount-raw-no-chroot shouldn’t have

+main (){
+   root_check
+   if [ "$ANON_BUILD_INSTALL_TO_ROOT" = "1" ]; then
+      true "${green}INFO: Skipping script, because
ANON_BUILD_INSTALL_TO_ROOT=1: $BASH_SOURCE${reset}"
+      exit 0
+   else
+      mount_raw_nochroot
+   fi
+}
+
+main "$@"

The main point of sourceing a file in most cases is to import a bash
function which is then used from multiple bash scripts.

(Except when using a logic "if sourced, don’t execute (up to script that
sources it. if executed (not sourced), execute it. (We have examples of
this in Whonix source code.))

So just source mount_raw_nochroot and then use it from
2350_zerofree-raw.

The duplicate source code issue is not solved yet. mount-raw /
unmount-raw still have the full copy of the source code.

Maybe the easiest would be if the mount-raw / unmount-raw would also
source mount-raw-no-chroot / unmount-raw-no-chroot and then run their
functions.


#51

Thanks for your feedback. Unfortunately I have trouble understanding what you mean here. I am afraid this is beyond my technical skills yet. I don’t know how I can help here.


#52

onion_knight:

As usual, the 1700 script fails during unmounting (device busy)… Since it happens before the new 2350 script, and nothing else was changed, I am assuming it is not related to the zeroing function. Last time it happened I did a check with lsof and it showed that a single process, gpg-agent had actually some files still open in the Workstation image (see screenshot below). Has anyone already seen that?

Attempting to fix this.

Included in 14.0.0.7.0-developers-only but haven’t finished a build
myself yet.


#53

Above works.

It works around an interesting maybe heisenbug.

+ /home/user/Whonix/help-steps/umount_kill.sh /home/user/whonix_binary/Whonix-Gateway_image/
-> Attempting to kill any processes still running in '/home/user/whonix_binary/Whonix-Gateway_image/' before un-mounting
Okay, the following pids are still running inside '/home/user/whonix_binary/Whonix-Gateway_image/', which will now be killed.
  PID TTY      STAT   TIME COMMAND
 8486 ?        S      0:00 /bin/bash /etc/X11/Xsession.d/20power_savings_disable_in_vms
 8488 ?        S      0:00 sleep 60
un-mounting /home/user/whonix_binary/Whonix-Gateway_image/dev

And also attempted to really fix the bug.

https://github.com/Whonix/power-savings-disable-in-vms/compare/093ae0438c295d678e567da31b9cbe79bf1a09f5...a8454e788cbe414c290f8edc62e9989abd1d1967#diff-52b97294a5ecaed249ad90c4623aba65


#54

zerofree is now implemented.

Please test building:
14.0.0.7.3-developers-only


#55

https://github.com/Whonix/Whonix/compare/14.0.0.7.0-developers-only...whonix:14.0.0.7.3-developers-only


#56

Maybe this bug below is related?

In Qubes R4, with sys-whonix set as updateVM, when you run sudo qubes-dom-update in dom0, you see:

/usr/lib/qubes/qubes-rpc-multiplexer: 14: /etc/profile.d/20_power_savings_disable_in_vms.sh: shopt: not found

(repeats twice)

dom0 update procedure still works though.


#57

Unrelated but fixed in git.


#58

Has been deployed.


#59

Fantastic! @Patrick thanks a lot for making this real! Sorry that I could not take it through the end. I’ll make sure to have a look on the code and see how it looks now, hopefully with time my technical skills will progress and I’ll be able to contribute more thoroughly :slight_smile:


#60

this is an excellent development. thank you @onion_knight and the whonix team.


#61

Please have a look.


#62

Qubes uses fstrim.

https://github.com/QubesOS/qubes-linux-template-builder/blob/master/cleanup_image#L24

If we run that before and/or after zerofree (or vice versa) could we further reduce Whonix image sizes?