[HOME] [DOWNLOAD] [DOCS] [NEWS] [SUPPORT] [TIPS] [ISSUES] [Priority Support]

Reducing size of ova images


#44

Let’s stay still away from virt-sparsify. At the moment it may be ok, but we don’t know how the implementation of virt-sparsify changes over the years.

This is because we don’t have safeguards in place for sanity testing. That is automated checking if modifications to the image are made, https://www.whonix.org/wiki/Verifiable_Builds, deterministic builds.


#45

Just a little update: the new functions/scripts worked fine with the Whonix-Gateway, I am trying with the Workstation (building from scratch). It takes time. I’ll keep you updated.


#46

OK, seems to work now, except that my build attempts ALWAYS fail during the unmount-raw function of script 1700 (I have to reboot the machine and launch the remaining scripts manually)… But this is most probably unrelated to this topic.

Should I do a few more tries (maybe on a ext4 partition, not NTFS), or I can already commit the new scripts?


#47

I did a few more tries, both on NTFS and ext4 partitions.

I confirm zeroing works now! .ova files are shrunk as expected. I will update the code on git later today. Please test and review!

As usual, the 1700 script fails during unmounting (device busy)… Since it happens before the new 2350 script, and nothing else was changed, I am assuming it is not related to the zeroing function. Last time it happened I did a check with lsof and it showed that a single process, gpg-agent had actually some files still open in the Workstation image (see screenshot below). Has anyone already seen that?


#48

I put everything on GitHub. Please check and review the code.


#50

mount-raw-no-chroot shouldn’t have

+main (){
+   root_check
+   if [ "$ANON_BUILD_INSTALL_TO_ROOT" = "1" ]; then
+      true "${green}INFO: Skipping script, because
ANON_BUILD_INSTALL_TO_ROOT=1: $BASH_SOURCE${reset}"
+      exit 0
+   else
+      mount_raw_nochroot
+   fi
+}
+
+main "$@"

The main point of sourceing a file in most cases is to import a bash
function which is then used from multiple bash scripts.

(Except when using a logic "if sourced, don’t execute (up to script that
sources it. if executed (not sourced), execute it. (We have examples of
this in Whonix source code.))

So just source mount_raw_nochroot and then use it from
2350_zerofree-raw.

The duplicate source code issue is not solved yet. mount-raw /
unmount-raw still have the full copy of the source code.

Maybe the easiest would be if the mount-raw / unmount-raw would also
source mount-raw-no-chroot / unmount-raw-no-chroot and then run their
functions.


#51

Thanks for your feedback. Unfortunately I have trouble understanding what you mean here. I am afraid this is beyond my technical skills yet. I don’t know how I can help here.


#52

onion_knight:

As usual, the 1700 script fails during unmounting (device busy)… Since it happens before the new 2350 script, and nothing else was changed, I am assuming it is not related to the zeroing function. Last time it happened I did a check with lsof and it showed that a single process, gpg-agent had actually some files still open in the Workstation image (see screenshot below). Has anyone already seen that?

Attempting to fix this.

Included in 14.0.0.7.0-developers-only but haven’t finished a build
myself yet.


#53

Above works.

It works around an interesting maybe heisenbug.

+ /home/user/Whonix/help-steps/umount_kill.sh /home/user/whonix_binary/Whonix-Gateway_image/
-> Attempting to kill any processes still running in '/home/user/whonix_binary/Whonix-Gateway_image/' before un-mounting
Okay, the following pids are still running inside '/home/user/whonix_binary/Whonix-Gateway_image/', which will now be killed.
  PID TTY      STAT   TIME COMMAND
 8486 ?        S      0:00 /bin/bash /etc/X11/Xsession.d/20power_savings_disable_in_vms
 8488 ?        S      0:00 sleep 60
un-mounting /home/user/whonix_binary/Whonix-Gateway_image/dev

And also attempted to really fix the bug.

https://github.com/Whonix/power-savings-disable-in-vms/compare/093ae0438c295d678e567da31b9cbe79bf1a09f5...a8454e788cbe414c290f8edc62e9989abd1d1967#diff-52b97294a5ecaed249ad90c4623aba65


#54

zerofree is now implemented.

Please test building:
14.0.0.7.3-developers-only


#55

https://github.com/Whonix/Whonix/compare/14.0.0.7.0-developers-only...whonix:14.0.0.7.3-developers-only


#56

Maybe this bug below is related?

In Qubes R4, with sys-whonix set as updateVM, when you run sudo qubes-dom-update in dom0, you see:

/usr/lib/qubes/qubes-rpc-multiplexer: 14: /etc/profile.d/20_power_savings_disable_in_vms.sh: shopt: not found

(repeats twice)

dom0 update procedure still works though.


#57

Unrelated but fixed in git.


#58

Has been deployed.


#59

Fantastic! @Patrick thanks a lot for making this real! Sorry that I could not take it through the end. I’ll make sure to have a look on the code and see how it looks now, hopefully with time my technical skills will progress and I’ll be able to contribute more thoroughly :slight_smile:


#60

this is an excellent development. thank you @onion_knight and the whonix team.


#61

Please have a look.


#62

Qubes uses fstrim.

https://github.com/QubesOS/qubes-linux-template-builder/blob/master/cleanup_image#L24

If we run that before and/or after zerofree (or vice versa) could we further reduce Whonix image sizes?


#63

Hi Patrick. I didn’t know this method. I can give it a try.
Cheers


#64

Hello

Back to the forum after a long summer pause…

I did a try with a Whonix-Gateway (current, 14) build and did not notice any difference.

I proceeded as follows:

  • I ran manually steps 1100 to 2300
  • After rebooting the machine I mounted the .raw image using /home/user/Whonix/help-steps/mount-raw
  • Then I ran sudo /sbin/fstrim -v /home/user/Whonix-Gateway_image
  • After that I proceeded with the last build-steps as usual

The final .ova size is 937 MB, exactly the same size as without the fstrim command.

By the way, I notice that using the current github version results in a 14.0.0.8.1-19 version, whereas the official download version is currently 14.0.0.7.4, and only weighs 850 MB. Is there any substantial difference between the two versions, and why this size increase?

One more little thing: the values for .ova files must be updated here according to their new value: https://www.whonix.org/wiki/Template:DownloadTable#