Whonix Desktop Installer with Calamares - field report

onion_knight · May 9, 2019, 3:18pm

So after a few days playing with Calamares and my “Whonix Desktop” VM and ISO files, I thought it would be time to report back my experience.

See also

For the discussion related to the Whonix Host/Desktop project.

In short, the results are quite encouraging.

What I did:

I built a hardened debian 15 VM on debian buster (“Whonix-Desktop VM”;
I run a few custom chroot bash scripts to customize the Whonix-Desktop VM, installed virt-manager, lib-virt, etc, copied the Gateway and Workstation VMs, installed non-free firmware and created an ISO file out of it (see link above for complete description)
On top of that I installed the Calamares installer to try it out (in buster repos)

After a few tests I have now a working ISO file that correctly installs the “Whonix-Desktop” on an external drive (also with encryption!) which is bootable and running, at least on qemu/kvm (didn’t have the opportunity to test it on real hardware yet).

So it’s pretty amazing I would say, but there are still a few shortcomings that need to be addressed.

Here are the few things I learnt while experiencing with Calamares:

I did not have time to try a UEFI install on a GPT disk, only BIOS on MBR
I found the documentation related to calamares quite sparse, but maybe because it is self explanatory to developers (the syntax is very clear)
On Debian Buster, the default calamares package does not provide configuration files for modules which must be written by hand, following examples that can be found on github
Installing the calamares-settings-debian package provides all configuration files used by the debian-live team for their new Buster live images with Calamares installer (link)
Only after installing calamares-settings-debian was I able to have a completely bootable system, so I think it would be a good idea to depart from their experience and configuration files and modify them to our needs (for instance to change the branding)
The way I understand it, the default installing sequence provided by calamares-settings-debian merely copies the content of the filesystem.squashfs file into the target and then executes the needed operations (creating new user, configuring fstab, bootloader, etc.), but does not install packages (it does remove the live-boot related packages though). A side-note: it also needs the rsync package otherwise the installer will fail.
The advantage of this is obvious, as the system is created in a matter of minutes (no need to redownload and recreates everything).
The problem is that it also copies the already existing ‘user’ and his home directory, and maybe other stuff that we don’t need, very unclean
Debian-live installer does not have this problem as its live-user is created at boot-time (its filesystem.squashfs file does not contain anything under its /home directory)
It is my understanding that they achieve this behavior by building their iso using the live-build tools which are capable of creating complete debian live-systems and have options that enable to run a lot of stuff at boot-time only (and thus these specific configurations are not copied into the target by the installer)
Not sure if we can use live-build to create a hardened Whonix ISO from scratch without having to entirely rewrite the VM build process. Lot of work.
I did find a workaround by installing the live-config package, which enables to append parameters at boot time (or apply them by default) that can reprouce the behavior of debian live-systems. I think it works by reading script files in /lib/live/config which are basically a series of test (such as if no user, then launch the live system with default debian user, etc.)
The workaround I used was “simply” to copy the user .config files into /etc/skel on the Whonix-Desktop vm and then userdel user. Now the ISO automatically launches an X session with a user user (full name: Debian Live User) that it creates at boot time (note: I also had to suppress the vboxsf group as it 1000 GUID somehow seemed to conflict with the live-config scripts, preventing them to create the live-user)
The encryption does not seem to use LVM (is it a problem?)

While it’s quite encouraging and definitely a proof of work, it is still quite experimental, and a lot of configuration work remains to be done, such as:

Better understand the calamares installer sequence (taking the calamares-settings-debian as a base) in order to configure it exactly to our needs
Make sure we don’t copy anything unwanted into the target, make sure the new user’s environment on the install target behaves exactly as it is supposed to (also correct groups, etc.)
Fine-tune the overall ISO environment so it behaves exactly as we need (does the live-user needs sudo rights? Are the settings specific to the hardened debian version environment also applied in live-mode?)

@Patrick et al. what do you think of this? Are you still interested in pursuing this matter further? If yes, what would be the next steps, how could I help further?

Below some screenshots taken from the Whonix-Host Hardened Debian ISO installing itself on an external virtual hard drive (all done in qemu/kvm):

boot screen with isolinux (BIOS mode)

Calamares installer (default debian 10 branding, I did not try to change anything yet)

rebooting into the installed system

The newly created user does not belong to the libvirt-qemu and kvm groups, there is an option to configure that in /etc/calamares/modules/users.conf I don’t remember if I modified this file accordingly or not… In any case, it is one of the settings that must be addressed

Patrick · May 9, 2019, 3:49pm

For Whonix host builds… Can we replace/skip [yes we can skip] https://github.com/Whonix/Whonix/blob/master/build-steps.d/1300_create-raw-image and run live-build instead? Rather than 1300_create-raw-image we’d have another build step that runs instead. Don’t worry about the build steps, I can sort that out.

My question is, is it possible to mount an iso image similar to it being possible mounting a raw image for read/write? Once a bootable (raw, iso, anything) “base” (just Debian) image exists, it is easy for the other Whonix build steps to take over. [which can then install a Whonix meta package]

Or has an iso to be created in on run? I.e. to we have to leave it all to live-boot and once the iso is finalized we cannot mount it, chroot it, modify it, unchroot it, unmount it, deploy it?

More answers to come later.

Looks amazing!

For sure!

Certainly not a blocker. I find without LVM even easier to grasp and recover from boot medium.

onion_knight · May 9, 2019, 6:58pm

Great!
I really think that we are close to be production ready, we have an actual proof of work, we “only” need to make sure everything works well and of course integrate everything in the host build process in a clean and professional way (I did everything by hand with dirty bash chroot scripts or directly on the cli).

Patrick:

For Whonix host builds… Can we replace/skip [yes we can skip] https://github.com/Whonix/Whonix/blob/master/build-steps.d/1300_create-raw-image and run live-build instead? Rather than 1300_create-raw-image we’d have another build step that runs instead. Don’t worry about the build steps, I can sort that out.

My question is, is it possible to mount an iso image similar to it being possible mounting a raw image for read/write? Once a bootable (raw, iso, anything) “base” (just Debian) image exists, it is easy for the other Whonix build steps to take over. [which can then install a Whonix meta package]

Or has an iso to be created in on run? I.e. to we have to leave it all to live-boot and once the iso is finalized we cannot mount it, chroot it, modify it, unchroot it, unmount it, deploy it?

I think it shouldn’t be a problem at all. live-build can be run integrally (lb build) or can be broken down into different steps that can be run separately, all of them completely customizable. It also allows for the creation of disk images. The ISO creation comes last in the build process. As quoted from man live-build:

LIVE-BUILD COMMANDS
       We divide live-build into high level  ("porcelain")  commands  and  low
       level ("plumbing") commands.

       Here  is  the  complete  list of all available live-build commands. See
       their man pages for additional documentation.

HIGH-LEVEL COMMANDS (PORCELAIN)
       We separate the porcelain commands into  the  main  commands  and  some
       ancillary user utilities.

   Main porcelain commands
       lb config(1)
           creates configuration for live-build

       lb bootstrap(1)
           executes  the  first  build  stage, creating (bootstraping) a basic
           Debian root filesystem

       lb chroot(1)
           executes the second build stage, building the live OS filesystem

       lb installer(1)
           executes the third  build  stage,  obtaining  installer  components
           (optional)

       lb binary(1)
           executes the fourth build stage, generating a binary image

       lb source(1)
           executes   the   fifth  build  stage,  generating  a  source  image
           (optional)

       lb clean(1)
           cleans up system build directories

Documentation is quite good and abundant, see for example

https://live-team.pages.debian.net/live-manual/html/live-manual/about-manual.en.html

But the question is do we need live-build? What for? In your opinion, what specific needs would cover live-build tools that we don’t currently cover with the current Whonix Host build-steps?

We already have a bootable raw image, that can be converted into a bootable BIOS/UEFI (“burnable”) ISO file with mksquashfs and xorriso commands (see my script above). What would be the purpose of live-build?

In my experience, the only real problem I was faced with was avoiding the copy of the existing user and his home directory into the target during the install part with Calamares. My solution was simply to suppress the user and his directory and let the live-config tools take care of the rest, which automatically creates a live-user. Quite basic, but functional.

If it’s not acceptable, maybe we can simply avoid creating the user during the Hardened Debian VM build? Or maybe we could play with the live-config scripts so they would ignore the existing user and his home directory and login with the live-user at boot time (just thinking out loud, I don’t know if it’s possible)? Or maybe even easier to somehow configure the Calamares installer so it would ignore the existing user and its home directory during the unpack step of the install process?

This being said, there might be a ton of things that I am missing. Such as making sure that the live system AND the installed system do honor the various hardening settings that are the main purpose of the “hardened” Whonix version. Also making sure that we don’t copy useless stuff into the target (I am thinking about logs, files owned by the initial user, etc.).

I could easily verify that with my current build, if you or anyone knowledgeable enough would provide me with a check-list of the hardened settings that we expect to be working (apparmor stuff, etc.). I must say that I didn’t take the time to real dig into what makes the hardened version special compared to a stock debian version.

Patrick · May 10, 2019, 1:30am

I would not know. I was only addressing this:

Certainly good to avoid the extra complexity of interacting with another build system we don’t need it.

You mean https://github.com/onions-knight/whonix-stuff/blob/master/ok/whonix-iso-maker.sh?

I am eager to have that build step 2550_create_iso to create a bootable iso added to Whonix source code ASP.

Then I’d implement --target iso which would run the build process as usual.

[And I would add skipping build-steps that don’t apply; 1150_export-libvirt-xml; 2375_build_rpi_fs; 2400_convert-raw-to-qcow2 2500_convert-raw-to-vdi; 2500_convert-raw-to-vdi; 2600_create-vbox-vm; 2700_create-vm-text; 2800_create-report]

Could you add licensing comment on top of the script please?
And/or add license to repository?
Or send pull request to https://github.com/Whonix/Whonix/tree/master/help-steps [with the usual license header]?
#create_squashfs_ok is currently out commented. Why’s that?
Also other functions are out commented currently which is a bit confusing.

[not sure yet if help-step or build-steps.d but it is easily moved later on]

Once it is in Whonix/help-steps I can easily make it use variables rather than hardcoded strings such as for CHROOT_FOLDER, ISO_FILE, and whatnot. Also re-use code for (un)mount-raw (if possible, sensible, probably yes).

The would be step one. A hardened debian iso which can be live booted.
Would look probably like this: sudo ~/Whonix/whonix_build --flavor hardened-debian --target iso --build

But that’s not a Whonix-Duo Installer ISO yet. So step two would be to implement --flavor whonix-duo-xfce-kvm which would install a different meta package whonix-duo-xfce-kvm. (1700_install-packages)

Therefore a new package whonix-duo-xfce-kvm needs to be added to anon-meta-packages. What packages whonix-duo-xfce-qcow2 should depend on? Could you send a pull request for anon-meta-packages please or let me know which packages it should depend on?

I guess the usual KVM dependencies qemu-kvm libvirt-daemon-system libvirt-clients virt-manager,
and calamares?
Anything else?

Then we’d have a bootable Whonix-Duo Live Installer ISO with calamares installer, KVM dependencies, but not Whonix KVM images yet. I am not sure yet how to best get the Whonix KVM images installed on Whonix-Duo (host) during the build process. See my idea to package Whonix KVM images as [Help Welcome] KVM Development - staying the course - #289 by Patrick deb packages.
sudo apt-mark hold whonix-gateway-xfce-qcow2
sudo apt-mark hold whonix-workstation-xfce-qcow2

whonix-duo-xfce-kvm package dependencies:

qemu-kvm libvirt-daemon-system libvirt-clients virt-manager
calamares
whonix-gateway-xfce-qcow2 whonix-workstation-xfce-qcow2

The “more final” build process to create Whonix-Duo-KVM would then be:

build Whonix-Gateway XFCE qcow2 / Whonix-Workstation XFCE qcow2
build packages whonix-gateway-xfce-qcow2 whonix-workstation-xfce-qcow2
build --flavor whonix-duo-xfce-kvm

(I’d automate this with a step outside of whonix_build, there’d be a wrapper calling whonix_build multiple times.)

Does that make sense so far?

Patrick · May 10, 2019, 1:59am

Sounds acceptable. [as long as the end-result is ok]
But I did not understand “suppress the user”.

Certainly doable by modification of anon-base-files.postinst. I can work on a variable that skips the creation of user user. Easy.

As a temporary measure even build config

export SKIP_SCRIPTS+=" anon-base-files.postinst "

could skip the whole anon-base-files.postinst maintainer script.

Yes.

Would not worry much. Could be called “out of scope” for now. Hardened Debian still is underdeveloped. As long as package hardened-debian-xfce is installed, the job of Whonix host is done. Anything missing in hardened debian can be called “future work”.

That would be up to Calamares. But if that happens, that is not that bad. It’s non-deterministic contents created on the user’s machine. Not during the build process of the iso.
What’s really bad is creating let’s say a random seed or Tor entry guards and shipping that in an iso which then gets shared by thousands of users.
But there might be one unsupported use case:
Boot Live; then install; and assume the iso live session activities did not leak into the install, right?

There is not too much to it yet:

onion_knight · May 10, 2019, 7:38am

Patrick:

You mean https://github.com/onions-knight/whonix-stuff/blob/master/ok/whonix-iso-maker.sh?

I am eager to have that build step 2550_create_iso to create a bootable iso added to Whonix source code ASP.

Then I’d implement --target iso which would run the build process as usual.

[And I would add skipping build-steps that don’t apply; 1150_export-libvirt-xml; 2375_build_rpi_fs; 2400_convert-raw-to-qcow2 2500_convert-raw-to-vdi; 2500_convert-raw-to-vdi; 2600_create-vbox-vm; 2700_create-vm-text; 2800_create-report]

Could you add licensing comment on top of the script please?

And/or add license to repository?

Or send pull request to https://github.com/Whonix/Whonix/tree/master/help-steps [with the usual license header]?

#create_squashfs_ok is currently out commented. Why’s that?

Also other functions are out commented currently which is a bit confusing.

Great, I’ll try to do that when I have sometime (maybe not today).

Yes, the script definitely needs some cleaning, the uncommented step is a mistake. Also the -e boot options has to be removed (it excludes the /boot directory from the copied squashfs file, Calamares needs it to make the target system bootable (it won’t reinstall the kernel by default).

I will make a list of all added packages for the “Whonix Desktop” image. Basically, it’s KVM/virt-manager dependencies + free/non-free firmware (needed for wifi support) + network manager related packages (user-friendly wifi setup) + Calamares dependencies + some theming.

I did not install these additional packages with the --no-install-recommends option, didn’t want anything to break and spend hours to debug what recommended package was missing for the sake of sparing a few hundreds MB.

On a side note, I think ideally the Calamares installer should have an option asking the user whether he would like to install non-free packages or not on the target system. If yes = keep it as it is, if no = remove the non-free packages (this is taken care of during the packages step of the installer).

I am sure it is easily doable, though I don’t know how yet.

OK

Yes

I ran userdel -r user with root

Patrick:

Certainly doable by modification of anon-base-files.postinst. I can work on a variable that skips the creation of user user . Easy.

As a temporary measure even build config
export SKIP_SCRIPTS+=" anon-base-files.postinst "
could skip the whole anon-base-files.postinst maintainer script.

OK, seems the easiest way to me right now as I have no idea how to implement all the other options I was thinking of (or if it is even possible).

Indeed, that would be really bad. But as far as I know, the live session activities do not leak as the installer only copies the filesystem.squashfs file which does not get modified by the file session.This file is just a squashfs’ed image of the .raw disk image packaged into an ISO.

So my understanding right now is that if the master .raw image is clean, then its filesystem.squashfs image should be clean too, and so should be the newly installed system. But I will do some more testing to verify this claim.

Patrick · May 10, 2019, 10:37am

On reflection hardened debian based builds should not even have anon-base-files installed. It’s not a dependency in anon-meta-packages. Hardened debian and Whonix host is going to need its own base files package.

Freedom Software purists won’t be satisfied by that. They don’t want to download any iso including anything nonfreedom in the first place let alone boot anything nonfreedom including. Therefore would be futile. Instead I’ll be providing two meta packages and two build flavors pure and impure. Relatively minor extra effort for me. [However, I probably won’t be uploading purist builds. That should be done by someone else or self-made builds.]

Anything else on there should be (no) free vs nonfree discussion, please move here:

Sounds awesome!

HulaHoop · May 10, 2019, 2:46pm

No. You’ll be missing the volume snapshotting feature that LVM provides, but that’s it. This can be compensated by having a filesystem that support this instead.

https://www.reddit.com/r/archlinux/comments/8xqujf/fulldisk_encryption_luks_vs_lvm_on_luks/

Patrick · May 12, 2019, 7:30am

https://github.com/Whonix/shared-folder-help/commit/43d28a01dbece96048d10d8b8bda5aae63213f47

Patrick · May 12, 2019, 7:35am

https://github.com/Whonix/Whonix/commit/21e53c63fd85fb0bdb16c937730103bab3698269

https://github.com/Whonix/Whonix/commit/4dad36875fe3b8d2b6f0a5024fc3626688332cc5

Patrick · May 12, 2019, 8:26am

freedom vs nonfreedom build siwtches are now implemented.

Example pure build only containing Freedom Software.

sudo ./whonix_build --target qcow2 --flavor whonix-host-xfce --freedom true --build --redistribute

results in installation of package whonix-host-xfce-kvm-freedom

vs Example containing nonfreedom software.

sudo ./whonix_build --target qcow2 --flavor whonix-host-xfce --freedom false --build --redistribute

results in installation of package whonix-host-xfce-kvm-nonfreedom

Patrick · May 12, 2019, 8:27am

https://github.com/Whonix/Whonix/commit/d74ec50c6da27cdf066cacd87ce903b39306e9f1

https://github.com/Whonix/Whonix/commit/006dfd933e7902a5cbecc386171ea777d1c5c987

https://github.com/Whonix/Whonix/commit/26802c5566ae9a97cb05d8c01c712fe486e2c354

Patrick · May 12, 2019, 8:33am

https://github.com/Whonix/Whonix/commit/61d26248c569c19b04794df2e4e54cb89b183f2a

Patrick · May 12, 2019, 8:36am

I’ll be moving as much as possible from the two new build steps to packages. First was vboxsf group issue, next done is package insatllation.

https://github.com/Whonix/Whonix/commit/7897954192072c7b7e7228a29c60195268da6866

I plan on moving as much as possible from function configure-kvm() to package GitHub - Kicksecure/libvirt-dist: Libvirt XML Files for Derivative Linux Distributions KVM - https:/www.kicksecure.com/wiki/KVM / https://www.whonix.org/wiki/KVM.

Patrick · May 12, 2019, 9:13am

https://github.com/Whonix/Whonix/commit/4a03bbe9662a92ebb6d83fddd357c8eb63c5d713

https://github.com/Whonix/whonix-libvirt/blob/master/usr/lib/whonix-libvirt/install

/usr/lib/whonix-libvirt/install is currently not (yet?) idempotent, meaning

it cannot be re-run without error
if it breaks in the middle, it cannot recover when run again

Not yet added to postinst but soon.

I am not sure what’s best. It could be easily made idempotent but then we would keep re-running its commands on each time whonix-libvirt gets upgraded. I guess the best solution is to make it idempotent but run it only at initial installation. That’s what I’ll be going for unless there are better suggestions.