Whonix-Host Operating System (OS) ISO

onion_knight via Whonix Forum:

See pull request.

Merged.

Added packages as dependencies for whonix-host-xfce-kvm-freedom. Not sure it’s the right place to have them?

Not sure if perfect but good enough for now. Could be split into a
different package later on that makes clear that it’s non-essential.
Supposing these packages should only be there on an Whonix-Host XFCE
build but not inside VMs.

Do you want me to create a ticket on phabricator?

Yes.

^ Kicksecure-Hosts and Whonix-Hosts will now have grub-live installed by default. I.e. installed Whonix-Host should have the grub live boot menu entries.


When booting the host for the first time it might be interesting to boot into recovery mode. In Whonix VMs I could see that /home/user does not exist yet at that time (no pam mkhomedir run yet), which is expected. Might be interesting for installed Whonix-Host.

Using qemu-img now instead of cp --sparse=always to hopefully fix sparse file issue.

https://github.com/Whonix/Whonix/commit/7cb5ca4e7f52de5cfd7557655fc05e629e565dd1

See TODO in commit above.

Should we use these qemu-img command line parameters?

  • -p - progress - that one shouldn’t hurt.
  • -o cluster_size=2M
  • -o preallocation=metadata

Or would these cause the issue?

15.0.1.0.8-developers-only includes above fix as well as grub, /etc/issue and /etc/motd branding specific to host, gateway, workstation. Untested.

I’ve just built 15.0.1.0.8-developers-only

/etc/motd and /etc/issue are empty (no content inside, applies to Whonix-Host and gw/ws).

Besides, something went completely wrong, VMs were unbootable (“Boot failed: not a bootable disk”).

It seems that this was caused by a faulty qemu-img convert command in 1800_copy_vms_into_raw

Changing -f raw to -f qcow2 and rebuilding Whonix-Host seemed to fix the issue.

Se my pull request:
https://github.com/Whonix/Whonix/pull/434

More testing later.

1 Like

Merged, thanks! :slight_smile:

This is expected.
Changed files: https://github.com/Whonix/Whonix/commit/f59f94616188d1c3d3fa69b60ec62cdc2ea6aa19
Implementation:






motd and issue should now be dynamically created from drop-in configuration folder. Any idea why this isn’t working?

All right, I am making some progress with EFI installation (all tests done in KVM for now, not real hardware).

Previously, Calamares would fail at the very beginning, unable to create an EFI partition. Turns out it needed dosfstools to do so, which is installed now on the ISO. So fixed.

Then it failed to install grub-efi-amd64 at module bootloader. This is because it needs module sources-media in order to have a working apt sources list to download the package. So it seems we will need this module, which creates a temporary sources.list in the chrooted installed environment, as well as module sources-media-unmount which removes it at the end.

So now theoretically all works well, Calamares installs grub-efi, doesn’t complain and completes the installation. Problem: it won’t boot unless the Whonix-Host ISO is attached:

There is something broken somewhere, or conflicts with something, although Calamares installation logs don’t show anything alarming. I also did an installation with a vanilla live Debian 10 XFCE system with Calamares to compare between the two. I didn’t notice anything out of the extraordinary, logs seem pretty much the same.

I am kind of stuck at this stage. I can’t figure what’s going wrong. I will research some more, I’m sure it must be something really stupid blocking me.

EDIT: comparison of efibootmgr -v on plain Debian 10 XFCE and Whonix-Host right after Calamares installation:

Debian 10

BootCurrent: 0004
Timeout: 0 seconds
BootOrder: 0004,0002,0001,0000,0003
Boot0000* UiApp	FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(462caa21-7614-4503-836e-8ab6f4662331)
Boot0001* UEFI QEMU DVD-ROM QM00005 	PciRoot(0x0)/Pci(0x6,0x0)/Sata(0,65535,0)N.....YM....R,Y.
Boot0002* UEFI Misc Device	PciRoot(0x0)/Pci(0x8,0x0)N.....YM....R,Y.
Boot0003* EFI Internal Shell	FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(7c04a583-9e3e-4f1c-ad65-e05268d0b4d1)
Boot0004* Debian	HD(1,GPT,1155be0b-252a-46cb-9cd2-735846abcf0c,0x1000,0x96000)/File(\EFI\Debian\shimx64.efi)

Whonix-Host

BootCurrent: 0001
Timeout: 0 seconds
BootOrder: 0001,0004,0002,0000,0003
Boot0000* UiApp	FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(462caa21-7614-4503-836e-8ab6f4662331)
Boot0001* UEFI QEMU DVD-ROM QM00005 	PciRoot(0x0)/Pci(0x6,0x0)/Sata(0,65535,0)N.....YM....R,Y.
Boot0002* UEFI Misc Device	PciRoot(0x0)/Pci(0x8,0x0)N.....YM....R,Y.
Boot0003* EFI Internal Shell	FvVol(7cb8bdc9-f8eb-4f34-aaea-3ee4af6516a1)/FvFile(7c04a583-9e3e-4f1c-ad65-e05268d0b4d1)
Boot0004* Whonix-Host	HD(1,GPT,aa03e154-6750-4877-9cc5-9d1fc075184b,0x1000,0x96000)/File(\EFI\Whonix-Host\shimx64.efi)

We see that Whonix-Host bootorder doesn’t default on entry 4. But even after correcting the order manually, it still doesn’t boot…

1 Like

Still no success. I tried:

  • Removing all grub packages and configuration files from the Whonix-Host master raw file, then reburning the ISO
  • Running Calamares installer with default settings.conf of Debian (all default modules)
  • Manually changing the efi boot order
  • Manually reinstalling grub (efi) on the installed host
  • Making sure that blkid UUID correspond to grub/efibootmgr entries
  • Trying it on real hardware

All to no avail. Still stuck to the “Minimal BASH-like line editing” mode.

1 Like

I don’t know much about that stuff.

Does Whonix-Host installed have a /boot partition?

Is there file /boot/grub/grub.cfg?

Could you please compare /boot from a working EFI system with Whonix-Host installed? There are tools which can compare whole folders such as meld. Using sudo find /boot to get a list of all files and compare those working vs non-working.

Which is “50%” done. Meaning, EFI Bios boots until grub-efi.

Maybe that can help to debug. Here is a list of commands there:

Could you try please command ls?
For inspiration what other commands are useful see file /boot/grub/grub.cfg from any system that has working EFI booting.
There might also be some commands online how to manually boot.

Anyone we can ask? Grub people on IRC?

irc://irc.gnu.org/grub

Or Help-grub Info Page.

Yes. And btw live-mode works (but again, it boots only if the Whonix-Host ISO is attached).

user@host:~$ cat /boot/grub/grub.cfg 
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
  set have_grubenv=true
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="0"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}
function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

if [ x$feature_default_font_path = xy ] ; then
   font=unicode
else
insmod part_gpt
insmod ext2
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root  83cbe41d-2c9b-4986-b803-4af882749597
else
  search --no-floppy --fs-uuid --set=root 83cbe41d-2c9b-4986-b803-4af882749597
fi
    font="/usr/share/grub/unicode.pf2"
fi

if loadfont $font ; then
  set gfxmode=auto
  load_video
  insmod gfxterm
  set locale_dir=$prefix/locale
  set lang=en_US
  insmod gettext
fi
terminal_output gfxterm
if [ "${recordfail}" = 1 ] ; then
  set timeout=30
else
  if [ x$feature_timeout_style = xy ] ; then
    set timeout_style=menu
    set timeout=5
  # Fallback normal timeout code in case the timeout_style feature is
  # unavailable.
  else
    set timeout=5
  fi
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/05_debian_theme ###
set menu_color_normal=cyan/blue
set menu_color_highlight=white/blue
### END /etc/grub.d/05_debian_theme ###

### BEGIN /etc/grub.d/10_linux ###
function gfxmode {
	set gfxpayload="${1}"
}
set linux_gfx_mode=1024x768
export linux_gfx_mode
menuentry 'Whonix GNU/Linux' --class whonix --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-83cbe41d-2c9b-4986-b803-4af882749597' {
	load_video
	gfxmode $linux_gfx_mode
	insmod gzio
	if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
	insmod part_gpt
	insmod ext2
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root  83cbe41d-2c9b-4986-b803-4af882749597
	else
	  search --no-floppy --fs-uuid --set=root 83cbe41d-2c9b-4986-b803-4af882749597
	fi
	echo	'Loading Linux 4.19.0-8-amd64 ...'
	linux	/boot/vmlinuz-4.19.0-8-amd64 root=UUID=83cbe41d-2c9b-4986-b803-4af882749597 ro  spectre_v2=on spec_store_bypass_disable=on tsx=off tsx_async_abort=full,nosmt mds=full,nosmt l1tf=full,force nosmt=force kvm.nx_huge_pages=force random.trust_cpu=off intel_iommu=on amd_iommu=on efi=disable_early_pci_dma slab_nomerge slub_debug=FZP page_poison=1 mce=0 pti=on vsyscall=none extra_latent_entropy quiet resume=UUID=adfbad2f-cd8a-4fc4-a4cb-14e551d4b5f8
	echo	'Loading initial ramdisk ...'
	initrd	/boot/initrd.img-4.19.0-8-amd64
}
submenu 'Advanced options for Whonix GNU/Linux' $menuentry_id_option 'gnulinux-advanced-83cbe41d-2c9b-4986-b803-4af882749597' {
	menuentry 'Whonix GNU/Linux, with Linux 4.19.0-8-amd64' --class whonix --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.19.0-8-amd64-advanced-83cbe41d-2c9b-4986-b803-4af882749597' {
		load_video
		gfxmode $linux_gfx_mode
		insmod gzio
		if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
		insmod part_gpt
		insmod ext2
		if [ x$feature_platform_search_hint = xy ]; then
		  search --no-floppy --fs-uuid --set=root  83cbe41d-2c9b-4986-b803-4af882749597
		else
		  search --no-floppy --fs-uuid --set=root 83cbe41d-2c9b-4986-b803-4af882749597
		fi
		echo	'Loading Linux 4.19.0-8-amd64 ...'
		linux	/boot/vmlinuz-4.19.0-8-amd64 root=UUID=83cbe41d-2c9b-4986-b803-4af882749597 ro  spectre_v2=on spec_store_bypass_disable=on tsx=off tsx_async_abort=full,nosmt mds=full,nosmt l1tf=full,force nosmt=force kvm.nx_huge_pages=force random.trust_cpu=off intel_iommu=on amd_iommu=on efi=disable_early_pci_dma slab_nomerge slub_debug=FZP page_poison=1 mce=0 pti=on vsyscall=none extra_latent_entropy quiet resume=UUID=adfbad2f-cd8a-4fc4-a4cb-14e551d4b5f8
		echo	'Loading initial ramdisk ...'
		initrd	/boot/initrd.img-4.19.0-8-amd64
	}
	menuentry 'Whonix GNU/Linux, with Linux 4.19.0-8-amd64 (recovery mode)' --class whonix --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.19.0-8-amd64-recovery-83cbe41d-2c9b-4986-b803-4af882749597' {
		load_video
		gfxmode $linux_gfx_mode
		insmod gzio
		if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
		insmod part_gpt
		insmod ext2
		if [ x$feature_platform_search_hint = xy ]; then
		  search --no-floppy --fs-uuid --set=root  83cbe41d-2c9b-4986-b803-4af882749597
		else
		  search --no-floppy --fs-uuid --set=root 83cbe41d-2c9b-4986-b803-4af882749597
		fi
		echo	'Loading Linux 4.19.0-8-amd64 ...'
		linux	/boot/vmlinuz-4.19.0-8-amd64 root=UUID=83cbe41d-2c9b-4986-b803-4af882749597 ro single  spectre_v2=on spec_store_bypass_disable=on tsx=off tsx_async_abort=full,nosmt mds=full,nosmt l1tf=full,force nosmt=force kvm.nx_huge_pages=force random.trust_cpu=off intel_iommu=on amd_iommu=on efi=disable_early_pci_dma slab_nomerge slub_debug=FZP page_poison=1 mce=0 pti=on vsyscall=none extra_latent_entropy
		echo	'Loading initial ramdisk ...'
		initrd	/boot/initrd.img-4.19.0-8-amd64
	}
}

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/11_linux_live ###
function gfxmode {
	set gfxpayload="${1}"
}
set linux_gfx_mode=1024x768
export linux_gfx_mode
menuentry 'LIVE mode USER (For daily activities.) GNU/Linux, with Linux 4.19.0-8-amd64' --class live --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.19.0-8-amd64-advanced-83cbe41d-2c9b-4986-b803-4af882749597' {
	load_video
	gfxmode $linux_gfx_mode
	insmod gzio
	if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
	insmod part_gpt
	insmod ext2
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root  83cbe41d-2c9b-4986-b803-4af882749597
	else
	  search --no-floppy --fs-uuid --set=root 83cbe41d-2c9b-4986-b803-4af882749597
	fi
	echo	'Loading Linux 4.19.0-8-amd64 ...'
	linux	/boot/vmlinuz-4.19.0-8-amd64 root=/dev/disk/by-uuid/83cbe41d-2c9b-4986-b803-4af882749597 ro  spectre_v2=on spec_store_bypass_disable=on tsx=off tsx_async_abort=full,nosmt mds=full,nosmt l1tf=full,force nosmt=force kvm.nx_huge_pages=force random.trust_cpu=off intel_iommu=on amd_iommu=on efi=disable_early_pci_dma slab_nomerge slub_debug=FZP page_poison=1 mce=0 pti=on vsyscall=none extra_latent_entropy boot=live plainroot union=overlay ip=frommedia noeject nopersistence quiet resume=UUID=adfbad2f-cd8a-4fc4-a4cb-14e551d4b5f8
	echo	'Loading initial ramdisk ...'
	initrd	/boot/initrd.img-4.19.0-8-amd64
}

### END /etc/grub.d/11_linux_live ###

### BEGIN /etc/grub.d/20_linux_xen ###

### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/30_uefi-firmware ###
menuentry 'System setup' $menuentry_id_option 'uefi-firmware' {
	fwsetup
}
### END /etc/grub.d/30_uefi-firmware ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###

Seems pretty much the same. Just compared with a Debian 10 XFCE install with Calamares.

grub> ls
(proc) (hd0) (hd0,gpt3) (hd0,gpt2) (hd0,gpt1) (cd0)

This answer here allowed me to boot into Whonix-Host from minimal grub menuentry:

set prefix=(hd0,gpt1)/efi/Whonix-Host
set root=(hd0,gpt1)
insmod linux
insmod normal
normal

But it doesn’t survive reboot. sudo update-grub doesn’t help either.
My guess is that at this point there is just a little something wrong somewhere that caused a faulty grub configuration… Just need to find it.

1 Like

UEFI booting in KVM needs all kinds of tweaks to work such as enrolling the signing keys. Never got it to work

1 Like

Well it just worked out of the box with Debian 10 and Calamares installer.

2 Likes

Tails recently managed to implement EFI and Secure Boot as per:
https://tails.boum.org/news/test_4.5-rc1/

Maybe that could help?

1 Like

Great, let me have a look.

2 Likes

In fact we already have an ISO file that boots on EFI and BIOS systems.

What we could not achieve is having an EFI bootable installed Whonix-Host system. Reasons still unknown.

@Patrick, would you maybe consider already releasing Whonix-Host ISO as it is, without EFI support for Calamares installer? This way people could already start to test it and maybe we could gather some feedback on how to fix this issue.

The way it is now I fear that otherwise we will once again be stuck for a long time. I have fewer time available right now.

1 Like

Just built Whonix-Host-XFCE-15.0.1.2.2.
Bad news: encrypted install (BIOS) is broken. That means, unable to boot the system after grub menu

Volume group "luks" not found 
Cannot process volume group luks

And then
ALERT! /dev/mapper/luks-xxxxxx-xxx does not exist. Dropping to a shell!

I have NO IDEA why it’s broken. I see nothing unusual in the encrypted disk, nor during the installation process. It used to work before. Of course, I have erased all previous copies of working Whonix-Host ISO, so I couldn’t even tell which versions used to work nor compare them…

Non-encrypted BIOS install still works (both persistent and live mode).

We must have broken something somewhere during the last weeks or so…

1 Like

I think I found the reason. live-boot packages must be removed on the installed host during Calamares packages module. Otherwise Calamares will fail to update initramfs (update-initramfs is disabled live system is running without media mounted on /lib/live/mount/medium).

@Patrick did you modify this file?

As far as I remember, we used to remove all live-boot packages.

See also /usr/lib/x86_64-linux-gnu/calamares/modules/initramfs/README.md:

## initramfs module

This module is specific to Debian based distros. Post installation on Debian
the initramfs needs to be updated so as to not interrupt the boot process
with a error about fsck.ext4 not being found.

## Debian specific notes

If you're using live-build to build your ISO and setup the runtime env
make sure that you purge the live-\* packages on the target system
before running this module, since live-config dpkg-diverts update-initramfs
and can cause all sorts of fun issues.

Now the question is how do we achieve a live-bootable installed Host if we have to remove all live-build packages…

There might be a workaround here, which is calling directly mkinitramfs instead of update-initramfs

I tried replacing

return_code = target_env_call(["update-initramfs", "-k", "all", "-c",
                                   "-t"])

with

return_code = target_env_call(["mkinitramfs", "-o", "/boot/initrd.img-$(uname -r)"])

in /usr/lib/x86_64-linux-gnu/calamares/modules/initramfs/main.py

No error during Calamares install. But still no luck, won’t boot.

EDIT the command generated an initramfs wrongly named 'initrd.img-$(uname -r)'. Replacing the old initramfs with this new one works. Using “/boot/initrd.img-4.19.0-8-amd64” in main.py also worked (meaning encrypted install Whonix-Host boots in persistent and live-mode).

So it’s just a question of correct python coding I guess…

2 Likes

Ok, it’s fixed.

Now we have bootable BIOS persistent/live-mode encrypted/non-encrypted Whonix-Host Installed.

We have to modify /usr/lib/x86_64-linux-gnu/calamares/modules/initramfs/main.py as follows:

from libcalamares.utils import target_env_call
import os
release = os.uname()[2]
initrdname = "/boot/initrd.img-"
fullpath = str(initrdname+str(release))

def run():
    """ Generate an initramfs image.
    :return:
    """
    return_code = target_env_call(["mkinitramfs", "-o", fullpath])

    if return_code != 0:
        return (
            "Failed to run update-initramfs on the target",
            "The exit code was {}".format(return_code)
            )

This file is part of calamares package (not calamares-settings-debian)

Please see pull request (I put it in live-config-dist I’ll let you sort it out):

1 Like

Will answer other points soonish.

Could you please report a bug against calamares upstream?

I don’t think it’s a bug.

See /usr/lib/x86_64-linux-gnu/calamares/modules/initramfs/README.md :

## initramfs module

This module is specific to Debian based distros. Post installation on Debian
the initramfs needs to be updated so as to not interrupt the boot process
with a error about fsck.ext4 not being found.

## Debian specific notes

If you're using live-build to build your ISO and setup the runtime env
make sure that you purge the live-\* packages on the target system
before running this module, since live-config dpkg-diverts update-initramfs
and can cause all sorts of fun issues.