Whonix for arm64 / Raspberry Pi (RPi)

Patrick · May 2, 2022, 1:31pm

https://forums.whonix.org/t/whonix-for-arm64-raspberry-pi-rpi/1788/180

endersandman89 · May 3, 2022, 6:30pm

Aw.,…

Edit to my last post:
my mistake, actually, fstab shows dev by uuid for / (root).

Patrick · August 7, 2022, 10:05am

github.com/derivative-maker/derivative-maker

The ARM final build size is too large for typical Whonix Image

opened 11:55AM - 06 Aug 22 UTC

closed 10:10AM - 07 Aug 22 UTC

duraki

Upon installing Kickstarter OS in VirtualBox and following Whonix documentation …for building phyisolated image for Raspberry Pi 3, the final build size is taking too large: ``` galaxy@devil. ~/build/derivative-binary/16.0.5.3 ls -la | grep Whonix-Gateway-RPi-16.0.5.3.arm64.raw -rw------- 1 pwned staff 107374182400 Aug 6 12:31 Whonix-Gateway-RPi-16.0.5.3.arm64.raw ``` 107GB doesn't seems normal for a typical SD Card based boot. The following is excerpt of my `history`: ``` 2 sudo echo $(hostname -I | cut -d\ -f1) $(hostname) | sudo tee -a /etc/hosts 3 sudo /usr/bin/test -x /usr/bin/test ; echo $? 4 df -h 5 git 6 sudo apt install git time curl apt-cacher-ng lsb-release fakeroot dpkg-dev fasttrack-archive-keyring 8 su - 9 apt install --no-install-recommends sudo adduser 10 sudo apt install --no-install-recommends sudo adduser 11 addgroup --system console 12 sudo apt install --no-install-recommends iucode-tool intel-microcode 13 apt install sudo git -y 15 sudo apt install sudo git -y 16 git clone --depth=1 --branch 16.0.5.3-stable --jobs=4 --recurse-submodules --shallow-submodules https://github.com/derivative-maker/derivative-maker.git 17 cd derivative-maker/ 18 sudo ./derivative-maker --target raw --flavor whonix-gateway-rpi --build --arch arm64 --kernel linux-image-arm64 --headers linux-headers-arm64 ``` Upon build completion, the derivative-binary directory is created with a versioned build from the pipeline.

Patrick · May 5, 2023, 3:21pm

github.com/derivative-maker/derivative-maker

Issue while building Whonix for arm64

opened 06:51PM - 31 Mar 23 UTC

xebastine

I was trying the latest stable release of Whonix for arch64. The following code …does not end cause of a possible error in monero : ``` git clone --depth=1 --branch 16.0.9.0-stable --jobs=4 --recurse-submodules --shallow-submodules https://github.com/Whonix/derivative-maker.git ... Cloning into '/Users/abcd/work/derivative-maker/packages/kicksecure/open-link-confirmation'... remote: Enumerating objects: 36, done. remote: Counting objects: 100% (36/36), done. remote: Compressing objects: 100% (29/29), done. remote: Total 36 (delta 2), reused 20 (delta 0), pack-reused 0 Receiving objects: 100% (36/36), 35.50 KiB | 1.42 MiB/s, done. Resolving deltas: 100% (2/2), done. Cloning into '/Users/jkattakkayamsure/work/derivative-maker/packages/kicksecure/monero-gui'... ```

Patrick · October 25, 2023, 2:33pm

grml-debootstrap now merged arm64 build support.

Patrick · December 8, 2023, 9:41am

grml-debootstrap bug "VM build failing if combining --vmefi with --arch arm64 which maybe can be worked around in derivative-maker.

github.com/grml/grml-debootstrap

VM build failing if combining `--vmefi` with `--arch arm64`

opened 09:34AM - 08 Dec 23 UTC

adrelanos

From a user perspective it is valid to combine `--vmefi` with `--arch arm64`. … ``` + mkfs.fat -n EFI /dev/mapper/loop0p1 mkfs.fat 4.2 (2021-01-31) + MKFS_OPTS=' -F -L LINUX' + einfo 'Running mkfs.ext4 -F -L LINUX on /dev/mapper/loop0p3p3' + einfon 'Running mkfs.ext4 -F -L LINUX on /dev/mapper/loop0p3p3\n' + '[' '' '!=' yes ']' + '[' einfon = ebegin ']' + printf ' %s*%s Running mkfs.ext4 -F -L LINUX on /dev/mapper/loop0p3p3\n' 'e[32;01m' 'e[0m' e[32;01m*e[0m Running mkfs.ext4 -F -L LINUX on /dev/mapper/loop0p3p3 + LAST_E_CMD=einfon + return 0 + return 0 + mkfs.ext4 -F -L LINUX /dev/mapper/loop0p3p3 mke2fs 1.47.0 (5-Feb-2023) The file /dev/mapper/loop0p3p3 does not exist and no size was specified. ``` In that case an unexpected code path will be used. `if` is `if [ -n "$VMEFI" ]; then` which means that code path will be used instead of the `if [ "$ARCH" = 'arm64' ]; then` code path. It's awesome that grml-debootstrap recently learned VMEFI and ARM64 support. Now we can iterate based on that. I don't think this should be solved on the command line parsing level, i.e. unset VMEFI if using `--arch amd64`? Seems better to refactor, simplify all the related code. But that seems a different request than fixing this. Therefore created a separate issue for that: * https://github.com/grml/grml-debootstrap/issues/258

Patrick · December 8, 2023, 9:41am

related grml-debootstrap arm64 issues:

github.com/grml/grml-debootstrap

--vmefi too small size of ESP partition / grub partition? / make size of ESP configurable (EFI)

opened 11:51AM - 20 Oct 23 UTC

adrelanos

``` if [ -n "$VMEFI" ]; then parted -s "${TARGET}" 'mklabel gpt' pa…rted -s "${TARGET}" 'mkpart ESP fat32 1MiB 101MiB' parted -s "${TARGET}" 'set 1 boot on' parted -s "${TARGET}" 'mkpart bios_grub 101MiB 102MiB' parted -s "${TARGET}" 'set 2 bios_grub on' parted -s "${TARGET}" 'mkpart primary ext4 102MiB 100%' ``` Could this lead to issues over time? Too small? As per: https://unix.stackexchange.com/questions/623304/size-of-efi-system-partition-esp calamares [uses](https://github.com/calamares/calamares/blob/calamares/src/modules/partition/partition.conf) `300M`. [Roderick W. Smith](https://www.rodsbooks.com/linux-uefi/) ([rEFIt](http://refit.sourceforge.net/) boot manager developer) recommends: > , I recommend creating an ESP that's 550MiB in size. That's why I would go with. Quote https://www.rodsbooks.com/efi-bootloaders/principles.html > Although the EFI specification is mute on the subject of the ESP's size, most OSes make it fairly small—Macs ship with 200MiB ESPs, and the Windows 7 installer creates one of just 100MiB. (That value has been raised to a bit over 200MiB on newer versions of Windows.) Some users, however, have found that some EFIs have bugs that cause problems with FAT32 ESPs that are under 512MiB (537MB) in size. One very common problem is files that can't be read by the EFI. The Linux mkdosfs command defaults to using FAT16 for partitions of up to 520MiB (546MB). Therefore, adding a margin of safety to protect against MiB/MB confusion and rounding errors, I recommend creating an ESP that's at least 550MiB in size. If you must use a smaller ESP and if you encounter mysterious problems, try converting it to FAT16; most ESPs will work fine with this, and it may eliminate your problems. On the other hand, this may cause the Windows installer to fail should you need to install this OS. https://www.rodsbooks.com/gdisk/advice.html#ESP_Sizing with more detailed reasoning: > 550 MiB. I'd like to use the grml-debootstrap created raw image as a base to create an ISO from it so it can be written to USB or DVD. (Using grml-debootstrap, grml-live or other tools.) (https://github.com/grml/grml-debootstrap/issues/215) Does that change the size requirements?

github.com/grml/grml-debootstrap

refactoring `--vmefi` `--arch arm64`

opened 09:37AM - 08 Dec 23 UTC

adrelanos

The following code should be refactored, simplified. ``` if [ -n "$VMFILE"… ]; then qemu-img create -f raw "${TARGET}" "${VMSIZE}" fi if [ -n "$VMEFI" ]; then parted -s "${TARGET}" 'mklabel gpt' parted -s "${TARGET}" 'mkpart ESP fat32 1MiB 101MiB' parted -s "${TARGET}" 'set 1 boot on' parted -s "${TARGET}" 'mkpart bios_grub 101MiB 102MiB' parted -s "${TARGET}" 'set 2 bios_grub on' parted -s "${TARGET}" 'mkpart primary ext4 102MiB 100%' else # arm64 support largely only exists for GPT if [ "$ARCH" = 'arm64' ]; then einfo "Setting up GPT partitions for arm64" parted -s "${TARGET}" 'mklabel gpt' parted -s "${TARGET}" 'mkpart EFI fat32 1MiB 10MiB' parted -s "${TARGET}" 'set 1 boot on' parted -s "${TARGET}" 'mkpart LINUX ext4 10MiB 100%' else parted -s "${TARGET}" 'mklabel msdos' if [ "$FIXED_DISK_IDENTIFIERS" = "yes" ] ; then einfo "Adjusting disk signature to a fixed (non-random) value" MBRTMPFILE=$(mktemp) dd if="${TARGET}" of="${MBRTMPFILE}" bs=512 count=1 echo -en "\\x41\\x41\\x41\\x41" | dd of="${MBRTMPFILE}" conv=notrunc seek=440 bs=1 dd if="${MBRTMPFILE}" of="${TARGET}" conv=notrunc eend $? fi parted -s "${TARGET}" 'mkpart primary ext4 4MiB 100%' parted -s "${TARGET}" 'set 1 boot on' fi fi ``` Maybe the VMEFI and ARM64 code paths could be merged. * 1) Use the same labels. `EFI` and `ESP` are currently labels. * 2) Use the same sizes. * `1MiB 101MiB` (or the larger sizes discussed in #221) * Or use variables if you prefer. (I prefer simpler code for something simple as this over complex code that saves a bit of disk space.) * 3) Use `'mkpart bios_grub 101MiB 102MiB'` and `'set 2 bios_grub on'` also for `--arm64` if that works. Not sure if there are any ARM64 legacy BIOS booting systems, but all of these changes are for code simplification purposes so this code and related code that comes later can be simplified. Also maybe the grub installation code can be refactored but perhaps that can be discussed separately. Why? This would fix or simplify the following issues: * https://github.com/grml/grml-debootstrap/issues/257 * #221

Patrick · December 22, 2023, 6:14pm

VM --arch arm64 builds might now be functional.

It might now even be possible to cross-build arm64 images while the build machine is running Intel / amd64.

I am writing “might” because I only superficially tested that is booting using QEMU and I am not using any ARM64 hardware myself. Since there is no dedicated ARM64 maintainer, this might break in the future. This is because the amount of architectures, platforms I can support is limited by being only 1 person.

While I added a (Kicksecure) CI test for arm64 builds, which will hopefully prevent the build process from breaking again and going unnoticed for a long time, the boot process might break in the future because we don’t have CI testing yet that does not only building but actual booting and testing.

related: Continuous Integration (CI) Whonix Testing - Automated Test Suite

Patrick · December 22, 2023, 8:14pm

Rasberry PI (RPI) support · Issue #114 · grml/grml-debootstrap · GitHub

Patrick · June 11, 2024, 1:29pm

Patrick · June 11, 2024, 8:13pm

The good news: “some newer” RPi (don’t ask me about the details) now support booting using UEFI (Tiano Core) and are compatible with a standard arm64 GRUB2. According to above blog post, even the standard Debian installer from debian.org can nowadays install on RPi. Raspian and Armbian are now optional for RPi.

The bad news: This generate-recipe.py (part of Debian RaspberryPiImages - Debian Wiki) build script shows that each RPi needs a few different things. Such as a different kernel.

# Arch, kernel, DTB:
if version == '1':
    arch = 'armel'
    linux = 'linux-image-rpi'
    dtb = '/usr/lib/linux-image-*-rpi/bcm*rpi-*.dtb'
elif version == '2':
    arch = 'armhf'
    linux = 'linux-image-armmp'
    dtb = '/usr/lib/linux-image-*-armmp/bcm*rpi*.dtb'
elif version in ['3', '4']:
    arch = 'arm64'
    linux = 'linux-image-arm64'
    dtb = '/usr/lib/linux-image-*-arm64/broadcom/bcm*rpi*.dtb'

Also a different wireless (WiFi) / bluetooth firmware package.

This means there can likely be no universal bootable image that works on all RPi hardware versions.

AyoXeN · August 2, 2024, 11:23am

Hi
I wanted to ask what is the current status of Whonix Gateway on RPi. Has anyone found a way to make it work? If not - what is missing? I want to help make it work but don’t know where to start.

Patrick · August 2, 2024, 2:19pm

No progress and none should be expected, unless contributed.

This is non-trivial stuff:

A tool is required that is capable of securely creating bootable RPi images that can be integrated into derivative-maker (image build script). The next required task:

Rasberry PI (RPI) support · Issue #114 · grml/grml-debootstrap · GitHub needs to be implemented upstream.

nowni · October 24, 2024, 8:25pm

I’ve managed to get Whonix with multiple workstations working on an Arm RockPro64. This device is fully supported by Debian with FOSS mainline kernel drivers.

I used libvirt/Qemu-KVM images compiled from source but with the following modifications to the XML files:

remove genid (not supported)
remove memoryBacking (not sure if this needed or supported or what it’s doing)
set type machine=‘virt’
added <os> loaders of type=‘pflash’ to /usr/share/AAVMF/AAVMF_{CODE,VARS}.fd
remove pvsinlock (not supported)
remove vmport (not supported)
remove kvmclock (not supported)
remove <pm> (ACPI S3 not supported)
remove spice/graphics video options (for a headless server system, kloak doesn’t seem supported in a virsh console but maybe that doesn’t matter for what I’m doing)
the disk is running on a HDD because SD card storage is too volatile

I set the minimum RAM requirements for the Gateway (450MB) and is working fine. I think you need a minimum of 4GB RAM total so the host and VMs have enough or else you will be swapping a lot.

Seems to work just fine for me. Not sure what the problem is for the pi, as long as you can run libvirt it shouldn’t matter what the host is running I think?

Patrick · October 25, 2024, 7:22am

This is about host operating system support. A bootable image that can run on ARM64 / RPi. A dedicated Whonix-Gateway box. For purpose of Build Documentation: Physical Isolation maybe. Perhaps a Dev/Torified Wi-Fi Hotspot - Whonix.

This is difficult as per, why, see Whonix for arm64 / Raspberry Pi (RPi) - #214 by Patrick.