Whonix on Mac M1 (ARM) - User Support (still unsupported at time of writing)

Unsupported Platforms
13

Okay, latest update.

I’ve made some good progress. I’m now running a Debian buster VM on QEMU on the Mac M1.
I ran this to get to the installer:

qemu-system-aarch64 \
         -machine virt,accel=hvf,highmem=off \
         -cpu cortex-a72 -smp 8 -m 4G \
         -device intel-hda -device hda-output \
         -nographic \
         -device usb-ehci \
         -device usb-kbd \
         -device virtio-net-pci,netdev=net \
         -device virtio-mouse-pci \
         -netdev user,id=net,ipv6=off \
         -drive "if=pflash,format=raw,file=./edk2-aarch64-code.fd,readonly=on" \
         -drive "if=virtio,format=raw,file=./hdd.raw,discard=on" \
         -drive "file=../debian/debian-10.9.0-arm64-netinst.iso,id=cdrom,if=none,media=cdrom" \
         -device virtio-scsi-device -device scsi-cd,drive=cdrom \
         -boot order=d

followed the usual Debian installation steps. Then I can just remove the SCSI cdrom device and use QEMU to boot from the hdd.raw file itself which now contains a bootable Debian. I did have to manually get the QEMU UEFI to load the grubaa64.efi but that’s a bit beside the point… (I am keeping track of all the commands I am running, and what each part does so I can write a nice post for future M1 users).

This works much better with the grml-debootstrap package, it now finishes building in a few minutes. Thanks @Patrick! In order to get a successful run, I’ve used the below command:

sudo KERNEL='none' NOKERNEL='true' UPGRADE_SYSTEM='no' GRUB_INSTALL='no' grml-debootstrap \
        --debopt "--verbose --include=initramfs-tools,eatmydata,apt-transport-tor,python3.7,gpg,gpg-agent" \
        --arch arm64 \
        --filesystem ext4 \
        --force \
        --hostname host \
        --nopassword \
        --release buster \
        --keep_src_list \
        --verbose \
        --vmfile \
        --vmsize 25G \
        --packages ./grml_packages \
        --target ./debian.raw

I’ve made this based on these whonix build steps.

This provides me with a debian.raw file. I understand that this does not have grub (if I don’t use GRUB_INSTALL='no' the build fails), nor a kernel (same here, if I don’t pass KERNEL='none' NOKERNEL='true') the build fails.

The reason for these failures (as far as I can tell) is that grml-debootstrap does not natively support arm64.

I know that I now need to get arm64 grub and an arm64 kernel onto this VM file. So far I’ve tried two things (neither of which worked).

  1. Mount the debian.raw file and try run grub-install.
sudo  mkdir -p "/mnt/debootstrap.grub"
sudo kpartx -asv ./debian.raw
sudo mount -o rw,suid,dev "/dev/mapper/loop0p1" "/mnt/debootstrap.grub"
sudo grub-install /mnt/debootstrap.grub --target arm64-efi
sudo sync
sudo umount /mnt/debootstrap.grub
sudo kpartx -d ./debian.raw >/dev/null
sudo rmdir /mnt/debootstrap.grub
sudo chown user:user ./debian.raw

Unfortunately I didn’t see any relevant grub files in the /mnt/debootstrap.grub/boot folder.

So, then I tried…

  1. Copy the files I have from my working Debian VM’s /boot folder into the mounted Debian filesystem’s boot folder (i.e. /mnt/debootstrap.grub/boot). This meant my debian.raw file now has (what looks like) the files necessary for booting. However, running this in a similar qemu command as above yields a UEFI error.

I know UEFI generally looks for an EFI partition (I have some experience in hackintoshing). However, I’m not really use how to go about making this partition in a file. Maybe that’s the reason the QEMU UEFI BIOS cannot find the required grubaa64.efi file?

I’m still continuing my trials. Just wanted to leave another update as I progress.

Note: @HulaHoop, seems I can use edk2-aarch64-code.fd to boot arm64 debian. Working well for me so far with the Debian VM I created using the net installer. I suppose this then should work when I can get a correct build from grml-debootstrap. Thanks though!

EDIT: quick update. Figured out I need to get that new partition setup and also probably make changes to /etc/fstab.

Some helpful resources I am going to use next to try get my debian.raw booting are:

Also seems it’s not too dissimilar to what this script does: https://gitlab.com/whonix/Whonix/-/blob/master/build-steps.d/2375_build_rpi_fs#L22

So, will report back once I’ve exhausted my efforts there.

And, finally, I suppose once I do have a way of getting this working we feed that back into a new script in the build-steps.d folder.

2 Likes
14

Quick follow-up, some success!

My assumptions were correct. Once I manually added grub and the kernel, I can boot from my newly created Debian arm64 image file, using QEMU on the M1.

If anyone is curious how I did that, you can follow the gist below.

Note: this is really rough around the edges. I’ll clean it up and add comments, but first I want to get Whonix booting.

So, I guess now I need to go through the rest of the Whonix build steps and adjust as needed in order to get Whonix booting similarly on my Mac host. Thanks for all the help so far!

2 Likes
15

Or much better. Add ARM support to grml-debootstrap.

1 Like
16

Ladies and gentlemen we got somewhere.

Screenshot 2021-04-27 at 21.40.24
Screenshot 2021-04-27 at 21.40.242158×1660 438 KB

I added a new step into the build-steps.d folder which is basically a cleaned up / parameterised version of my gist above. Also needed to make a few small changes to other scripts.

Still to do:

  1. Ensure I haven’t broken builds of other archs / flavours. Do you guys have some CI actually?
  2. Get networking working between the Gateway and the Workstation using the vmnet-mac QEMU networking mode.
  3. Clean it all up, contribute back in terms of a GitHub PR and docs.

@Patrick Regarding grml-debootstrap, I will revert on that issue you created and contribute back there also. I think that might take a bit longer though, so I want to first get Whonix working fully on the M1.

1 Like
17

Alright, more progress. We have networking from the Workstation! And, also the workstation built perfectly with the changes I made to the build-steps.d scripts.

Screenshot 2021-04-28 at 03.20.32
Screenshot 2021-04-28 at 03.20.322696×1652 811 KB

I have noticed the systemcheck is failing due to the onion-grater service. But, I cannot for the life of me figure out why. When running /usr/lib/onion-grater by itself it works fine… Is this a known issue by any chance in the latest built of Whonix? I am basing my work off 15.0.1.7.2.

My QEMU commands look like this now by the way, just in case anyone else is following along. I’ll publish my build script changes soon also.

qemu-system-aarch64 \
         -machine virt,accel=hvf,highmem=off \
         -cpu cortex-a72 -smp 4 -m 2G \
         -device intel-hda -device hda-output \
         -device virtio-gpu-pci \
         -device usb-ehci \
         -device usb-kbd \
         -device usb-tablet \
         -device virtio-net-pci,netdev=external \
         -device virtio-net-pci,netdev=internal \
         -netdev user,id=external,ipv6=off,net=10.0.2.0/24 \
         -netdev socket,id=internal,listen=:8010 \
         -display cocoa \
         -drive "if=pflash,format=raw,file=./edk2-aarch64-code.fd,readonly=on" \
         -drive "if=pflash,format=raw,file=./edk2-vars-whonix.fd,discard=on" \
         -drive "if=virtio,format=raw,file=./Whonix-Gateway-XFCE.raw,discard=on"

qemu-system-aarch64 \
         -machine virt,accel=hvf,highmem=off \
         -cpu cortex-a72 -smp 8 -m 4G \
         -device intel-hda -device hda-output \
         -device virtio-gpu-pci \
         -device usb-ehci \
         -device usb-kbd \
         -device usb-tablet \
         -device virtio-net-pci,netdev=internal \
         -netdev socket,id=internal,connect=127.0.0.1:8010 \
         -display cocoa \
         -drive "if=pflash,format=raw,file=./edk2-aarch64-code.fd,readonly=on" \
         -drive "if=pflash,format=raw,file=./edk2-vars-work.fd,discard=on" \
         -drive "if=virtio,format=raw,file=./Whonix-Workstation-XFCE.raw,discard=on"

Still lots to do overall, but getting there.

1 Like
18

Great progress!

Likely caused by sandboxing parameters in onion-grater’s systemd unit file. Similar to this platform specific issue:

Try commenting out this line:

https://github.com/Whonix/onion-grater/blob/master/lib/systemd/system/onion-grater.service#L54

After changing that file:

sudo systemctl daemon-reload && sudo systemctl restart onion-grater && sudo systemctl --no-pager status onion-grater

If that doesn’t help, try out commenting out all onion-grater systemd sandboxing.

19

Used to but Travis CI is shutting down.

But there was never any free (or affordable) CI which supports sudo/root, Debian based, supporting device-mapper. All of that would be required to even build a VM image. Let alone of booting a VM and testing if it’s functional. That would be difficult due to CI’s being based on virtualization and requirement for nested virtualization. Any contribution to improve CI support would be most welcome!

20

That indeed worked, the onion-grater service now runs fine, thanks! I had the same issue with sdwdate and resolved it the same way.

I see you mentioned this is only happening on certain architectures. Would you recommend I comment out those line then for the arm64 builds of Whonix? At least this way other users wouldn’t have the issue. Obviously, we should re-enable it once we can.

I see you’ve opened a superuser issue, hopefully someone else can help us debug it, I don’t have any experience there myself.

Regarding CI, I understand the difficulty finding such a provider. We would basically need to run our own CI infrastructure to get all of those features.

1 Like
21

Opened an initial PR: https://github.com/Whonix/Whonix/pull/439 - would appreciate some early feedback, or even if someone wants to trial it already.

I want to still do some more changes:

  1. Ensure those above mentioned services will run out-of-the-box.
  2. Update docs for building and running these images on an M1 Mac.
1 Like
22

Looks good overall!

It’s OK because it’s highly unlikely to break anything existing.

Seems pretty perfect.

Some nitpicks:

cp $binary_image_raw $orig_img
mkdir --parents $mpoint_efi
rm $orig_img

Few more.

Not a big deal. I could add the nitpick on top (but then hope the quoting doesn’t break it although very unlikely).

Optional, non-blocker:

That would be Whonix for macOS: Download and Installation?

That would be

(Please disagree should I suggest some wrong stuff. I am certainly not married to any of these opinions. :))

1 Like
23

Thanks for the reply and comments! I’ve made another commit which should address those.

Regarding the RPI file, I actually implemented my checks in such a way that that build should not be affected. I wasn’t aware that it wasn’t working anymore. I can remove it in my PR if you want?

Once you’re happy with PR, feel free to merge it. I may do some follow-up PRs which get the arm64 build into better shape (currently the .qcow2 files it produces doesn’t work, but at least .raw does).

Thanks for the doc links, I’ll get editing those soon also.
Really appreciate all your feedback, no problem with disagreeing! :slight_smile:

1 Like
24

Merged! :slight_smile:

Thank you, this might be great for arm64 support generally!
(I.e. unrelated to Mac hardware)

Btw now that you’ve found your way around Whonix’s build script to the level of porting to a newer platform, perhaps you’ll have suggestions on how it could have been better structured / documented to simplify contributions / easier to understand. (Ideally in a separate forum thread.)

https://github.com/Whonix/Whonix/commit/ebcd1dda74ad06c28a094bd28919d40bc8286fed

1 Like
25

Great, thanks Patrick!

Yes, I think in its current incarnation it should work for any future arm64 machines (provided QEMU arm64 works). I do expect we’ll see arm64 chips in Linux workstations soon seeing as Linus (and others) are pushing for this.

Regarding feedback on the build scripts, I actually found them quite approachable. They are well designed such that I was able to integrate without needing to understand every one, rather I just needed to know where to slot in. The RPI one definitely helped with that.

I’d be happy to actively contribute and improve M1 support, so as I go about doing that (via the repo itself and docs), I’ll be sure to provide any further feedback I have. Thanks for you help, and great project! :slight_smile:

1 Like
26

Alright, I’ve made some changes to the wiki pages:

I’m not too used to this Wiki markdown language, so please do feel free to clean up any of the formatting. I’m sure I’ll get better at it. While I’m not super happy with the state of the docs, I think it’s better to have something in there for now at least, especially while it’s fresh on my mind and I have some time. Will update them as I go along.

For some reason I don’t see an “Edit” option on this page: Build Configuration - Whonix - I’d like to especially arm64 it under “Platforms Choice”. Is this restricted?

Thanks again!

1 Like
27

Great!

Removed protection from Build Configuration - Whonix. Can now be edited.

Whonix ™ for macOS: Download and Installation could also be split into two different pages.

  • Intel based Mac
  • M1 based Mac

Not sure that would make sense? Might depend on:

Will it be possible / is it planned to make this work with either/and/or KVM / virt-manager / VirtualBox?

1 Like
28

Oracle hasn’t announced any such plans. That doesn’t seem likely at this time at all. That could be years, if ever.

https://forums.virtualbox.org/viewtopic.php?t=98742

https://www.virtualbox.org/ticket/20192

Potential build speed up coming to mind for developer builds:

…but probably miss out on platform specific packages:
Existing Ports of and Porting Whonix to other Architectures

1 Like
29

Thanks Patrick, I updated the Build Configuration page now with some small changes.

Regarding the macOS page, yes, I think it could do with a larger restructure. I’ll tackle this probably next weekend, but didn’t have much time this weekend so just got the Apple Silicon steps in quickly so they are not only on my machine / in my head. :wink:

If VirtualBox releases ARM compatibility at some point, I would definitely port it there. It would be much more user friendly than QEMU. I’m also considering https://getutm.app/ - it uses QEMU under the hood so would be easier to get it working there and more user friendly.

KVM is a linux hypervisor, so I don’t see it ever working on macOS. HVF is the macOS implementation and the QEMU commands I added into the Whonix wiki use HVF already.

1 Like
30

That’s a handy flag for the build script, will keep it in mind, thanks!

@Patrick would it be possible to add those arm64 images to https://download.whonix.org/ at some point? I think we could then already make an “easier” version of the Apple Silicon setup.

2 Likes
31

Is in -XX-<git commit hash>.raw the git commit hash part inconvenient? (For intermediate documentation writing?) The only reason I originally implemented that is preventing users to build from git master (or other arbitrary git commits) and then wondering why their build is different from git tag releases.

File names are configurable but I guess setting an environment variable for that is also inconvenient.

Instead to clarify to users “caution, not building from a tag”, the git commit hash could be replaced by “untagged”. That would also sufficiently indicate “caution, custom build”.

(Nothing fundamentally wrong with builds from non-tags as long as knowing that.)

(Going to answer other parts later.)

32

That’s an incredible development. Thanks for creating this. It would be a pleasure for me to handle aarch64 KVM Builds. Will you be reachable for any bug troubleshooting in case something breaks down the line?

2 Likes