Nope has nothing to do with baremetalâs IP management and everything to do with NAT network config for hypervisors. So it has to stay this way to avoid creating different settings for KVM vs Vbox.
Whonix-Host (currently running inside VirtualBox) is now networking is now by network manager. eth0 got automatically assigned 10.0.2.15.
Now Whonix-Host networking (currently running inside VirtualBox) is functional and both gateway and workstation are starting.
Speed in nested virtualization is too slow to test if networking inside VMs is functional but I will assign more RAM and try again. Can also say more when I build a new image that doesnât require all the manual changes I did during experimentation.
However, maybe likely this will work?
For QEMU support both of these shouldbe removed?
gateway (actually works without removing this):
<vcpu placement='static' cpuset='0'>1</vcpu>
workstation:
<vcpu placement='static' cpuset='1'>1</vcpu>
Removal was required for me inside VirtualBox. Otherwise I was getting cannot set CPU affinity on process
error when trying to start workstation.
Whonix tag 15.0.1.3.2-developers-only
should build.
Though, only tested with --remote-derivative-packages true
. And had a minor hiccup which hopefully was just a one time thing.
dpkg-deb (subprocess): decompressing archive member: lzma error: compressed data is corrupt
dpkg-deb: error: <decompress> subprocess returned error exit status 2
dpkg: error processing archive /var/cache/apt/archives/calamares_3.2.4-3_amd64.deb (--unpack):
cannot copy extracted data for './usr/lib/x86_64-linux-gnu/calamares/modules/keyboard/libcalamares_viewmodule_keyboard.so' to '/usr/lib/x86_64-linux-gnu/calamares/modules/keyboard/libcalamares_viewmodule_keyboard.so.dpkg-new': unexpected end of file or stream
Errors were encountered while processing:
/var/cache/apt/archives/calamares_3.2.4-3_amd64.deb
Managed to make Whonix-Host in VirtualBox coexist with KVM networking file Whonix-External.xml.
nmcli con show
NAME UUID TYPE DEVICE
virbr0 ecc61543-fa15-4a43-9732-fa99065ff801 bridge virbr0
virbr2 7b784b6c-44e4-4589-8fba-9ab6f322e810 bridge virbr2
Wired connection 1 2ed217e7-866f-3760-926a-11dedcca3182 ethernet eth0
nmcli con down Wired\ connection\ 1
Connection âWired connection 1â successfully deactivated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/1)
virsh -c qemu:///system net-start Whonix-External
Network Whonix-External started
nmcli con up Wired\ connection\ 1
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/7)
I.e. the trick is to let libvirt (whonix-libvirt-install.service
) (virsh -c qemu:///system net-start Whonix-External
) run before network manager.
Network manager can deal with it but libvirt cannot.
I donât see any solution for this with systemd drop-ins in this case because:
/lib/systemd/system/network-manager.service
has Before=network.target
while
/lib/systemd/system/libvirtd.service
has After=network.target
.
Yes can be safely ignored/removed.
A lot of progress has been done lately on the Whonix-Host project.
I think we are quite close to be release-ready, at least in beta state. I also thought a nice sum-up of what blockers/issues remain would be nice to set priorities
Whonix-Host to-do list as of May 29, 2020
We currently have 21 open issues on phabricator:
https://phabricator.whonix.org/project/view/26/
Letâs break them down (subjective triage by myself):
1. Severity: blocker
needs to be addressed before official release
-
anti-forensics / amnesia testing of Whonix-Host in Live mode
*â T910 anti-forensics / amnesia testing of Whonix-Host in Live mode
Note: this is a routine test that should be successful but that I was just too lazy to run. I will take care of it -
Whonix Host Firewall for Whonix Host
â T942 Whonix Host Firewall for Whonix Host
Note: discussed here. Donât know what remains to be done?
Whonix-Host Firewall -
instructions how to burn Whonix-Host ISO image to DVD or USB
â T969 instructions how to burn Whonix-Host ISO image to DVD or USB
Not sure if itâs technically a blocker, but should be ready for official release date. Work in progress -
add Whonix-Host EFI booting support
â T978 add Whonix-Host EFI booting support
EFI booting support for INSTALLED Whonix-Host still not working (works for ISO). No progress here. We should probably ask for upstream help (Calamares github?) -
co-install grub-efi-amd64 and grub-pc by default on Whonix-Host ISO
â T979 co-install grub-efi-amd64 and grub-pc by default on Whonix-Host ISO
Note: this is a blocker only if the possibility of installing Whonix-Host without Internet access is a requirement. Otherwise the correct grub version is downloaded by Calamares during installation
2. Severity: not a blocker
small issue/might be implemented in a future release
-
Packaging USBKill
â T552 Packaging USBKill
âIts purpose is to trigger protection events that prevents adversaries from siphoning files/installing malware/running a mouse jiggler. It creates a USB whitelist of allowed devices of which anything else plugged into the machine causes it to erase its RAM and immediately shutdown. This can be adjusted to exclude all devices.â -
Package and test wiperam for Debian
â T901 package and test wiperam for Debian -
disable removable drives auto-mounting - XFCE only
â T902 disable removable drives auto-mounting - XFCE only -
emergency shutdown on USB removal
â T905 emergency shutdown on USB removal -
encrypt Whonix-Host disk after first boot of Whonix-Host
â T906 encrypt Whonix-Host disk after first boot of Whonix-Host
Note: needed only when (and if) Whonix-Host without Calamares will be available -
resize Whonix-Host disk at first boot of Whonix-Host
â T907 resize Whonix-Host disk at first boot of Whonix-Host
Note: needed only when (and if) Whonix-Host without Calamares will be available -
installing Whonix-Host without installer (calamares)
â T909 installing Whonix-Host without installer (calamares)
Note: needed only when (and if) Whonix-Host without Calamares will be available -
grub boot password
â T940 grub boot password -
Whonix Images Quick Rebuild
â T974 Whonix Images Quick Rebuild
3. Severity: unknown (needs triage)
Maybe a blocker, maybe not, maybe a non-issue. Needs advising
-
Clock Drift Detection
â T550 Clock Drift Detection -
sdwdate connectivity check host support
â T915 sdwdate connectivity check host support -
Whonix-Host Tor configuration and anon-connection-wizard (ACW); ipv6 disable; ipv4 forward disable
â T981 Whonix-Host Tor configuration and anon-connection-wizard (ACW); ipv6 disable; ipv4 forward disable -
connect to public Tor network by default
â T983 connect to public Tor network by default
To sum-up, as of today, and pending decision on âneed triageâ issues, there are only 2 identified blocking issues:
- Implementation of Whonix-Host firewall
- Fixing EFI booting problem on installed Whonix-Host
Alternatively, to speed things up, we could decide that the inability of installing Whonix-Host on a EFI machine is not a blocker (as the main innovation we offer is the ability to boot an GRUB/EFI ISO with Whonix VMs pre-installed and pre-configured).
Of course, many other issues will probably pop up once we will have more test timeâŚ
Doesnât need to be blocked. Forensics testing would be OK for beta.
Blocker indeed. It needs to be developed.
Right.
Yes, also does not need to be a blocker. In beta testing can be called a known issue. Perhaps by then we find out or someone else finds out.
Same as above. As long as efi booting is broken, this is not too bad either.
Indeed.
Actually blocker. Without Tor host config, sdwdate and apt is broken by default.
Blocker since this needs to be documented.
Yes.
Greetings!
Coming back from a looooong pauseâŚ
But still alive and willing to complete this project.
I just built Whonix-Host from source, from stable branch (15.0.1.3.4).
Good news: what did work still works apparently (didnât run extensive tests, but on the surface looks good):
- Boot ISO in BIOS mode: OK
- Boot ISO in EFI mode: OK
- Run VM machines in ISO live mode: OK
- Install in BIOS mode: OK
- Run VM machines in host persistent (post-installation) mode: OK
âBadâ news (but expected): what didnât work, still doesnât work:
- Whonix-Host (post-installation) fails to boot in EFI, although installation reports no error
- Whonix-Host Firewall and Whonix-Host Tor configuration and anon-connection-wizard are still not implemented (as far as I know)
I have no idea how to work out the Firewall and Tor configuration issues. I hoped someone more knowledgeable would take up the challenge
Meanwhile, I will try to solve the EFI issue. I will contact the debian-live team on Gitlab, see if they can help. I will post the link here once itâs posted.
EDIT: I cannot post any issue there, only pull requests.
In the meantime, if we want the EFI installation to work, we must add the modules
sources-media
and
sources-media-unmount
in /etc/calamares/settings.conf
otherwise it will fail to download the package grub-efi-amd64
during the installation process and the installation will abort.
See pull request: Update settings.conf.dist by onions-knight ¡ Pull Request #4 ¡ Kicksecure/live-config-dist ¡ GitHub
Welcome back! We missed you around here
Yeah grub-efi
is definitely a hard dep AFAICT.
Yay! Welcome back!
Whonix-Host Firewall is rather difficult to invent. â Whonix-Host KVM Firewall
Maybe now after working on other things Iâll have new inspiration.
They use mailing lists.
https://lists.debian.org/debian-live/
Merged. Now included in 15.0.1.5.2-developers-only
.
A bit non-ideal to download during installation but we can perfect that later (somehow cache the package during build process).
EFI booting might be broken because of this. In a VM with grub-pc installed:
sudo apt install grub-efi-amd64
The following NEW packages will be installed:
efibootmgr grub-efi-amd64 grub-efi-amd64-bin grub-efi-amd64-signed mokutil shim-helpers-amd64-signed shim-signed shim-signed-common
shim-unsigned
Most of these packages seem to be Recommends:
.
When using
sudo apt install grub-efi-amd64 --no-install-recommends
The following NEW packages will be installed:
grub-efi-amd64 grub-efi-amd64-bin
These Recommends:
in Debian packaging terms might actually be required for successful EFI booting.
Since Whonix source code doesnât mention grub-efi
or other packages in the build script or in anon-meta-packages, I guess that could be why EFI booting is broken.
Thank for the warm welcoming back
Great suggestion.
I installed grub-efi-amd64-bin
(wouldnât allow me to install grub-efi-amd64
) and all its dependencies (7 additional packages, forgot to write down which ones exactlyâŚ) in the Host raw file and reburnt the ISO.
Still stuck at âMinimal BASH-like editingâŚâ / âgrub>â prompt after installation.
Itâs something really small and stupid that is messing (missing?) somewhere. We will eventually find out.
Installation works, and the Host EFI installed machine actually boots into GRUB if repaired using grub> commands or if it is booted with the ISO still in (I know, strange. Same behavior last time I checked in May).
Of course, all these tests are performed in QEMU/KVM (virt-manager) for now, but if it doesnât work here at this level it wonât work on real hardware.
New functionality added for overlayfs may be useful for live mode:
Hello and happy new year!
GOOD NEWS for EFI install mode!
I think I have FINALLY found the culprit!
-> Just tried it with version 15.0.1.5.4 (built myself).
It appears that unless specified otherwise, Calamares installation in EFI mode creates a directory on the EFI partition using the default branding name, i.e. in our case âWhonix-Hostâ:
user@host:/mnt$ ls -lR
.:
total 4
drwxr-xr-x 4 root root 4096 Jan 22 14:05 EFI
./EFI:
total 8
drwxr-xr-x 2 root root 4096 Jan 22 14:05 boot
drwxr-xr-x 2 root root 4096 Jan 22 14:05 Whonix-Host
./EFI/boot:
total 0
./EFI/Whonix-Host:
total 5212
-rwxr-xr-x 1 root root 108 Jan 22 14:05 BOOTX64.CSV
-rwxr-xr-x 1 root root 1206824 Jan 22 14:05 fbx64.efi
-rwxr-xr-x 1 root root 209 Jan 22 14:05 grub.cfg
-rwxr-xr-x 1 root root 1533296 Jan 22 14:05 grubx64.efi
-rwxr-xr-x 1 root root 1261192 Jan 22 14:05 mmx64.efi
-rwxr-xr-x 1 root root 1322936 Jan 22 14:05 shimx64.ef
But for some reason, the installed machine is unable to boot if the directory containing the bootloader files is not named âDebianâ.
It seems to be an obscure bug on which few relevant information is readily available online.
-> renaming /EFI/Whonix-Host into /EFI/Debian solves the problem.
I will upload a pull request on gitlab in a few moments.
Apart from that, didnât run extensive tries, but everything else seemed to work as expected.
All tested on KVM so far (not on real hardware).
Happy New Year to you too. I always love the moments of epiphany when finally hunting down an annoying bug. This is great progress
Yes, exactly. I wasnât very actively working on it for months, and decided to give it a try with a fresh head. Then I tried to change the name of this directory and bingo!
I am currently testing (and writing this post) using Whonix-Workstation on Whonix-Host installed on real hardware (high capacity USB stick)! Works fine.
Only thing, had to change back the domain name from qemu to kvm, as it was set to qemu for testing purposes in KVM (for nested virtualization). Just a reminder that we need to it back in the release candidate. Otherwise the VMs are unbearably slow.
See post 237 by @Patrick:
EDIT: I see that per use qemu if kvm is unavailable for easier Whonix development using ne⌠¡ Kicksecure/libvirt-dist@b77262f ¡ GitHub there should be a check during the first installed Host boot for kvm capabilities (If I understand correctly). As I used KVM with virt-manager to install Whonix-Host on the USB stick (easier to debug), that might explain why the VMs xml files were set to qemu on first boot (also in KVM). I should try to run the entire installation process on real hardware to see if kvm domain is correctly set. Then we wouldnât need to change anything.
So now as per our roadmap we âonlyâ lack Whonix-Host Firewall. Whonix-Host Tor configuration and anon-connection-wizard and we are good to go.
Any news on the development of these features? How can it be helped?
Yay!
Welcome back again and happy new year!
EDIT: I see that per use qemu if kvm is unavailable for easier Whonix development using ne⌠¡ Kicksecure/libvirt-dist@b77262f ¡ GitHub there should be a check during the first installed Host boot for kvm capabilities (If I understand correctly).
whonix-libvirt /usr/lib/whonix-libvirt/install
is started by /lib/systemd/system/whonix-libvirt-install.service
https://github.com/Whonix/whonix-libvirt/blob/master/usr/lib/whonix-libvirt/install
That script creates a .done
file:
/var/lib/whonix-libvirt/install.done
whonix-libvirt-install.service
wonât run /usr/lib/whonix-libvirt/install
again if the .done
file exists.
ConditionPathExists=!/var/lib/whonix-libvirt/install.done
If you have suggestions on how to improve that please let me know. Would be good to support this use case somehow. Why not. Installation of Whonix-Host in a VM on USB can be a good idea how beginners/testers since that guarantees really installation to USB. Internal boot disk which the user currently booted from remains unchanged. We also need to keep things comfortable for developers/testers.
( There is also a use case of sometimes running an operating system from hardware and sometimes inside a VM. Suppose you just made a full dd
backup and now want to try if the backup is functional. Assign the USB drive to a VM and boot it.)
What would be a sane way to implement this? At every boot iterate over all installed libvirt XLMâs (multiple gatewayâs, workstationâs) and change back/forth from qemu to kvm? That seems surprising / intrusive? Users who set up some non-Whonix VM to qemu to notice that these where just changed to kvm after reboot.
A bit hard to script. kvm to qemu includes âremove <pvspinlock state='on'/>
from XML fileâ but qemu to kvm would mean âre-add <pvspinlock state='on'/>
â. Re-adding is more difficult because it needs to be added in the correct position.
XML files donât support comments so comment in/out isnât possible either.
Neither we should delete the userâs local version in /etc/ and copy over the original from /usr/share because then user modifications would be lost.
So now as per our roadmap we âonlyâ lack Whonix-Host Firewall. Whonix-Host Tor configuration and anon-connection-wizard and we are good to go.
Yes.
Any news on the development of these features?
Unfortunately not.
How can it be helped?
If anyone could implement any of that, that would help.
As for Whonix-Host KVM Firewall I never got any idea how to filter traffic by âVM nameâ. Simplified: How do we allow Whonix-Gateway to use the internet but prohibit everyone else?
Ideally the user could configure a list of VM names which have networking permitted. The default list would only include Whonix-Gateway
.
More advanced:
- allow Whonix-Gateway (default)
- allow Whonix-Gateway 2 (custom, multiple Whonix-Gateway)
- allow Kicksecure VM
- allow
debian-tor
user (host operating system Tor process)
After the simplified question is solved, implementing the advanced stuff might actually be easy.