After much tinkering, this solution appears to be stable with the following setup: Debian 13 host, KVM with virt-manager, several WS. Does not conflict with a host-based kill switched VPN.
The issue
The session mode XMLs use a UDP tunnel for the internal network where one workstation gets one tunnel. Adding another WS means another tunnel, which consequently gives the Gateway another internal interface. The stock firewall only handles one (INT_IF="eth1"), so the WS2 traffic on eth2 gets dropped silently as a result. The old system mode fix which was a shared Whonix-Internal libvirt network does not and cannot work in session mode.
Using a shared host bridge solves this problem using a single bridge, a single internal interface, an unmodified gateway firewall and an arbitrary number of Workstations. But when qemu-bridge-helper was tried in the rootless thread (/t/rootless-virtual-machines-with-kvm-and-qemu/20952/34) , it threw an error. That error was because of the XML interface type and not the helper itself.
Using <interface type='network'> asks libvirt to look up a managed network object; since session mode doesn’t have one, the lookup fails before the helper can be invoked. Whereas <interface type='bridge'> is a different code path and it permits the use of >1 WS that all use a single Gateway.
Host Setup
The bridge has to pre-exist before libvirt can attach VMs to it. Session mode libvirt can’t make bridges so this is a one time host operation:
sudo ip link add virbr-whonix type bridge 2>/dev/null || true
sudo ip link set virbr-whonix up
The bridge must also be whitelisted for qemu-bridge-helper which should already be setuid root from qemu-system-common (ensure bridge.conf is 0644, with 0640 the helper seems to fail):
sudo mkdir -p /etc/qemu
printf 'allow virbr-whonix\n' | sudo tee /etc/qemu/bridge.conf
sudo chmod 0644 /etc/qemu/bridge.conf
On a persistent system, this has to be done once. If running from the KS live system with the VMs not set up persistently these commands are idempotent and can be ran every session / automated in a batch script.
XML Changes
For the Gateway, replace the UDP block(s) with passt (first, becomes eth0) and bridge (second, becomes eth1) like this:
<interface type='user'>
<backend type='passt'/>
<model type='virtio'/>
</interface>
<interface type='bridge'>
<source bridge='virbr-whonix'/>
<model type='virtio'/>
</interface>
Gateway sees one internal eth1 interface irrespective of how many Workstations attach to the bridge. This lets you use the stock Gateway firewall with no modifications.
Each Workstation’s XML, replace the UDP block with a bridge block:
<interface type='bridge'>
<source bridge='virbr-whonix'/>
<model type='virtio'/>
</interface>
There is no need to use <mac> elements because libvirt handles it for you.
Cloning a Workstation
To actually add a second Workstation, one must duplicate both the disk image and the XML definition.
Copy the disk image:
cp --sparse=always /path/to/Whonix-Workstation.qcow2 /path/to/Whonix-Workstation-2.qcow2
(for filesystems that dont support sparse files like exFAT, a standard cp works fine but it will write the full virtual size).
Copy and edit the XML file:
cp Whonix-Workstation.xml Whonix-Workstation-2.xml
Open Whonix-Workstation-2.xml and change:
- The
<name>tag:<name>Whonix-Workstation-2</name> - The
<source file=...>tag inside the<disk>block to point to the new.qcow2file. - Delete the
<uuid>...</uuid>line entirely. - Delete the
<mac address='...'/>line entirely (if present)
Leave the <interface type='bridge'>. Adding Workstations 3,4,…,N is just repeating this process.
Guest config
Gateway: ifupdown brings eth1 up at 10.152.152.10. Default firewall uses INT_IF=eth1 and handles all the routing, Tor binds normally, and the guest does not know nor care whether or not eth1 is wired to a UDP tunnel or a bridge. However, one caveat is that iface eth0 inet dhcp requires a DHCP client binary, which was intentionally removed to reduce the attack surface. The fix is to use systemd-networkd for eth0 only like so:
sudo sed -i '/auto eth0/s/^/#/' /etc/network/interfaces.d/30_non-qubes-whonix
sudo sed -i '/iface eth0/s/^/#/' /etc/network/interfaces.d/30_non-qubes-whonix
sudo sh -c 'printf "[Match]\nName=eth0\n\n[Network]\nDHCP=ipv4\nIPv6AcceptRA=no\n" \
> /etc/systemd/network/05-external.network'
sudo systemctl enable --now systemd-networkd
Ifupdown continues to manage eth1, networkd manages eth0 (DHCP thru passt) & the change persists.
Workstations: Require a unique IP. One must edit /etc/network/interfaces.d/30_non-qubes-whonix inside the guest:
address 10.152.152.11 --> 10.152.152.12 (WS2)
10.152.152.13 (WS3)
10.152.152.XY (WSX)
Gateway & DNS remain as 10.152.152.10. Restart networking with sudo systemctl restart networking or reboot, changes are persistent.
Miscellaneous gotchas
bridge-nf-call-iptablesshould be 0. On most systems the module shouldn’t be loaded so effectively 0. To be certain one may runsudo sysctl -w net.bridge.bridge-nf-call-iptables=0.- If bridge state appears as
UNKNOWNit should be cosmetic given that the VMs communicate just fine with one another.
Security
The shared L2 between the Workstations is about the same as other existing architectures: VBox internal network has a shared L2, system mode Whonix-Internal network has a shared L2. The Workstations can see each other at L2.
As I have mentioned, the Gateway firewall remains completely stock.
The bridge has no IP, no routing, no NAT, no DHCP, no services. There ought not be a path from any VM to the host or the internet except through the Gateway’s Tor. The bridge traffic should bypass iptables (br_netfilter unused or 0) so the host’s firewall rules cannot interfere with internal VM comms.
The qemu-bridge-helper binary is setuid root as is standard Debian packaging, it drops privileges to CAP_NET_ADMIN as soon as it creates the tap device. There is a theoretical privesc here but I judge it to be basically irrelevant.
I can’t think of any other security caveats at the moment.
Testing
From each WS as normal user:
export UWT_DEV_PASSTHROUGH=1
curl -m 25 https://check.torproject.org # expect: Tor html page
curl -m 8 10.152.152.10:9051 # expect: "510 Command filtered"
On the Gateway:
ip -br a # eth0: inet (passt), eth1: 10.152.152.10/24 (or something similar)
sudo ss -tunlp | grep -E ':9040|:5300' # two tor lines (TCP 9040, UDP 5300)
systemctl is-active tor@default # active
On the host:
bridge link show # should be one tap per VM, all master virbr-whonix
ip addr show virbr-whonix # no inet (isolated)
Should work perfectly and allow multiple WS with qemu:///session without headaches.
