Rootless virtual machines with KVM and QEMU

KVM
1

cc @Patrick @HulaHoop

2 Likes
[Help Welcome] KVM Development - staying the course
No Internet Connection inside Whonix-Workstation KVM with NordVPN with Kill-Switch on Host
2

It’s possible but iirc the network stack of rootless KVM (qemu:///session) only support very basic networking option with SLiRP. Can it be configured to run Whonix as-is?

Possibly related:

1 Like
3

Also worth noting is that, at least on Ubuntu, even with QEMU/KVM connection (Libvirt uri: qemu:///system ) QEMU aren’t run as root, but as a separate user “libvirt-qemu”.

1 Like
4

Lets see, hopefully one day qubes-os gonna be having KVM based qubes which gonna help alot.

5

You can change the user in the libvirt config. Even auto chown for self-generating files like snapshots or images is possible.

By default it uses root though, if I’m not mistaken. At least on Debian I believe.

1 Like
6

Pretty sure that’s not the case. The Qemu processes are run as user libvirt-qemu on my computer as we speak. You can ps -ef |grep qemu on host to see if this was true.

7

This is looks like a must in case of Kicksecure, as the current way of importing Whonix to KVM cant be done on user side (nor sysmaint):

8

Severity: Critical.

This makes KVM port more or less unsupported.

The absence of porting KVM to rootless makes Whonix KVM (and Kicksecure KVM) incompatible with a default Kicksecure installation, which comes with user-sysmaint-split.

Right now:

  • Installation commands: Such as sudo virsh -c qemu:///system net-define Whonix_external*.xml maybe can be run in sysmaint session.
  • Start commands: Such as sudo virsh start Whonix-Gateway shouldn’t be run in sysmaint session and cannot be run in user session.

In effect, Kicksecure hosts are incompatible unless unrestricted admin mode is enabled,which would be silly to suggest, since KVM seems to support rootless.

Hence porting to rootless is crucial.

Please kindly fix ASP. @HulaHoop

9

From researching this, apparently it is the networking that throws a wrench in the possibility of easily running rootless QEMU, but it seems to have been resolved and documented. Since I am not running the appropriate host setup to experiment I ask @nurmagoz to please help test these instructions and report back on a Kicksecure host

A solution is in these articles to run libvirt under a qemu:session vs the qemu:system default connection. passst is superior to SLIRP if we have to go down the emulated network route.

https://bbs.archlinux.org/viewtopic.php?id=262025

https://www.gns3.com/gns3-cannot-work-with-qemu

Thanks to this Reddit article (https://old.reddit.com/r/archlinux/comments/b9emxp/qemusystemx86_64_does_not_execute_how_can_i/ek47btb/), I found the issue.

I opened an issue ( qemu-system-x86_64 fails to run with regular user after following arch wiki article (#515) ¡ Issues ¡ QEMU / QEMU ¡ GitLab ) to qemu developers (where you can see the resolution.

And I edited the Arch Wiki article ( QEMU - ArchWiki ) that led to this issue.

10

I went ahead and experimented with this.

File → Add connection → QEMU user session

Experimented with adding KS under this connection type which is straight forward enough:

  1. Use tar to decompress the archive:
    tar -xvf Kicksecure*.libvirt.xz

  2. Import the Kicksecure ™ image. Kicksecure ™ works with the network named default out of the box:
    virsh -c qemu:///session define Kicksecure*.xml

  3. Moving the Kicksecure ™ Image File
    mv Kicksecure*.qcow2 /home/user/.local/share/libvirt/images/Kicksecure.qcow2

Change path in KS xml to redirect to /home/user/.local/share/libvirt/images

The networking is where things get hairy and despite following the most recent and correct instructions for Debian, I end up with:

“Network not found: no network with matching name ‘default’”

So let’s explain how networking is supposed to work on a session connection. You cannot directly create any useful type of connection under this unprivileged mode. VMM would burp out an error that you don’t have permission to create a bridge. The only possible workaround is to use the qemu-bridge-helper binary to act as a liaison between the VMs running as users and the bridge networks created under the system connection.

You would need to add the virtual bridge interface names to a whitelist, then adjust the permissions for that whitelist file and the qemu-bridge-helper binary and enable IP forwarding as a sysctl option and Bob’s your uncle. Except all these widely documented steps don’t seem t be working and I get the aforementioned error

This is the most recent documentation that has improved on earlier referenced version. Perhaps someone can pull up their sleeves and give this a go and tell me what we’re missing here.

11

Yeah i tested this and we dont need sudo to use virsh because we added “kvm” user already:

maybe we can remove “sudo” from the commands in the wiki.

This is not as easy as its written, user need commands gymnastics in order to avoid the path issue:

/var/lib/libvirt/images/

E.g:

  • Create new (user level) path on user side e.g ~/VMs/Whonix.
  • Move or copy the images to the new path.
  • Change the paths inside the .xml for both workstation and gateway to the new created path.
  • As per libvirt documentation:

The directories containing disk images must either have their ownership set to match the user/group configured for QEMU, or their UNIX file permissions must have the ‘execute/search’ bit enabled for ‘others’.

The simplest option is the latter one, of just enabling the ‘execute/search’ bit. For any directory to be used for storing disk images, this can be achieved by running the following command on the directory itself, and any parent directories

chmod o+x /path/to/directory

In particular note that if using the “system” instance and attempting to store disk images in a user home directory, the default permissions on $HOME are typically too restrictive to allow access.

This means Libvirt disk image files and their parent directories must be accessible to the user/group under which QEMU runs, and parent directories need execute/search permission, so it means has access to these paths:

/home/user
/home/user/VMs
/home/user/VMs/Whonix
/home/user/VMs/Whonix/*.qcow2

Note: preferable to install acl package and use setfacl in order to only give access to libvirt-qemu for reading .qcow2 files (not to every other user).

e.g of usage:

setfacl -m u:libvirt-qemu:--x /home/user

then when you run:

virsh -c qemu:///system start Whonix-Gateway

Will give:

error:Failed to create file /var/lib/libvirt/dnsmasq/virbr1.macs,new: no such file or directory

And lets go into another gymnastics of commands..

Thats crazy to have it done from normal user just to make a hypervisor running whonix or x distro.

So without a method to have the KVM smoothly running by default (can be installed and configured by default in KS/Whonix), KVM can be considered incompatible in KS/Whonix.

12

Please ignore the storage path problems. This part is easily solved by using an alternative path under the user directory and is inconsequential. For now go ahead and follow the first steps when moving the image file and adjusting the xml to reflect that new path.

What we need to solve is the networking issue because without it, the VMs are useless without connectivity.

We should not touch the wiki instructions until this is completely solved.

13

Then we can mark the Whonix KVM (and Kicksecure KVM) port as unsupported.

Because Whonix is based on Kicksecure, if the Whonix KVM maintainer does not have a host setup to ensure Whonix support on Kicksecure hosts, as well as compatibility in source code, documentation, and testing, then Whonix KVM (and Kicksecure KVM) is effectively unsupported.

14

From sysmaint:

This is needed since we are using qemu:///system not qemu:///session

sudo mkdir -p /var/lib/libvirt/dnsmasq
sudo chown root:root /var/lib/libvirt/dnsmasq
sudo chmod 755 /var/lib/libvirt/dnsmasq

Then on user side:

virsh -c qemu:///system net-start Whonix-External
virsh -c qemu:///system net-start Whonix-Internal

Then run Whonix-Gateway and Workstation normally.

Some might say what about using SLIRP (instead of virbr0) with qemu:///session isnt it for usermode?

Well its too much restricted to the level it can break many things e.g current bridge (check here).

Unless something new does resolve this issue (which i didnt find), qemu:///system is going to be used.

Sadly yes and no solution to this seems to be.

Not really that much of helpful as we need more than just one VM to have internet, its like:

Whonix-Workstation → Whonix-Internal → Whonix-Gateway → Whonix-External (Internet/Tor)

I dont think anybody explained how to solve this on the internet.

1 Like
15

I went ahead and completely ported KS and Whonix to work under QEMU session as documented below. I had to use passt and (internal) network tunneling to replace the current nework architecture, since qemu-bridge-helper was useless.

It’s great news because now no sudo is needed at any point in the setup or runtime process.

This may also help solve the OpenVPN on the host issue that’s been reported before as incompatible and the lingering dnsmasq problem too (both need to be confirmed)..

1 Like
16

Great progress!

The idea is get fully rid of qemu:///system and exclusively use qemu:///session.)

Likely not, because your current version is still using IP addresses.

Ported to unix domain socket files. Please test and take over:
No Internet Connection inside Whonix-Workstation KVM with NordVPN with Kill-Switch on Host - #16 by Patrick

17
  • For qemu commandlines to be accepted in the file, the XML namespace needs to be included in the first line of the settings file (or the QEMU commands block)
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  • One will run into Apparmor issues when using QEMU commands since they are treated as arbitrary and not whitelisted by default
    libvirt: QEMU command-line passthrough
  • Testing the git branch settings for sockets triggered this bug which I don’t even know how to begin to diagnose. I think we are better off using what we currently know works
    Error starting domain: internal error: process exited while connecting to monitor: 2026-06-03 qemu-system-x86_64: -device {“driver”:“virtio-vga”,“id”:“video0”,“max_outputs”:1,“bus”:“pcie.0”,“addr”:“0x1”}: PCI: slot 1 function 0 not available for virtio-vga, in use by virtio-net-pci,id=(null)
18
19

*The reason I picked vhost-user over user is performance. I am wary of allowing untrusted access to shared memory though. The GW should be trusted but KS VMs shouldn’t. Perhaps we can opt for just ‘user’ for the latter.

Since 11.1.0 (QEMU and KVM only) you may prefer to use the passt backend with the more efficient and performant type=‘vhostuser’ rather than type=‘user’. All the options related to passt in the paragraphs below here also apply when using the passt backend with type=‘vhostuser’; any other details specific to vhostuser are described here.

This is a compelling alternative, because passt provides all of its network connectivity without requiring any elevated privileges or capabilities, and vhost-user uses shared memory to make this unprivileged connection very high performance as well.
libvirt: Domain XML format

20

Attempting to remove the dnsmasq-base package out right would rip the guts out of the libvrit install.[1] After a brief search it was suggested that disabling ‘default’ network should disable dnsmasq and it was indeed the case as confirmed by ‘netstat -tulpen’. So by not creating or using a network that would use dnsmasq’s services it is never used and can be safely ignored.

[1]
sudo apt purge dnsmasq-base
The following package was automatically installed and is no longer required:
libvirt-daemon-config-nwfilter
Use ‘sudo apt autoremove’ to remove it.

REMOVING:
dnsmasq-base* libvirt-bin* libvirt-daemon-config-network* libvirt-daemon-driver-lxc* libvirt-daemon-driver-network* libvirt-daemon-system*

Summary:
Upgrading: 0, Installing: 0, Removing: 6, Not Upgrading: 0
Freed space: 1,873 kB

Continue? [Y/n]

1 Like