Rootless virtual machines with KVM and QEMU

KVM
21

According to this post [1], one of the “drawbacks” that users who want to inspect inter guest traffic lament over is the lack of easy ability to do so. It is essentially opaque to the host since it doesn’t make use of network interfaces. Correct me if I’m wrong, but the original complaint was that the VPN killswitch broke WS connectivity while leaving the GW intact. This will not happen here.

Finally, it’s worth mentioning an unpleasant quirk of the UDP tunnels. They don’t use Linux interfaces, making it impossible to do simple packet capture on VM-to-VM links. QEMU does provide a traffic capture CLI command3 that saves packets into a file, and there might be a Wireshark plugin out there that strips the UDP header automatically (or you could create custom decoding rules). netlab uses a workaround: it politely tells you how to replace a UDP tunnel with a Linux bridge.

[1] Point-to-Point Links in Virtual Labs « ipSpace.net blog

22

So to avoid getting lost after these updates, the 2 actionable items we need to decide on:

  1. The preferred standard dir for images and the command to create it with the appropriate permissions

  2. Decide on vhost-user vs user - factoring in performance differences between both

UPDATE: A speed test running in WS Tor Browser gave pretty identical results. I’d prefer going for user in his case since I’m iffy about using shared memory for anything, it could very well be coded with security in mind, but still any bugs can be pretty bad.

***

https://blogs.oracle.com/linux/introduction-to-virtio-part-2-vhost

VSock sockets are not shared nor connectable to each other by KVM and need host side sorcery with socat to route packets between them. Unsure how the security situation woulD compare between this setup and just using natively supported tunnels. Socat would need to be compared with peer apps on security merits and further host side confinement for it would be needed.

***

PS. The current setup I documented has cleared all the leaktests

23

<interface type='user'> versus <interface type='vhostuser'>:

  • Seems like only vhostuser supports unix domain socket files but requires shared memory.
  • user doesn’t need shared memory but doesn’t support unix domain socket files.

That’s a mess. Need to find some way to configure libvirt / qemu command line so it fulfills both requirements, unix domain socket files + no shared memory.

Path:

If avoiding AppArmor (and sVirt [1]) issues is a consideration, it’s best to keep standard paths, i.e. keep using: $HOME/.local/share/libvirt/images

/opt or /srv is probably the wrong choice.

If it should be something else… The system wide standard path chosen by libvirt is inside /var/lib. The standard path is /var/lib/libvirt/images. So that would indicate to use some newly invented path inside /var/lib or /var/lib/libvirt.

The problem with any folder outside of $HOME is that rebooting into sysmaint session is required. Creating folders inside /srv or /opt requires sudo/root (sudo mkdir) and then makign the folder accessible to yourlinuxusername (sudo chmod).

Any setup instructions that require sudo/root and booting into sysmaint mode are more complicated than telling users to edit the libvirt XML files and go with standard paths. I think.

/home/user/.local/share/libvirt/images and a libvirt feature request to support $HOME seems to be the most simple, standard compliant and reliable solution.

Has been added to the PR in commit 2.

Tested, confirmed issue?

Potentially ask:

  • ChatGPT
  • Claude Code
  • Upstream libvirt mailing list.

I think our first priority here needs to be being fail-safe and leak-proof.

Inspection of inter guest traffic can be treated as a lower priority. That should be possible with both IP and Unix domain socket files. One can always add a proxy in between.

If there’s a host IP address, it seems like host VPN software (or corridor) can break it.

Unix domain socket files will be unaffected. So that’s the gold standard. Also, Unix domain socket files are subject to standard Linux permissions, so there may be an opportunity to harden access to them.

Unix domain socket files for VM-to-VM internal traffic likely fix VPN killswitch issues reliably once and for all. With Unix domain socket files, there is no host IP address or host network interface to interfere with. From the perspective of any firewall rules on the host, it’s just yet another file on the file system, so it does not notice it.

[1] Depending on whether sVirt is functional today or at some point in the future.

24

“passt exchanges packets with qemu via UNIX domain socket, using the socket back-end in qemu. This is supported since qemu 7.2.”

***

Yes it’s an important consideration, but I’m under the impression that sVirt is now smart enough to confine disk images in different locations either by default or can be easily told to do so. This can be easily checked.

***

Yes trying to use these options creates weird conflicts that I cannot easily figure out or find answers for online. This is a complete show stopper for me. That is why I will push the working settings in a separate branch.

***

Tunneling options are all documented as inter VM only and cannot connect to the outside world unless one of the VMs is configured as a gateway and has an appropriate NIC configured for external access.

According to this post, tunneling uses sockets (though not UNIX sockets)

25

Claude (and ChatGPT) figured out the issue - maybe. Updated branch pushed just now.

Might be helpful before and after:

virsh domxml-to-native qemu-argv ./usr/share/libvirt-dist/xml/Whonix-Gateway.xml

virsh domxml-to-native qemu-argv ./usr/share/libvirt-dist/xml/Whonix-Workstation.xml
26

Confirmed with Apparmor that SVirt not in use when starting as qemu session. Don’t know if this is because we run images outside /var/lib/libvirt

sudo aa-status

Also confirmed to be the case

***

EDIT:

Gemini corroborates what I suspected:

sVirt is not utilized when running VMs via qemu:///session.

The qemu:///session URI connects to a libvirtd/virtqemud instance running under your exact user account. Consequently, the VM processes (and their disk images) share your user’s Linux permissions, bypassing mandatory access controls like SELinux or AppArmor.

qemu:///session (User Sessions): sVirt is disabled. VMs are isolated strictly by standard discretionary access control (file permissions). Because all VMs run as your user, a compromised VM process could theoretically access the files of any other VM running under the same user session.

qemu:///system (System Sessions): sVirt is enabled. VMs run with elevated root privileges, but sVirt enforces Mandatory Access Control (MAC) using SELinux or AppArmor. It gives each VM process and its virtual disks a unique, randomized security label (e.g., svirt_t:s0:c34,c44), preventing one compromised VM from accessing another VM’s data on the host.

***

Also attempting to start qemu:///session VMs with seclabel apparmor results in:
“Error starting domain: unsupported configuration: Security driver model ‘apparmor’ is not available”

We are essentially sacrificing inter vm isolation, on the otherhand qemu is on the whole running in a less privileged manner than under qemu:///system

Theoretically we can manually recreate what was lost and install the profiles, however it will need a lot of troubleshooting and modification in cases where custom setups are concerned

27

With the new Claude changes, the VMs can boot now at least but networking is still dead

28

While libvirt VM settings do not handle relative file paths, the storage pools do. By pointing the XML at the storage pool you avoid the issues with hardcoded paths entirely. mv commands can also use ‘~’ which makes the install completely portable.

Tested successfully and added to Dev wiki

29

sVirt:

We don’t know if sVirt was ever functional even without any changes, with qemu:///system. [Help Welcome] KVM Development - staying the course - #593 by Patrick has never been addressed. Could you check please?

Related upstream documentation:

This strongly implies sVirt with qemu:///session is non-functional.

No upstream feature request could be found.

As for inter-VM security and host protection:

As per What is sVirt? | Richard WM Jones, it seems that,

  • rootless libvirt KVM without sVirt (since unavailable) is less secure than,
  • root libvirt KVM with sVirt.

As for host protection:

As per HowTo: Use the unprivileged "user session" in virt-manager for rootless virtualization with Qemu and KVM - #5 by boredsquirrel - Fedora Discussion

libvirt access equals root access.

Command sudo adduser user libvirt (add account user to group libvirt) breaks Kicksecure user-sysmaint-split security model.

storage pool:

  <source pool="default" volume="Kicksecure.qcow2"/>

That’s a really good solution you’ve found there.

30

Yes it was and has always been auto enabled and applied if apparmor was installed on the host. This is verifiable by running sudo aa-status. When in effect, each VM when running under qemu:///system has a AA profile that is auto-generated with its respective uid appended to the profile name at run time.

Confirmed as unfunctional and apparently always has been. Indeed, the lack of sVirt enforcement under this mode comes across as a result of an inherent design limitation of the framework - unable to impose any finer user access granularity when running as an unprivileged user.

Yes, but a lot has happened since that post. Most of the virtual hardware had been modernized and revamped as virtio as opposed to the emulated models that were bug prone.

I reason that VMs running under session without sVirt is a better proposition than running as root with privileged access to libvirt with it. It seems that trying to restrict a root process with an LSM is a losing proposition.

GNOME Boxes runs as qemu:///session and indeed the prevailing wisdom is that it is recommended that desktop users run libvirt in that manner.

Only when running under the libvirt group which is what it uses under system mode to seamlessly access functionality such as networking. NB that libvirt can run as unprivileged under session mode and is run under a user’s account instead in that case.

The whole raison d’etre of porting is to not resort to that.

I guess one can open a feature request asking for them to somehow extend sVirt to session mode, but I wouldn’t hold my breath or consider it a blocker at this point.

Please specify the last shipped branch of the XML so I can go ahead and push all my changes to it and get this going. I think the install instructions should only be updated once this new port is made available to users.

1 Like
31

Same branch. No new changes from my side.

32

Went ahead and merged all working changes straight to the branch

Made a feature request for sVirt to support qemu session

1 Like
[Help Welcome] KVM Development - staying the course
33

I have went through the steps and tested them on real device, last resort was a working WS/GW but it didnt happened without adjusting and improving the steps:

Minor Notes

Wont needed commands

[user ~/Downloads]% virsh -c qemu:///session net-autostart "default"

error: failed to get network 'default'

error: this function is not supported by the connection driver: virNetworkLookupByName

zsh: exit 1     virsh -c qemu:///session net-autostart "default"

[user ~/Downloads]% virsh -c qemu:///session net-autostart default  

error: failed to get network 'default'

error: this function is not supported by the connection driver: virNetworkLookupByName

zsh: exit 1     virsh -c qemu:///session net-autostart default

[user ~/Downloads]%

Not needed, as this is only for qemu:///system.

Path must be created beforehand

mv Whonix-Gateway*.qcow2 ~/.local/share/libvirt/images/Whonix-Gateway.qcow2

mv Whonix-Workstation*.qcow2 ~/.local/share/libvirt/images/Whonix-Workstation.qcow2

Path wont be created automatically, so it must be created before hand:

mkdir -p ~/.local/share/libvirt/images

How to edit WS/GW xml (missing step)

  • For GW:

virsh -c qemu:///session edit Whonix-Gateway

  • For WS:

virsh -c qemu:///session edit Whonix-Worstation

Remove <blkiotune> From both GW and WS

Remove this entire block of text from both of them:

  <blkiotune>
    <weight>250</weight>
  </blkiotune>

More clear step on how to change GW/WS connection

Default GW:

<interface type='network'>
      <mac address='xx:xx:xx:xx:xx:xx'/>
      <source network='Whonix-External'/>
      <model type='virtio'/>
      <driver name='qemu'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='xx:xx:xx:xx:xx:xx''/>
      <source network='Whonix-Internal'/>
      <model type='virtio'/>
      <driver name='qemu'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>

Going to be changed to:

    <interface type='user'>
      <mac address='xx:xx:xx:xx:xx:xx'/>
      <model type='virtio'/>
      <driver name='qemu'/>
      <backend type='passt'/>
      <ip family='ipv4' address='172.17.5.4' prefix='24'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <interface type='udp'>
      <mac address='xx:xx:xx:xx:xx:xx'/>
      <source address='127.0.0.1' port='5577'>
        <local address='127.0.0.1' port='6688'/>
      </source>
      <model type='virtio'/>
      <driver name='qemu'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>

WS configs just in case:


    <interface type='udp'>
      <mac address='xx:xx:xx:xx:xx:xx'/>
      <source address='127.0.0.1' port='6688'>
        <local address='127.0.0.1' port='5577'/>
      </source>
      <model type='virtio'/>
      <driver name='qemu'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>

Virt-Manager GUI default is qemu:///system not session

WS and GW wont be shown in the default opening of Virt-Manager, because it assumes your VMs on normal QEMU/KVM or qemu:///system, so you need to change that to QEMU/KVM user session:

Go to virt-manager → File → Add Connection → Switch Hypervisor from QEMU/KVM to QEMU/KVM user session → Connect

Bad usability: This change wont stay if you shutdown virt-manager, so you need to do that every-time you access virt-manager.

Solution to make the change permanent run:

echo "export LIBVIRT_DEFAULT_URI='qemu:///session'" >> ~/.zshrc
source ~/.zshrc

(If someone is using Bash instead of Zsh, change .zshrc to .bashrc, This tells your entire user profile including the GUI to always target your local user session by default).

Critial Notes

libvirt: error : cannot limit core file size of process 7077 to 18446744073709551615: Operation not permitted

When trying to run GW:

Error starting domain: internal error: Process exited prior to exec: libvirt:  error : cannot limit core file size of process 7077 to 18446744073709551615: Operation not permitted

Traceback (most recent call last):

  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 71, in cb_wrapper

    callback(asyncjob, *args, **kwargs)

    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 107, in tmpcb

    callback(*args, **kwargs)

    ~~~~~~~~^^^^^^^^^^^^^^^^^

  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 57, in newfn

    ret = fn(self, *args, **kwargs)

  File "/usr/share/virt-manager/virtManager/object/domain.py", line 1384, in startup

    self._backend.create()

    ~~~~~~~~~~~~~~~~~~~~^^

  File "/usr/lib/python3/dist-packages/libvirt.py", line 1390, in create

    raise libvirtError('virDomainCreate() failed')

libvirt.libvirtError: internal error: Process exited prior to exec: libvirt:  error : cannot limit core file size of process 7077 to 18446744073709551615: Operation not permitted

Why its happening: The giant number 18446744073709551615 represents RLIM_INFINITY (unlimited). When libvirt tries to set the VM process maximum core dump file size to “unlimited”, the system blocks it because unprivileged users not allowed (part of the protection in debian) to raise their own hard resource limits (setrlimit).

Since there is no root access to modify system-wide limits, then we need to tell libvirt to stop trying to adjust this limit by forcing libvirt to skip resource limits.

So we need to override the memory/core limits for QEMU inside user configuration directory:

mkdir -p ~/.config/libvirt/

nano ~/.config/libvirt/qemu.conf

Copy/Paste:

max_core = 0
max_processes = 0
max_files = 0

Then stop and start the daemon:

systemctl --user stop libvirtd.service 2>/dev/null

systemctl --user stop virtqemud.service 2>/dev/null

pkill -9 -f libvirtd

pkill -9 -f virtqemud

libvirt.libvirtError: unsupported configuration: unable to open vhost-vsock device

If you get this error when trying to run WS:

Error starting domain: unsupported configuration: unable to open vhost-vsock device

Traceback (most recent call last):

  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 71, in cb_wrapper

    callback(asyncjob, *args, **kwargs)

    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 107, in tmpcb

    callback(*args, **kwargs)

    ~~~~~~~~^^^^^^^^^^^^^^^^^

  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 57, in newfn

    ret = fn(self, *args, **kwargs)

  File "/usr/share/virt-manager/virtManager/object/domain.py", line 1384, in startup

    self._backend.create()

    ~~~~~~~~~~~~~~~~~~~~^^

  File "/usr/lib/python3/dist-packages/libvirt.py", line 1390, in create

    raise libvirtError('virDomainCreate() failed')

libvirt.libvirtError: unsupported configuration: unable to open vhost-vsock device

Go to virt-manager → Press on Whonix-Workstation → Then click on Open from the top bar → Press on the light bulb (Virtual hardware details) → Right-Click on VirtiO VSOCK then remove it.

WS will start normally.

@HulaHoop check if something needs to be improved further.

@Patrick i think its a good idea if we can add KVM and these configs easy by default (Windows Pro edition has hyper-v by default but user must pay for this version, KS (maybe Whonix host later) do this for free to the community).

1 Like
34

Wont needed commands

Referring to the wiki, which hasn’t been updated yet?

mkdir -p ~/.local/share/libvirt/images

Valid point. Wiki needs update.

Orchestrating the wiki changes is a bit hard. Made a copy of /KVM instead.

Please stop huge edits in /KVM until settled.

Documentation edits for qemu:///user can be done here:

  • https://www.kicksecure.com/wiki/KVM_Testers_Only_Version
  • https://www.whonix.org/wiki/KVM_Testers_Only_Version

Why invent this?

Why?

How to reproduce? Is this a Kicksecure user session?

1 Like
35

Of course this won’t work. qemu:///session cannot directly access or interact with network bridges because it is unprivileged. qemu-bridge-helper is designed to help with that, but I couldn’t get it to work.

Unecessary. First, when the XML is imported libvirt creates this new subdirectory under home in this location. No manual creation needed. Then the image is moved there in the next step

OK so this may be need to be specified when editing them since the default bash varable is set to the qemu:///system connection

These changes are already in the code and documented on the wiki at Dev/KVM

Did not need to that on my end. Importing the XML created a new session connection with the machines listed on there by default every time I opened VMM

Have absolutely no idea what this is and haven’t seen this either. It would be good to create a separate thread about it, listing your setup like Host OS and hardware specs and try to find a solution for this bug. An unreproducible problem should not be documented on the wiki as if it is a matter of fact bug that everyone needs to work around

We don’t include a vsock virtual device. Note this is not to be confused with vhostuser which I did not end up using either. Nor does the UNIX socket solution researched by Patrick have anything to do with vsock.

Vsock by default is a host guest communication channel that could be harnessed by qemu-guest-agent. It cannot be used for inter guest communication unless the vsock-bridge (a tool by its author) is used to connect two vms’ vsockets

2 Likes
36

Yes

Sure, will do that.

Done by default when the VM imported, not something added from my end.

Yes, everything tested on KS host.

1 Like
37

mv Whonix-Gateway*.qcow2 ~/.local/share/libvirt/images/Whonix-Gateway.qcow2

Command will fail without having that path before hand, because mv doesnt create missing directories.

yeah but wasnt clear from GW or WS only or both.

Interesting, you are using KS as a host? better test only on KS to have reproducible results on our ends.

Using KS as a host, PC specs are: OMEN laptop 16, i7 (hybrid), 48G RAMs, with dedicated GPU Nvidia RTX 4060, Hard desk is 1 TB MP700 PRO PCIe GEN 5 M.2 SSD. But i believe its irrelevant because if you search the error you will find out its related to user session not hardware.

Nice, just document it in case someone will fall into it.

1 Like
38

Both instructions have been added to both wikis.

Notes:

  • Whonix KVM wiki needs general trimming and changes/testing based on the change from qemu system to session.

  • Internal changes of XML not added to the instructions, because user expect Whonix and KS to come working by default on session instructions. Instructions should come by default in new images XML:

** UDP, localhost, passt network configs
** Removal of <blkiotune>

Rest of the configs which should be done in OS, should come in KS and Whonix-WS by default for qemu session.

1 Like
39

@Patrick When successfully starting with sockets (by verifying their creation in the specified host directory) Tor connectivity to the outside world is broken. Tor error logs seem to be complaining about not being able to access the controlport.

The stream option causes permanent Tor breakage while removing the dgram setting restores connectivity in GW. Both use sockets, but that’s not the point. After some research, it seems socat or something like it is needed to forward a program’s TCP port to a unix socket for it to be usable.

TL;DR the problem is with Tor’s confusion with socket interaction not QEMU having bugs or not being able to use the socket feature.

1 Like
40

These 2 links don’t seem related?

The inside of the VM shouldn’t even notice any changes to the networking. This should be purely a host change. Or what is changing from the inside-VM perspective?

That sounds more as if there is no network interface(s) inside the VM anymore?

Inside the VM, check:

sudo ifconfig

Before and after host VM settings changes. Then compare the two.