According to this post [1], one of the “drawbacks” that users who want to inspect inter guest traffic lament over is the lack of easy ability to do so. It is essentially opaque to the host since it doesn’t make use of network interfaces. Correct me if I’m wrong, but the original complaint was that the VPN killswitch broke WS connectivity while leaving the GW intact. This will not happen here.
Finally, it’s worth mentioning an unpleasant quirk of the UDP tunnels. They don’t use Linux interfaces, making it impossible to do simple packet capture on VM-to-VM links. QEMU does provide a traffic capture CLI command3 that saves packets into a file, and there might be a Wireshark plugin out there that strips the UDP header automatically (or you could create custom decoding rules). netlab uses a workaround: it politely tells you how to replace a UDP tunnel with a Linux bridge.
So to avoid getting lost after these updates, the 2 actionable items we need to decide on:
The preferred standard dir for images and the command to create it with the appropriate permissions
Decide on vhost-user vs user - factoring in performance differences between both
UPDATE: A speed test running in WS Tor Browser gave pretty identical results. I’d prefer going for user in his case since I’m iffy about using shared memory for anything, it could very well be coded with security in mind, but still any bugs can be pretty bad.
VSock sockets are not shared nor connectable to each other by KVM and need host side sorcery with socat to route packets between them. Unsure how the security situation woulD compare between this setup and just using natively supported tunnels. Socat would need to be compared with peer apps on security merits and further host side confinement for it would be needed.
***
PS. The current setup I documented has cleared all the leaktests
<interface type='user'> versus <interface type='vhostuser'>:
Seems like only vhostuser supports unix domain socket files but requires shared memory.
user doesn’t need shared memory but doesn’t support unix domain socket files.
That’s a mess. Need to find some way to configure libvirt / qemu command line so it fulfills both requirements, unix domain socket files + no shared memory.
Path:
If avoiding AppArmor (and sVirt [1]) issues is a consideration, it’s best to keep standard paths, i.e. keep using: $HOME/.local/share/libvirt/images
/opt or /srv is probably the wrong choice.
If it should be something else… The system wide standard path chosen by libvirt is inside /var/lib. The standard path is /var/lib/libvirt/images. So that would indicate to use some newly invented path inside /var/lib or /var/lib/libvirt.
The problem with any folder outside of $HOME is that rebooting into sysmaint session is required. Creating folders inside /srv or /opt requires sudo/root (sudo mkdir) and then makign the folder accessible to yourlinuxusername (sudo chmod).
Any setup instructions that require sudo/root and booting into sysmaint mode are more complicated than telling users to edit the libvirt XML files and go with standard paths. I think.
/home/user/.local/share/libvirt/images and a libvirt feature request to support $HOME seems to be the most simple, standard compliant and reliable solution.
Has been added to the PR in commit 2.
Tested, confirmed issue?
Potentially ask:
ChatGPT
Claude Code
Upstream libvirt mailing list.
I think our first priority here needs to be being fail-safe and leak-proof.
Inspection of inter guest traffic can be treated as a lower priority. That should be possible with both IP and Unix domain socket files. One can always add a proxy in between.
If there’s a host IP address, it seems like host VPN software (or corridor) can break it.
Unix domain socket files will be unaffected. So that’s the gold standard. Also, Unix domain socket files are subject to standard Linux permissions, so there may be an opportunity to harden access to them.
Unix domain socket files for VM-to-VM internal traffic likely fix VPN killswitch issues reliably once and for all. With Unix domain socket files, there is no host IP address or host network interface to interfere with. From the perspective of any firewall rules on the host, it’s just yet another file on the file system, so it does not notice it.
[1] Depending on whether sVirt is functional today or at some point in the future.
“passt exchanges packets with qemu via UNIX domain socket, using the socket back-end in qemu. This is supported since qemu 7.2.”
***
Yes it’s an important consideration, but I’m under the impression that sVirt is now smart enough to confine disk images in different locations either by default or can be easily told to do so. This can be easily checked.
***
Yes trying to use these options creates weird conflicts that I cannot easily figure out or find answers for online. This is a complete show stopper for me. That is why I will push the working settings in a separate branch.
***
Tunneling options are all documented as inter VM only and cannot connect to the outside world unless one of the VMs is configured as a gateway and has an appropriate NIC configured for external access.
According to this post, tunneling uses sockets (though not UNIX sockets)
Confirmed with Apparmor that SVirt not in use when starting as qemu session. Don’t know if this is because we run images outside /var/lib/libvirt
sudo aa-status
Also confirmed to be the case
***
EDIT:
Gemini corroborates what I suspected:
sVirt is not utilized when running VMs via qemu:///session.
The qemu:///session URI connects to a libvirtd/virtqemud instance running under your exact user account. Consequently, the VM processes (and their disk images) share your user’s Linux permissions, bypassing mandatory access controls like SELinux or AppArmor.
qemu:///session (User Sessions): sVirt is disabled. VMs are isolated strictly by standard discretionary access control (file permissions). Because all VMs run as your user, a compromised VM process could theoretically access the files of any other VM running under the same user session.
qemu:///system (System Sessions): sVirt is enabled. VMs run with elevated root privileges, but sVirt enforces Mandatory Access Control (MAC) using SELinux or AppArmor. It gives each VM process and its virtual disks a unique, randomized security label (e.g., svirt_t:s0:c34,c44), preventing one compromised VM from accessing another VM’s data on the host.
***
Also attempting to start qemu:///session VMs with seclabel apparmor results in:
“Error starting domain: unsupported configuration: Security driver model ‘apparmor’ is not available”
We are essentially sacrificing inter vm isolation, on the otherhand qemu is on the whole running in a less privileged manner than under qemu:///system
Theoretically we can manually recreate what was lost and install the profiles, however it will need a lot of troubleshooting and modification in cases where custom setups are concerned
While libvirt VM settings do not handle relative file paths, the storage pools do. By pointing the XML at the storage pool you avoid the issues with hardcoded paths entirely. mv commands can also use ‘~’ which makes the install completely portable.
Yes it was and has always been auto enabled and applied if apparmor was installed on the host. This is verifiable by running sudo aa-status. When in effect, each VM when running under qemu:///system has a AA profile that is auto-generated with its respective uid appended to the profile name at run time.
Confirmed as unfunctional and apparently always has been. Indeed, the lack of sVirt enforcement under this mode comes across as a result of an inherent design limitation of the framework - unable to impose any finer user access granularity when running as an unprivileged user.
Yes, but a lot has happened since that post. Most of the virtual hardware had been modernized and revamped as virtio as opposed to the emulated models that were bug prone.
I reason that VMs running under session without sVirt is a better proposition than running as root with privileged access to libvirt with it. It seems that trying to restrict a root process with an LSM is a losing proposition.
GNOME Boxes runs as qemu:///session and indeed the prevailing wisdom is that it is recommended that desktop users run libvirt in that manner.
Only when running under the libvirt group which is what it uses under system mode to seamlessly access functionality such as networking. NB that libvirt can run as unprivileged under session mode and is run under a user’s account instead in that case.
The whole raison d’etre of porting is to not resort to that.
I guess one can open a feature request asking for them to somehow extend sVirt to session mode, but I wouldn’t hold my breath or consider it a blocker at this point.
Please specify the last shipped branch of the XML so I can go ahead and push all my changes to it and get this going. I think the install instructions should only be updated once this new port is made available to users.