[Help Welcome] KVM Development - staying the course

I think its best if we don’t supply pre-configured shared folders and leave it up to the user to decide if they want to enable them, and if so, where they want to set it up.

Your instructions are easy to follow, but its best if the ones with the “generic” markings were added to inform the user what variables he can change to suit their needs.

I am working on the readme for inclusion in the tar packages and will post it on Git soon to get your opinion.

Please check if the xmls now validate after my changes. I checked and the machines and network files validate. I will remove the blocker bugs as I resolved them yesterday, except the ones that apply.

I’ve finished the readme and will now see if I can get qemu-arm running with minimal changes and document that.

EDIT:
QEMU ARM differs a lot and needs its own template. It requires linking to a kernel and is somewhat different in the way its machines are structured.
https://fedoraproject.org/wiki/Architectures/ARM/HowToQemu#How_To:_Running_Fedora-ARM_under_QEMU

Should we make a different template for each configuration even qemu-x86? It requires barely naything different from the kvm one, but I’m suggesting this out of a design principle.

I’m thinking about cleaning up the kvm wiki page, not dev/kvm since we don’t need the GUI to setup a vm anymore. This info should be archived and not deleted probably.

Patrick if you can re-purpose the script you wrote to dynamically update the names in the xml for both the vm and disk image with the new version number that would be great. It would save us from having to go in there and edit it everytime there is a new release, while at the same time keeping a convenient naming scheme for the users.

At the moment we are following this naming convention to differ from Virtualbox: Whonix-Gateway-9.0

Why? I would like to reduce the steps someone has to figure out on an internet page to as few as possible to make it as simple as possible. I prefer instructions with a working example that can be used for mindless copy and paste. We can add some “but yes, if you’re a special person you can get your own very special paths, but if you just want to use it, just copy and paste this” additions and footnotes to make the geeks happy as well.

Your instructions are easy to follow, but its best if the ones with the "generic" markings were added to inform the user what variables he can change to suit their needs.
These instructions are inspired by Murphy's law, "anything can go wrong, will go wrong". Supposed to create as little support requests as possible.
Please check if the xmls now validate after my changes.
Unfortunately doesn't work. Updated Dev/KVM. Maybe you have a more recent version of virsh than I have?
  • is supposed to connect two words with each other.
    _ is supposed to separate name and version number from each other.

(Having in mind that someone might want to script it for some purpose, aggregate links, etc.)

Why do we need a different naming scheme than VirtualBox? I am reluctant here, because I think such things can look messy and cause confusion.

Yes.

But if we’re doing this at all is up to you. Just for adding kvm support to Whonix, you don’t have to maintain qemu/arm as well. Just if you’re feeling this won’t take too much from you.

Deprecated instructions are usually moved here:

I’ll make this a new build step. build-steps.d/1150_export-libvirt-xml (?), which gets only run if, well if what? If someone chooses --libvirt, --kvm or --qemu? Or should this be already the case if someone is using --qcow2?

Why should we dynamically adjust the name of the VM at all? For VirtualBox, $VMNAME is either Whonix-Gateway or Whonix-Workstation. That name gets visible in the window title bar. If we hardcode the build version number like Whonix-Workstation 9.1, then that number will never be updated even if the user runs apt-get dist-upgrade. I think this will be very confusing. Didn’t add this for now.

Can you refresh my mind please… Why did we decide again to add version numbers to .qcow2 images? What was the advantage of that?

I just imaging the developer documentation. “Yeah, we copy the xml from libvirt folder to a temporary location, then adjust the path to add the version number to the qcow2 image. Then we make a compressed tarball that includes the xml as well as the qcow2 image.” Sounds rather complicated, so we should better have a strong reason for dynamically editing the xml for every version.

Proposal:

  • we ship an archive named Whonix-Gateway-9.1.libvirt.xz
  • it includes Whonix-Gateway.qcow2
  • it includes whonix_gateway_kvm.xml that points to /var/lib/libvirt/images/Whonix-Gateway.qcow2
  • it maybe includes whonix_gateway_qemu.xml
  • instructions include advice to “cp ./Whonix-Gateway.qcow2 /var/lib/libvirt/images/”

Anyhow, I got started with the script:
https://github.com/Whonix/Whonix/blob/master/build-steps.d/1150_export-libvirt-xml
Please have a look. It includes a lot out commented code that would have to be activated for dynamically editing the image path in the temporary xml file.

I think throwing out dynamic image numbers things would be much simpler and would avert alot of problems. No need to patch for a

generic image name, static but different domain names hardcoded into xml dpending on what they are kvm, qemu etc.

Lets use the scheme: Whonix_Gateway.qcow2 instead
and
whonix_gateway_kvm.xml whonix_gateway_qemu-x86-64.xml

What caused us to panic and abandon the import script was that we tried to include many things in it that are out of scope. This mission creep made it seem like a nightmare to do this in a quick and easy fashion. I admit typing in a list of commands to get the thing up and running is a chore. New users might panic.

I propose a new import script design that is based on the Keep It Simple design:

What it is and its purpose:

  1. Scripts purpose is to provide an automated way to register a vm by feeding virsh the import commands and telling it which xml to apply that data from, no matter what the xml file name is as long as its in the same directory as the script. The xmls are run separately and one at a time. By putting each containment type into a different xml that makes things much easier.

2.We are only concerned with a bear minimum subset of virsh commands to do the import process

  1. It will NOT attempt to change or patch anything into the xml according to alternative config, the users can do this for themselves by directly editing the XML. However there is one exception to the patching:

One Exception: Script obtains the absolute path of the directory it resides in (and hence the image file) and patches that in the xml for the storage path parameter. The image file name is spared. Should the user wish to rename the image from its generic name, they would need to modify the xml and change it so that virsh can read this info correctly.

  1. 2 scripts one for each tar needed but can essentially be the same.

The start of the script would display the security warning - but this is excluded in the future should someone want to use it as a generic import tool.

A second warning before we start the import would be to say: please remove any machine using the same name from libvirt. This can be done by using virt-manager or virsh using the undefine command.

Note: Deleting the storage associated with the machines being removed is not necessary. You can configure them with a different name and re-run this script to import them later.

Benefits to this approach over past ones:

-No need to path the hell out of one xml to run it as a different configuration
-No need for an export function as the images and xmls are already in their original folder which makes this setup portable. Moving it to a new machine, a user would simply run the script that defines the machine there.
-No copying or moving of anything.
-Solves the overwrting and dupe probem as new version reside in their own newly unzipped directories and more than one xml can be told to use the same disk img.
-users avoid dupe problem with network as they simply don’t define its script if they already done so in the past.

+++++++++++++++++++++++

On shared folders:
Feel free to add the automouting code inside Whonix for a specific directory inside. That way should any user enable it from outside it should work in a convenient fashion. We must tell users in the readme the tag name however that they would use.

While KVM folder sharing is developed with security in mind and is very robust, I think that shipping the Whonix configuration to allow it by default is a bad idea. We don’t want an adversary counting on that to be the way things are on every Whonix install out of the box.

Furthermore, at the beginning of the script be can add the option to [EDIT] the xml in nano, as long as its placed in the same directory as the script. If a user mistakenly edits the image path, this can be overwritten by the [IMPORT] process.

The beauty of this design is that its machine agnostic and can potentially work with any other vm when its placed in the same folder as the disk img and template.

Made some changes.

use Whonix-Gateway instead of Whonix_Gateway; use Whonix-Workstation instead of Whonix_Workstation. (We’re using Whonix-XXX notation with dash over underscore everywhere already.)
https://github.com/Whonix/Whonix/commit/c9ed99739d3ef97184f1b6b5aef1adb1c8768d99

renamed: whonix_gateway_qemu-x86-x64.xml → Whonix-Gateway_qemu-x86-x64.xml
renamed: whonix_workstation_kvm.xml → Whonix-Workstation_kvm.xml
renamed: whonix_workstation_qemu-x86-x64.xml → Whonix-Workstation_qemu-x86-x64.xml
renamed: whonix_network.xml → Whonix_network.xml

Removed containment from domain name. I think it is really confusing to have the containment (kvm, qemu, etc.) in the name of the VM. Users who experiment with these files or later change from one to the other will still keep the existing domain name and will be confused about the conflict. It would be up to virt-manager/kvm to automatically add the type of the containment to the window title.
https://github.com/Whonix/Whonix/commit/457728ed54d8e1d7fb262652a5192e7fbed8a9e7

[quote=“HulaHoop, post:190, topic:166”]-No need for an export function as the images and xmls are already in their original folder which makes this setup portable. Moving it to a new machine, a user would simply run the script that defines the machine there.
-No copying or moving of anything.[/quote]
Portable running the VM from a folder is something I like and feel that users do like as well. Portable software is so much better to understand than when stuff is moved “somewhere”. However…
This would conflict with AppArmor / selinux, because it does not have access to the custom folder of the user. I think manually copying the files to the default directory is the most compatible and canonical way.

On shared folders: Feel free to add the automouting code inside Whonix for a specific directory inside. That way should any user enable it from outside it should work in a convenient fashion. We must tell users in the readme the tag name however that they would use.

While KVM folder sharing is developed with security in mind and is very robust, I think that shipping the Whonix configuration to allow it by default is a bad idea. We don’t want an adversary counting on that to be the way things are on every Whonix install out of the box.


I am not sure shared folders would make anything worse. My strategy is: make it as simple as possible by default so we can grow the number of general Linux [or even more general, Libre Software] users (not just Whonix users). Thereby get more attention, money, programmers and so forth. Thereby making it more secure. And have an advanced security guide to make the geeks happy as well. Making it most inconvenient/secure/geek friendly by default will only get the geeks and scare newbies away. But the newbies are the ones who are required to make general Libre Software finally a real success.

Removed containment from domain name. I think it is really confusing to have the containment (kvm, qemu, etc.) in the name of the VM. Users who experiment with these files or later change from one to the other will still keep the existing domain name and will be confused about the conflict. It would be up to virt-manager/kvm to automatically add the type of the containment to the window title.

I think having containment type in domain name is necessary for those who would like to have and experiment with multiple types of isolation. The user wouldn’t need to change anything in their xml to get qemu instead of kvm but simply import its file instead. The naming convention you are using now will introduce direct conflicts for a multi-type scenario, so please change it back.

Portable running the VM from a folder is something I like and feel that users do like as well. Portable software is so much better to understand than when stuff is moved "somewhere". However... This would conflict with AppArmor / selinux, because it does not have access to the custom folder of the user. I think manually copying the files to the default directory is the most compatible and canonical way.

A portable scenario should not cause any problems when using MAC. sVirt applies its policy to an image when a qemu/kvm process runs it. In order to run it, libvirt needs to know the path of the image. Should the user change the location of the vm folder, they would run import and it would update the path to its new place. Libvirt would recognize it and run the vm processes (with its associated image file) with its confinement rules when initialized.

I am not sure shared folders would make anything worse. My strategy is: make it as simple as possible by default so we can grow the number of general Linux [or even more general, Libre Software] users (not just Whonix users). Thereby get more attention, money, programmers and so forth. Thereby making it more secure. And have an advanced security guide to make the geeks happy as well. Making it most inconvenient/secure/geek friendly by default will only get the geeks and scare newbies away. But the newbies are the ones who are required to make general Libre Software finally a real success.

Ok then I’ll add shared folders in the xml later. Including it into the documentation is a good idea.

After we are done with this please consider making a generic copy of the import script - that excludes: shared folder creation and our security warnings, so it can be used by others who want to import vms in general into KVM. But that comes later not now.

Patrick you were right we have to move the image files to /var/lib… or else there will be MAC conflicts.

I mistakenly though this wasn’t needed because It turns out this isn’t necessary only if an image directory is defined as a storage pool for libvirt, but that only happens if you are attaching an image to a vm created thru virt-manager gui. But we are using virsh so this won’t work.

So alternate step:

Have the script look if an image file with the specified name in the path tag exists, and copy it in a sparse aware manner. Copy, rather than move, just in case the user wants to experiment and create another domain, so it gets its own image as long as the user changes the name in the xml and the file

I think having containment type in domain name is necessary for those who would like to have and experiment with multiple types of isolation. The user wouldn’t need to change anything in their xml to get qemu instead of kvm but simply import its file instead. The naming convention you are using now will introduce direct conflicts for a multi-type scenario, so please change it back.[/quote]
We would be kinda trying to work around a missing KVM feature again. When you import a VM in VirtualBox, it will tell you, that you cannot have two VMs with the same name in a graphical GUI. Rather, a new random id is generated to identify the VM. Those few who want special things, who want to use different containments (there is little point in using qemu, if you’re capable of using kvm) or those who want multiple Whonix-Workstations, should manually change the domain name.

A portable scenario should not cause any problems when using MAC. sVirt applies its policy to an image when a qemu/kvm process runs it. In order to run it, libvirt needs to know the path of the image. Should the user change the location of the vm folder, they would run import and it would update the path to its new place. Libvirt would recognize it and run the vm processes (with its associated image file) with its confinement rules when initialized.
As far as AppArmor is concerned, paths are hardcoded and as far I know not automatically adjusted to custom paths unless the AppArmor profile in /etc/apparmor.d/ gets edited.
Ok then I'll add shared folders in the xml later. Including it into the documentation is a good idea.
Good!

I’ve given up on the script. It’s not a good use of time. I’d be better off using that time getting better in python and including the missing feature into virtmanager. Maybe some day.

Makes sense. Was worth a shot so,but thanks for trying to make it work. Life is too short for doing things like this. I may try to contact the guy upstream, whose name was on the virt-manager import/export ticket. But chances are this will never materialize

Now here is what needs to be done before we get a final release ready:

-We must revert the shared folder commit, because virt-manager complains about the folder not existing on the host, blurts a cryptic error message and refuses to start the vm. We don’t want to scare and discourage people so I’m going to remove it.

  • I will add the tag you used for automounts in the readme documentation for those who want to use the automount feature to type that in.

-Update the naming in readme.

-remove the script from libvirt directory on git so we don’t ship it, so no one gets confused.

If you haven’s done so, please include the internal directory you will automount and share, in the image build.

Used the spice example to enable it in xml.
http://froebe.net/blog/2013/02/10/howto-windows-7-32bit-and-64bit-sound-with-kvm-libvirt-and-the-spice-client-2/

Turns out we were never using it because it wasn’t enabled in the xml, even if spice-vdagent was installed it wasn’t the real thing.

Tried it now and its very fast with the acceleration parameters enabled.

It still needs spice-vdagent to be activated when vdagent is installed in Whonix. Maybe you will enable it be default?
From a security perspective, qxl is qemu and that is fenced off from the host by MAC.

More Links on Spice - for me to flick thru you don’t need to look at them if you’re not interested:
http://elmarco.fedorapeople.org/manual.html

http://www.ovirt.org/Testing/Spice