[Help Welcome] KVM Development - staying the course

I guess if we want to Do This The Right Way ™, we also should support users who wish to use multiple Whonix-Workstations.

Any suggestion how we do that?

Why don’t we let the user choose a name for the VMs and image names instead? Why attempt to autodetect it? We could default to Whonix-Gateway / Whonix-Workstation if user presses enter. And those who want to use a qemu Whonix-Gateway, simply import it again and name it “Whonix-Gateway qemu”. And those who want a second Whonix-Workstation just import it again and choose “ws2” as name.

I guess if we want to Do This The Right Way (TM), we also should support users who wish to use multiple Whonix-Workstations.

Any suggestion how we do that?

Why don’t we let the user choose a name for the VMs and image names instead? Why attempt to autodetect it? We could default to Whonix-Gateway / Whonix-Workstation if user presses enter. And those who want to use a qemu Whonix-Gateway, simply import it again and name it “Whonix-Gateway qemu”. And those who want a second Whonix-Workstation just import it again and choose “ws2” as name.

Yes I definitely agree.
I had a design in mind that addresses the issues you bring up and is much more flexible but will require a radical rewrite and departure from the way we are doing things at the moment. Its much better to avoid hardcoded paths and settings as much as possible in order to keep this extendable. Now I don’t know much to determine if this is possible or a nightmare of complexity, for that I need your opinion.If you decide to go down that route even if it takes longer, there is no rush as long as we get this right and keep it open to future improvements in a way as easy as possible.

The perceived advantages will be discussed below in more detail after the sequence diagram.

[code]1. Script choose mode →

  1. If GUI chosen check for dep.
    a) if not present prompt: Your system does not appear to have the necessary packages to allow a script based GUI, please install any of the following and re-launch the script in this mode: zenity, yad, kdialog, gxmessage, xmessage [whatever else you want to inlcude here];
    b) would take them back to [1] to choose another interface.

  2. #Security warning#:
    IMPORTANT: As a security precaution you are advised to shutdown any running Whonix instance currently attached to the internal virtual network. This is required to prevent cross contamination of the new machines being imported now, in the event that a powerful adversary has taken control over the ones in use.

Note: This is not required if you intend to create a new virtual network for the machines you are importing now.


[As an aside: This should be warned of at least in documentation somewhere for Virtualbox since it imports and adds new machines to the any internal network with the same name.]

  1. parallel variable settings panel that will allow the configuration of numerous settings. can be expanded over time for higher control over vm settings:
    $path of image to work on !By default current dir
    $path of xml !By default current dir
    $name of xml
    $vm name
    $disk img name !Must be equivalent to name specified in $path of image whatever that is
    $virtual net name !Default set to ‘Whonix’
    $container type
    $architecture ! Next to this option specify that its qemu only like this: (QEMU only)

Each variable can be arbitrarily set by the user with the exception of a few things that must be the same as specified. sed could replace the content in between the xml tag category in question and doesn’t have to find and replace a specific (hardcoded) expression.

  1. If duplicate detected in these variables only: img name in destination dir., vm name then prompt for

Dulicate domain vm name:
#Domain Duplicate Detected#
The current domain name is already in use. Please go back and change it to something else.
a) This is the only option provided in the event that a dupe domain is detected from virsh output. This check should precede the dupe checks that follow it and is a necessary condition to continue beyond that point.

*** Step placeholder here

img name overwrite dialogue:
#Image Overwrite#
A disk image with the same name currently exists in the destination directory. Would you like to overwrite it?
a) proceed to [6] Import
b) return to [4] to allow changing of img name

(***) at this point when attempting to register the virtual network, virsh will complain that there is a duplicate in that name. We should assume that the user made a conscious decision to choose this vnet to connect their machines to, and shoud therefore not change anything and silently discard he warning.

  1. Import[/code]

Perceived Advantages:

  1. This could potentially be done in one script that the user would run twice once for the gw and once for the ws. Or two copies of the script one with each machine could be supplied for those who only want to download one machine or the other.
  2. It allows the user much more freedom to customize the machine name and settings during the import process to conventions that suit them.
  3. Allows the bypassing of the whole current mess of arbitrarily naming the file & machine according to settings selected.
  4. This technique may be extended to allow more fine grained control over settings, for example amount of RAM devices attached etc.
  5. Allows a multiple workstations scenario

Should you choose it, this could be developed as to become a mainstream solution for seamless vm importing in libvirt. Minus the Whonix specific warnings this could be used generically for any vm and so could be offered for packaging in distro repos.

A simpler export script counterpart could be achieved, one that converts and sanitizes problematic xml tags in a machines configuration. This could be the xml of either an original or cloned machine. A user would need to only run this cleanup routine on a machine’s xml file to prepare it for proper portability. This would mimic the sanitization steps outlined on StackExchange but in an automated way for machine and network config files.

You mean skip adding the virtual network then?

This is not implemented yet, but should be.

How can the script check, if the virtual network already exists?

virsh outputs an error message. I am not sure, but can a script detect the error output of a command and act accordingly?

virsh net-autostart ./whonix_network.xml

For this we specify the network domain name as the component to be autostarted. So it becomes:

virsh net-autostart Whonix

The error output of what command?

It’s best to use exit codes or alternatively commands that are supposed to be used in scripts. Parsing other output can more easily break in later version.

For this we specify the network domain name as the component to be autostarted. So it becomes:

virsh net-autostart Whonix


Doesn’t work for me. First problem, it can be re-run and done multiple times.

[code]virsh net-define ./whonix_network.xml
Network vboxnet5 defined from ./whonix_network.xml

virsh net-define ./whonix_network.xml
Network vboxnet6 defined from ./whonix_network.xml[/code]

And the command you suggested doesn’t work. Perhaps because VBox is installed as well.

[code]virsh net-autostart Whonix ; echo $?
error: failed to get network ‘Whonix’
error: An error occurred, but the cause is unknown

1[/code]

I guess something like “vboxnet6” needs to be imported.

EDIT:
actually the auto-start command is a flag that is used after starting a network to tell livbirt to always automatically start this network upon its own running.

In virt-manager by clicking Start Network, or in virsh net-start. This command takes one mandatory argument, the network name. When starting a virtual network, libvirt will automatically set iptables and dnsmasq. However, transient networks are created and started at once.
virsh net-start Whonix

Will test and report back.

EDIT:

the tag for DNS domain name service on internal net, could be done without and so I will remove it on git. Even with another network name: whonix2, virsh blocks it from starting because it has the exact same network subnet range as its predecessor whonix. We have to include parameters to configure the network xml portion.

From the virt-manager GUI, creation of a network is not allowed if the subnet range is identical to an existing one.

From commandline, to make it work, I changed bridge name, network name and subnet mask

Writing that script generally still seems very wrong, like reinventing the wheel. I am not sure it would be accepted into distros. The GUI part should better be implemented in virt-manager. The CLI part, no idea, perhaps a good feature request for libguestfs or virt tools?

I am still wondering this is so apparently a missing feature and so complicated in comparison to VBox.

Is there at least a feature request for the apparently missing import/export GUI feature?

Maybe using virt-install would work better. See:
http://www.linux-kvm.com/content/copy-guest-another-system

Can virt-convert help us?

Have you seen this answer about cloning?

Other search terms:
“kvm” “ovf”
“kvm” “ofv”
"libvirt “ovf”
“libvirt” “ofv”
libvirt import export
kvm import export
virt-manager import export

I think there is something that can do all that. After all something as advanced as KVM surely has a way to automate all this since its designed for enterprise I will give it a look.

Yeah, KVM even supports live migration! So there must be an easy way to export/deploy/import VMs as well.

What Won’t Work:

This handled reproducing a vm from xml files.

They deprecated it I can’t believe this >:( they claim that no one used xml and relied no the crap that is ovf and so killed it off.

http://blog.wikichoon.com/2014/04/deprecating-little-used-tool-virt-image1.html

This one creates vm from scratch - like the gui counterpart in virt-manager. Is of no use to us since we want a unified configuration applied to all Whonix users everywhere

What Does Work:

It turns out that the solution that is now available is the virt-v2v tool’s import function :

https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Virtualization_for_Servers/2.2/html/Administration_Guide/virt-v2v-scripts.html#import-run-converted-vm

Handles xml for machines and networking. Last time I suggested it you refused because you thought that it would run and modify the vms - which would ruin verifiable builds. That may be true when virt-v2v its preparing a vm image and converting it, but as far as the importing is concerned none of that happens. The way the documentation is worded is confusing and this poinr is easily missed.
We already have the ‘appliance’ ready and just need to use an automated foolproof way to get it into libvirt and up and running. This is what this tool does for use and it supports networks configuration files as well as I confirmed from the manual.

During importing, it handles the mapping of virtual networks to the interfaces of whatever vm is assigned to it all of this data is applied on the fly from a virt-v2v conf file.

We can rely on it to do the rest

[quote=“HulaHoop, post:169, topic:166”]virt-install(1): provision new virtual machines - Linux man page
This one creates vm from scratch - like the gui counterpart in virt-manager. Is of no use to us since we want a unified configuration applied to all Whonix users everywhere[/quote]
Not sure.

In context of http://www.linux-kvm.com/content/copy-guest-another-system they are using.

sudo virt-install --name acce1 --ram=1024 --os-type=windows --os-variant=win2k3 --hvm --virt-type=kvm --import --disk=/kvm/images/rmtDC-clone.img --network bridge=virbr0

And the virt-install man page says.

--import Skip the OS installation process, and build a guest around an existing disk image. [...]

But I am not seeing an option to import an existing xml file using virt-install. Which we need for our required settings such as disable kvmclock. Maybe you find one.

virt-v2v sounds good. Can you figure out these questions?

Alternative idea…

virt-image:

Looks great:

But how could we use it:

Can you figure this out?

But I am not seeing an option to import an existing xml file using virt-install. Which we need for our required settings such as disable kvmclock. Maybe you find one.

Same here, and that’s why this is a no go.

virt-v2v sounds good. Can you figure out these questions? https://www.whonix.org/wiki/Dev/KVM#virt-v2v

virt-v2v assumes that you have an already attached vm in libvirt and uses that as input for the export->conversion->import process
Problem is, we don’t have a machine in libvirt to begin with and so for that reason its not useful for our purpose.

Front-end management tools for KVM like oVirt are beasts which is good in other situations but overkill to do something as simple as importing a vm. They require setting up network shares for storage pools even if imports are done locally AFAIK.

Before the last step GUI import, the user has to go thru commandline with virt-v2v

Alternative idea...

virt-image:

Looks great:

But how could we use it:

Can you figure this out?

Fedora was planning to work on a point and click appliance import solution in virt-manager but the effort stalled and never happened. Features/VirtAppliances - Fedora Project Wiki
The Thincrust appliance creator is of no advantage because we already have the vm template created. Also I don’t think it can configure timers either, take a look at the example recipes fed to the tool :
http://thincrust.org/ace-examples.html

While virt-image would have been the perfect solution, unfortunately it has been deprecated upstream and being pulled from distro repos. I had posted the link describing this and the absurd reasoning behind it a few posts above. We cannot base our solution on it because of that.

libguestfs only deals with modifying and injecting drivers/files in guests and is used by virt-v2v during the conversion process when transforming images used on other virtualizer platforms. It does not handle import or export in any way.

I am open to the idea of communicating with virt-manager to add this. And keep our fingers crossed that they will listen. Currently there is no GUI solution to accomplish this.

If there is anything I forgot to answer tell me.

Arguments against virt-v2v: virt-v2v use is still commandline based and will actually require editing of configuration files: http://linux.die.net/man/5/virt-v2v.conf
Lets admit that whoever can do this, might as well do it as it is now.

It is my opinion that we shelve the automated script idea completey and should instead do the following:

1) ship multiple templates according to containment systems we want to support, one for qemu, one for kvm 2) A readme file with advice :

The security warning I was telling you about the other day, that involves cross contamination.

and

includes commands for how to:

[a.] how to copy the sparse file properly to any location a user wants if they wish [b.] how to introduce the sparse file into privileged dirs like /var/lib.. [c.] how to import xml for machines and network [d.] how to autostart the isolated network [e.] how to edit xml files in the case that they want to use non-default paths or change virtually anything they want. [f.] how to setup shared folders and configure them to automount

The build process would be scripted to update the version number of the name and disk img xml fields and thats it. So revert back to just:
whonix_gateway_version no.
or the naming scheme with the dashes, instead of underscores, so no confusion arises when libvirt detects virtualbox machines

The absence of an easy way to import a machine is too big of an issue. We cannot compensate for a shortcoming as big as this coming from upstream. It would take tremendous effort as we have seen and some people might not be happy that it doesn’t do X. So I say we leave it as described above.

Patrick please reread my new posts before responding. If you read them a few hours back they would have been different, because I updated them according to any new info I came across which lead me to a different conclusion. By the time you respond they would have taken their final form however.

The opinions here superseded those on the wiki and may differ.

[quote]virt-v2v sounds good. Can you figure out these questions? https://www.whonix.org/wiki/Dev/KVM#virt-v2v[/quote]

virt-v2v assumes that you have an already attached vm in libvirt and uses that as input for the export->conversion->import process
Problem is, we don’t have a machine in libvirt to begin with and so for that reason its not useful for our purpose.


Maybe this is not a problem. For VBox builds, we also can only use export, because we added a VBox VM to the build system. (One that you could in theory boot.) So we can import the KVM VM during the build if that helps to export it. Would that be any help?

Good point noting that. I get e-mail notifications for new posts, but not for edited posts. I also get e-mail notification for new pull requests and new github comments, but not for commits added to an existing pull request.

Tried to simply kvm shared folder instructions. Please review.

Agreed with your post Whonix Forum.