[Help Welcome] KVM Development - staying the course

KVM
101

That’s wrong. There are multiple configurations preventing Whonix-Gateway from connecting directly to clearnet.

This is probably not a good thing as an ISP can know if its customers are running Whonix if and when an update is made using apt.
Agreed.
I think this is very important to maintain homogenous anonymity with the normal TBB crowd.
Agreed.
Can you please consider torrifying all the gateway traffic to mask this?
This already is the case. At least for VirtualBox / bare metal version. Looks like a grave Whonix KVM related bug.
102

That is more than fast enough.

So does whonixcheck still have the same error or does it work now?

since disabling transproxy, timesync no longer works.
Made a note under blockers: https://www.whonix.org/wiki/Dev/KVM#Sdwdate_breaks_when_Transparent_Proxy_is_disabled
As for 8.2, the images are working fine and decompressed fine.
Terrific.
Please add links for the qcow2 images on the download page for them to be accessible to more users.
I don't know how to best add it to that page. There is quite a lot to be added to this page now and in future: https://www.whonix.org/wiki/Dev/Download_Wizard No idea how to organize so much information best.
103
That's wrong. There are multiple configurations preventing Whonix-Gateway from connecting directly to clearnet.

Ok I see what you mean. I assumed that this wasn’t happening when I saw Transproxy disabled in the gateway firewall config file.

So does whonixcheck still have the same error or does it work now?

Same error unfortunately.

don't know how to best add it to that page. There is quite a lot to be added to this page now and in future: https://www.whonix.org/wiki/Dev/Download_Wizard No idea how to organize so much information best.

I mean this page: Download Whonix (FREE)

104

Ok I see what you mean. I assumed that this wasn’t happening when I saw Transproxy disabled in the gateway firewall config file.

Same error unfortunately.[/quote]
This is really strange. Now I am out of ideas.

[quote] don't know how to best add it to that page. There is quite a lot to be added to this page now and in future: https://www.whonix.org/wiki/Dev/Download_Wizard No idea how to organize so much information best.[/quote]

I mean this page: Download Whonix (FREE)


I mean that page as well.

105

Sorry correction: timesync does indeed still function even if transproxy disabled. This is confirmed by running it on the clean image for 8.2. My unclean shutdowns and hard resets in the older vm have something to do with it breaking permanently, but that has nothing to do with the transproxy. ld be removed from blockers. So I think this should be removed from blockers, I am sure about this.

Leak still applies in 8.2 however.

106

Removed.

107

HulaHoop, looks like zweeble made considerable progress! :slight_smile: You may with to check this out:

108

Now that the leak issue has been clarified and discovered to be not an issue, I want to ask what I should concentrate on next to bring official support closer.

Should I dump the xml config files for gateway and workstation with desired parameters and post them here?

109

Here is the link for reference:

Yes, please post to the wiki.

110

Is Dev/KVM - Whonix up to date so far?

111

The main blockers seem to be solved, but I need input on that.
Other things that are solved but still as blockers like the virtio-blk, shared folders etc. For virtio-blk I had edited grub parameters for it to work. This will probably need a permanent fix in the repo to solve.

I did some digging on the xml solution we thought of and found out that any iser who hopes to run their machine will need to edit their copy of the configuration file to have their own path to the image file and to mask the cpu to match their own cpu model. Disk UUID I’m not sure of, does the image have a uniform UUID or does it depend on an arbitrary one assigned to it by each system its used on?

Either way, ts clear to me that a click and set model like the ova import feature is not possible on KVM right now unfortunately. Should we instead stick with adding instructions for simple manual edits to be done to the xml like clock settings, graphics ram and the like, but we rely still on manual steps for the large part.

112

No worries, I’ll answer the other points you raised later when I processed them. :slight_smile: For now only this…

Added some stuff to Dev/KVM - Whonix. Unfortunately, it raises new issues.

113

That implies that there are two different types of blockers (main blockers and blockers). Seems confusing. Either a blocker or not a blocker. Anyway.

but I need input on that.
https://www.whonix.org/wiki/Dev/KVM#Instructions_include_how_to_set_clock_to_UTC.3F and https://www.whonix.org/wiki/Dev/KVM#How_to_use_-rtc_clock.3Dvm_.3F are solved.

But there is a new blocker:

KVM instructions have not been updated yet including new subnet from Whonix Forum. And is there anything else from Whonix Forum that should result in a change of the Dev/KVM or KVM wiki page?

Other things that are solved but still as blockers like the virtio-blk, shared folders etc.
I don't know why virtio-blk is considered a blocker. Please add the reason to the wiki. I don't know the problem with shared folders, please add it to Dev/KVM as well. While I am not considering those as blockers (yet, maybe you explain), since you're taking the lead on KVM support, you're free to move those items to blockers.
For virtio-blk I had edited grub parameters for it to work.
You mean https://www.whonix.org/wiki/Dev/KVM#System_gets_unbootable.3F and http://grumpymole.blogspot.nl/2007/05/ubuntu-how-to-edit-grub-boot-parameters.html ?

That info comes from a Ubuntu centric page. Does it work in Debian as well? Have you tested that solution already?

This will probably need a permanent fix in the repo to solve.
How could that change look like?
I did some digging on the xml solution we thought of and found out that any iser who hopes to run their machine will need to edit their copy of the configuration file to have their own path to the image file and to mask the cpu to match their own cpu model.
I see.
Disk UUID I'm not sure of, does the image have a uniform UUID or does it depend on an arbitrary one assigned to it by each system its used on?
As for KVM disk identifiers, maybe KVM assigns a disk identifier on its own to any image, no idea, you tell me. As for the uuids (" ls -l /dev/disk/by-uuid"), those aren't fixed in Whonix 8 unfortunately (Whonix 8.1 has a different one than 8.2 and so forth), but those will be fixed in Whonix 9 and above. [Has been implemented to ease verifiable builds.]
Either way, ts clear to me that a click and set model like the ova import feature is not possible on KVM right now unfortunately.
Yeah.
Should we instead stick with adding instructions for simple manual edits to be done to the xml like clock settings, graphics ram and the like, but we rely still on manual steps for the large part.
I still preferred this: https://www.whonix.org/wiki/Dev/KVM#Script.2Fautomate_creation_of_VM_description_files.3F

Same in other words: doing it similarly as https://github.com/Whonix/Whonix/blob/Whonix8/build-steps.d/2600_create-vbox-vm does.

Same in easier words:

With VirtualBox, we’re doing it like this…

## Create a new VM.
VBoxManage createvm --name "Whonix-Gateway" --register

## Adjust RAM.
VBoxManage modifyvm "Whonix-Gateway" --memory 768

## Add eth0 and  configure as NAT.
VBoxManage modifyvm "Whonix-Gateway" --nic1 nat

I.e. are creating all VM settings using the VBoxManage command line utility. No steps using mice required.

Now, KVM can be used in that way as well. It’s all in the man page / docs. Creating the VM settings would be as simple as copying and pasting these commands. And once we’re there, I am happy to script these commands to automate them.

114

Before making any changes because of virtio, I think we really need to find the answer to this one first:

Otherwise we might make changes, that don’t even really enable virtio, because we have no way to check it really is enabled.

115

Semantics mistake, I see you label them as blockers vs non-blockers rather than major vs minor. Either way what I meant was that these ones I thought were resolved.

I might try figuring out the virtio-blk workaround I posted, but its really a non-priority, if we can get everything else working.

For subnet range please describe whats the problem in layman terms. As I said before, I see that no matter what subnet I assign on Libvirt, the guest is free to assign any address that suits it at will. I don’t see a problem with things as they are now. Gateway uses default NAT with no problems.

For the export of a vm I believe I found a solution in a tool called virt-v2v. It is the kvm equivalent for export/import of a vm “appliance” and its configuration. Part of the package is virt-p2v that converts a physical host into a vm image but its not relevant to what we’re doing here.

virt-v2v is supposed to handle the virtio drivers part of the conversion, enabling it on the Linux guests it converts, which might solve the problem without an further intervention or editing of grub files on the image. If it doesn’t work out of the box with virt-v2v, then lets shelve it and forget about it.

Even though it was made by Redhat for RHEL, it is packaged for and works with other distros. This utility can be scripted too. Importing an image is handled by it as well.

https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Virtualization_for_Servers/2.2/html/Administration_Guide/virt-v2v-scripts.html#converting-guests

If it does work, we can then discuss the details of what virtual hardware should be.

EDIT:

Non-important Link for reference about someones experience with virt-v2v involving other non-RHEL distros:
http://lists.ovirt.org/pipermail/users/2014-January/020283.html

Note that Sparse file allocation needs to be specified to avoid this turning into a massive image, should work with v2v exports too:
https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/V2V_Guide/Preperation_Before_the_P2V_Migration-Define_a_Host_Profile_in_virt_v2v.conf.html

116

For me, blockers means “unfit for release”, stuff like “as long as this is unsolved, users are put at risk and/or users won’t ever come back” [theoretic reason for definition: very poor performance].

Major/minor is something I don’t understand well. Does it mean importance? Implementation difficulty? Implementation effort? A mixture of these?

For subnet range please tell me more whats the problem in layman terms. I see that no matter what subnet I assign on Libvirt, the guest is free to assign any address that suits it at will. I don't see a problem with things as they are now. Gateway uses default NAT with no problems.
I didn't follow https://www.whonix.org/forum/index.php/topic,328.0.html because I haven't tried myself in KVM. What was it that causes you trouble (whonixcheck), that zweeble did different in https://www.whonix.org/forum/index.php/topic,328.0.html? KVM instructions have not changed since then.
1. How about we see if everyone who uses an image has the same uuid?

On the host run and paste result here for the 8.2 workstation image:
virsh domuuid [domain-id or domain-name]


Sure, we can try.

Interestingly also shows VBox VMs…

virsh list

Shows powered off VMs as well…

virsh list --all

Since the following command does not take an argument for vm image…

virsh domuuid Whonix-Workstation 493a78d1-829a-4ad7-9183-25430ba61239

I am quite sure, this is an non issue. That ID is only used to correlate “Whonix-Workstation” with the unique ID “493a78d1-829a-4ad7-9183-25430ba61239”. It most likely has nothing to do with harddrive uuids. I guess adding extra hdds to the VM won’t ever change that ID.

Most likely virsh has a command to add images to VMs using VM name and image path.

For VBox for example the following can be used.

VBoxManage storageattach "Whonix-Gateway" --storagectl "Whonix-Gateway" --type hdd --port 0 --medium "/home/user/VirtualBox VMs/Whonix-Gateway/Whonix-Gateway.vdi"

I would really wonder if virsh/KVM had no similar command.

2. I need to know how to extract a tar image to the privileged directory: /var/lib/libvirt/images/ That way the image path should be the same for everyone.
Done. That information has been added to https://www.whonix.org/wiki/KVM#KVM_Setup_Instructions.
3. Host cpu model fallback is a problematic instruction that is bound to conflict with others' machines since it assumes the same cpu model for everyone. I need to disable that.
If that's possible. Please consider creating a TODO item(s) on Dev/KVM. Do we want a KVM specific development forum where only single KVM specific points are discussed?
117

Hi Patrick I have edited my post alot yesterday so please check out the tool virt-v2v. Its a more refined solution. If you set up a test KVM deployment and export a machine, I would be willing to test the results.

118

Why me? I still think for KVM support I need a lot help. Seems to cause only support overhead while suiting very few users.

It would be interesting to see how it handles virtio, though. Then we might be able to re-use that solution for Whonix.

I don’t think virt-p2v / virt-v2v are good solutions to deploy official Whonix images for many reasons:

  • required booting a VM and running the tool inside
  • therefore, builds can not be fully automated
  • therefore, there is always some unprofessional “boot the VM, then click this and this” type instructions, that may be done wrong
  • booting a VM results in starting of lots of services (Tor etc.) that create private files (such as /var/lib/tor storing Tor guards relays)
  • Such files should not be deployed and cleaning them up is difficult. It’s better not to start any services while building the image.
  • Also such files mess up Verifiable Builds - Whonix.

There is no way to avoid the hard work. Someone has to bite the bullet and read the actual virsh man page virsh(1) — libvirt-clients — Debian bullseye-backports — Debian Manpages. Then implement an alternative to https://github.com/Whonix/Whonix/blob/master/build-steps.d/2600_create-vbox-vm using virsh. I.e. see command
VBoxManage storageattach “Whonix-Gateway” --storagectl “Whonix-Gateway” --type hdd --port 0 --medium “/home/user/VirtualBox VMs/Whonix-Gateway/Whonix-Gateway.vdi”
and translate it into virsh style. Otherwise easy KVM support does not come any closer.

119
Why me? I still think for KVM support I need a lot help. Seems to cause only support overhead while suiting very few users.

This wasn’t meant as a disrespect. The reason I suggested you, is that A. you know more than me B. your machineis the dev machine and I have no way to give you whatever I would have created to test for feedback. Anyhow thanks for saying why this wouldn’t work.

We don’t know how many people use KVM, so we shouldn’t assume they are few. Linux users tend to be proficient in using computers in general and so you wouldn’t have seen complaints from that usergroup in general. Also those who are happy campers don’t tend to write to say that they are, its only when shit hits the fan :wink:

I will see what I can do with virsh, I hope I can get far this time round.

120

Alright so I read the manual for virsh more carefully this time and here is what it’s about, it tells a user how to control and manage a vm running under virsh, but doesn’t really have much to do in the way of creating a vm. The creation step is hinted at to involve dumpxml.

I did some digging around and I think I found the answer this time. Check the second answer with 9 upvotes to see what I mean:

virt-clone is supposed to deal with duplication and also do it in a way that makes it work for other machines host machines, bypassing uuid and folder path problems:

Theoretically if this works all a user needs to do is to import 3 xml files, 2 vm and 1 virtual network, through commandline and be done.

Manual I found:

Patrick please understand that I am not asking you to do anything here, just need to know if the tools I mention here is a step in the right direction and satisfy the verifiable builds property of Whonix.