ProxyVM + AppVM Development

Qubes-Whonix
41

[quote=“nrgaway, post:39, topic:512”]What stage was that within?

./whonix-install sources
./whonix-install build
./whonix-install template whonix-gateway[/quote]

It was within: ./whonix-install build

[hr]

Yes. I believe so.

[hr]

Is this for Fedora AppVM or chroot-wheezy?

In the Fedora AppVM:

sudo ls -lsa /home/user/.gnupg
total 24 4 drwx------ 2 root root 4096 . 4 drwx------ 17 user user 4096 .. 12 -rw------- 1 root root 9188 gpg.conf 0 -rw------- 1 root root 0 pubring.gpg 4 -rw------- 1 root root 600 random_seed 0 -rw------- 1 root root 0 secring.gpg

In chroot-wheezy:

cd /home/user/qubes-builder/chroot-wheezy/home/user
sudo ls -lsa .gnupg
ls: cannot access .gnupg: No such file or directory

[hr]

Didn’t see it in above places: “/home/user/.gnupg” or “/home/user/qubes-builder/chroot-wheezy/home/user/.gnupg”

42

You’re welcome, Patrick! After laying out my general understanding, let me know if you’d still like me to pose a specific question to qubes-devel.

[hr]

Yes. In general, TorVM seems to run a startup script (startup.sh) that auto generates a new /etc/tor/torrc file and dynamically populates variables.

See nrgaway’s post: Whonix Forum

[hr]

Either way…

  • Split off networking discussion.
  • Split off build discussion.
  • Or keep things integrated here until an initial ProxyVM + AppVM build is achieved.

I don’t mind.

I’m guessing there will likely be other development issues beyond networking and code building for this initial ProxyVM + AppVM development work.

Whatever works for you guys.

43

Give me till the end of the weekend on the IP issue. Now you have detailed what each package is used for in the Wiki I can manually make changes on my end and when it starts working where I can get a connection to outside world, etc I will have a better idea how to handle this.

As for the torrc, I will just set up a udev rule that will trigger a search and replace on the torrc file; that’s how torvm does it, so it may not be that difficult. It looks like the udev rule is also responsible for activating the virtual interface.

Now, on to that build error. I get it too with a fresh VM install. I kind of remember I had to create a trust key when I first started out, so I will get back to you when I figure out what to do again :slight_smile:

44

[quote=“WhonixQubes, post:37, topic:512”]White trying to build, I got a fatal build error:

I will try running through it again and see if this fatal error reoccurs.[/quote]

Looks like .gnupg is owned by root, so change permissions to user and it will work

sudo chown -R user.user ~/.gnupg

I added this to the whonix-install script; not sure when that directory gets created so if you get error on build, just run build again and the problem should be resolved.

But since you have older version of whonix-install, just run the command

45

Great.

Ok.

Good to know it wasn’t specific to my machine.

[quote=“nrgaway, post:44, topic:512”]Looks like .gnupg is owned by root, so change permissions to user and it will work

sudo chown -R user.user ~/.gnupg

I added this to the whonix-install script; not sure when that directory gets created so if you get error on build, just run build again and the problem should be resolved.

But since you have older version of whonix-install, just run the command[/quote]

Yup. Will do! Thanks for your awesome work on this project! :slight_smile:

46

Sounds good on the IP topic.

[hr]

Reading user’s ~/.gnupg in a build script sounds like a bug anyhow.

For Whonix something like this is in use…

gpg \
   --no-options \
   --no-default-keyring \
   --homedir "$some_folder_variable" \
   ...

Depending on what you are doing, you may need a temporary/empty/non-empty folder.

47

Interesting idea!

Three things coming to my mind, you probably know this already, just in case:

*idempotent is defined here as per debian policy (6. Package maintainer scripts and installation procedure — Debian Policy Manual v4.6.2.0) “Maintainer scripts idempotency”. In short: it shouldn’t matter if one manually re-runs the script several times in a row and it would be best if it could recover if the script was terminated in the middle (unlikely, but possible due to power failure, user error, etc.).

48

Hi!

If Whonix gateway IP address is hardcoded in too much places, you can just leave it as it is and use some NAT rules to hide it from Qubes infrastructure. Something like this:

  1. Workstation uses IP assigned by Qubes.
  2. Gateway uses IP assigned by Qubes, and has additional dummy interface (modprobe dummy) with IP 10.152.152.10
  3. Gateway: iptables -t nat -A PR-QBS-SERVICES -d 10.152.152.10 -j REDIRECT (PR-QBS-SERVICES is called from PREROUTING, specifically for this purpose)
    This way whatever is sent to 10.152.152.10, it will be intercepted by the gateway.
    The same approach is already used in Qubes for updates proxy - VM can send traffic to 10.137.255.254 and it will be intercepted by updates proxy, wherever it is. Details here: https://wiki.qubes-os.org/wiki/SoftwareUpdateVM#Technicaldetails

This wont be as simple for workstation IP, but I guess it is much less problematic to change its IP.

Also I’m thinking how to make installation process as simple as possible. Is it possible to take a clean wheezy installation, add some apt repo and just install some metapackage, which will pull all the required stuff? If yes, IMHO it is the way to go. If no, can you still separate scripts for building clean wheezy from whonix? This way review would be much easier and your work will be reusable for other templates. In fact the current situation is almost good (at least in linux-template-builder), the commit history just contains some mess around it - perhaps I should look at the final diff (origin/master…nrgaway/master), not each commit separately…

Regarding AppVM/StandaloneVM - I assume you need some configuration/data persistence. Using StandaloneVM is rather inconvenient - at least because of disk space, but also more VMs to update. The easiest way would be to store such files in /rw (/rw/config? /rw/whonix?) and simply place symlinks in original location pointing at those files. Then add some startup script which will initialize /rw on the first boot. Keep in mind that template’s private.img isn’t accessible for VMs, so files to initialize /rw should be stores somewhere else.

For example I have this in /rw/config/rc.local in my UsbVM (where bluetooth adapter is attached):

rm -rf /var/lib/bluetooth 
ln -s /rw/config/var-lib-bluetooth /var/lib/bluetooth
49

Great idea about solving the IP question by using iptables!

I just updated the wiki a few moments ago to answer some questions (you were not the first to ask, and I wanted to do this for some time now).

This page is supposed to answer the feasibility of installing Whonix from a apt repository:

Chroot-scripts get into the way, but likely there are solutions for that:

50

[quote=“marmarek, post:48, topic:512”]Hi!

If Whonix gateway IP address is hardcoded in too much places, you can just leave it as it is and use some NAT rules to hide it from Qubes infrastructure. Something like this:

  1. Workstation uses IP assigned by Qubes.
  2. Gateway uses IP assigned by Qubes, and has additional dummy interface (modprobe dummy) with IP 10.152.152.10
  3. Gateway: iptables -t nat -A PR-QBS-SERVICES -d 10.152.152.10 -j REDIRECT (PR-QBS-SERVICES is called from PREROUTING, specifically for this purpose)
    This way whatever is sent to 10.152.152.10, it will be intercepted by the gateway.
    The same approach is already used in Qubes for updates proxy - VM can send traffic to 10.137.255.254 and it will be intercepted by updates proxy, wherever it is. Details here: https://wiki.qubes-os.org/wiki/SoftwareUpdateVM#Technicaldetails

This wont be as simple for workstation IP, but I guess it is much less problematic to change its IP.

Also I’m thinking how to make installation process as simple as possible. Is it possible to take a clean wheezy installation, add some apt repo and just install some metapackage, which will pull all the required stuff? If yes, IMHO it is the way to go. If no, can you still separate scripts for building clean wheezy from whonix? This way review would be much easier and your work will be reusable for other templates. In fact the current situation is almost good (at least in linux-template-builder), the commit history just contains some mess around it - perhaps I should look at the final diff (origin/master…nrgaway/master), not each commit separately…

Regarding AppVM/StandaloneVM - I assume you need some configuration/data persistence. Using StandaloneVM is rather inconvenient - at least because of disk space, but also more VMs to update. The easiest way would be to store such files in /rw (/rw/config? /rw/whonix?) and simply place symlinks in original location pointing at those files. Then add some startup script which will initialize /rw on the first boot. Keep in mind that template’s private.img isn’t accessible for VMs, so files to initialize /rw should be stores somewhere else.

For example I have this in /rw/config/rc.local in my UsbVM (where bluetooth adapter is attached):

rm -rf /var/lib/bluetooth ln -s /rw/config/var-lib-bluetooth /var/lib/bluetooth [/quote]

In some of my more recent builds, I have already separated out Whonix from Wheezy installation. Whonix now get installed as a flavor, like fedora uses for minimum install templates. So linux-template-builder can now create a Jessie only, Wheezy only, Jessie+Whonix-gateway/workstation (untested), Wheezy+Whonix-gateway/workstation, or ability to add in any other custom template flavors since I added hooks for that.

The commit history is a mess since I had to try many different configurations to get things to work. I will build a clean commit history once everything is working.

I have not had much time this weekend to work on the IP challenges, but will take a look at your solution as well now. It seems very interesting.

The requirement of needing a standalone AppVM is only for testing purposes and will not be a requirement once I have this working. I still need to work on the ability to update the main template though and it MUST be updated via a tor / Whonix connection, not via clearnet.

One other thing I am working on is the ability to install Whonix using Salt-Stack’s salt-minion. This should allow Whonix to be installed on any Linux platform (using my custom formulas). I can complete this once I get everything else working.

51

[quote=“marmarek, post:48, topic:512”]Hi!

… In fact the current situation is almost good (at least in linux-template-builder), the commit history just contains some mess around it - perhaps I should look at the final diff (origin/master…nrgaway/master), not each commit separately…[/quote]

Just wanted to further clarify that the whonix branch within linux-template-builder is a WIP branch with dirty commits. It is not meant for use other than testing out different builds. I will clean the commit history and merge into a fresh development branch once my testing is complete which will allow a clean merge or ability to cherry pick.

52

Hi Marek. Welcome! :smiley:

[quote=“marmarek, post:48, topic:512”]If Whonix gateway IP address is hardcoded in too much places, you can just leave it as it is and use some NAT rules to hide it from Qubes infrastructure. Something like this:

  1. Workstation uses IP assigned by Qubes.
  2. Gateway uses IP assigned by Qubes, and has additional dummy interface (modprobe dummy) with IP 10.152.152.10
  3. Gateway: iptables -t nat -A PR-QBS-SERVICES -d 10.152.152.10 -j REDIRECT (PR-QBS-SERVICES is called from PREROUTING, specifically for this purpose)
    This way whatever is sent to 10.152.152.10, it will be intercepted by the gateway.
    The same approach is already used in Qubes for updates proxy - VM can send traffic to 10.137.255.254 and it will be intercepted by updates proxy, wherever it is. Details here: https://wiki.qubes-os.org/wiki/SoftwareUpdateVM#Technicaldetails

This wont be as simple for workstation IP, but I guess it is much less problematic to change its IP.[/quote]

I’m wondering how this solution would end up affecting the user-level functionality of a ProxyVM Gateway + Dynamic AppVMs.

For example, based on my current understanding with Qubes Fedora-based ProxyVMs and AppVMs…

Let’s say the ProxyVM receives an IP address of 10.137.2.5.

Let’s say 3 AppVMs are networked (NetVM/backend) to the ProxyVM and receive the following IP addresses:

  • AppVM-1 IP: 10.137.3.6
  • AppVM-2 IP: 10.137.3.7
  • AppVM-3 IP: 10.137.3.8

Inside the ProxyVM, the following vif interfaces would be dynamically created for connectivity with the AppVMs (where 51,52,53 are used as example Xen VM IDs):

  • vif51.0 IP: 10.137.3.1 (backend for AppVM-1)
  • vif52.0 IP: 10.137.3.1 (backend for AppVM-2)
  • vif53.0 IP: 10.137.3.1 (backend for AppVM-3)

So the vifs in the ProxyVM use 10.137.3.1 as the IP which is used for the networking gateway address with AppVM eth0 interfaces.

My concerns with the iptables static redirect approach are:

  • a) Would the Whonix-Gateway ProxyVM act like a “native” Qubes ProxyVM where other AppVMs can dynamically make use of its network connectivity?

  • b) Would multiple Whonix-Workstation AppVMs be able to dynamically utilize a Whonix-Gateway ProxyVM via the VM Manager GUI?

Just on the surface, without having to do manual network configuration work on the user’s part, it seems the iptables redirect hack might limit the implementation to using a single AppVM with the ProxyVM, and specifically only using a Whonix AppVMs with a Whonix ProxyVM.

Trying to think ahead for Joanna’s intention to integrate Whonix along with TorVM as a native Qubes user experience in the official Qubes installer.

P.S. As a reference, here is the post where I laid out more of my comprehensive understanding of Qubes networking:

53

Question: Are you going to implement that iptables exception suggested by Marek for all versions of Whonix? I’d urge you to reconsider as spoofing IP addresses is very easy and can fool firewalls.

54

I don’t think so, only in worst case scenario as Marek suggested. There are many places that have a hard coded IP address today, but I imagine it will not be too difficult to lower that number considerably.

So far I have been successful using a search and replace on required files. Once I can prove everything is working with the qubes install scripts we will know exactly what steps are required to get things going and at that point can figure out the best way to proceed.

I am very close to getting the ProxyVM working. I had it going a few hours ago, just rebuilding another version with the changes I applied manually to it to get it working.

55

Had good progress tonight.

Was able to get ProxyVM working, although I had a few manual tweaks ([url=https://github.com/nrgaway/qubes-builder/blob/master/whonix-install.NOTES]https://github.com/nrgaway/qubes-builder/blob/master/whonix-install.NOTES[/url]) to apply in order for it to work.

Whonixcheck mostly completed complaining about hostname. Guess I should rename it host. Time sync failed though. I was able to apt-get update repo in gateway, and have an AppVM connect via proxy port 9050.

I was also able to get the AppVM to connect without a proxy if I used torvm firewall settings([url=https://github.com/nrgaway/linux-template-builder/blob/master/scripts_debian/custom_wheezy_whonix-gateway/extra-whonix-files/usr/lib/whonix/start-whonix-proxy.sh]https://github.com/nrgaway/linux-template-builder/blob/master/scripts_debian/custom_wheezy_whonix-gateway/extra-whonix-files/usr/lib/whonix/start-whonix-proxy.sh[/url], which seem to forward the vif ports via NAT to the forward facing eth0 interface. Note that I am not using this configuration file anywhere, I just ran it to see if it would work and it did allow the AppVM to not use a proxy (confirmed with check.torproject.org).

Where is Whonix firewall rules and how / when does it get started? I will look into the firewall more tomorrow.

56

Yes. Unfortunately, some upstream packages don’t allow variables. And please keep in mind, that search and replace in /etc/* would cause an interactive dpkg conflict resolution dialog on upgrade.

Once I can prove everything is working with the qubes install scripts we will know exactly what steps are required to get things going and at that point can figure out the best way to proceed.
Sounds good. Feel free to postpone the IP discussion to a comfortable time.
Whonixcheck mostly completed complaining about hostname. Guess I should rename it host.
Yes.
Where is Whonix firewall rules and
whonix-gw-firewall: /usr/bin/whonix_firewall - https://github.com/Whonix/whonix-gw-firewall/blob/master/usr/bin/whonix_firewall
how / when does it get started?

By an eth0 pre-up hook in /etc/network/interfaces (https://github.com/Whonix/whonix-gw-network-conf/blob/master/etc/network/interfaces.whonix#L24) before eth0 gets up.

57
I don't think so, only in worst case scenario as Marek suggested. There are many places that have a hard coded IP address today, but I imagine it will not be too difficult to lower that number considerably.

I see. But lest say it comes to the worse, will that feature just affect qubes or is it a change in Whonix itself for all versions?

58

[quote=“nrgaway, post:55, topic:512”]Had good progress tonight.

Was able to get ProxyVM working…

I was also able to get the AppVM to connect…[/quote]

Awesome. Really exciting to see you make steady progress! :smiley:

59

My guess (although others my know better) is that, if utilized, it would be specific to the Whonix Qubes implementation only. I say that without knowing for sure, and just with a surface-level overview of what was mentioned being involved.

We will at some point have to figure out how and where we want to package and deliver a Whonix Qubes ProxyVM + AppVM solution.

The way we choose to accomplish this packaging and delivering might also have an affect on which components make sense to bake into Whonix proper and which to keep outside of it.

Right now, a lot of things with ProxyVM + AppVM development are still up in the air, TBD, not final, and subject to change/optimization.

60

[quote=“WhonixQubes, post:52, topic:512”]My concerns with the iptables static redirect approach are:

  • a) Would the Whonix-Gateway ProxyVM act like a “native” Qubes ProxyVM where other AppVMs can dynamically make use of its network connectivity?

  • b) Would multiple Whonix-Workstation AppVMs be able to dynamically utilize a Whonix-Gateway ProxyVM via the VM Manager GUI?[/quote]

Yes and yes.

ProxyVM will still be available at its original (Qubes-assigned) IP, so all the routing will be still working. The only thing that changes is that the traffic directed for some special IP (10.152.152.10) will be intercepted there. And perhaps - if you want to support normal AppVMs (which know nothing about tor) - it should intercept all the traffic, same as TorVM. That trick with iptables is to not change IP in many config files. A second though - when the IP will be set on some interface (dummy0), perhaps even iptables rule will not be needed there - thanks to routing, traffic should be simply routed to that dummy0 local interface. But still - if you want to allow usage of normal AppVM (which is not configured to proxy traffic through specific IP), you should add some iptables rule to intercept all the traffic.

Workstation IP should still be assigned dynamically by Qubes tools, so will be unique (no conflicts). And should use ProxyVM IP (10.137.x.y) as its default route. But can assume that Whonix Gateway is available at 10.152.152.10.