ProxyVM + AppVM Development

I don’t know how far you have gotten in getting Whonix working as an appvm but I thought I would share my code to build it as a appvm / proxyvm if you were interested.

I couldn’t figure out how to get the HVM to go full screen (greater than 1280x800 or something like that) and was also having the same issue of TorTransport not reachable so I decided to build Whonix as an AppVM and set the Gateway up as a Qubes Proxy VM using the Qubes template builder.

I have successfully been able to build both the workstation and gateway as an AppVM which can then be used to create a ProxyVM.

To build, just clone my repo git@github.com:nrgaway/qubes-builder.gitin a fedora appvm (with lots of private user space… I used 40GB; 15-20 may work) and run ./wheezy-install in the qubes-builder directory. It should set up all the fedora depends you need and start building. I tried to make the build process as simple as possible for myself and it will patch qubes and whonix where needed.

The original template builder had issues build a wheezy template, so I added tweaks to make it build a wheezy template. The Whonix gateway / workstation build options are as follows:

sudo ~/Whonix/whonix_build \
  --build "$1" \
  --64bit-linux \
  --current-sources \
  --enable-whonix-apt-repository \
  --whonix-apt-repository-distribution stable \
  --bare-metal \
  --skip-verifiable \
  --skip-sanity-tests

The build process is still WIP. For instance, when Whonix is building, you have to manually interject twice at this point. The first time it will fail on grub-pc even though I marked it as a held package, so you need to uninstall grub-pc at that point (instructions in wheezy-install.README). Also for some reason I cant umount the directory after installing Whonix (working on that issue today) so you may need to shutdown the development appvm (or umount -l /qubes-builder/qubes-src/linux-template-builder/mnt) and comment out the part in Makefile that calls qubeize.

Once the template is built, install it in dom0 (instructions in README). Run it as a normal VM (not custom). Also, at least for now, when creating your AppVM or ProxyVM from the template, choose as a standalone, as it is easier to debug. The first time you start the appvm/proxyvm, wait until the vm stops (look in logs for this indication) since thats when Whonix does its initialization thing. Then restart it again and use Terminal to access (All Whonix programs also available from menu).

I am now currently working on configuring the ProxyVM so it actually works :slight_smile: This will take tweaking since it expects to have a eth0 of 10.152.152.x and ProxyVM has one of 10.137.X.X (most likely 10.137.3.X if this is the 3rd netvm you installed). I already wrote a script for torrc as follows that needs to be run, and have to search the Whonix code to see where it is relying on an interface of eth0 (proxyvm creates a new interface (vifX.0) for each appvm connecting.

I am unsure at this point what else needs to be tweaked, but will dig into it further over the weekend.

#!/bin/bash

# TODO: Obtain automatically
IP=10.137.3.1

cp /usr/share/tor/tor-service-defaults-torrc /etc/tor/torrc
sed -i 's/10.152.152.10/'$IP'/g' /etc/tor/torrc
cat /etc/tor/torrc
service tor restart

Anyway, I can be available on IRC or here to help you though the build process if you are interested. I only started using Qubes a few weeks ago after learning about your release of Whonix on it so there may be better ways of doing things than what I have done. Since I only started coding this three days ago, expect build errors; although I have been able to build with only manually interject 3 times. Will get better as time progresses.

I also am going to be adding encryption the the workstation appvm as part of the Whonix build process so it will be available to other platforms as well.

Split this off into a new topic…

Wow nrgaway, you are an angel from above! :smiley:

I haven’t yet begun any development on this, but, you’ve gotten further faster than I likely would have, and are really upping the Qubes + Whonix game for us now. Bravo! This is awesome!

I’ve got a couple things to finish up and then I’m going to dive into testing your work for ProxyVM + AppVM support.

If you’re willing, I’d be happy to work with you to get this further tested, developed, and supported as a primary implementation of Qubes + Whonix.

Joanna (Qubes founder) expressed interest to me for supporting Whonix as an alternative that gets natively built-in to the Qubes installer. So this native ProxyVM + AppVM work of yours could be a big step in making this awesome leap for us.

Feel free to further use this thread like a development log as you make improvements.

I will certainly be diving into testing out your work and posting my experiences and feedback here.

Again… Bravo nrgaway! Great job so far! :smiley:

Sure, I can use this thread as a log.

One thing I wanted to point out in case others want to build using my repo. My repo is not signed at this point and therefore can not be verified. Also I have been unable to verify the Whonix submodules at this point. So only use for development purposes until we can create a build that is verifiable.

Also forgot to mention, when cloning the ‘qubes-builder’ package from my repo, you need to checkout the ‘whonix’ branch or you wont be able to see the wheezy / whonix related tweaks :slight_smile:

wheezy-install has been renamed to whonix-install

Be careful using ./wheezy-install clean since it cleans everything; l lost 3 hours of work since I did not commit my changes in linux-template-builder :frowning:

Guess I better add a clean-all option

Sorry about that. In Whonix 9 that must be.

  --whonix-apt-repository-distribution wheezy

(Stable is the Whonix 8 repository.) (By running “sudo whonix_repository” you can also fix that after having installed.)

  --bare-metal

Please make that.

--install-to-root

(It’s the same, but more logical when using it in VMs.)

  --skip-verifiable

You likely want to combine this with.

--minimal-report

Thanks Patrick. I have implemented those changes and have build a workstation and currently building a gateway with them.

I have a few more questions I hope can be answered:

  1. What does the ‘Initial Boot’ of Whonix actually do? When building on a ‘–install-to-root’ VM, as I am with Qubes, is it possible just to run that script in setup since there is no grub within Qubes? Maybe just point me to the boot script location and I can see what it is doing.

  2. Are there many packages in the Gateway that depend on eth1 being 10.152.152.10? I have already created a script to modify the torrc on the fly and was wondering if there are more packages such as firewall I need to look at. If there is a huge list, there is no need to list everything, I can begin the process of auditing each package then (which I don’t want to do if I don’t have to).

Isn’t related to grub.

The context is:

Package:

During the build process it runs this:
https://github.com/Whonix/whonix-initializer/blob/master/usr/lib/anon-dist/chroot-scripts-post.d/80_cleanup

On first boot it does this:
https://github.com/Whonix/whonix-initializer/blob/master/usr/lib/first_run_initializer

For your use case it would be best if you could avoid installing the whonix-initializer package altogether, I think. But this is a bit difficult. I have it on my list to re-think this. Unfortunately whonix-initializer is a dependency of whonix-shared-packages-dependencies.

I wouldn’t mind if you remove whonix-initializer or skip installing it. But then you would have to manually keep care of having installed packages that whonix-shared-packages-dependencies would install. The whole packaging Recommends vs Depends vs --[no-]install-recommends is a mess or I am overlooking something. But probably not. In essence…

  • Recommends doesn’t ensure the package really gets installed
  • When using Recommends, we cannot use --no-install-recommends obviously.
  • When using Recommends with --install-recommends (which is the default), lots of irrelevant extra packages, that are set as Recommends in packages we recommend get installed as well.
    Haven’t see or found a great solution to this.

The most reliable and always most up to date way to get a full list is running this command in Whonix’s source code.

grep -r 10.152.152.10 *

See also chapter “generic packaging” (just that one):

Thanks.

For testing purposes, I will to a mass search and replace of those IP addresses within the Whonix source code before installing and replace the IP address with one I know to work.

As for the first-run-initializer maybe I should just run that in a chroot right after installing whonix. I will give that a shot.

Then you need to commit it. Or…

Or just create the done file. (From https://github.com/Whonix/whonix-initializer/blob/master/usr/lib/first_run_initializer_gui.)

done_file="/root/.whonix/first_run_initializer.done" touch "$done_file"

Hi nrgaway,

I’m now getting to your work and am going to start testing it out and playing with it.

I will re-read through everything you’ve written and attempt a build.

Excited to get into this! Will be in touch! :smiley:

Tonight’s build is borked.

Everything builds, but I changed the order in which things build. I was building Whonix first and then added Qubes, but then I changed the build order a few days ago and I think that affects Qubes starting since I can not connect to the ProxyVM (and the lib modules directory has disappeared).

So I am currently changing the build order back to the original way. Will update here when that’s complete and I am able to connect to ProxyVM again.

I’m guessing that when preparing an --install-to-root Whonix installation, that means it expects to have nothing of importance already loaded. I was thinking it was to be able to just update an existing machine to Whonixify it :slight_smile:

Anyway, things are building at the moment; hope that it completes with success tonight.

Ok, its building nicely again.

Whonix server starts quickly now as proxyvm. I automatically search and replace all ip addresses when it loads each time from 10.152.152.10 to its real IP and if its IP changes, it will notice and change everything again.

tor wont bind to interface. Actually interface is not appearing as it should so gotta figure out that for both eth0 and xen virtual interfaces. Must be some configuration issue I am not aware of yet. Will look into it more tomorrow.

Remember to make sure you checkout whonix branch when building. Also it is important that you create a STANDALONE ProxyVM that uses the debian-7-whonix-gateway template at this point so when networking starts working we can update things.

The git clone commands are throwing fatal errors:

Input:

git clone git@github.com:nrgaway/qubes-builder.git

Output:

Cloning into 'qubes-builder'... The authenticity of host 'github.com (192.30.252.129)' can't be established. RSA key fingerprint is 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'github.com,192.30.252.129' (RSA) to the list of known hosts. Permission denied (publickey). fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists

Resolved by using:

git clone https://github.com/nrgaway/qubes-builder

Also in:

./whonix-install sources
git clone git@github.com:nrgaway/linux-template-builder.git
git clone git@github.com:nrgaway/gui-agent-linux.git

The git clone commands are throwing the same fatal errors.

I will try modifying these git clone commands and see how far it gets then.

I am building over a Tor connection, so maybe GitHub requires a different URL format with Tor?

the URL starting with git most likely does not work since you do not have commit access to my repo I am guessing. Using ‘git clone GitHub - nrgaway/qubes-builder: Qubes Builder’ would be the correct way for you to proceed. I updated the install script as well and pushed the changes.

I had a very quick look at GitHub - nrgaway/qubes-builder: Qubes Builder.

  • Glad to see another bash coder working on Whonix! Interesting stuff going on with Whonix lately!
  • Just some random thoughts that occured to me. I don’t claim to understand what you’re doing there.
  • Please consider using $(…) rather than backticks ....
  • Please consider running your scripts through shellcheck (available from Debian package sources [jessie] as well as online service http://www.shellcheck.net/ but I guess locally installed makes much more sense for using it more than once).
  • Please avoid using https://github.com/adrelanos/nothing_to_see.git - it’s really just for me. I do sometimes weird stuff there. Such as pushing a tag, such as tag 9. Then I am testing it. And if it’s not as perfect as I want, I delete it in my local as well as in the remote repository. Now, when you got tag 9 at that point, you won’t get the “newer” git tag. This is a security feature by git and rightly so. You would have to know that there is a “newer” git tag 9. Otherwise we could run into some pretty strange inconsistent results while we think “we use the same tag”, but we actually do not. (And if you must use it, what I do not hope… Delete a tag before using it, git fetch again, and use the “newer” one. But I hope we find another solution.)

Thanks!

Also…

When cloning the repo, it only seems to include the “master” branch:

git branch

Output:

* master

Alternatively, I could just clone the “whonix” branch this way:

git clone -b whonix https://github.com/nrgaway/qubes-builder

just do this

cd qubes-builder
git fetch
git checkout whonix

It should get the remote branch. To make sure, just look at whonix-install; it should contain the https git links now.

[quote=“Patrick, post:15, topic:512”]I had a very quick look at GitHub - nrgaway/qubes-builder: Qubes Builder.

  • Glad to see another bash coder working on Whonix! Interesting stuff going on with Whonix lately!
  • Just some random thoughts that occured to me. I don’t claim to understand what you’re doing there.
  • Please consider using $(…) rather than backticks ....
  • Please consider running your scripts through shellcheck (available from Debian package sources [jessie] as well as online service http://www.shellcheck.net/ but I guess locally installed makes much more sense for using it more than once).
  • Please avoid using https://github.com/adrelanos/nothing_to_see.git - it’s really just for me. I do sometimes weird stuff there. Such as pushing a tag, such as tag 9. Then I am testing it. And if it’s not as perfect as I want, I delete it in my local as well as in the remote repository. Now, when you got tag 9 at that point, you won’t get the “newer” git tag. This is a security feature by git and rightly so. You would have to know that there is a “newer” git tag 9. Otherwise we could run into some pretty strange inconsistent results while we think “we use the same tag”, but we actually do not. (And if you must use it, what I do not hope… Delete a tag before using it, git fetch again, and use the “newer” one. But I hope we find another solution.)[/quote]

I am not a bash shell coder by trade; mostly I do python so thanks for the tips. I was looking for some type of shell checker within an IDE, but have not found one yet. I will try that site.

I started to use nothing to see, since the latest 9 code was not on the main site. Which repo should I be using, since I expect to push some changes back to you and will need to be using something more current than the main repo I would think.

I think I used the 9 branch otherwise verify failed, since verification is automatic.

FYI, nrgaway:

Correct me if I’m wrong Patrick.

The stable production repo is: https://github.com/Whonix/Whonix

The public development repo is: GitHub - adrelanos/Whonix: Whonix is an operating system focused on anonymity, privacy and security. It's based on the Tor anonymity network, Debian GNU/Linux and security by isolation. DNS leaks are impossible, and not even malware with root privileges can find out the user's real IP. https://www.whonix.org

Patrick’s personal development repo is: GitHub - adrelanos/nothing-to-see: Temporary Git Tags for Whonix

Reference: https://github.com/Whonix/Whonix/issues/276

Patrick can further advise you on specifics, and pushing changes, I’m sure.

From https://github.com/nrgaway/qubes-builder/blob/master/whonix-install#L88.

[code]# Patch anon-meta-packages to not depend on grub-pc

XXX: Seems like the error disappears, but then whonix updates to original code?

pushd qubes-src/Whonix/packages/anon-meta-packages/debian/
sed -i ‘s/ grub-pc,//g’ control
#git commit -am ‘removed grub-pc depend’
[/code]

Sorry about that. This is a weird dependency. The grub-pc should be already get installed in build-steps.d/1300_create-raw-image build step by grml-debootstrap (./grml_packages) which is fine for VM builds. For --install-to-root users it’s unnecessary, since they already have a booting system. Created https://github.com/Whonix/Whonix/issues/342 for it. I’ll fix that soon for Whonix 10 / anon-meta-packages 0.6 and above. Will update it’s status in the github ticket.