ProxyVM + AppVM Development

[quote=“nrgaway, post:17, topic:512”]just do this

cd qubes-builder
git fetch
git checkout whonix

It should get the remote branch. To make sure, just look at whonix-install; it should contain the https git links now.[/quote]

Thanks! I’m still a newbie with git. I left out the fetch command.

Will give it a go.

FYI,

Most of the whonix related installation step are in:
qubes-src/linux-template-builder/scripts_debian

specifically qubes-src/linux-template-builder/scripts_debian/custom_wheezy_whonix*

I am not aware of an IDE. Kate seems most feature rich. Please tell me if you found one.

Seems you were quite fast. It’s pushed for a while. Not after the release this time, I think. Please ping me if I forget to push the tag to Whonix/Whonix.

The master branch should be always up to date in Whonix/Whonix as well as in adrelanos/Whonix. The only cases where adrelanos/Whonix might be more current, when there are merge conflicts (seldom) or when I haven’t locally merged github merges yet. (But in that case using Whonix/Whonix would be safer anyhow.) The adrelanos/nothing-to-see might even lag behind, because I am mostly using it to transfer code to a different machine.

Let’s say that “pretty” git tags, such as 9, 9.1 in nothing-to-see aren’t reliable. Sometimes I re-use them as said above.

Do you need more current, developers-only git tags than 9 and 9.1? In that case, I could start a “developers-only git tags” thread, where I post about tags with very briefly status reports such as “entirely untested” or “that one at least builds”. Or something like that.

[quote=“nrgaway, post:22, topic:512”]FYI,

Most of the whonix related installation step are in:
qubes-src/linux-template-builder/scripts_debian

specifically qubes-src/linux-template-builder/scripts_debian/custom_wheezy_whonix*[/quote]
I am overlooking that folder linux-template-builder somehow on GitHub - nrgaway/qubes-builder: Qubes Builder. Please post a link.

By the way, I want to make Whonix as re-usable friendly as possible. Having that in mind while developing. So I want to spare you from as much sed and patching escapades as possible by providing .d style config folders (Configuration Files - Kicksecure), (build) options, hooks and whatever required to make that process comfortable.

[But should we reach points where patching is required or even simpler, then I suggest, that using debian packaging’s “patches” mechanism might be more appropriate than sed. However, just saying. Since you are doing the code, implementations details are of course up to you.]

[quote=“Patrick, post:24, topic:512”][quote author=nrgaway link=topic=537.msg4357#msg4357 date=1411604187]
FYI,

Most of the whonix related installation step are in:
qubes-src/linux-template-builder/scripts_debian

specifically qubes-src/linux-template-builder/scripts_debian/custom_wheezy_whonix*
[/quote]
I am overlooking that folder linux-template-builder somehow on GitHub - nrgaway/qubes-builder: Qubes Builder. Please post a link.[/quote]

Located in a different repo:

https://github.com/nrgaway/linux-template-builder

Code also makes use of this repo:

https://github.com/nrgaway/gui-agent-linux

[quote=“Patrick, post:24, topic:512”][quote author=nrgaway link=topic=537.msg4357#msg4357 date=1411604187]
FYI,

Most of the whonix related installation step are in:
qubes-src/linux-template-builder/scripts_debian

specifically qubes-src/linux-template-builder/scripts_debian/custom_wheezy_whonix*
[/quote]
I am overlooking that folder linux-template-builder somehow on GitHub - nrgaway/qubes-builder: Qubes Builder. Please post a link.[/quote]

[url=https://github.com/nrgaway/linux-template-builder/tree/whonix/scripts_debian]https://github.com/nrgaway/linux-template-builder/tree/whonix/scripts_debian[/url] (whonix branch)

As for the development tags, we can just try going with what you have right now and if that does not work I will let you know. I just needed up to date Whonix 9 repos at the time.

[quote=“Patrick, post:25, topic:512”]By the way, I want to make Whonix as re-usable friendly as possible. Having that in mind while developing. So I want to spare you from as much sed and patching escapades as possible by providing .d style config folders (Configuration Files - Kicksecure), (build) options, hooks and whatever required to make that process comfortable.

[But should we reach points where patching is required or even simpler, then I suggest, that using debian packaging’s “patches” mechanism might be more appropriate than sed. However, just saying. Since you are doing the code, implementations details are of course up to you.][/quote]

At this point I am using whichever patch mechanism is easiest to implement just so I can get Whonix working in Qubes. Once I figure out everything that needs patching (when Whonix starts working on Qubes), I will then look at what I needed to do to make it work and come up with a plan to implement these patches within Whonix build or at least make sure they will survive version release updates.

Just to clarify, I do not expect what you see now to be final as it can break too easily.

One thing that I hope we can think about now though is how to handle the static interfaces and network address you are currently using. I wonder if there is a was you can have a file located in /etc/whonix that can contain the WAN ip address and private LAN ip address and have the scripts use those, then we can update them as need in qubes. In Qubues, the IP address is assigned by XEN and I have to currently do a search and replace on the whole file system (usually only once; first boot):

#!/bin/bash

# TODO: Obtain automatically
#
# - Create a /etc/whonix-netvm-gateway, initially populated with 10.152.152.10
# - then the loop below will compare /etc/whonix-netvm-gateway with $IP and
#   so a sed search and replace if it differs using whonix-netvm-gaeway as source
#   and $IP as target. That way if xen changes the IP, it will get noticed and be able
#   to sed replace again
# - update whonix-netvm-gateway with IP once sed is complete

IP=`xenstore-read qubes-netvm-gateway`
IP_PART=$(echo $IP | cut -f 1,2,3 -d".")

LAST_IP="$(cat /etc/whonix-netvm-gateway)"
LAST_IP_PART=$(echo $LAST_IP | cut -f 1,2,3 -d".")

if ! [ "$LAST_IP" == "$IP" ]; then
    # Lets not modify binary files
    find / -xdev -type f -print0 | xargs -0r file | grep 'ASCII text' | awk -F: '{print $1}' | \
        xargs -d'\n' -r sed -i "s/$LAST_IP/$IP/g"
    find / -xdev -type f -print0 | xargs -0r file | grep 'ASCII text' | awk -F: '{print $1}' | \
        xargs -d'\n' -r sed -i "s/$LAST_IP_PART./$IP_PART./g"

    echo "$IP" > /etc/whonix-netvm-gateway
    service tor restart
fi
echo "$IP" > /etc/resolv.conf
echo "$IP_PART.254" >> /etc/resolv.conf

Another FYI, just in case you did not know.

This is being build in a Fedora 20 build environment.

First a version of wheezy is bootstrapped, then that bootstraped image is used to build Whonix (Whonix is cloned in the ~user directory and a --install-to-root directive is issued).

The reason its being built that way is so it can be built along with the rest of the Qubes system. The qubes-builder package will build a complete Qubes system, including creating an ISO, as well as creating all the template virtual machines you want (Fedora, Archlinux, Debian 8 and now Debian 7 and Whonix soon)

Interesting. I wasn’t yet aware it was building all of this in the process. If we are just aiming to build Whonix templates, I wonder if we can, or if it’d be worthwhile considering script maintenance, to ultimately ignore building all of this extra stuff in a final version.

Yes, Whonix networking internals seems to be a fundamental issue for improving and adapting Whonix to the native Qubes ProxyVM + AppVM architecture.

I am currently documenting ideas how to solve the IP stuff…

Can you make IPs static? Or will you always have to cope up with dynamic IPs? Static IPs would make it much simpler for everyone.

Let’s use Dev/Qubes - Whonix for brainstorming.

Ah damn it, this is a hard problem.

The problem is, it’s not just scripts that use static IPs. The scripts are actually the easy part. On Dev/Qubes - Whonix I documented the places where you could easily overrule IPs by using simple .d folder config snippets. But for example Tor’s config does not support variables yet. So at the moment I don’t have great ideas how to solve this. See below… That would be a much simpler solution.

I guess the simplest would be if you could somehow use the same IPs as Whonix does?
Or at least use static IPs rather than dynamic ones?
Or can Whonix Debian somehow use the same IPs as Whonix Qubes?
(I’d be willing to change the IPs for Whonix 10 again if that was any help if we could find some range that is acceptable for both Debian and Qubes if the current choices are unacceptable.)

[quote=“Patrick, post:32, topic:512”]I guess the simplest would be if you could somehow use the same IPs as Whonix does?
Or at least use static IPs rather than dynamic ones?
Or can Whonix Debian somehow use the same IPs as Whonix Qubes?
(I’d be willing to change the IPs for Whonix 10 again if that was any help if we could find some range that is acceptable for both Debian and Qubes if the current choices are unacceptable.)[/quote]

One of the issues here is that Qubes uses different dynamic C-class IP ranges.

For example:

10.137.2.X
10.137.3.X
10.137.4.X
10.137.5.X
10.137.6.X
etc

So even if the final dynamic host id (X) could be accounted for in an overlapping range between Qubes and Whonix, the C-class is dynamic.

Could you ask on qubes-dev please if there is some way to make it static or has this been already discussed?

I did a little digging to see how TorVM for Qubes hadles this issue. Seems they use a combination of a template that is used to genertate the configuration on each boot as well as using interface names instead of IP addresses in the iptables rule set.

Here is the template that is used:

# qubes-tor: Default configuration
###################################################################
# This file is AUTO-GENERATED on startup by the qubes-tor service #
#                                                                 #
# Place your own tor settings in /rw/config/qubes-tor/torrc       #
###################################################################

DataDirectory DATA_DIRECTORY
SocksPort "QUBES_IP:TOR_SOCKS_ISOLATED_PORT IsolateClientAddr IsolateSOCKSAuth IsolateDestPort IsolateDestAddr"
SocksPort "QUBES_IP:TOR_SOCKS_PORT IsolateClientAddr IsolateSOCKSAuth"
TransPort "QUBES_IP:TOR_TRANS_PORT IsolateClientAddr IsolateDestPort IsolateDestAddr"
DNSPort "QUBES_IP:53 IsolateClientAddr IsolateSOCKSAuth"
ControlPort TOR_CONTROL_PORT
AutomapHostsOnResolve 1
VirtualAddrNetwork "VIRTUAL_ADDR_NET"

Here is the startup.sh code

#!/bin/sh
# 
# The Qubes OS Project, http://www.qubes-os.org
#
# Copyright (C) 2012-2013 Abel Luck <abel@outcomedubious.im>
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
#

# run only if qubes-tor service enabled
[ -r /var/run/qubes-service/qubes-tor ] || exit 0

killall tor &> /dev/null

# defaults torrc variables - overridable by user
QUBES_IP=$(xenstore-read qubes-ip)
TOR_TRANS_PORT=9040 # maximum circuit isolation
TOR_SOCKS_PORT=9050 # less circuit isolation
TOR_SOCKS_ISOLATED_PORT=9049 # maximum circuit isolation
TOR_CONTROL_PORT=0 # 0 = disabled
VIRTUAL_ADDR_NET=172.16.0.0/12
DATA_DIRECTORY=/rw/usrlocal/lib/qubes-tor
RUNDIR=/var/run/tor
TOR_USER=`id -u -n _tor 2>/dev/null || id -u -n toranon 2>/dev/null || echo root`

VARS="QUBES_IP TOR_TRANS_PORT TOR_SOCKS_PORT TOR_SOCKS_ISOLATED_PORT TOR_CONTROL_PORT VIRTUAL_ADDR_NET DATA_DIRECTORY"

# command line arguments - not overrideable
DEFAULT_RC=/usr/lib/qubes-tor/torrc
DEFAULT_RC_TEMPLATE=/usr/lib/qubes-tor/torrc.tpl
USER_RC=/rw/config/qubes-tor/torrc
PID=$RUNDIR/qubes-tor.pid


# $1 = space delimited vars
# $2 = template file
function replace_vars()
{
	for var in $1; do
		expressions+=("-e s|$var|${!var}|g")
	done

	sed "${expressions[@]}" $2
}

function setup_firewall
{

	echo "0" > /proc/sys/net/ipv4/ip_forward
	/sbin/iptables -F
	/sbin/iptables -P INPUT DROP
	/sbin/iptables -P FORWARD DROP
	/sbin/iptables -P OUTPUT ACCEPT
	/sbin/iptables -A INPUT -i vif+ -p udp -m udp --dport 53 -j ACCEPT
	/sbin/iptables -A INPUT -i vif+ -p tcp -m tcp --dport $TOR_TRANS_PORT -j ACCEPT
	/sbin/iptables -A INPUT -i vif+ -p tcp -m tcp --dport $TOR_SOCKS_PORT -j ACCEPT
	/sbin/iptables -A INPUT -i vif+ -p tcp -m tcp --dport $TOR_SOCKS_ISOLATED_PORT -j ACCEPT
	if [ "$TOR_CONTROL_PORT" != "0" ]; then
		/sbin/iptables -A INPUT -i vif+ -p tcp -m tcp --dport $TOR_CONTROL_PORT -j ACCEPT
	fi
	/sbin/iptables -A INPUT -i vif+ -p udp -m udp -j DROP
	/sbin/iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
	/sbin/iptables -A INPUT -i lo -j ACCEPT
	/sbin/iptables -A INPUT -j REJECT --reject-with icmp-host-prohibited

	# nat rules
	/sbin/iptables -t nat -F
	/sbin/iptables -t nat -P PREROUTING ACCEPT
	/sbin/iptables -t nat -P INPUT ACCEPT
	/sbin/iptables -t nat -P OUTPUT ACCEPT
	/sbin/iptables -t nat -P POSTROUTING ACCEPT
	/sbin/iptables -t nat -A PREROUTING -i vif+ -p udp -m udp --dport 53 -j DNAT --to-destination $QUBES_IP:53
	/sbin/iptables -t nat -A PREROUTING -i vif+ -p tcp -m tcp --dport $TOR_SOCKS_ISOLATED_PORT -j DNAT --to-destination $QUBES_IP:$TOR_SOCKS_ISOLATED_PORT
	/sbin/iptables -t nat -A PREROUTING -i vif+ -p tcp -m tcp --dport $TOR_SOCKS_PORT -j DNAT --to-destination $QUBES_IP:$TOR_SOCKS_PORT
	/sbin/iptables -t nat -A PREROUTING -i vif+ -p tcp -j DNAT --to-destination $QUBES_IP:$TOR_TRANS_PORT

	# completely disable ipv6
	/sbin/ip6tables -P INPUT DROP
	/sbin/ip6tables -P OUTPUT DROP
	/sbin/ip6tables -P FORWARD DROP
	/sbin/ip6tables -F

	for iface in `ls /proc/sys/net/ipv6/conf/vif*/disable_ipv6 2> /dev/null`; do
		echo "1" > $iface
	done
}

# function to print error and setup firewall rules to prevent traffic leaks
function exit_error()
{
	echo "qubes-tor: $1" 1>&2
	setup_firewall
	exit 1
}

# double check we've got an ip address
if [ X$QUBES_IP == X ]; then
	QUBES_IP="127.0.0.1"
	exit_error "Error getting qubes ip"
fi


# make the data directory if it doesn't exist
if [ ! -d "$DATA_DIRECTORY" ]; then
	mkdir -p $DATA_DIRECTORY || exit_error "Error creating data directory"
fi
chown -R $TOR_USER:$TOR_USER $DATA_DIRECTORY

if [ ! -d "$RUNDIR" ]; then
	mkdir -p $RUNDIR || exit_error "Error creating run directory"
fi
chown -R $TOR_USER:$TOR_USER $RUNDIR

# pass the -f option only when config file exists
if [ -r "$USER_RC" ]; then
    USER_RC_OPTION="-f $USER_RC"
fi

# update the default torrc file with current values
(replace_vars "$VARS" $DEFAULT_RC_TEMPLATE) > $DEFAULT_RC  || exit_error "Error writing default torrc: $DEFAULT_RC"

# verify config file is useable
/usr/bin/tor \
	--defaults-torrc $DEFAULT_RC \
	$USER_RC_OPTION --verify-config \
	--user $TOR_USER \
|| exit_error "Error in Tor configuration"

# start tor
/usr/bin/tor \
	--defaults-torrc $DEFAULT_RC \
	$USER_RC_OPTION \
	--user $TOR_USER \
	--RunAsDaemon 1 \
	--Log "notice syslog" \
	--PIDFile $PID \
|| exit_error "Error starting Tor!"

# if we get here tor is running
setup_firewall

And finally this is the torrc file generated:

# qubes-tor: Default configuration
###################################################################
# This file is AUTO-GENERATED on startup by the qubes-tor service #
#                                                                 #
# Place your own tor settings in /rw/config/qubes-tor/torrc       #
###################################################################

DataDirectory /rw/usrlocal/lib/qubes-tor
SocksPort "10.137.2.25:9049 IsolateClientAddr IsolateSOCKSAuth IsolateDestPort IsolateDestAddr"
SocksPort "10.137.2.25:9050 IsolateClientAddr IsolateSOCKSAuth"
TransPort "10.137.2.25:9040 IsolateClientAddr IsolateDestPort IsolateDestAddr"
DNSPort "10.137.2.25:53 IsolateClientAddr IsolateSOCKSAuth"
ControlPort 0
AutomapHostsOnResolve 1
VirtualAddrNetwork "172.16.0.0/12"

Aspects of this have been discussed.

Based on what’s been said and what I’ve personally seen inside of the Qubes system, I don’t think there is any built-in way to switch to static networking for ProxyVM networking.

https://groups.google.com/d/topic/qubes-users/RFXoZ3zt-PE

Xen network devices are simple point-to-point connections (no multi-node "internal networks" like in VirtualBox). It consists of two ends: 1. frontend - ethN interface in one VM 2. backend - vifX.N interface in other VM (X is VM Xen ID, N is the same as in frontend).

With HVMs, I was able to manually do static networking, but ProxyVM networking is a different animal, fundamentally based on dynamic networking.

Here’s how I understand it:

  • All VMs are given an underlying ID number by Xen.

  • Xen VM network interfaces connect through frontend/backend interfaces, which can either be an “eth” or “vif” inside the VM’s OS.

  • Qubes AppVMs use frontend eth interfaces that connect to ServiceVM/ProxyVM backend vif interfaces.

  • Qubes AppVM eth interfaces that are frontends to ServiceVMs/ProxyVMs are named: ethN (where N is the incremented interface). Typically just a single eth0 interface in AppVMs unless customized.

  • Qubes ServiceVM/ProxyVM vif interfaces that are backends to AppVMs are named: vifX.N (where X is Xen VM ID and N is same as ethN. Typically vif.0

  • Qubes ServiceVMs/ProxyVMs dynamically generate multiple vif interfaces as AppVMs boot up and connect to them.

  • Qubes ServiceVM/ProxyVM IP addresses are allocated as: 10.137..

  • Qubes AppVM IP addresses are allocated as: 10.137..

As an example layout:

netvm <-- firewallvm (ProxyVM): XENID=2, IP=10.137.1.3
          ^      ^
          |      | <-- fedoravm (AppVM): XENID=5, IP=10.137.2.4, ProxyVM=vif5.0
          |      | <-- debianvm (AppVM): XENID=7, IP=10.137.2.5, ProxyVM=vif7.0
          |      | <-- othervm (AppVM): XENID=10, IP=10.137.2.6, ProxyVM=vif10.0
          | 
          | <-- whonix-gateway (ProxyVM): XENID=15, IP=10.137.1.7, ProxyVM=vif15.0
                     ^
                     | <-- whonix-workstation-1 (AppVM): XENID=21, IP=10.137.3.8, ProxyVM=vif21.0
                     | <-- whonix-workstation-2 (AppVM): XENID=24, IP=10.137.3.9, ProxyVM=vif24.0
                     | <-- whonix-workstation-3 (AppVM): XENID=32, IP=10.137.3.10, ProxyVM=vif32.0

One can observe the real-time allocation of “vif” interfaces between ProxyVMs and AppVMs inside their respective VM terminals with “sudo ifconfig”.

So everything seems inherently dynamic in the native ProxyVM + AppVM networking architecture of Qubes.

The Qubes VM Manager GUI is able to create and destroy multiple ProxyVMs and AppVMs on demand, so the networking architecture is dynamic along with this.

The IP and backend portion is done in dom0 by the Qubes system, so attempting to force static networking upon Qubes would likely:

a) Require modifying the underlying Qubes system in dom0, which is not user friendly.

b) Break the intended fluid GUI-based dynamic creation and removal of ProxyVMs and AppVMs, which is not user friendly, and a big reason for going beyond our current HVM setup.

White trying to build, I got a fatal build error:

+ echo main/binary-amd64/Packages + rm -f dists/wheezy/Release.gpg + gpg -abs --no-default-keyring --secret-keyring /home/user/qubes-builder/keys_debian/wheezy-secring.gpg --keyring /home/user/qubes-builder/keys_debian/wheezy-pubring.gpg -o dists/wheezy/Release.gpg dists/wheezy/Release gpg: can't access `/home/user/.gnupg/trustdb.gpg': Permission denied gpg: fatal: can't init trustdb: trust database error secmem usage: 1408/1408 bytes in 2/2 blocks of pool 1408/32768 make[2]: *** [dist-build-dep] Error 2 --> build failed! make[1]: *** [packages] Error 1 make: *** [core-vchan-xen-vm] Error 1 [user@fedoravm qubes-builder]$

I will try running through it again and see if this fatal error reoccurs.

FYI. Got this same fatal error again.

Haven’t explored any deeper into it.

What stage was that within?

./whonix-install sources
./whonix-install build
./whonix-install template whonix-gateway

Also, are you sure you checked out the whonix branch?

I am assuming this is happening during the build stage which is building (compiling) the actual debian qubes modules needed for the AppVM. It also bootsstraps debain and uses it to build those modules within the ~/qubes-builder/chroot-wheezy directory

I clone the repo last night and was able to build the gateway from a fresh install base within a Fedora-20-x86 based AppVM with 40Gig Private space allocated.

I am wondering what the permissions of `/home/user/.gnupg/trustdb.gpg’ are and its parent directory? I am also wondering if I had to add the Qubes master key as a trusted source. Maybe check that and that the ‘trustdb.gpg’ actually exists.

echo ‘427F11FD0FAA4B080123F01CDDFA1A3E36879494:6:’ | gpg --import-ownertrust

I will fire up a fresh AppVM too to see where the problems comes from.

Great explanation on the Qubes IPs, thanks!

Dynamic IPs will make setting up Tor hidden services real cumbersome, I think. Because each time the workstation’s IP changes, /etc/tor/torrc has to be edited. How does TorVM solve this?

Perhaps since you’re discussing build issues here, should we start a separate topic about IPs?