ProxyVM + AppVM Development

Qubes-Whonix
61

[quote=“marmarek, post:60, topic:512”][quote author=WhonixQubes link=topic=537.msg4504#msg4504 date=1411958050]
My concerns with the iptables static redirect approach are:

  • a) Would the Whonix-Gateway ProxyVM act like a “native” Qubes ProxyVM where other AppVMs can dynamically make use of its network connectivity?

  • b) Would multiple Whonix-Workstation AppVMs be able to dynamically utilize a Whonix-Gateway ProxyVM via the VM Manager GUI?
    [/quote]

Yes and yes.

[/quote]

Awesome clarification. Thanks Marek!

Yes, if we can, then I think it makes sense to offer Tor routing through Whonix-Gateway to other types of AppVMs connecting to it as well…which doesn’t limit the user’s flexibility and sets a consistent expectation/experience for using Whonix along side of TorVM in Qubes.

62

Seems like I have got the networking working now :slight_smile:

Whonix firewall now starts on boot and will bind tor to the second interface IP.

Tor TransPort transparent proxy is working from an AppVM as well and software can also connect via a socks port.

I ran whonixcheck and it returned all PASS for each tested item. I guess now I just need to know what other kinds of tests can I run to make sure the firewall is good; both on the gateway and AppVM (I will build the workstation once gateway is complete)

PS.
There were some upstream Qubes changes that broke the build, so I am fixing that now. Whoever has been making changes has been writing for Jessie so I had to allow some backports, so my repo is also broken at the moment. Jessie is so unstable, so that is why I am not considering building Whonix on top of it right now. For example, I just updated the standard Debian 8 supplied with Qubes today and it now had broken packages from the update.

63

Congratulations!

See: Leak Tests

64

If you can provide patches already made for building wheezy template, I can merge them right now (i.e. before final whonix template version). This way I’ll be able to test my changes also on wheezy, which should minimize risk of regressions.

65

This is awesome news! :smiley:

I literally was just in the process of building your old code again a few hours ago before seeing this news. So great timing!

Looking forward to learning exactly how you achieved the networking and overall solution.

And after you get things smoothed out, and we test, then we will likely want to talk about packaging and distribution (involving Qubes devs too).

Great job, nrgaway! :smiley:

66

Good to hear. Looking forward to the potential of being able to run Whonix through the VM manager rather than the current method.

67

[quote=“marmarek, post:64, topic:512”][quote author=nrgaway link=topic=537.msg4710#msg4710 date=1412938717]
There were some upstream Qubes changes that broke the build, so I am fixing that now. Whoever has been making changes has been writing for Jessie so I had to allow some backports, so my repo is also broken at the moment. Jessie is so unstable, so that is why I am not considering building Whonix on top of it right now. For example, I just updated the standard Debian 8 supplied with Qubes today and it now had broken packages from the update.
[/quote]

If you can provide patches already made for building wheezy template, I can merge them right now (i.e. before final whonix template version). This way I’ll be able to test my changes also on wheezy, which should minimize risk of regressions.[/quote]

I can get Wheezy to build with the upstream changes to core-agent-linux and the VM will boot but I can not connect a terminal client anymore. I see you are using systemd now and have made sure it is re-installed again before Qubes packages are installed.

There use to be an ‘/etc/init.d/qubes-core startup’ file that would execute the ‘/usr/lib/qubes/setup-ip’ file. I am in the process of determining why the VM will load with success but unable to connect to it with a terminal.

Once I can figure out how to access the VM again, I will be able to cleanup the code and prepare it for you to merge the Wheezy portion.

68

[quote=“WhonixQubes, post:65, topic:512”][quote author=nrgaway link=topic=537.msg4710#msg4710 date=1412938717]
Seems like I have got the networking working now :slight_smile:
[/quote]

This is awesome news! :smiley:

I literally was just in the process of building your old code again a few hours ago before seeing this news. So great timing!

Looking forward to learning exactly how you achieved the networking and overall solution.

And after you get things smoothed out, and we test, then we will likely want to talk about packaging and distribution (involving Qubes devs too).

Great job, nrgaway! :D[/quote]

It ended up being a simple fix for the vif+ interface. The issue I was having was that I need to be able to run Whonix firewall before when eth0 came up initially, but if there were no guest VM connected to the Whonix Gateway at the time of boot, the tor scripts would fail to be able to bind to any interface.

So based on something @mammamek said about using a dummy interface, I got the idea to just use a dummy interface ‘eth1’ that uses the ‘/usr/lib/qubes/setup-ip’ ip address (same ip address the vif+ interfaces use, for example 10.137.3.1) and then tor configurations load properly. The dummy interface is used for no other purpose and is not routing any packets.

# Create a dummy eth1 interface so tor can bind to it if there
# are no DOMU virtual machines connected at the moment
INTERFACE="eth1"
/sbin/ip link add $INTERFACE type dummy

# Now, assign it the netvm-gateway IP address
ip=$($XENSTORE_READ qubes-netvm-gateway 2> /dev/null)
if [ x$ip != x ]; then
    netmask=$($XENSTORE_READ qubes-netvm-netmask)
    gateway=$($XENSTORE_READ qubes-netvm-gateway)
    /sbin/ifconfig $INTERFACE $ip netmask 255.255.255.255
    /sbin/ifconfig $INTERFACE up
    /sbin/ethtool -K $INTERFACE sg off
    /sbin/ethtool -K $INTERFACE tx off
fi

I am still trying to find the proper spot to hook this code into. I was just replacing the ‘setup-ip’ script qubes uses since ‘/etc/init.d/qubes-core’ called it very early in the ‘init’ sequence and was looking for a better spot to implement it from when the upstream changes broke my build; which I am currently working on fixing.

I was hoping to actually just create a udev rule to execute the script but I was having an issue where the udev code was not executing for some reason unless I restarted udev after booting. I am not that familiar with udev, so I may have missed a configuration step.

But once I can get the VM to boot again, and find the proper spot to execute the dummy interface creation it won’t take long to finish things off. I am trying to emulate the behaviour of the original whonix implementation and start the firewall right when eth0 becomes active so the VM is protect with the firewall rules when the eth0 goes online.

69

I have managed to get going again using the qubes-rc2 repos instead of the marmarek repos. The marmarek repos contain updates that changed some of the debian modules to use systemd for init scripts instead of sysvinit. I did update Whonix to use systemd but there must be some script that is not starting since I can not access the VM with a terminal. The other issue using systemd is that sysvinit will re-install itself during a dist-upgrade even if its pinned not to do so or marked as held. I have not found a solution to this issue yet.

Again, I want to state that whonixcheck reports success for all tests, although I am still having issues with other stuff.

These are issues I am currently having:

  1. sdwdate fails on boot. I see that it tries to access network before tor is started. Is it supposed to use clearnet? It also starts before tor on regular version of Whonix

  2. sdwdate also fails after boot when tor becomes active. Is it supposed to connect to 4 sites that seem random when using a browser to check connection status (https://www.1984.is, https://www.privacyinternational.org, https://chavez.indymedia.org, https://chavez.indymedia.org).

Would this not expose whonix users as whonix users if they connect to those sites via clearnet? Or is the firewall supposed to prevent this (which it may, but since tor is not enabled at the time I am not sure how it can complete with success)

  1. initial whonix check fails on boot stating it can not find the pid ‘/var/run/tor/tor.pid’. I am assuming maybe that tor may be still boot strapping at this point? A manual whonixcheck works when logging into a terminal though.

  2. It seems like I can not run connection manager within terminal. It keeps wait for its own window id or something. You can run ‘sudo whonixsetup’ manually from a terminal window.

  3. Not sure if I can run the whonix apps; I had problems when I tried before, but still need to test. If they can open a window of sorts they should work.

70
1. sdwdate fails on boot. I see that it tries to access network before tor is started.
It also starts before tor on regular version of Whonix

It might start before Tor yes, but it Whonix-Default/Download-Version, it waits until Tor is fully bootstrapped. This is implemented in https://github.com/Whonix/sdwdate-plugin-anon-shared-con-check.

Is it supposed to use clearnet?
Inside it Whonix it can't use clearnet by design. It explicitly uses proxy settings for stream isolation. This is implemented in https://github.com/Whonix/sdwdate-plugin-anon-shared-streamiso. [Off-topic: However, sdwdate is created with being able to use it over clearnet. So one day there will be a Whonix Host Additions package, that will likely recommend sdwdate.]
Is it supposed to connect to 4 sites that seem random when using a browser to check connection status
3 random picks from 3 different pools. Only more when some servers fail. With fail-max and so forth.
Would this not expose whonix users as whonix users if they connect to those sites via clearnet?
Yes, but it's not using cleranet.
Or is the firewall supposed to prevent this (which it may, but since tor is not enabled at the time I am not sure how it can complete with success)
The firewall is supposed to prevent this. But also without firewall, due to using explicit proxy settings in sdwdate, it cannot connect without using the Tor proxy.
3. initial whonix check fails on boot stating it can not find the pid '/var/run/tor/tor.pid'. I am assuming maybe that tor may be still boot strapping at this point?
No, when Tor is bootstrapping, the pid is designed to exist. (By upstream Debian package.) Otherwise it would be a grave bug.
A manual whonixcheck works when logging into a terminal though.
Seems more like a race condition with the particular Whonix-Qubes boot process. With the default/download version, /etc/init.d/whonixcheck includes a "Required-Start: $all". But if Tor does not get started during the usual boot process, perhaps because of the $XENSTORE_READ magic script, then this might happen. In other words... If Tor fails to bind, because the IP in its default config does not exist, then sysvinit will consider Tor as "done" and go on with starting whonixcheck. Ideally the $XENSTORE_READ magic script also was an init script, that runs before Tor. Perhaps using something like "Should-Start: tor" in the init script.
4. It seems like I can not run connection manager within terminal. It keeps wait for its own window id or something.
5. Not sure if I can run the whonix apps; I had problems when I tried before, but still need to test. If they can open a window of sorts they should work.
Using default KDE or some other desktop environment? The package involved is: https://github.com/Whonix/msgcollector

The hook is:

ps aux | grep msgdispatcher

Should include.

/usr/lib/msgcollector/msgdispatcher --x
71

Thanks Patrick. Your last post should give me enough info to investigate for the next few days to iron stuff out!

72

In regards to timesync not working it just appears that it is unable to connect. I can run the regular curl without a problem from the command line but I get errors if trying to run

user@host:~$ curl.anondist-orig http://pay.reddit.com/r/usenet#
curl: (6) Couldn't resolve host 'pay.reddit.com'

user@host:~$ curl.anondist-orig http://198.41.209.142/r/usenet#
curl: (7) couldn't connect to host

So I guess I must have some issues with the firewall still. Any suggestions on trouble shooting it?

Also, I am not sure if it matters but I did receive an error in one of the modules since TEMP_DIR environment variable was not set

/usr/lib/anon-shared-helper-scripts/tor_bootstrap_check.bsh: line 86: error: command not found
    if [ "$TEMP_DIR" = "" ]; then
       error "Variable TEMP_DIR is empty."
    fi

The command ‘error’ is not on my system. Is it a function? I came across the error when manually running /usr/lib/anon-shared-helper-scripts/te_pe_tb_check when trying to track down this timesync issue.

EDIT:

I commented out the if [ “$TEMP_DIR” = “” ] section in ‘/usr/lib/anon-shared-helper-scripts/tor_bootstrap_check.bsh’ and now when I boot Whonixcheck no longer gives me the pid not found error and reports back correctly so either I need to set TEMP_DIR somewhere or ensure the command or function ‘error’ can be found.

73

On Whonix-Gateway? On Whonix-Gateway curl.anondist-orig won’t work out of the box, because Whonix-Gateway does not enable transparent proxying for its own traffic by default. (It is possible, though: Whonix-Gateway ™ Traffic: Transparent Proxying)
Using existing open Tor SocksPorts should work however out of the box, ex:

curl.anondist-orig --socks5-hostname 127.0.0.1:9050 check.torproject.org

TEMP_DIR is not a system environment variable, but an variable, that the script that sourcees the anon-shared-helper-script is supposed to set…

“error” is not implemented on purpose. It is only a way to make the script stop and complain loudly. (That works great when a fully featured trap ERR is implemented as for example whonixcheck does.)

The TEMP_DIR variable gets set by https://github.com/Whonix/whonixcheck/blob/Whonix9/usr/lib/whonixcheck/preparation#L141.

The whonixcheck startup process is sysvinit → https://github.com/Whonix/whonixcheck/blob/Whonix9/etc/init.d/whonixcheck → https://github.com/Whonix/whonixcheck/blob/Whonix9/usr/lib/whonixcheckdaemon → https://github.com/Whonix/whonixcheck/blob/Whonix9/usr/bin/whonixcheck.

I try to come up with a hypothesis why the TEMP_DIR variable might not be set.

74

My original hypothesis was, that somehow the Qubes specific boot process, does not let whonixcheck use mktemp or mktemp fails and returns “” what the anon-shared-helper-script would complain about. But later in the boot process, mktemp can work as expected.

But I usually try to always have ports and other unusual occurrences in mind. I deliberately messed up the mktemp command with mktemp___ for debugging to check if whonixcheck would notice that, loudly complain and exit. It’s not the case.

ERROR: ############################################################################### 
## whonixcheck script bug. 
## No panic. Nothing is broken. Just some rare condition has been hit. 
## Try again later. There is likely a solution for this problem. 
## Please see Whonix News, Whonix Blog and Whonix User Help Forum. 
## Please report this bug! 
## 
## BASH_COMMAND: TEMP_DIR="$(mktemp___ --directory)" 
## exit_code: 127 
## 
## Experts only: 
## bash -x whonixcheck --verbose 
## for verbose output. Clean the output and report to Whonix developers. 
#########################################################################
## BASH_COMMAND: TEMP_DIR="$(mktemp --just-fail-for-debugging --directory)" 
## exit_code: 1

What would be possible in theory though, that mktemp has a hypothetical bug, exiting 0 and still not write a temp dir to stdout, i.e. end up with an empty variable. To check that hypothesis, even though unlikely… Right below https://github.com/Whonix/whonixcheck/blob/Whonix9/usr/lib/whonixcheck/preparation#L141 add this.

   if [ "$TEMP_DIR" = "" ]; then
      error "Variable TEMP_DIR is empty!"
   fi

(No need to do a new build or this. Can be done on live system. Reboot should be sufficient.)

[hr]

For further debugging purposes, whonixcheckd (init script) can also be configured to run in verbose mode. File:
https://github.com/Whonix/whonixcheck/blob/Whonix9/etc/default/whonixcheckd

Add:

debugging="true"

A log will then be created.

/home/user/whonixcheck.log

Search that log for TEMP_DIR. See if it got set after initial start of whonixcheck after boot.

For me it includes.

[code]++ mktemp --directory

  • TEMP_DIR=/tmp/tmp.a60Zf6Ui30
    [/code]
75

FYI:

For those interested and following this thread, nrgaway and marmarek are currently making additional progress over on the qubes-devel mailing list…

Current thread: New Wheezy Template / Whonix Template

[b]https://groups.google.com/d/topic/qubes-devel/lsSGUS12T9g[/b]

Things seem to be going quite well! :smiley:

76

Good news… I finally have a build that runs without errors and both the gateway and workstation are working in Qubes as a ProxyVM and AppVM!

The Transport Internet access is also working from an AppVM. Any AppVM can be used to connect to Whonix Gateway or use the Workstation for better security.

Everything needs to be leak tested and there are still some issues with running apps. Some apps run and some don’t from the Qubes ‘Start Menu’. They all run if you start them manually from the terminal.

Three apps I will list as an example that do not work from ‘Start Menu’ are:

  1. Whonix Repository: Tries to start and exits
  2. Configuration Manager (whonixsetup): It won’t show and will actually shutdown (power off) the VM
  3. Tor Browser: Same as #1. Maybe a restart of VM will help.

I get a lot of invalid image error messages when syncing apps to Qubes ‘Start Menu’. Will be looking into that.

Whonix check and time sync do work and show properly.

So I guess others should get there hands on this soon and help testing and not use it for everyday use yet. It can be built from my repo and within a few days most likely Qubes repo.

I still recommend running both the Gateway and Workstation as a standalone VM mostly since I have not tested them as templates yet. Main reason though is updating the template; it may end up updating via clearnet or not at all. Have not tested yet.

The local time issue must be looked at as well to see if there is a way to disable it.

There also still an issue with Wheezy that when you open the terminal the first time, you have to go into setting to change the colors otherwise it show black on black. Looking into that too now since I now have time to look into more configuration related issues.

If you decide to build this, grab my ‘qubes-builder’ repo and git fetch; git checkout wheezy to use wheezy branch, not whonix. Then you just have to run the ./README.whonix file to build.

77
1. Whonix Repository: Tries to start and exits 2. Configuration Manager (whonixsetup): It won't show and will actually shutdown (power off) the VM

About 2.) The usual way in Whonix-Debian/Default-[…] is showing a menu using dialog and check if the answer to the disclaimer is “yes”, and if it’s not, shutdown. This is implemented here:
https://github.com/Whonix/whonixsetup/blob/master/usr/lib/whonixsetup_/ft_disclaimer

My hypothesis is, that somehow through the Qubes menu there is no terminal attached?

Might be a bug in Qubes. To falsify/confirm, could you create a test script to the menu please that check whether running in a terminal or not? Perhaps a loop that waits until terminal gets available? This would help (check for stdin):

I could help with that test script if necessary.

But perhaps Whonix scripts that use terminal input should check if they’re really running in a terminal and otherwise wait for the terminal forever?

About 1.) Probably similar to 2.) [Using dialog while no terminal is attached.]

78

[quote=“Patrick, post:77, topic:512”][quote]1. Whonix Repository: Tries to start and exits
2. Configuration Manager (whonixsetup): It won’t show and will actually shutdown (power off) the VM[/quote]

About 2.) The usual way in Whonix-Debian/Default-[…] is showing a menu using dialog and check if the answer to the disclaimer is “yes”, and if it’s not, shutdown. This is implemented here:
https://github.com/Whonix/whonixsetup/blob/master/usr/lib/whonixsetup_/ft_disclaimer

My hypothesis is, that somehow through the Qubes menu there is no terminal attached?[/quote]

I think you are right. Qubes make a call to the VM to start x-window programs. If you want an actual terminal you call the gnome-terminal application.

Might be a bug in Qubes. To falsify/confirm, could you create a test script to the menu please that check whether running in a terminal or not? Perhaps a loop that waits until terminal gets available? This would help (check for stdin): http://stackoverflow.com/questions/9863508/how-do-i-detect-whether-stderr-is-connected-to-a-terminal-from-within-a-script I could help with that test script if necessary.

But perhaps Whonix scripts that use terminal input should check if they’re really running in a terminal and otherwise wait for the terminal forever?

About 1.) Probably similar to 2.) [Using dialog while no terminal is attached.]

I will look into the script. Maybe a wrapper can be designed to start those types of apps within gnome-terminal.

79

Repo:
https://github.com/Whonix/whonix-repository

Startup sequence:
/usr/share/applications/whonix_repository.desktop → /usr/lib/whonix-repository → /usr/bin/whonix_repository

[hr]

Repo:
https://github.com/Whonix/whonixsetup/tree/master/usr/lib

Startup sequence:
/usr/share/applications/gateway-firsttimesetup.desktop → /usr/lib/whonixsetup → /usr/bin/whonixsetup

[hr]

We can work around it in Whonix, but I think it should be reported against Qubes as well.

For the report against Qubes, a small test script
/usr/share/applications/teststdin.desktop → /usr/bin/teststdin
that reads from stdin (which will fail as I expect) or uses dialog (which I am more surely expect to fail).

80

[quote=“nrgaway, post:78, topic:512”][quote author=Patrick link=topic=537.msg4983#msg4983 date=1414343605]

[quote]1. Whonix Repository: Tries to start and exits
2. Configuration Manager (whonixsetup): It won’t show and will actually shutdown (power off) the VM[/quote]

About 2.) The usual way in Whonix-Debian/Default-[…] is showing a menu using dialog and check if the answer to the disclaimer is “yes”, and if it’s not, shutdown. This is implemented here:
https://github.com/Whonix/whonixsetup/blob/master/usr/lib/whonixsetup_/ft_disclaimer

My hypothesis is, that somehow through the Qubes menu there is no terminal attached?[/quote]

I think you are right. Qubes make a call to the VM to start x-window programs. If you want an actual terminal you call the gnome-terminal application.[/quote]

Indeed Qubes currently ignores Terminal=true option in desktop file, only Exec= line is handled. I’ll fix it shortly.