Unable to update Whonix

maltfield · April 26, 2018, 4:02pm

I’m unable to update my whonix-gw TemplateVM in Qubes R3.2 due to network issues.

I haven’t been able to connect to tor in my whonix workstation AppVM for at least several days, but I noticed in the Qubes VM Manager that there’s an update available for whonix-gw TemplateVM. But updating whonix-gw fails.

I’m getting sporadic “Err … 500 Unable to connect” errors when attempting to fetch packages from, for example, deb.qubes-os.org, deb.whonix.org, etc.

Note that I’m connecting to the internet through an uncensored ISP in the US. Tor is not blocked, and Orbot/Orfox works fine on my phone.

I kicked-off a traceroute to ‘whonix.org’ in a disposable vm overnight (which is going out straight to my IPS; no tor or vpn), and I found what looks like significant packet loss over the transatlantic line between the US and the UK en-route to the whonix servers in France.

Has anyone else had issues updating Whonix recently? Is there anything I can do to further debug this?

maltfield · April 26, 2018, 4:04pm

adding screenshot of traceroute in second post, because I wasn’t allowed to add two images in a single post as a new user:

packetLossToWhonix

entr0py · April 26, 2018, 7:21pm

AFAIK, traceroute doesn’t work over Tor nodes.

Make sure sys-whonix has Tor connectivity. Try running whonixcheck in sys-whonix proxyVM. You can also update sys-whonix but realize that updates will only persist until you shut down that VM.
Make sure whonix-gw templateVM is set to use sys-whonix as its netVM.

From the screenshot, it looks like whonix-gw has no connectivity at all.

maltfield · April 26, 2018, 9:31pm

Thanks for the fast response!

whonixcheck fails on sys-whonix with “Whonixcheck gave up waiting. / Tor Circuit: not established”

user@host:~$ whonixcheck 
[INFO] [whonixcheck] sys-whonix | Whonix-Gateway | whonix-gw Template-Based ProxyVM | Thu Apr 26 20:47:16 UTC 2018
dmesg: read kernel buffer failed: Operation not permitted
[INFO] [whonixcheck] Tor Bootstrap Result: Bootstrapping for 0 seconds. 50 % done. Tor Circuit: not established. Tor reports: NOTICE BOOTSTRAP PROGRESS=50 TAG=loading_descriptors SUMMARY="Loading relay descriptors"
...
[INFO] [whonixcheck] Tor Bootstrap Result: Bootstrapping for 117 seconds. 50 % done. Tor Circuit: not established. Tor reports: NOTICE BOOTSTRAP PROGRESS=50 TAG=loading_descriptors SUMMARY="Loading relay descriptors"
[ERROR] [whonixcheck] Tor Bootstrap Result:
Whonixcheck gave up waiting.
Tor Circuit: not established
Bootstrapping 50 % done. Tor reports: NOTICE BOOTSTRAP PROGRESS=50 TAG=loading_descriptors SUMMARY="Loading relay descriptors"
...
user@host:~$

You’re right, whonix-gw has no connectivity at all. sys-whonix also has no connectivity at all.

First, I verified that sys-firewall has access to www.google.com (I’m using the IP to eliminate DNS)

[user@sys-firewall ~]$ ip r
default via 10.137.1.1 dev eth0 
10.137.1.1 dev eth0  scope link 
10.137.2.10 dev vif17.0  scope link  metric 32735 
10.137.2.30 dev vif6.0  scope link  metric 32746 
10.138.2.2 dev vif9.0  scope link  metric 32743 
[user@sys-firewall ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:5e:6c:06 brd ff:ff:ff:ff:ff:ff
    inet 10.137.1.8/32 brd 10.255.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe5e:6c06/64 scope link 
       valid_lft forever preferred_lft forever
4: vif6.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 32
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
    inet 10.137.2.1/32 scope global vif6.0
       valid_lft forever preferred_lft forever
    inet6 fe80::fcff:ffff:feff:ffff/64 scope link 
       valid_lft forever preferred_lft forever
5: vif9.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 32
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
    inet 10.137.2.1/32 scope global vif9.0
       valid_lft forever preferred_lft forever
    inet6 fe80::fcff:ffff:feff:ffff/64 scope link 
       valid_lft forever preferred_lft forever
9: vif17.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 32
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
    inet 10.137.2.1/32 scope global vif17.0
       valid_lft forever preferred_lft forever
    inet6 fe80::fcff:ffff:feff:ffff/64 scope link 
       valid_lft forever preferred_lft forever
[user@sys-firewall ~]$ 
[user@sys-firewall ~]$ dig +short www.google.com
172.217.0.68
[user@sys-firewall ~]$ time curl "http://172.217.0.68/"
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

real	0m0.071s
user	0m0.010s
sys	0m0.023s
[user@sys-firewall ~]$

I tried this from both whonix-gw & sys-whonix; it failed for both.

Here’s whonix-gw

root@host:~# ip r
default via 10.137.3.1 dev eth0 
10.137.3.1 dev eth0  scope link 
root@host:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:5e:6c:03 brd ff:ff:ff:ff:ff:ff
    inet 10.137.3.5/32 brd 10.255.255.255 scope global eth0
       valid_lft forever preferred_lft forever
root@host:~# time curl -v "http://172.217.0.68"
* Rebuilt URL to: http://172.217.0.68/
* Hostname was NOT found in DNS cache
*   Trying 172.217.0.68...
* Immediate connect fail for 172.217.0.68: Connection timed out
* Closing connection 0
curl: (7) Couldn't connect to server

real    2m1.054s
user    0m56.424s
sys     1m4.622s
root@host:~#

And here’s sys-whonix

root@host:~# ip r
default via 10.137.2.1 dev eth0 
10.137.2.1 dev eth0  scope link 
10.137.3.5 dev vif18.0  scope link  metric 32734 
root@host:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:5e:6c:08 brd ff:ff:ff:ff:ff:ff
    inet 10.137.2.10/32 brd 10.255.255.255 scope global eth0
       valid_lft forever preferred_lft forever
3: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether f2:2b:09:ea:99:dd brd ff:ff:ff:ff:ff:ff
4: eth1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 96:50:df:fc:ac:99 brd ff:ff:ff:ff:ff:ff
    inet 10.137.3.1/32 brd 10.255.255.255 scope global eth1
       valid_lft forever preferred_lft forever
5: vif18.0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 32
    link/ether fe:ff:ff:ff:ff:ff brd ff:ff:ff:ff:ff:ff
    inet 10.137.3.1/32 scope global vif18.0
       valid_lft forever preferred_lft forever
root@host:~# time curl -v "http://172.217.0.68"
* Rebuilt URL to: http://172.217.0.68/
* Hostname was NOT found in DNS cache
*   Trying 172.217.0.68...
* Immediate connect fail for 172.217.0.68: Connection timed out
* Closing connection 0
curl: (7) Couldn't connect to server

real    2m1.157s
user    0m55.272s
sys     1m5.687s
root@host:~#

I also confirmed that the whonix-gw templateVM is using sys-whonix as its netVM

[user@dom0 ~]$ qvm-ls whonix-gw sys-whonix
--------------+----+---------+-------+-------+-----------+--------------+-------+
         name | on |   state | updbl |  type |  template |        netvm | label |
--------------+----+---------+-------+-------+-----------+--------------+-------+
  [whonix-gw] |  * | Running |   Yes |   Tpl |       n/a |   sys-whonix | black |
 {sys-whonix} |  * | Running |       | Proxy | whonix-gw | sys-firewall | black |
[user@dom0 ~]$

If sys-whonix is simply configured to use sys-firewall as its NetVM, and I’ve confirmed that sys-firewall has no connectivity issues, then why does sys-whonix have connectivity issues?

0brand · April 27, 2018, 12:23am

Hi maltfield

Does your system keep the correct time? If sys-whonix time if off by to much, connections to Tor will not be possible.

https://whonix.org/wiki/Troubleshooting#Clock_Fix

You can use this site to correct system time.

https://timeanddate.com/worldclock/timezone/utc

If that does not help the problem could be a slow entry guard. You can create a test sys-whonix with a new entry guard and see if you have better connectivity.

https://whonix.org/wiki/FAQ#Whonix_has_Slowed_Tor_Connections_Dramatically.21

Patrick · April 27, 2018, 10:24am

curl -> mistake -> use curl.anondist-orig

(related to uwt, stream isolation)

run under user clearnet

maltfield · April 28, 2018, 3:36am

indeed, I do have connectivity to the clearnet from sys-whonix

user@host:~$ time curl.anondist-orig "http://172.217.0.68"
curl: (7) Failed to connect to 172.217.0.68 port 80: No route to host

real    0m1.100s
user    0m0.013s
sys     0m0.027s
user@host:~$ su - clearnet
clearnet@host:~$ time curl.anondist-orig "http://172.217.0.68"
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

real    0m0.074s
user    0m0.010s
sys     0m0.033s
clearnet@host:~$

Thanks for the tip, but why ‘curl.annondist-orig’? There’s no man page for this command, the command itself is a binary, and the ‘–help’ arg just shows the same info as curl. I tried googling it, and couldn’t find the distinction.

Can you link me to a document describing the difference between curl, curl.annondist, and curl.annondist-orig please?

maltfield · April 28, 2018, 4:35am

So the time on dom0 was accurate, but sys-whonix was off by 2 minutes. I fixed the clock on sys-whonix, but whonixcheck still failed.

For some reason I cannot clone sys-whonix from the Qubes VM Manager. The option for “Clone VM” is greyed-out, even when sys-whonix and whonix-gw are both shutdown.

I spent some time fumbing with the instructions per the link listed above for regenerating the Tor State File, and it eventually worked. At first it didn’t, but I remember hitting ‘x’ in arm, and it finally got a successful tor circuit.

I’m very unclear as to what I did, though.

First of all, is it always safe to delete the contents of /var/lib/tor? Is there exclusively ephemeral data in that directory?

And why did I read that one should delete their Tor State File when changing physical locations?

Thank you!

maltfield · April 28, 2018, 5:16am

update: when poking around in my torrc, I realized that I had my tor configuration setup to use an obfs4 pluggable transport.

I had set that up weeks ago when I was surfing on a public access point that was censoring content and using DPI to block Tor.

I commented-out the “UseBridges 1” line in my torrc, and it connected much easier!

My best guess is that the bridges I have defined in the torrc file are stale.

facepalm thanks for the help!

I’d still really like to know more about the curl commands in whonix and the /var/lib/tor dir in general

Patrick · April 28, 2018, 6:50am

curl is a symlink to curl.anondist which is a symlink to uwt which then
runs curl under torsocks which makes it use Tor running on localhost
for stream isolation.

Use curl.anondist-orig to use the original curl.

Could this please be documented somewhere under
Dev/anon-ws-disable-stacked-tor - Whonix or so?
Trying to use curl rather than curl.anondist-orig is a common mistake
when trying to debug Whonix network issues.

It is already under

but not clear enough and not a good reference to answer a forum post
like this one.

Michael Altfield:

user@host:~$

Mistake. User clearnet.

sudo su clearnet

Also needs to be documented.

Patrick · April 28, 2018, 6:59am

Michael Altfield:

First of all, is it always safe to delete the contents of /var/lib/tor?

No, it changes Tor entry guards likely which could be the intention of
an adversary.

We have documentation on entry guards here:

Does that documentation need any improvements?

And why did I read that one should delete their Tor State File when changing physical locations?

Same as above. You need to know a bit about Tor entry guards to see why.

Patrick · April 28, 2018, 7:10am

Michael Altfield:

I’d still really like to know more about the curl commands in whonix and the /var/lib/tor dir in general

Your interested is appreciated!

0brand · April 28, 2018, 10:18am

What version of Qubes are you using? R3.2 ?

I don’t have my 3.2 box with me but I don’t think AppVMs and ProxyVMs can be cloned in 3.2. Just TemplateVMs.

I’ll get it done

Maybe entry guards technical and non-technical would be good to have?

I can work on that to.