whonix-gateway not reachable

whus141 · June 3, 2019, 2:42pm

Hello everyone,

I’m using whonix on Qubes 4.0. Since i installed updates (dom0 and whonix templates) a few days ago the control/socks port of sys-whonix-14 (my whonix-gateway, based on whonix-gw-14 template) is not reachable from my whonix workstation anon-whonix

Running whonixcheck -v in anon-whonix (template whonix-ws-14):

[INFO] [whonixcheck] anon-whonix | Whonix-Workstation | whonix-ws-14 TemplateBased AppVM | Mon Jun 3 14:08:20 UTC 2019
[INFO] [whonixcheck] Input Detection: INPUT_AUTO=true CLI=true GUI=false
stdin connected to terminal. Using cli output. Not using gui output.
Alternatively, if want to run from command line, but still use the graphical user interface for input, you could add to command line: --gui
[INFO] [whonixcheck] Root Check Result: Ok, not running as root.
[INFO] [whonixcheck] Pin torproject,.org certificate: disabled.
[INFO] [whonixcheck] whonix_build_version: 3:2.5-1
[INFO] [whonixcheck] whonix-workstation-packages-dependencies-cli: 9.8-1
[INFO] [whonixcheck] /etc/whonix_version: 14
[INFO] [whonixcheck] Spectre Meltdown Test: skipping since spectre_meltdown_check=false, ok.
If you wish to enable this test, run:

sudo spectre_meltdown_check=true whonixcheck
[INFO] [whonixcheck] Whonix firewall systemd unit check Result: Ok.
[INFO] [whonixcheck] Qubes qubes-db Test Result: Connection to local qubes-db daemon succeeded, ok.
[INFO] [whonixcheck] Qubes Settings Test Result: Ok. (GATEWAY_IP: 10.137.0.46)
[INFO] [whonixcheck] Qubes Settings Test Result: Ok, qubes_vm_type is AppVM.
[INFO] [whonixcheck] Check Kernel Messages Test Result: Found nothing remarkable, ok.
[INFO] [whonixcheck] check network interfaces Result: Ok.
[INFO] [whonixcheck] Check Package Manager Running Result: None running, ok.
[INFO] [whonixcheck] Tor Check Result: Not running on Whonix-Gateway, ok.
[INFO] [whonixcheck] Tor Config Check Result: Tor config ok.
[INFO] [whonixcheck] Tor Pid Check Result: Not running on Whonix-Gateway., ok.
[WARNING] [whonixcheck] Tor SocksPort Reachability Test Result: Unreachable! (curl exit code: 28 | curl status message: [28] - [Operation timeout. The specified time-out period was reached according to the conditions.])
[ERROR] [whonixcheck] Tor Connection Result:
Tor’s Control Port could not be reached!

Troubleshooting:

Confirm that Whonix-Gateway is running.

Run whonixcheck on Whonix-Gateway and confirm success.

Rerun whonixcheck here in this Whonix-Workstation.

(Technical information:)
(tor_circuit_established_check_exit_code: 277)
(tor_bootstrap_timeout_type: )
(tor_bootstrap_status: )
(check_socks_port_open_test: 28)
(Tor Circuit: not established)

Output of whonixcheck in sys-whonix-14 looks normal:

[INFO] [whonixcheck] sys-whonix-14 | Whonix-Gateway | whonix-gw-14 TemplateBased ProxyVM | Mon Jun 3 14:43:03 UTC 2019
[INFO] [whonixcheck] Connected to Tor.
[INFO] [whonixcheck] Whonix APT Repository: Enabled.
When the Whonix team releases STRETCH updates,
they will be AUTOMATICALLY installed (when you run apt-get dist-upgrade)
along with updated packages from the Debian team. Please
read https;://www.whonix.org/wiki/Trust to understand the risk.
If you want to change this, use:
sudo whonix_repository
[INFO] [whonixcheck] Debian Package Update Check: Checking for software updates via apt-get… ( Documentation: https;://www.whonix,.org/wiki/Update )
[INFO] [whonixcheck] Debian Package Update Check Result: No updates found via apt-get.

I also tried to set sys-whonix-14 as netvm of a appvm based on fedora-29 template. No connection, unable to ping out.

What I tried so far:

Update template using:

sudo qubes-dom0-update --enablerepo=qubes-templates-community --action=upgrade qubes-template-whonix-gw-14

Reinstall template using:

sudo qubes-dom0-update --enablerepo=qubes-templates-community --action=reinstall qubes-template-whonix-gw-14

Run:

sudo qubesctl state.sls qvm.anon-whonix

in dom0, (shortened)result:
Succeeded: 13
Failed: 0

Any ideas what could be the problem? I have no idea how to debug networking at Xen-level for possible network misconfigurations.

awokd · June 3, 2019, 6:02pm

Someone on qubes-users had a similar problem when they updated Qubes to
kernel 4.19.45-1. Try changing your .cfg to default boot the previous
kernel, or upgrade to the one in current-testing (4.19.47-1).

whus141 · June 3, 2019, 7:43pm

Thanks for your help.
Now I updated to 4.19.47-1. But unfortunately the problem remains, nothing changed.

awokd · June 3, 2019, 10:20pm

Now I updated to 4.19.47-1. But unfortunately the problem remains, nothing changed.

Try also 4.14.119-2 by editing your xen.cfg default= line if using UEFI
boot, or selecting it from the Grub boot menu. The two might be
unrelated of course, but I’m running 4.19.47-1 with the latest stable
patches in whonix-gw and whonix-ws with no problems. Are you on any
testing repositories?

whus141 · June 4, 2019, 11:58am

The oldest kernel version I have is 4.14.116-1. I played around with it.
Setting the kernel version of sys-whonix-14 to 4.14.116-1 fixed the problem. This works with 4.14.116-1 and 4.19.47-1 kernels running in dom0 and anon-whonix.
So the problem seems to be in the kernel of sys-whonix-14.
I’m very happy to have a working whonix setup again. Thanks.
Otherwise it would be nice to be able to use a recent kernel.

awokd · June 4, 2019, 6:08pm

Setting the kernel version of sys-whonix-14 to 4.14.116-1 fixed the problem.

Good, that will make troubleshooting easier. I wonder why my sys-whonix
works at 4.19.47-1, but not yours. What version of the whonix-gw-14
template do you have installed? Mine is 4.0.1-201901231238. Did you
install any patches from testing?

whus141 · June 5, 2019, 10:27am

Running sudo dnf info qubes-template-whonix-gw-14:

Last metadata expiration check: 22:27:32 ago on Tue Jun 4 13:50:32 2019.
Installed Packages
Name : qubes-template-whonix-gw-14
Arch : noarch
Epoch : 0
Version : 4.0.1
Release : 201807171801
Size : 2.0 G
Repo : @System
From repo : qubes-dom0-cached
Summary : Qubes template for whonix-gw-14
URL : http://www.qubes-os.org
License : GPL
Description : Qubes template for whonix-gw-14

As far as I can say/remember I made no changes to the template, no additional packages and no change of repository configuration. Even if I did these changes should have been deleted through running:

sudo qubes-dom0-update --enablerepo=qubes-templates-community --action=reinstall qubes-template-whonix-gw-14

Or am I wrong about how this works?

awokd · June 5, 2019, 11:14am

That is an older version of the template. Did you perform that reinstall
recently? It should work, but might try it manually if it’s not
updating. You can clone whonix-gw to a temporary template and point
sys-whonix to it to manually reinstall without having to do it in the clear.

whus141 · June 5, 2019, 3:03pm

I manually remoced qubes-template-whonix-gw-14 using dnf and reinstalled it. Now my version is:

sudo dnf info qubes-template-whonix-gw-14
Last metadata expiration check: 2:40:05 ago on Wed Jun 5 14:16:09 2019.
Installed Packages
Name : qubes-template-whonix-gw-14
Arch : noarch
Epoch : 0
Version : 4.0.1
Release : 201901231238
Size : 1.9 G
Repo : @System
From repo : qubes-dom0-cached
Summary : Qubes template for whonix-gw-14
URL : http://www.qubes-os.org
License : GPL
Description : Qubes template for whonix-gw-14

I installed all updates in whonix-gw-14 (didn’t modified repo-settings, so should be stable).

The situation is unchanged: When sys-whonix-14 is booted with 4.14.116-1 kernel eveything works fine, when unsing 4.19.47-1 running whonixcheck in anon-whonix fails with the old error.

awokd · June 5, 2019, 5:46pm

The situation is unchanged: When sys-whonix-14 is booted with 4.14.116-1 kernel eveything works fine, when unsing 4.19.47-1 running whonixcheck in anon-whonix fails with the old error.

How about Firefox in a Fedora based AppVM connected to sys-whonix on
4.19, does that work now? If not, I am running out of suggestions. Maybe
reinstall Qubes 4.0.1 if you’ve been upgrading from 4.0, unless someone
has a better idea.

whus141 · June 6, 2019, 8:53am

fedora-based appvm is not able to access the internet though sys-whonix-14 with 4.19 kernel.

I think that you are right about it being related to problems with updating from 4.0 to 4.0.1.
sudo dnf info qubes-release says:

Installed Packages
Name : qubes-release
Arch : noarch
Epoch : 0
Version : 4.0
Release : 8
Size : 157 k
Repo : @System
From repo : qubes-dom0-cached
Summary : Qubes release files
License : GPLv2
Description : Qubes release files such as yum configs and various /etc/ files
: that define the release.

But when running qubes-dom0-update it says “No new updates available”.

Patrick · June 6, 2019, 9:44am

Do you know anything related to such issues due to kernel upgrade? @marmarek

adw · June 7, 2019, 3:27am

4.0.1 is just 4.0 with all routine updates. There is no separate action of upgrading from 4.0 to 4.0.1. You can install either one, then fully update, and it results in the same system.

I think @awokd just meant that something might have gotten broken between your original 4.0 installation and the present. If so, he is suggesting that you might wish to reinstall. If you were to reinstall, it would make sense to use the most recent ISO, which is 4.0.1.

Perhaps this:

github.com/QubesOS/qubes-issues

Xorg crash on Qubes 4.0 with kernel 4.19

opened 07:53PM - 31 Mar 19 UTC

closed 11:38PM - 22 Jun 19 UTC

AndreasKieling

T: bug C: kernel P: critical r4.0-dom0-stable r4.1-dom0-cur-test

### Qubes OS version: R4.0 ### Affected component(s) or functionality: Xorg Lightdm Graphical Login Radeon Driver --- ### Steps to reproduce the behavior: Have Radeon Evergreen GPU, Start R4.0 with 4.19 kernel from repo. Try starting. ### Expected or desired behavior: Lightdm starts and login works. ### Actual behavior: Xorg in crash loop. Logging is completely useless except dmesg, which weirdly isnt combined in journal. Error message about radeon failed to create GEM object ### General notes: Works normally on 4.14 kernel with same settings. --- ### I have consulted the following relevant [documentation](https://www.qubes-os.org/doc/): Everything except module code. ### I am aware of the following related, [non-duplicate](https://www.qubes-os.org/doc/reporting-bugs/#new-issues-should-not-be-duplicates-of-existing-issues) issues:

awokd · June 7, 2019, 3:43am

I think @awokd just meant that something might have gotten broken between your original 4.0 installation and the present. If so, he is suggesting that you might wish to reinstall. If you were to reinstall, it would make sense to use the most recent ISO, which is 4.0.1.

Yes, sorry, meant that or maybe he is running one of the earlier RC
versions. Not sure what else could be different at the networking level.
Here is the other who was having the same problem with the newer kernel
and sys-whonix:
[qubes-users] Kernel 4.19.43 breaks sys-whonix too?.

marmarek · June 8, 2019, 12:13am

No idea. I’ve just tried: fully updated sys-whonix with 4.19.47 kernel and anon-whonix with 4.19.47 and it works for me (whonixcheck -v in anon-whonix says so). qubes-template-whonix-gw-14 package version is 4.0.1-201807171801.
Is your anon-whonix connected directly to sys-whonix? Do you see any failed services in sys-whonix (unlikely, as whonixcheck would detect that)? Check also if you see vifXX.0 interface in sys-whonix (some number instead of XX).

BTW looking at iptables -t nat in sys-whonix, I see redirection rules for 10.0.0.0/8 duplicated for each port (only for that IP range, 192.168.0.10 and 10.152.152.10 are listed only once). And also 10.152.152.10 is already handled by 10.0.0.0/8 so one or another is not needed.
Additionally, according to iptables-extensions manual, --to-ports is not necessary if you don’t change the port. It would allow to compact all those rules to very few - one for each port ranges. Like this:

iptables -t nat -A PREROUTING -i vif+ -d 10.152.152.10 -p tcp --dport 9152:9189 -j REDIRECT

Patrick · June 8, 2019, 3:08am

WORKSTATION_DEST_SOCKSIFIED="\
10.137.0.0/8,\
10.138.0.0/8"

Is working.

But even with only

WORKSTATION_DEST_SOCKSIFIED="10.137.0.0/8"

DispVMs are also working? How come?

And also somehow…

iptables --wait -t nat -A PREROUTING -i vif+ -d 10.137.0.0/8 -p tcp --dport 9168 -j REDIRECT --to-ports 9168

Results in:

-A PREROUTING -d 10.0.0.0/8 -i vif+ -p tcp -m tcp --dport 9168 -j REDIRECT --to-ports 9168

How come?

Nice. Done.

marmarek · June 8, 2019, 3:18am

Oh, this is from where 10.0.0.0/8 come. /8 is a netmask, and everything not covered by it is ignored. 10.137.0.0/8 basically says “compare just first 8 bits of the address”. If you want to cover 10.137.*.*, but not other 10.x.*.*, then it should be 10.137.0.0/16.

marmarek · June 8, 2019, 3:24am

I’m not sure if fix Qubes vs Non-Qubes prerouting IP range · Whonix/whonix-firewall@fbf4ad4 · GitHub is correct. On my qubes system, /lib/systemd/system/anon-ws-disable-stacked-tor_autogen__var_run_tor_socks.service redirects /var/run/tor/socks to 10.152.152.10:9050

Patrick · June 8, 2019, 3:31am

Awesome! Fixed.

Indeed. That would have lead to breakage.

It’s hard to use the actual Qubes-Gateway IP there:

it’s defined in a source file
unknown at build time
dynamic per Qubes installation

re-added:

flergusen · June 14, 2019, 1:33pm

I bumped into this a couple of weeks ago, I too think it was after a Qubes kernel update. I didn’t investigate until today, but what I found out was that no IP address is set on vif interfaces when attached, it seems like /usr/lib/qubes/setup-ip is not triggered (I put ‘thouch /tmp/thing’ right under the hash bang and no such file was created a new VM and vif was attached).

With ‘dmesg -w’ I saw UDEV/KERNEL lines with add+online and offline+remove, but none with bind or unbind.

With ‘udevadm info --attribute-walk /sys/class/net/vifXX.0’ I saw ATTRS(state)==“InitWait” where it should be “Connected”

Manually setting the same IP on the attached vifXX.0 as on the other interfaces, ‘ip address add dev vifXX.0 10.137.0.17/32’, enables the Whonix gateway to route traffic from connected VM through Tor.

I have not yet tried with a regular Debian (Stretch) template as netvm to see if the problem is the same. There are no issues with a netvm with Fedora 29 or 30 as template.