Unable to get VPN Before Tor (User -> VPN -> Tor -> Internet) working, network interfaces are reset

Support
1

Hello everyone.

I have some slight trouble getting Whonix with “VPN Before Tor” (User → VPN → Tor → Internet) working.

I have followed this guide Connecting to a VPN before Tor exactly.

My setup is:

Whonix workstation → Whonix gateway → Internet.

I am using TUNNEL_FIREWALL inside the Whonix gateway.
And it works.

However, not for long.

The Whonix gateway starts up and I can see from ifconfig it creates the tun0 interface, gets an IP, there are no errors, this uses the openvpn@openvpn service.

It then connects to Tor with the tor@default service, again this is fine, no errors.

However then, after perhaps 120 to 240 seconds, I can see this:

pcnet32 0000:00:03.0 eth0: link up, 100Mbps, full-duplex
pcnet32 0000:00:08.0 eth1: link up, 100Mbps, full-duplex

When I see this, at the exact same time, the connection on the Whonix workstation goes down.

The above messages (pcnet32 …) seem to be similar to what would happen, if you unplugged the network cable and plugged it back in.

However I cannot find any errors, not in the openvpn log, not in the tor log, not in dmesg, ifconfig also seems fine.

I mean, I can see that it can no longer connect in journalctl -xe, both openvpn and tor are unable to connect. But there is nothing in the logs to indicate why the problem occurred, why was the network on eth0 and eth1 reset?

Doing systemctl restart tor@default and systemctl restart openvpn@openvpn restores the connection, but it doesn’t take long before this happens again, that the connection dies. So it is not usable as it is now.

I am at a loss for what I should do about this. I have tried reinstalling. I am following the guide exactly.

I’m using Virtualbox, and I have several other virtual machines that have no network problems.

Any help in debugging this?

Addition:

I did some debugging myself. Still no solution, but perhaps an experienced Whonix user can pinpoint the problem?

If I let it boot, and then immediately run whonixcheck, it tries to do timesync, which it reports as:
“Timesync status: not done”

This repeats, for around 120 seconds, and then just as it says “Whonixcheck gave up waiting”, the network interfaces are reset, as mentioned above.

The curious part is, that from the Whonix workstation, as I mentioned, everything works fine.

Can I just disable sdwdate? Is that safe? What should I do?

2

Hi rob75

I’m not sure this is a Whonix issue.

Could you please configure your VPN in a Debina virtual machine. Do you have problems with the same issue? (Rational)

No. sdwdate is very important for keeping time.

Its likely sdwdate is not able to time sync because of the connection issue. In other words ,VPN issue is not because of sdwdate.On another note, I don’t think this is the problem but it would be a good idea just to check if your system keeps the correct time.

https://whonix.orgwiki/Troubleshooting#Clock_Fix

You can use this site for the accurate time

https://timeanddate.com/worldclock/timezone/utc

1 Like
3

Thanks for the help.

Yes, the VPN works fine, both on the host machine and in a different Debian virtual machine.

I believe this is a Whonix issue.

As I said I do have a functional Internet connection on the Whonix gateway, in the sense that it can route traffic from the Whonix workstation onto the Internet. I get a Tor IP, if VPN goes down, everything goes down. TUNNEL_FIREWALL seems to works as it should.

But then, just because sdwdate fails to run, the Whonix gateway decides to cycle both eth0 and eth1 such that the Internet connection is broken.

This is done by whonixcheck. If I disable both sdwdate AND whonixcheck, it works. This is probably not safe, so I would like a better solution.

How is this related to the VPN?

The clock is accurate. This is not the issue. If it wasn’t correct I can just set it myself, manually?

As an aside, why isn’t:

systemctl disable whonixcheck.service sufficient to disable whonixcheck.service?

When I do systemctl status whonixcheck.service right after disabling it, it is still enabled!

4

No. To prevent whonixcheck from autostarting.

sudo systemctl mask whonixcheck

https://whonix.org/wiki/Whonixcheck_Hardening

You can also disable the whonixcheck script .

sudo chmod -x /usr/bin/whonixcheck

Keep in mind sdwdate and whonixcheck are separate from each other. whonixcheck only checks system status. It would not be causing your issue.

See: https://whonix.org/wiki/Whonixcheck#Introduction

You can take a look at sdwdate logs to see if there is anything relevent.

tail -f /var/log/sdwdate.log

Something else to try is configuring Tor to wait for OpenVPN (end of instructions)? If not already done, can you try that.

1 Like
5

Ok, if I look at the log using tail -f, it waits for:

sdwdate - INFO - Running sdwdate fetch loop. iteration: 1
sdwdate - INFO - Requested URLs {some onion URL}

It waits here for a while.

Then suddenly, there is a lot more in the log as it tries another URL (some other onion URL).

Because then it seems to actually get a connection as it lists times and time_diff, etc.

Then at the end, it reports:

INFO - Success. Sleeping for …

But then I see this:

/usr/bin/whonix-gateway-firewall - OK: Skipping firewall mode detection since already set to ‘full’.
/usr/bin/whonix-gateway-firewall - OK: (Full torified network access allowed.)
/usr/bin/whonix-gateway-firewall - OK: Whonix firewall loaded.
pcnet32 0000:00:03.0 eth0: link up, 100Mbps, full-duplex
pcnet32 0000:00:08.0 eth1: link up, 100Mbps, full-duplex

Note that this happens JUST as my Internet connection on the Whonix workstation goes down.

There is probably something about these commands:

/usr/bin/whonix-gateway-firewall
/usr/bin/whonix-gateway-firewall
/usr/bin/whonix-gateway-firewall

They mess up the connection somehow.

It seems to me that sdwdate is broken in a VPN + Tor setup. Because everything works fine without it.

6

Hi rob75

I set up a VPN in Whonix-Gateway and had no issues. I used a “vanilla” Gateway with no modifications except for what was needed for VPN setup.

Are you using AppArmor by chance?

I see 2 other threads with pcnet32 0000:00:03.0 eth0: link up, 100Mbps, full-duplex and they are sdwdate/apparmor related.

Could you try configuring your VPN in a vanilla Whonix-gateway and see if that makes any difference. (if your not already using one)

1 Like
7

I had it running before without the VPN setup just for testing, no issues then.

It seems to be using AppArmor by default. I tried disabling it by systemctl mask, but it doesn’t change anything.

I guess I can try a vanilla setup without VPN, again. But I didn’t do anything beyond following the guide Connecting to a VPN before Tor

8

I can confirm now that if I disable VPN, the problem does not happen.

I disable VPN by setting VPN_FIREWALL=0 and masking the openvpn@openvpn service.

I know what you’re thinking, that this is then because there is something wrong with the VPN. But let me assure you this is not the case. There is nothing wrong with the VPN itself. It works fine on other computers, same configuration. It even works on the Whonix gateway.

Keep in mind that everything works fine as long as I systemctl disable sdwdate.

sdwdate does something, either directly or indirectly, with the network connections and/or iptables.

This is pretty obvious, since I get:

pcnet32 0000:00:03.0 eth0: link up, 100Mbps, full-duplex
pcnet32 0000:00:08.0 eth1: link up, 100Mbps, full-duplex

AFTER sdwdate has run.

So just to recap:

If I run without sdwdate:

dmesg |grep eth[0,1]

pcnet32 0000:00:03.0 eth0: link up, 100Mbps, full-duplex
pcnet32 0000:00:08.0 eth1: link up, 100Mbps, full-duplex

If I run with sdwdate:

dmesg |grep eth[0,1]

pcnet32 0000:00:03.0 eth0: link up, 100Mbps, full-duplex
pcnet32 0000:00:08.0 eth1: link up, 100Mbps, full-duplex
pcnet32 0000:00:03.0 eth0: link up, 100Mbps, full-duplex
pcnet32 0000:00:08.0 eth1: link up, 100Mbps, full-duplex

Notice that I get these two a second time then. That second time happens after sdwdate has run its course.

Again I believe this is related to this:

/usr/bin/whonix-gateway-firewall - OK: Skipping firewall mode detection since already set to ‘full’.
/usr/bin/whonix-gateway-firewall - OK: (Full torified network access allowed.)
/usr/bin/whonix-gateway-firewall - OK: Whonix firewall loaded.

9

This is normal behavior. Networking for all but Tor and sdwdate should be locked until time-sync is successful. One time-sync is complete, network access is allowed for apt-get etc. Since VPN is already connected I’m don’t think this is the culprit. I get the same messages.

I’m was thinking it could be something like AppArmor. You can enable Apparmor notification to see if anything relevant is in the logs.

https://whonix.org/wiki/AppArmor#Inspecting_and_Disabling_AppArmor_Notifications

Could you also please post the output of the last command. Preferably when VPN connnects, and disconnects. Make sure all sensitive information is redacted. note: While this is a pain in the but it will be very helpfull. Its good to have a second set of eyes to take a look.

sudo /usr/sbin/openvpn --rmtun --dev tun0
sudo /usr/sbin/openvpn --mktun --dev tun0 --dev-type tun --user tunnel --group tunnel
cd /etc/openvpn/
sudo -u tunnel openvpn /etc/openvpn/openvpn.conf

Also please post relevant Tor logs. (Tor bootstap success, any network failures, any errors) Redact any sensitive info.
cat /var/run/tor/log

One other thing to clarify. You said the Tor connects in Whonix-Gateway and Whonix-Workstation then connection drops? Are you able to use any networked applications in Whonix-Workstation before connection in Gateway goes down?

1 Like
10

I will clarify what happens:

The Whonix gateway starts, connects to the VPN successfully, no errors, the Tor service waits for VPN to finish (I’m using After=openvpn.service), it then starts, again with no errors.

Everything is great. Everything works.

However, in the meantime, sdwdate has been started and is now running in the background.

Everything still works. I can use the Whonix gateway to do apt-get update, I can use the Whonix workstation to browse. I can verify that I have a Tor IP.

But then, suddenly, just as sdwdate finishes, I get the dmesg messages I mentioned before (printed in tty1):

pcnet32 0000:00:03.0 eth0: link up, 100Mbps, full-duplex
pcnet32 0000:00:08.0 eth1: link up, 100Mbps, full-duplex

Then, nothing works. Tor cannot connect. The VPN is still up, however.

I have reviewed /var/log/audit/audit.log

I have two DENIED entries for /usr/lib/onion-grater for some PCI device.

I’m not sure what the purpose is of the test you suggest, keep in mind that both openvpn and the Tor service run without errors if I don’t let sdwdate start.

So if this test should make any sense, wouldn’t I also need to (manually) run sdwdate?

What exactly should I do? Keep tty1 running openvpn, tty2 running Tor manually, and tty3 running sdwdate? I would of course start them in the right order.

I don’t have logs for you yet, I will need to run these commands, write the output to some files, transfer them somehow off of the gateway, redact the information, and post it here. For now, all I have is just my own eyes looking at it.

Doesn’t it seem reasonable that there is just some simple mistake that causes this? I’m sure many people successfully use VPN + Tor on Whonix.

Anyway, if I perform the test as I suggested above, this is what happens:

No errors, warnings, or other failures when starting openvpn. I grep for these things with grep -i, and also look at anything suspicious.

No errors, warnings, or other failures when starting Tor. I grep for these things with grep -i, and also look at anything suspicious.

No errors, warnings, or other failures when starting sdwdate. I grep for these things with grep -i, and also look at anything suspicious.

After sdwdate has run its course, I get as mentioned:

pcnet32 0000:00:03.0 eth0: link up, 100Mbps, full-duplex
pcnet32 0000:00:08.0 eth1: link up, 100Mbps, full-duplex

Now I can no longer run apt-get update (on the gateway) or browse with Firefox (workstation).

And now, as I keep watching the output of openvpn and tor, I see openvpn getting [UNDEF] Inactivity timeout (–ping-restart), restarting), but it is unable to connect to the VPN server.

Tor also cannot connect (of course):

Tried for 120 seconds to get a connection to [scrubbed]:42. Giving up. (waiting for circuit).
Tried for 120 seconds to get a connection to [scrubbed]:42. Giving up. (waiting for circuit).
Tried for 120 seconds to get a connection to [scrubbed]:42. Giving up. (waiting for circuit).
etc.

I don’t think looking more carefully at the logs from before sdwdate runs makes any sense. Because, keep in mind, if sdwdate never runs, the VPN connection stays up forever, same with Tor. No problems at all.

But I will try to get the logs anyway …

11

I forgot to mention that I needed to make two modifications to get openvpn to work at all.

I get:

ERROR: Cannot ioctl TUNSETIFF tun: Operation not permitted (errno=1)

Setting tun to tun0 doesn’t affect the error.

To solve this I had to do setcap cap_net_admin=epi /usr/sbin/openvpn

I don’t know if this is safe, but even if it is not, I don’t see how it is related to the problem at hand.

It seems like Whonix + VPN is very unstable in general. I’ve tried reinstalling Whonix several times now, following very painstakingly the guide Connecting to a VPN before Tor

12

Nothing to worry about.(Tor control port filter proxy)

You already posted sdwdate.log when connection was reinstated (pcnet32). Wanted to also see VPN and Tor logs. But posting logs {vpn,sdwate,Tor} from when system starts (connects) and when connection goes down would be the most helpful.

cat /var/run/tor/log
cat /var/log/sdwdate.log

sudo /usr/sbin/openvpn --rmtun --dev tun0
sudo /usr/sbin/openvpn --mktun --dev tun0 --dev-type tun --user tunnel --group tunnel
cd /etc/openvpn/
sudo -u tunnel openvpn /etc/openvpn/openvpn.conf

Yes, reasonable.

I’m looking for any clues that could point to why the connection is dropping when sdwdate is used. I don’t have the luxury of having your box sitting in front of me. Log analysis is the only way this will be solved whether its by you, me or someone else. Its something I should have asked for in the beginning. Nothing of relevance in the logs is relevant.

Its very odd that there are 2 sets of:

pcnet32 0000:00:03.0 eth0: link up, 100Mbps, full-duplex

It looks like something is triggering the interface when sdwdate is used. Not necessarily a sdwdate specific issue. If there are duplicate outputs in any logs that could point to the source of issue.

What are you using in openvpn.conf tun0 or tun?

1 Like
13

I’m using tun0, but there is no difference between tun and tun0.

I just reinstalled, again.

I have two problems that I have solved without following the guide (two, including the one I mentioned already, so one new). I’m sorry for not mentioning this before. Perhaps my issue with eth0 and eth1 being reset is somehow related to that, although I doubt it:

Issue 1: ERROR: Cannot ioctl TUNSETIFF tun: Operation not permitted (errno=1)

My solution to 1: tun -> tun1 doesn’t help, so I did setcap cap_net_admin=eip /usr/sbin/openvpn

Issue 2: Unable to send UDP packets when running openvpn.

My solution to 2: setcap cap_net_admin=eip /bin/ip

I’m still working on getting logs…

14

There seems to be several mistakes in the Whonix wiki for setting this up. I will write a different post for this when I finally have it gone, in order to hopefully improve it and help other users. Once I have figured out how much I’m doing wrong (if anything) and how much is wrong with the wiki (if anything).

Not only do I need to add CAP_NET_ADMIN to both openvpn and ip, but I also need to add:

User=tunnel
Group=tunnel

to my /lib/systemd/system/openvpn@openvpn.service file. Otherwise openvpn doesn’t start as tunnel.

1 Like
15

When you say exactly…

The modifications are likely what caused this this particular issue.

Edit: We appreciate when users debug on there own. You also have to look at this from the point of the person helping you. Its important that he/she has all the information. Even if you think its not important. For ~ 95% of the users with Whonix/VPN issues, its because they didn’t follow the instructions as written. Thats were log come into play.

I have to say its great to see someone who is will to keep at it until a solution is found. So I’d would be more than happy to continue.

Did you add any VPN options to your openvpn.conf that was not in your providers .ovpn file?

Did you follow all steps exactly as written aside from what was already mentioned? The reason I ask is your should not have to manually add:

User=tunnel
Group=tunnel

to my /lib/systemd/system/openvpn@openvpn.service file.

If you can improve the VPN documentation that would be awesome. :wink: Keep in mind that these steps work for me with no issues. And work for other users as well.

1 Like
16

I will now reinstall once more, follow the guide step by step, and post where I encounter a problem.

Thanks for the help, and sorry for not pointing out those setopt modifications I did before this late.

17

So this is what happens, from the very scratch:

I import the image Whonix-Gateway-14.0.0.7.4.ova in VirtualBox, I use default settings, leave everything as is. Start the new imported box.

The GUI starts up. I select “Connect”. Next. It connects to Tor successfully. Finish.

whonixcheck starts up, checking the Tor connection, etc.

It gets to around 75% and hangs.

Eventually I get:

"WARNING: Debian Package Update Check Result: Could not check for software updates! (Timeout reached.) (apt-get code 124)

I run the suggested commands (apt-get update && apt-get dist-upgrade).

The commands run fine. Downloads, updates. No problem.

The commands finish with no errors.

Then whoixcheck pops up again with an error:

"ERROR: Tor Pid Check Result:
Tor not running.(tor_pid_message: Pid file /var/run/tor/tor.pid does not exist.)

etc."

Perhaps related to the dist upgrade, I reboot.

I manually run whonixcheck from the menu after it boots up again.

This time whonixcheck finishes with no errors.

I power off the machine and boot it again after setting RAM to 256, to force it into console only. I don’t do this until now, because dist-upgrade can fail when there is that little memory.

After booting, I run whonixcheck first.

Again I get:

[WARNING] [whonixcheck] Debian Package Update Check Result: Could not check for software updates! (apt-get code: 100)
Please manually check:
…

I run apt-get update && apt-get dist-upgrade manually.

This runs fine.

I run apt autoremove as it is listed as possible.

Now I start following the Tor + VPN guide.

Again I’m using: User → VPN → Tor → Internet

I edit: /etc/whonix_firewall.d/50_user.conf

And I add VPN_FIREWALL=1

Note that the file /etc/whonix_firewall.d/50_user.conf doesn’t exist from before, it gets created.

I reload the firewall: whonix_firewall

No errors.

I edit: /etc/sudoers.d/tunnel_unpriv

Comment in:

tunnel ALL=(ALL) NOPASSWD: /bin/ip
tunnel ALL=(ALL) NOPASSWD: /usr/sbin/openvpn *
Defaults: tunnel !requiretty

Now we get to the VPN setup.

The guide attempts to explain how to get a certificate and credentials. I’m just using a .ovpn file from my VPN provider. This .ovpn file I can usually just load using “openvpn openvpn.ovpn”, and it works fine (from e.g. Ubuntu, Debian, whatever). The .ovpn includes credentials, IP of the remote VPN server, certificate, etc.

So I skip:

Get VPN Certificate: Skip.
VPN Credentials: Skip.
VPN IP Address: Skip.

However, VPN Configuration File I cannot skip.

I need to add this to the .ovpn file.

script-security 2
user tunnel
iproute /usr/bin/ip_unpriv

OK, I add that, and I place the final file at /etc/openvpn/openvpn.conf

Note that in this file I now have:

client
dev tun0
proto udp
remote [redacted]
resolv-retry infinite
nobind
persist-key
persist-tun
script-security 2
user tunnel
iproute /usr/bin/ip_unpriv
remote-cert-tls-server
cipher AES-256-CBC
comp-lzo no
route-delay 5
verb 3
explicit-exit-notify 5
Then there is a bunch of stuff about certificates and keys.

It has everything else it needs to connect, the IP of the VPN server, certificate (in the same file), etc.

Configuration Folder Permissions:

I fix the permissions:

chown -R tunnel:tunnel /etc/openvpn
chown -R tunnel:tunnel /var/run/openvpn

systemd setup:

I create the systemd service file:

cp /lib/systemd/system/openvpn@.service /lib/systemd/system/openvpn@openvpn.service
systemctl enable openvpn@openvpn
systemctl start openvpn@openvpn
systemctl status openvpn@openvpn

I get:

“active (running)”

I do:

ifconfig, and see tun0 is up and has an IP.

I run: whonixsetup

Connect to tor, it gets successfully enabled, and whonixcheck starts up.

whonixcheck fails with:

/usr/lib/whonixcheck/cleanup.bsh: line 215: cd: …: Permission denied

I attempt to ignore this and just move on:

The guide asks for you to edit:

/etc/systemd/system/tor.service.d/50_user.conf

But this file doesn’t exist.

I assume they mean:

/etc/systemd/system/multi-user.target.wants/tor.service

I add:

After=openvpn.service

Under [Unit].

I reboot the Gateway.

Now I get the:

pcnet32 0000:00:03.0 eth0: link up, 100Mbps, full-duplex
pcnet32 0000:00:08.0 eth1: link up, 100Mbps, full-duplex

And we’re back to my opening post.

Edits:

I realize now that I didn’t have to do any setopt hack to get it working (well, it isn’t working, but to get this far, openvpn runs …), and I have no idea why!

1 Like
19

Yeah, that is an option if you have the ability to do that, thanks for the suggestion.

I want to get this to work though, especially since I spent so much time on it.

1 Like
20

I will now work on obtaining the requested logs.

Is the following test setup adequate?

I disable with systemctl mask openvpn@openvpn, tor, and sdwdate. I reboot.

Then, I run openvpn manually, and redirect all output to a log file.

Once openvpn is up and running I run tor manually, again I redirect all output to a log file.

I then run sdwdate manually, and redirect all output to a log file.

I allow the network to fail once sdwdate finishes and allow it to log for some time more into all three log files.

Then I will redact sensitive information and post it here. Will that work?

21

Hi rob75

You might not have seen Firewall settings “Hit Expand”

This is most likely your problem.

sudo nano /etc/whonix_firewall.d/50_user.conf

Add

VPN_FIREWALL=1

Comment in. As long as the comment “#” is removed your OK. Which It look likes you have done.

You might not have seen Firewall settings “Hit Expand”

sudo nano /etc/whonix_firewall.d/50_user.conf

Add.

VPN_FIREWALL=1

You didn’t press on “Expand on the right”

Additional Tweaks / Recommendations / Troubleshooting

If having problems with the connection / Tor is not fully bootstrapped

please press on → Expand on the right

Create a folder /etc/systemd/system/tor.service.d.

sudo mkdir /etc/systemd/system/tor.service.d

Create a file /etc/systemd/system/tor.service.d/50_user.conf.

sudo nano /etc/systemd/system/tor.service.d/50_user.conf

Add the following content.

[Unit] After=openvpn.service

Save.

Let me know if that works.

Looks good to me. Only needed if the previous info does not remedy the problem.

BTW I’ve missed a few thing my self i.e. not hitting expand

1 Like