Multiple Whonix workstations that can communicate with each other.

beaky · June 9, 2016, 7:27pm

ps Thank you Patrick for the script.

xloem · December 28, 2016, 2:19am

I am also trying to do this: https://tor.stackexchange.com/questions/13522/how-to-configure-whonix-gateway-for-communication-between-two-local-workstations
I’m disheartened that it apparently does not work.

It sounds like sys-whonix does not route packets the way sys-firewall does.

EDIT: indeed, /proc/sys/net/ipv4/ip_forward is 0 on sys-whonix and 1 on sys-firewall
EDIT: echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward on sys-whonix should resolve your problem. I’m still trying to figure out what firewall rule is filtering my pings

Patrick · December 28, 2016, 5:24pm

Xloem:

EDIT: indeed, /proc/sys/net/ipv4/ip_forward is 0 on sys-whonix and 1 on sys-firewall

Leave that as is. No need to change.

You need to open ports in the iptables firewall. Figure out how to do
inter-vm communication for non-Whonix AppVMs first. Then you should be
able to replicate that the same way using Qubes-Whonix.

xloem · December 29, 2016, 10:30am

Thanks, Patrick.

Like beaky, I have successfully set up inter-vm communication for non-Whonix AppVMs, following https://www.qubes-os.org/doc/firewall/#enabling-networking-between-two-qubes . However, the same set of steps does not work for Whonix VMs. Pings give no replies. A packet log on the whonix gateway shows no packets.

If I enable ip forwarding on the gateway, the pings do get replies, but the replies are “Packet filtered”. I need to understand it more to find the solution.

Patrick · December 29, 2016, 1:00pm

qubes-firewall-user-script indeed won’t work. You might have to edit /usr/bin/whonix_firewall (in a persistent way) forward rules.

entr0py · December 30, 2016, 12:22am

sys-whonix handles packets like any other Linux distribution - using routing tables and iptables. Perhaps you are referring to Tor’s handling of packets instead. Tor receives traffic through the INPUT filter, wraps it up, then sends it out through the OUTPUT filter. So strictly speaking, Tor is not a FORWARDing process.

Forgot about this thread but indeed, I have played around a bit. Since you are a Qubes-Whonix user, I originally wanted to suggest using a proxyVM as a mini-LAN router. While trivial to set up, it has a major flaw. It doesn’t allow IsolateClientAddr to work because all client IPs are MASQUERADEd - so only partial (weak) stream isolation.

To do what you want, you have to modify Whonix-Gateway templateVM (or use a bind dir?) which sometimes runs the risk of leading to future breakage or leaks. If you’re comfortable with that, here are some examples you can add to /usr/bin/whonix_firewall's IPv4 FORWARD section. Use one of the examples below:

### Enable 2-way inter-VM traffic. All protocols allowed, including ICMP (ping).
$iptables_cmd -I FORWARD 1 -i vif+ -o vif+ -j ACCEPT

### Enable 1-way inter-VM traffic. Limit to specific IPs. TCP only.
$iptables_cmd -I FORWARD 1 -i vif+ -o vif+ -m state --state RELATED,ESTABLISHED -j ACCEPT
$iptables_cmd -I FORWARD 2 -i vif+ -s <source.ip.address> -o vif+ -d <dest.ip.address> -p tcp -j ACCEPT

Also, enable FORWARD filter: [not leak tested with Whonix! Do so at your own risk!]

sudo mv /etc/sysctl.d/ipv4-forward-disable.conf /etc/sysctl.d/ipv4-forward-enable.conf
sudo nano /etc/sysctl.d/ipv4-forward-enable.conf
# Replace '0' with '1'

Reboot.

Patrick · December 29, 2016, 11:22pm

Not advisable to enable IP forwarding. Not leak tested. Always avoided this because without it, networking is so much simpler. Praise Tor’s DnsPort and TransPort.

entr0py · December 30, 2016, 12:44am

Well, that complicates things… [Are you saying it’s possible to do this without forwarding?] On principle, I would really rather use a separate proxyVM… But I think the only way to do that is to have multiple interfaces from one proxyVM connecting to Whonix-Gateway… (not ready to tackle that one.)

Patrick · December 30, 2016, 1:41am

No.

xloem · December 30, 2016, 12:34pm

I guess the right way to do this would be to set up a proxy on the gateway for the services / ports I want to use. For my use-case of SSH, I would connect to the gateway on port say 2122, and some process running on the gateway would copy the packets to the other workstation, changing the port to 22. This could be done with socat or netcat running as a service.

EDIT: hrm now I need to figure out how to make exceptions for the normal torification of the connections so that I can contact this local proxy

Patrick · December 30, 2016, 12:41pm

Better use a socket over qrexec instead.

Here is an example doing ssh through qrexec but different from what you want (inter-vm). Perhaps that helps.

https://www.whonix.org/wiki/Dev/Qubes#ssh_into_Qubes_dom0

There are also qrexec docs on Qubes website.

xloem · December 31, 2016, 12:20pm

Thank you so much for this! I’ve written up the solution as a 3-step process at https://tor.stackexchange.com/questions/13522/how-to-configure-whonix-gateway-for-communication-between-two-local-workstations/13546#13546 .

entr0py · December 31, 2016, 8:06pm

@xloem Thanks for the writeup! I had no idea how to do that. Might help with my problem.

xloem · January 1, 2017, 7:46am

entr0py, what’s this other problem you speak of?

entr0py · January 2, 2017, 2:34am

@xloem Thanks for your interest. Problem is a very low priority pet project: Physical Isolation is back! Qubes-Whonix style. I haven’t begun investigating yet. A good start would be learning how physical switches preserve source IP’s without masquerading an intermediary IP. Don’t know enough about networking in general - perhaps it’s done with ARP tables? The trivial, messy solution would be to assign a separate proxyVM to each WS.

In the past, I’ve used a configuration that is relevant to your original question and also affected by my issue:

 Whonix-GW                         
  ^^                               
  ||                               
  ||                               
 LAN proxyVM                       
  ^^   ^                           
  ||   +---+--------+---------+    
  ||       |        |         |       
 WS-A     WS-B     WS-C      WS-D

Here, A has internet access via Whonix-GW and also LAN access. B,C,D are restricted to LAN-only. Luckily, I only had one client connect to Whonix-GW. [Disclaimer: B,C,D should not be considered non-networked or air-gapped because leaks can happen through a compromised proxyVM or by exploiting WS-A via the LAN to leak through the proxyVM.]

xloem · January 4, 2017, 10:46am

I think that’s called Network Address Translation? The linux kernel can do it. ARP tables are for mapping MAC addresses to and from IP addresses.

An important question here is whether Whonix-GW separates streams based on virtual interface or based on IP address. If it’s the former, you’ll need separate internal netvms for each client.

The qrexec solution could be briefly messy if these are different physical boxes. You’d need to invent your own channel to send the process streams over.

Patrick · January 25, 2017, 11:22am

Talked to Marek at 33c3. In Qubes-Whonix (as opposed to Non-Qubes-Whonix) using direct IP is impossible to have inter VM communication without enabling IP forwarding. However, enabling IP forwarding is strongly discouraged. Therefore use the qrexec based solution.

qubenix · August 29, 2017, 3:56am

Ok, this qrexec method is great, but how can I use it for something other than ssh? ssh seems luck to have a ProxyCommand option, but most situations are not so lucky.

For instance say bitcoind is listening on 127.0.0.1:8332 in a dedicated workstation. How can I have curl from another workstation use this qrexec method? The only thing I could come up with so far is using this qrexec + ssh method from and then setting up an ssh listener in /rw/config/rc.local. However this seems hacky.