Improve apt-get Stream Isolation for Qubes Templates

entr0py · December 21, 2016, 7:00pm

Credit to @tasket @rustybird for highlighting the issue. (can’t find the post) Just remembered: https://github.com/rustybird/qubes-updates-cache

For Tor users, the remote network has some insight about what’s installed on the same physical computer, because there is no Tor circuit isolation between update requests from different clients. This can be prevented by waiting at least 10 minutes (or sending NEWNYM) between updating different clients. (The same is true of qubes-updates-proxy.)

Desired behavior: apt-get should use new circuits for each client VM and also use new circuits for each repository. In non-Qubes-Whonix, multiple VMs that perform apt-get update use different circuits because of IsolateClientAddr. In Qubes-Whonix, all TemplateVMs share the same circuits for apt-get (due to Update Proxy design?).

Current observed behavior: Circuit A is used to route traffic for all apt-get requests from all TemplateVMs going to vwakviie2ienjx6t.onion. Circuit B is used for all traffic going to sgvtcaew4bxjd7ln.onion. Circuit C is used for all traffic to clearnet repos, including qubes-os.org, whonix.org, and 3rd party repos. These potentially allows for some linkability between TemplateVMs - both by the Exit Node as well as by the repository server.

I don’t know enough about tinyproxy (or http proxies in general) to propose a solution. Some possibilities:

*1. (For non-Qubes-Whonix) Enable IsolateDestAddr for apt-get port. From /usr/share/tor/tor-service-defaults-torrc:

## Operating system updates: apt-get
## Not using IsolateDestAddr IsolateDestPort, because too much
## performance loss, too much load on Tor network and no gain
## in security.
SocksPort 10.137.1.1:9104

This was probably written pre-Qubes. Enabling IsolateDestAddr will cause an extra tor circuit per clearnet repo - not too much load. And no additional load if using .onion repos which are isolated by default. This would fix per-repo isolation but not per-client isolation.

! This doesn’t work for Qubes-Whonix because uwtwrapper is disabled for apt-get. Instead apt-get requests are sent to tinyproxy port 8082 and then sent to Tor via (TransPort???). Can we spin the packet around and have it route through a SocksPort?

*2. (For Qubes-Whonix) Somehow masquerade Source IP so that IsolateClientAddr is activated.

*3. (For Qubes-Whonix) Somehow enable SOCKSAuth so that some category of apt-get request is assigned a random SOCKS password.

Patrick · December 22, 2016, 3:43am

Yes.

If all TemplateVMs use the same sys-whonix as their NetVM, unfortunately yes.

entr0py:

## Operating system updates: apt-get
## Not using IsolateDestAddr IsolateDestPort, because too much
## performance loss, too much load on Tor network and no gain
## in security.
SocksPort 10.137.1.1:9104

This was probably written pre-Qubes.

Was written pre-Qubes indeed.

More than that.

When you connect run nslookup ftp.us.debian.org for several times, you get several results. DNS round robin. It’s actually multiple servers. Each apt-get package download is a separate fetch. So it would be one circuit per resolved ftp.us.debian.org IP when multiple packages are downloaded.

I agree we should revisit this. Can you please ask on tor-dev if it’s fine for Whonix to use IsolateDestAddr IsolateDestPort for its apt-get Tor SocksPort?

Yes.

I wouldn’t know how. That would be preferred. However, tinyproxy does not support socks proxy settings, which would be a lot better. It just uses normal networking ("speaks only trans"). It runs under its own linux user account at least which might make things easier.

The question you are asking is generally a hard one. How can one force an arbitrary application that does not support socks proxy settings through a socks proxy. Well, there there are proxifiers / socksifiers such as torsocks (and uwt scripting it). But that probably won’t work with tinyproxy.

sudo su
torsocks /usr/sbin/tinyproxy -d -c /etc/tinyproxy/tinyproxy-updates.conf

/usr/sbin/tinyproxy: Could not create listening socket.

One cannot socksify an application as well as at the same time have it open a listener port using a socksifier.

redsocks might be worth exploring.

(There is some information on redsocks here - although it’s on a different topic you can see the idea - https://www.whonix.org/wiki/Tunnels/Connecting_to_Tor_before_a_proxy#Transparent_Proxying_Method.(

Although all DNS would still go through Tor’s DnsPort. [Might be fingerprintable, because TCP traffic without previous DNS request.]

Do you think qrexec based updates proxy will help with this? @marmarek

marmarek · December 22, 2016, 5:16am

Do you think qrexec based updates proxy will help with this? @marmarek

Maybe. Connection to the “proxy” will arrive as qrexec call, stream
there will be “HTTP proxy” protocol. So, if you pass it to some “HTTP
proxy -> socks” filter (instead of tinyproxy), that would work. I think
socat can not do that, but maybe something else can?

You’ll have there name of source VM, so can use that to direct it to
isolated circuits.

Patrick · January 31, 2017, 1:59pm

Talked to Roger at 33c3: No, he prefers us not enabling that.

/usr/sbin/tinyproxy: Could not create listening socket.
One cannot socksify an application as well as at the same time have it open a listener port using a socksifier.

This might be possible in Debian stretch based Whonix 14. torsocks learned AllowOutboundLocalhost 1, which will be enabled by default in Whonix 14 in the uwt package.

nyxnor · July 7, 2022, 10:32pm

Any update on stream isolation when updating qubes?

nyxnor · July 8, 2022, 12:36am

I think tinyproxy config needs to change. It needs to bind certain client addresses to certain virtual hosts.

As it is right now, it does not show the client addr,

CONNECT   Jul [487354]: Connect (file descriptor 10): localhost [127.0.0.1]
CONNECT   Jul [487354]: Request (file descriptor 10): GET http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.1/vm/dists/bullseye/InRelease HTTP/1.1
INFO      Jul [487354]: No upstream proxy for deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion
INFO      Jul [487354]: opensock: opening connection to deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion:80

I think it is the way Qubes implemented Tinyproxy, as we are connecting directly to a localhost socket, we are localhost in the traffic, it can’t differ clients (different qubes templates).

github.com

QubesOS/qubes-core-agent-linux/blob/a90c3568d19026e85959fad1a16fd5cad88845cd/vm-systemd/qubes-updates-proxy-forwarder.socket#L9

      
        
            [Unit]
            Description=Forward connection to updates proxy over Qubes RPC
            ConditionPathExists=/var/run/qubes-service/updates-proxy-setup
            # don't start the forwarder when updates proxy itself is also enabled here,
            # otherwise it would most likely loop back to itself
            ConditionPathExists=!/var/run/qubes-service/qubes-updates-proxy
            
            
[Socket]
            ListenStream=127.0.0.1:8082
            BindToDevice=lo
            Accept=true
            
            
[Install]
            WantedBy=multi-user.target

Running 2 non-whonix templates and using sys-whonix as update vm:
Circuit reuse because tinyproxy does not pass the client addr.
The DNS request uses localhost ip, while the connection uses the gateway qube local ip.

$ tor-ctrl-stream -z -m

StreamId StreamPurpose StreamTarget StreamClient CircuitId CircuitPurpose
------------------------------------------------------------------------------------------------------
376 USER 10.228.239.209-(2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion:80) 10.137.0.8:40006 252 HS_CLIENT_REND
386 USER 10.240.194.32-(deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion:80) 10.137.0.8:47834 285 HS_CLIENT_REND
387 DNS_REQUEST deb.debian.org-(199.232.138.132:0) 127.0.0.1:57744 153 GENERAL
388 DNS_REQUEST deb.debian.org-([2a04:4e42:62::644]:0) 127.0.0.1:57744 153 GENERAL
389 USER deb.debian.org-(199.232.138.132:443) 10.137.0.8:45936 154 GENERAL
392 USER 10.240.194.32-(deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion:80) 10.137.0.8:47842 285 HS_CLIENT_REND
395 USER 10.240.194.32-(deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion:80) 10.137.0.8:47846 285 HS_CLIENT_REND

nyxnor · July 8, 2022, 10:51am

Stream isolation is fully function for different WS using the same GW.

Template stream isolation is a QubesOS issue that only pass 127.0.0.1 as the client address to sys-whonix tinyproxy.

Hello,
Our automated spam filter, Akismet, has temporarily hidden your post in How to identify which template requested to use the proxy? for review.
A staff member will review your post soon, and it should appear shortly.
We apologize for the inconvenience.

Patrick · July 8, 2022, 12:18pm

To understand my post, prerequisite knowledge from Tor manual:

SocksPort
TransPort
IsolateClientAddr
IsolateSOCKSAuth

Actually, I have to correct my former self. Now, I don’t think tinyproxy now having socks support will help a lot with improving apt-get stream isolation for Qubes Templates. All Qubes Template APT traffic would then end up in the same Tor SocksPort. A tiny improvement over all of that traffic ending up in Tor’s TransPort.

tinyproxy socks proxy support is useful for other things. Other than stream isolation. Namely, Whonix-Gateway firewall could perhaps be be simplified if tinyproxy was configured by the qubes-whonix package to use a Tor SocksPort.

Indeed. From the perspective of tinyproxy, traffic is “suddenly coming from nowhere”, in other words, technically from other VM, through qrexec, arriving at tinyproxy localhost.

tinyproxy receives no client addr. Neither tinyproxy has a concept similar to Tor such as IsolateClientAddr and IsolateSOCKSAuth.

Probably required:

qrexec feature request: tinyproxy would have to receive client addr which would require some qrexec modification.
tinyproxy feature request: tinyproxy would have to set a different Tor socks user name for each different client addr. This would result in Tor using a different Tor circuit for each Tor socks user name (IsolateSOCKSAuth) set by tinyproxy.

Not sure that is realistic?

Qubes: An older unrelated qrexec feature request https://github.com/QubesOS/qubes-issues/issues/5253 didn’t have any activity.
tinyproxy: How interested would tinyproxy contributors be to implement more granular, finer Tor stream isolation support?
- A feature request similar to the following:
  - Use different socks user name per account (Tor) (#9213) · Issues · gajim / gajim · GitLab
  - make use of stream isolation (#6359) · Issues · Legacy / Trac · GitLab

Help welcome with submitting feature request welcome and anything else. Otherwise I don’t think any progress will be made here.

The only other alternative that I can see is making Qubes-Whonix Templates networked by default to patch/fix the direct communication between APT and a Tor SocksPort. Otherwise if a proxy (tinyproxy) is wretched in between, granular stream isolation will be difficult to implement.

nyxnor · July 8, 2022, 12:43pm

qrexec feature request: tinyproxy would have to receive client addr which would require some qrexec modification.

I think this is the most realistic option because tinyproxy needs to get different clients first, as of know only getting 127.0.0.1.

My proposal is either use tinyproxy on a virtual network so it can receive the virtual network ip from templates and only that (but this may be insecure, idk)

Or Tinyproxy listening to connection on different ports for different templates, but this is cumbersome to configure on template clones and then having to modify its config port.

Patrick · August 30, 2022, 8:30am

A related issue:

github.com/unman/shaker

Cacher incompatible with sys-whonix (updates fail for whonix-ws and whonix-gw)

opened 02:56PM - 19 Aug 22 UTC

tlaurion

EDIT: - Working, undesired implementation cacher proxy, advertising itself as g…uaranteeing tor proxy even if false, is at: https://github.com/unman/shaker/issues/10#issuecomment-1221116802 - The only way whonix template updates could be cached as all other templates if repo defs modigied to compoy with apt-cacher-ng requirements is if a cacher-whonix version was created, deactivating whonix-gw's tinyproxy and replacing it with apt-cacher-ng so that sys-whonix would be the update+cache proxy. ---- When cacher is activated, whonix-gw and whonix-ws templates cannot be updated anymore, since both whonix-gw and whonix-ws templates implement a check through systemd qubes-whonix-torified-updates-proxy-check at boot. ~Also, cacher overrides whonix recipes applied at salt install from qubes installation, deployed when the user specifies that he wants all updates to be downloaded through sys-whonix.~ ~The standard place where qubes defines, and applies policies on what to use as update proxies to be used is under `/etc/qubes-rpc/policy/qubes.UpdatesProxy` which contains on standard install:~ Whonix still has policies deployed at 4.0 standard place: ``` $type:TemplateVM $default allow,target=sys-whonix $tag:whonix-updatevm $anyvm deny ``` And where cacher write those at standard q4.1 place https://github.com/unman/shaker/blob/3f59aacbad4e0b10e69bbbcb488c9111e4a784bc/cacher/use.sls#L7-L9 ~First thing first, I think both cacher and whonix should agree on where UpdatesProxy settings should be prepended/modified, which I think historically (and per Qubes documentation as well) it should be under `/etc/qubes-rpc/policy/qubes.UpdatesProxy` for clarity and not adding confusion.~ Whonix policies needs to be applied per q4.1 standard under Qubes. Not subject of this issue. @unman @adrelanos @fepitre --------- The following applies proper tor+cacher settings: https://github.com/unman/shaker/blob/3f59aacbad4e0b10e69bbbcb488c9111e4a784bc/cacher/change_templates.sls#L5-L13 Unfortunately, whonix templates implement a sys-whonix usage check which prevents templates to use cacher. This is documented over https://www.whonix.org/wiki/Qubes/UpdatesProxy, and is the result of `qubes-whonix-torified-updates-proxy-check` systemd service started at boot. Source code of the script can be found at https://github.com/Whonix/qubes-whonix/blob/98d80c75b02c877b556a864f253437a5d57c422c/usr/lib/qubes-whonix/init/torified-updates-proxy-check Hacking around current internals of both project, one can temporarily disable cacher to have torified-updates-proxy-check check succeed and put its success flag that subsists for the life of that booted Templatevm. We can then reactivate cacher's added UpdatesProxy bypass and restart qubesd, and validate cacher is able to deal with tor+http->cacher->tor+https on Whonix TemplatesVMs: 1- deactivate cacher override of qubes.UpdateProxy policy: ``` [user@dom0 ~]$ cat /etc/qubes/policy.d/30-user.policy #qubes.UpdatesProxy * @type:TemplateVM @default allow target=cacher ``` 2- restart qubesd ``` [user@dom0 ~]$ sudo systemctl restart qubesd [user@dom0 ~]$ ``` 3- Manually restart whonix template's torified-updates-proxy-check (here whonix-gw-16) `user@host:~$ sudo systemctl restart qubes-whonix-torified-updates-proxy-check` We see that whonix applied his state at: https://github.com/Whonix/qubes-whonix/blob/98d80c75b02c877b556a864f253437a5d57c422c/usr/lib/qubes-whonix/init/torified-updates-proxy-check#L46 ``` user@host:~$ ls /run/updatesproxycheck/whonix-secure-proxy-check-done /run/updatesproxycheck/whonix-secure-proxy-check-done ``` 4- Manually change cacher override and restart qubesd ``` [user@dom0 ~]$ cat /etc/qubes/policy.d/30-user.policy qubes.UpdatesProxy * @type:TemplateVM @default allow target=cacher [user@dom0 ~]$ sudo systemctl restart qubesd ``` 5- check functionality of downloading tor+https over cacher from whonix template: ``` user@host:~$ sudo apt update Hit:1 http://HTTPS///deb.qubes-os.org/r4.1/vm bullseye InRelease Hit:2 tor+http://HTTPS///deb.debian.org/debian bullseye InRelease Hit:3 tor+http://HTTPS///deb.debian.org/debian bullseye-updates InRelease Hit:4 tor+http://HTTPS///deb.debian.org/debian-security bullseye-security InRelease Hit:5 tor+http://HTTPS///deb.debian.org/debian bullseye-backports InRelease Get:6 tor+http://HTTPS///fasttrack.debian.net/debian bullseye-fasttrack InRelease [12.9 kB] Hit:7 tor+http://HTTPS///deb.whonix.org bullseye InRelease Fetched 12.9 kB in 7s (1,938 B/s) Reading package lists... Done Building dependency tree... Done Reading state information... Done ``` Problem with this is that qubes update processes will start templates and try to apply updates unattended, and this obviously won't work unattended. The question is then how to have whonix templates do a functional test for it to see that torrified updates **are possible** instead of whonix believing he is the only one providing the service? The code seems to implement curl check, but also doesn't work even if cacher is exposed on as a tinyproxy replacement, listening on 127.0.0.1:8082. Still digging, but at the end, we need to apply mitigation ( disable Whonix check) or proper functional testing from Whonix, which should see that the torrified repositories are actually accessible. ----- How to fix this? Some hints: 1- cacher and whonix should modify the policy at the same place to ease troubleshooting and understanding of what is modified on the host system, even more when dom0 is concerned. I think cacher should prepend `/etc/qubes-rpc/policy/qubes.UpdatesProxy` 2- Whonix seems to have thought of a proxy check override: https://github.com/Whonix/qubes-whonix/blob/685898472356930308268c1be59782fbbb7efbc3/etc/uwt.d/40_qubes.conf#L15-L21 @adrenalos: not sure this is the best option, and I haven't found where to trigger that override so that the check is bypassed? 3- At Qubes OS install, torrified updates and torrifying all network traffic (setting sys-whonix as default gateway) is two different things, the later not being enforced by default. Salt recipes are available to force updates through sys-whonix when selected at install, which dom0 still uses after cacher deployment: ![29-Q41_options_selection_done](https://user-images.githubusercontent.com/827570/156628021-d3ec34af-e620-48b8-aa39-45d84c968769.jpeg) So my setup picked up sys-whonix as the default gateway for cacher since I configured my setup to use sys-whonix proxy as default, which permits tor+http/HTTPS to go through after applying manual mitigations. But that would not necessarily be the case for default deployments but would need to verify, sys-firewall being the default unless changed. @unman: on that, I think the configure script should handle that corner case and make sure sys-whonix is the default gateway for cacher if whonix is deployed. Or your solution wants to work independently of Whonix altogether (#6) but there would be discrepancy between Qubes installation options, what most users use and what is available out of the box after installing cacher from rpm: https://github.com/unman/shaker/blob/3f59aacbad4e0b10e69bbbcb488c9111e4a784bc/cacher.spec#L50-L60 4- That is, cacher cannot be used as of now for dom0 updates either. Assigning dom0 updates to cacher gives the following error from cacher: `sh: /usr/lib/qubes/qubes-download-dom0-updates.sh: not found` So when using cacher + sys-whonix, sys-whonix would still be used by dom0 (where caching would not necessarily makes sense since dom0 doesn't share same fedora version then templates, but I understand that this is desired to change in the near future. Point being here: sys-whonix would still offer its tinyproxy service, sys-whonix would still be needed to torrify whonix templates updates and dom0 would still depend on sys-firewall, not cacher, on a default install (without whonix being installed). Maybe a little bit more thoughts needs to be given to the long term approach of pushing this amazing caching proxy forward to not break things on willing testers :) @fepitre: adding you to see if you have additional recommendations, feel free to tag Marek if you feel like so later on, but this caching proxy is a really long awaited feature (https://github.com/QubesOS/qubes-issues/issues/1957) which would be a life changer if salt recipes, cloning from debian-11-minimal and sepecializing templates for different use cases, and bringing a salt store being the desired outcome from all of this. Thank you both. Looking forward for a cacher that can be deployed as "rpm as a service" on default installations. We are close to that, but as of now, this doesn't work, still, out of the box and seems to need a bit of collaboration from you both. Thanks guys!

Patrick · September 5, 2022, 9:25am

github.com/tlaurion/shaker

WiP cacher-whonix

tlaurion:main ← tlaurion:cacher-whonix

opened 03:30PM - 31 Aug 22 UTC

tlaurion

+1444 -0

Done: - Duplicated and started to specialize cacher into cacher-whonix for both… cacher-whonix salt recipes and cacher-whonix.spec rpm build file - Have userinfo.html put in the right place so that apt-cacher-ng advertise an enforcing tor proxy to whonix templates - Have references to cacher replaced with cacher-whonix paths and salt proper placeholders - Remove dependence on clone; we only want to apply changes to sys-whonix - Removed configure statement dismissing whonix templates (host) so whonix templates have their repos changed to use apt-cacher-ng - Modify whonix-gw-16 directly, no clone dependency (removed and references deleted) Missing: - Actual testing.... - Force restart sys-whonix after deployment, so that network is interrupted but reestablished automatically - postun of cacher-whonix should disable apt-cacher-ng and reactivate tinyproxy under sys-whonix. Missing unconfigure script? - Change README to specify this is targeting whonix-gw-16 and sys-whonix directly @unman: please comment in code in PR to my fork so that this can go forward @adrelanos: feel free to comment under https://github.com/tlaurion/shaker/pull/1

Patrick · September 5, 2022, 9:25am

github.com/QubesOS/qubes-issues

remove tinyproxy from Whonix-Gateway (`sys-whonix`) and make Whonix Templates networked by default with Net qube set to `sys-whonix`

opened 09:22AM - 05 Sep 22 UTC

adrelanos

T: bug P: default

I am suggesting to remove tinyproxy from Whonix-Gateway (`sys-whonix`) and make …Whonix Templates networked by default with Net qube set to `sys-whonix`. motivation for this change suggestion: * The current implementation is still basically mostly as implemented by nrgaway many years ago. nrgaway is no longer active. No co-maintenance / help / resources / hyper-vigilant auditing eyes available to keep maintaining it. * I don't like the extra complexity of having Qubes UpdatesProxy specific firewall rules in Whonix-Gateway firewall. ([here](https://github.com/Whonix/whonix-firewall/blob/master/usr/bin/whonix-gateway-firewall#L332-L373)) * It makes leak testing more complicated. * Requires complex torified UpdatesProxy check. * Primarily goal of Whonix is to route all traffic over Tor, leakproofness. Therefore any complexity / exceptions in its networking configuration is to be avoided. * Now a more complex solution is suggested on top of the current complexity. (https://github.com/tlaurion/shaker/pull/1) advantages of this change would be: * Better leakproofness. * Simplifies Whonix-Gateway firewall. * [Improves Qubes-Whonix Templates stream isolation.](https://forums.whonix.org/t/improve-apt-get-stream-isolation-for-qubes-templates/3315) * Makes Qubes-Whonix more similar to Non-Qubes-Whonix and hence easier to audit and maintain. disadvantages if this change is implemented: * Qubes-Whonix Templates being networked by default. Why do I as maintainer of Qubes-Whonix make this ticket? * I am not sure I'd have the authority to decide to make this change. Making a Qubes Template networked by default might be perceived as quite a conceptual level change. And orchestrating this change should be discussed beforehand for sure anyhow. Other alternatives? * Getting tinyproxy out of sys-whonix somehow... But how would that be implemented? Maybe a third VM but the effort to maintain that would be too high on my side. Perhaps Qubes would like to maintain a standalone tinyproxy / UpdatesProxy / cacher VM?