implement /usr/lib/tor-controlport-filter-merger

[phabricator-migrator]

Information

ID: 617
PHID: PHID-TASK-cf4rmrvhkice5tpthdin
Author: Patrick
Status at Migration Time: resolved
Priority at Migration Time: Normal

Description

As per:
https://mailman.boum.org/pipermail/tails-dev/2017-January/011172.html

TODO:

• take (and review) https://git-tails.immerda.ch/tails/plain/config/chroot_local-includes/usr/local/lib/tor-controlport-filter?h=feature/12173-end-whonix-controlport-filter-fork
• replace https://github.com/Whonix/control-port-filter-python/blob/master/usr/lib/tor-controlport-filter with above
• invent /usr/lib/tor-controlport-filter-merger
  ** parse
  *** /etc/tor-controlport-filter-merger.d and
  *** /usr/local/etc/tor-controlport-filter-merger.d (for Qubes-Whonix)
  ** auto generate /etc/tor-controlport-filter.d/30_autogenerated.yml

Bonus:

• if possible and easy code wise, add some comments on top of 30_autogenerated.yml such as “Manual edits will be lost! This file has been auto generated by…”
• each merged key gets a comment from which source file it is coming from

Comments

---

joysn1980

2017-01-28 07:33:40 UTC

Patrick,

#1 -
Can you provide me few real yml files to be placed under
/etc/tor-controlport-filter-merger.d
and
/usr/local/etc/tor-controlport-filter-merger.d
which I can use to write the /usr/lib/tor-controlport-filter-merger?

#2 - Also let me know how do we prioritise the
a) Directories - /etc/tor-controlport-filter.d/*.yml has higher priority than /usr/local/etc/tor-controlport-filter.d/
b) Files - lexical order?

Thanks

---

Patrick

2017-01-28 15:52:27 UTC

! In T617#11897, @joysn1980 wrote:
#1 -
Can you provide me few real yml files to be placed under
/etc/tor-controlport-filter-merger.d
and
/usr/local/etc/tor-controlport-filter-merger.d
which I can use to write the /usr/lib/tor-controlport-filter-merger?

https://github.com/Whonix/control-port-filter-python/blob/master/etc/tor-controlport-filter.d/30_whonix.yml will go there.

With an optional (by user) combination of https://github.com/Whonix/control-port-filter-python/tree/master/usr/share/tor-controlport-filter/examples.

#2 - Also let me know how do we prioritise the
a) Directories - /etc/tor-controlport-filter.d/*.yml has higher priority than /usr/local/etc/tor-controlport-filter.d/
b) Files - lexical order?

Parse /etc/tor-controlport-filter.d/first. Parse /usr/local/etc/tor-controlport-filter.d/ second. Therefore the latter has higher priority.

---

joysn1980

2017-01-31 10:05:36 UTC

take (and review) https://git-tails.immerda.ch/tails/plain/config/chroot_local-includes/usr/local/lib/tor-controlport-filter?h=feature/12173-end-whonix-controlport-filter-fork

Tails changes have

1. -u option
2. safe_load
3. -interface

the change about merging is not present

replace https://github.com/Whonix/control-port-filter-python/blob/master/usr/lib/tor-controlport-filter with above

Here it is -
Replace existing version with TAILs version of tor-controlport-filter · joysn/control-port-filter-python@3da7a49 · GitHub

invent /usr/lib/tor-controlport-filter-merger

Here it is
control-port-filter-python/usr/lib/tor-controlport-filter-merger at master · joysn/control-port-filter-python · GitHub

1. This will create /etc/tor-controlport-filter.d/30_autogenerated.yml
2. This will add the following three lines at the top

## This file is part of Whonix.
## Manual edits to this file will be lost! 
## This file has been auto generated by...

Let me know if this is ok, or something more/modification is required.
3) There are couple of merges [typo in the path where this file to be created was fixed in subsequent merge]

---

Patrick

2017-01-31 18:35:28 UTC

So far so good.

filter_files_set1 = glob.glob(‘/etc/tor-controlport-filter.d/*.yml’)

filter_files_set2 = glob.glob(‘/usr/local/etc/tor-controlport-filter.d/*.yml’)

I think this needs to be tor-controlport-filter-merger.d?

---

Patrick

2017-01-31 21:24:13 UTC

(Manually overwrote this with tor-controlport-filter-merger.d in my tests.)

In /lib/systemd/system/tor-controlport-filter.service.d/30_cpfpy.conf please add above ExecStart=:

ExecStartPre=/usr/lib/tor-controlport-filter-merger

This needs some debugging. Fails for a unknown reason.

sudo service  tor-controlport-filter restart

Job for tor-controlport-filter.service failed because the control process exited with error code.
See “systemctl status tor-controlport-filter.service” and “journalctl -xe” for details.

sudo service tor-controlport-filter status

● tor-controlport-filter.service - Tor control port filter proxy
Loaded: loaded (/lib/systemd/system/tor-controlport-filter.service; enabled; vendor preset: enabled)
Drop-In: /lib/systemd/system/tor-controlport-filter.service.d
└─30_cpfpy.conf, 30_whonix_cpfpy.conf
Active: failed (Result: exit-code) since Tue 2017-01-31 21:10:56 UTC; 944ms ago
Docs: Tails - Design: specification and implementation
Process: 10297 ExecStartPre=/usr/lib/tor-controlport-filter-merger (code=exited, status=1/FAILURE)
Main PID: 9346 (code=killed, signal=TERM)

Jan 31 21:10:56 host systemd[1]: tor-controlport-filter.service: Control process exited, code=exited status=1
Jan 31 21:10:56 host systemd[1]: Failed to start Tor control port filter proxy.
Jan 31 21:10:56 host systemd[1]: tor-controlport-filter.service: Unit entered failed state.
Jan 31 21:10:56 host systemd[1]: tor-controlport-filter.service: Failed with result ‘exit-code’.
Jan 31 21:10:56 host systemd[1]: tor-controlport-filter.service: Service hold-off time over, scheduling restart.
Jan 31 21:10:56 host systemd[1]: Stopped Tor control port filter proxy.
Jan 31 21:10:56 host systemd[1]: tor-controlport-filter.service: Start request repeated too quickly.
Jan 31 21:10:56 host systemd[1]: Failed to start Tor control port filter proxy.
Jan 31 21:10:56 host systemd[1]: tor-controlport-filter.service: Unit entered failed state.
Jan 31 21:10:56 host systemd[1]: tor-controlport-filter.service: Failed with result ‘exit-code’.

---

Did run the merger and filter manually… Might have found a bug. Added 40_ricochet.yml and 30_whonix.yml. The resulting 30_autogenerated.yml …

cat /etc/tor-controlport-filter.d/30_autogenerated.yml 

## This file is part of Whonix.
## Manual edits to this file will be lost! 
## This file has been auto generated by...


---
- commands:
    ADD_ONION:
    - pattern: NEW:(\S+) Port=9878,\S+:(\S+)
      replacement: NEW:{} Port=9878,{client-address}:{}
    - pattern: (\S+):(\S+) Port=9878,\S+:(\S+)
      replacement: '{}:{} Port=9878,{client-address}:{}'
    DEL_ONION:
    - .+
    GETCONF:                                                                                                                                                  
    - DisableNetwork                                                                                                                                          
    GETINFO:                                                                                                                                                  
    - pattern: status/circuit-established status/bootstrap-phase net/listeners/socks                                                                          
      response:                                                                                                                                               
      - pattern: 250-status/bootstrap-phase=*                                                                                                                 
        replacement: 250-status/bootstrap-phase=NOTICE BOOTSTRAP PROGRESS=100 TAG=done
          SUMMARY="Done"
      - pattern: 250-net/listeners/socks=".*"
        replacement: 250-net/listeners/socks="127.0.0.1:9150"
    - status/circuit-established
    - version
    - pattern: net/listeners/socks
      response:
      - pattern: 250-net/listeners/socks=".*"
        replacement: 250-net/listeners/socks="127.0.0.1:9150"
    SIGNAL:
    - NEWNYM
  confs:
    __owningcontrollerprocess: null
  events:
    CONF_CHANGED:
      suppress: true
    SIGNAL:
      suppress: true
    STATUS_CLIENT:
      suppress: true
  match-exe-paths: '*'
  match-hosts:
  - '*'
  match-users: '*'
  name: 'merged_filter_files:  40_ricochet.yml 30_whonix.yml'

The filter does not work with that.

sudo -u tor-controlport-filter /usr/lib/tor-controlport-filter --debug --listen-interface eth1
IP address for interface eth1 : 10.137.11.1
Tor control port filter started, listening on 10.137.11.1:9051





10.137.11.80:42806 (filter: None) connected: loaded filter: None
Final rules:
commands: {}
events: {}
restrict-stream-events: false

---

joysn1980

2017-02-01 04:19:05 UTC

I think this needs to be tor-controlport-filter-merger.d?

Fix path for picking up the yml files · joysn/control-port-filter-python@70162e5 · GitHub

---

joysn1980

2017-02-01 04:25:44 UTC

The filter does not work with that.

This is what I tried
/etc/tor-controlport-filter.d/
has only one file - 30_autogenerated.yml [manually put it there]
and then

sudo service tor-controlport-filter restart
sudo journalctl -f -u tor-controlport-filter

Everything worked fine. Though I did not put the new tails version of tor-controlport-filter. Did you replace that and then you saw the error/bug?

What I am trying to find out is -
Is the issue with new tor-controlport-filter?
OR
Is the issue is with the merged config file - 30_autogenerated.yml

---

joysn1980

2017-02-01 10:34:56 UTC

OK, I got this one.

The issue is with the TAILs version of the tor-controlport-filter which we have originally merged some time back.

/etc/tor-controlport-filter.d/30_autogenerated.yml
and
with tor-controlport-filter
as in
https://github.com/Whonix/control-port-filter-python/blob/e504cabcdaf159f821f38edd4267112c71a190d7/usr/lib/tor-controlport-filter

[this is the last version before taking up the original tails version].
This works fine with this new auto merged config 30_automerged.yml file[generated from tor-controlport-filter-merger]

I fixed the issue with tails version
Matchers incorrectly spelled so the config file was not read properly · joysn/control-port-filter-python@6f488c1 · GitHub

The matchers were spelled incorrectly

-                ('exe-paths', client_exe_path),
 -                ('users',     client_user),
-                ('hosts', client_host),

Should be

 +                ('match-exe-paths', client_exe_path),
 +                ('match-users',     client_user),
 +                ('match-hosts', client_host),

So with this two fix, all should work fine.

---

Patrick

2017-02-02 18:12:25 UTC

Forwarded your patch Matchers incorrectly spelled so the config file was not read properly · joysn/control-port-filter-python@6f488c1 · GitHub to tails-dev mailing list for inclusion at upstream.

Merged.

Added a few minor commits on top.

• https://github.com/Whonix/control-port-filter-python/commit/a615611fbbbb3ebf3b6641a187f109d22da3fd51

• https://github.com/Whonix/control-port-filter-python/commit/5bf372be2708b1b86eff20e861607104597fbb80

• https://github.com/Whonix/control-port-filter-python/commit/6c78583df9350f47f4ee84531311ecdb5111072d

• https://github.com/Whonix/control-port-filter-python/commit/d05d1015c460812ae0faa1a699f95f248817d4c7

• ExecStartPre=/usr/lib/tor-controlport-filter-merger · Whonix/whonix-gw-network-conf@ac45946 · GitHub

---

Patrick

2017-02-02 22:30:38 UTC

https://github.com/Whonix/control-port-filter-python/commit/cf0c7afa93de78362236c5dc6539d74ccd1dec56

(That diff on github does not look great, however that diff looks quite sane in a better diff viewer.)

Also fully tested. Works great!

Do my changes look sane to you? If so, this ticket can be considered resolved.

---

Patrick

2017-02-02 22:42:49 UTC

https://github.com/Whonix/control-port-filter-python/commit/6fe2d272462ff4bdf7f33fea0d2db551cb232bfe

---

joysn1980

2017-02-03 03:19:16 UTC

Perfect Patrick. Thanks.

---

Patrick

2017-02-03 14:22:06 UTC

One more thing to do here.
s/match-//g
anonym replied.

• https://mailman.boum.org/pipermail/tails-dev/2017-February/011222.html
• https://mailman.boum.org/pipermail/tails-dev/2017-February/011224.html

Do you think you could git revert 6f488c14980e8b5c58a42374649c4d5725c8296e (so we share the same code with Tails) as well as remove match- prefixes in the merger code and configs?

---

Patrick

2017-02-03 14:21:56 UTC

! In T617#11999, @Patrick wrote:
Do you think you could git revert cf0c7afa93de78362236c5dc6539d74ccd1dec56 (so we share the same code with Tails) as well as remove match- prefixes in the merger code and configs?

My mistake. I meant git revert 6f488c14980e8b5c58a42374649c4d5725c8296e

---

joysn1980

2017-02-03 14:37:21 UTC

Fixed in
Remove 'match" from the config files · joysn/control-port-filter-python@f430cf3 · GitHub

Reverting the changes to tor-controlport-filter
Revert "Matchers incorrectly spelled so the config file was not read … · joysn/control-port-filter-python@d99eb8c · GitHub

---

Patrick

2017-02-03 19:45:00 UTC

Merged. Wondering about one thing…

   # Replace matched-exe-paths, matched-hosts and matched-users
   for i in range(len(merged_filter)):
      if 'match-exe-paths' in merged_filter[i].keys():
         merged_filter[i]['exe-paths'] = merged_filter[i].pop('match-exe-paths')
      if 'match-hosts' in merged_filter[i].keys():
         merged_filter[i]['hosts'] = merged_filter[i].pop('match-hosts')
      if 'match-users' in merged_filter[i].keys():
         merged_filter[i]['users'] = merged_filter[i].pop('match-users')

Why is that needed? Couldn’t we avoid that code block? If we just s/match-//g in all our configs, could we avoid that code block?

---

joysn1980

2017-02-04 03:00:38 UTC

This is possible when we are reading the file as “string” of characters [normally how we read a file]
But in this case we are reading our file as yaml which is a “dictionary” and each of the “match” we have to modified are the keys of the dictionary.

This is my understanding.

Again, we do not even need this code, if we modify our config file itself and replace these “keys” to the new expected “keys”

---

Patrick

2017-02-04 12:45:27 UTC

joysn1980 (Joy):

Again, we do not even need this code, if we modify our config file
itself and replace these “keys” to the new expected “keys”

Yes. Why not modify our config file and our merger example profiles?
Seems much better to me to use the same config file keywords as upstream
(Tails).

Basically s/match-//g for
etc/tor-controlport-filter-merger.d/30_whonix.yml and for
usr/share/tor-controlport-filter-merger/examples/.

Done that.

I have created a new git branch:
https://github.com/Whonix/control-port-filter-python/tree/keywords-as-tails

Just one commit:
https://github.com/Whonix/control-port-filter-python/commit/7b28225fbe6300631c89047a645a649fbed19208

Seems to work. I might not be fully understanding this.

Does that look good? If so, I am going to merge it into master.

---

joysn1980

2017-02-04 15:46:15 UTC

Sure Patrick. Everything looks good (as per my understanding)

---

Patrick

2017-02-04 17:00:46 UTC

Merged.