tool to onionize all APT sources

Patrick · February 20, 2022, 2:06pm

Patrick · February 20, 2022, 2:07pm

nyxnor commented 3 hours ago

Only problem I see is with fastrack not having that and in the meantime a workaround could be done.
Later would be great for it to also take care of debian.list and qubes-r4.list.

nyxnor · February 20, 2022, 2:32pm

Please remember the fasttrack guy about that issue, seems forgot for 5 months now.

Patrick · February 20, 2022, 2:42pm

I was wondering about this before but I didn’t thought I’d get to ever develop a solution for it, so no ticket created. Maybe you’d like to contribute?

dpkg -S /etc/apt/sources.list.d/*

anon-apt-sources-list: /etc/apt/sources.list.d/debian.list
dpkg-query: no path found matching pattern /etc/apt/sources.list.d/derivative.list
qubes-core-agent: /etc/apt/sources.list.d/qubes-r4.list

As for GitHub - Kicksecure/repository-dist an option --onion seems very doable to do. Probably best to have mandatory switches? One of the following three:

--clearnet (TLS only. For Kicksecure. Opt-in.)
--torified (I don’t really like that name --torified since --onion is also torified. But other names such as --torplus also seems misleading. Sounds like a fork of Tor. And --torplustls would be correct but a bit long and weird to type.
--onion

/etc/apt/sources.list.d/debian.list is a “managed” file. Managed for the benefit of the user. Package anon-apt-sources-list is updated whenever required, specifically on release upgrade to the next major version of Debian.

If that script was modified by the user or by a script (which looks to APT as if modified by the user) this would cause an interactive dpkg conflict resolution dialog which I can tell from support experience is very confusing for users, generating lots of support requests.

Configuration file `/etc/apt/sources.list.d/debian.list’
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ? Your options are:
Y or I : install the package maintainer’s version
N or O : keep your currently-installed version
D : show the differences between the versions
Z : background this process to examine the situation
The default action is to keep your current version.
*** interfaces (Y/I/N/O/D/Z) [default=N] ? N

Therefore maybe a new package anon-onion-sources-list would be better? The APT onionizer tool could then recommend to:

sudo apt install anon-onion-sources-list and
sudo apt purge anon-apt-sources-list.

Maybe even systemcheck --verbose would include a hint but that can be considered at the end.

/etc/apt/sources.list.d/qubes-r4.list is the worst offender currently. Not even using tor+. Not an anonymity risk but error handling would be better. File is managed by qubes-core-agent. Therefore also if that file was edited there would also be an interactive dpkg conflict resolution issue. Furthermore, this file is managed by Qubes, not Whonix. That file must not be installed in Non-Qubes-Whonix.

What could be done?

Request that file to be moved to a separate Qubes package, then create an alternative package with the onion version?
config-package-dev hide /etc/apt/sources.list.d/qubes-r4.list and create an alternative package with the onion version? Ideally Qubes would maintain that file/package after initial contribution to Qubes. Otherwise could become very messy with maintenance lapses.

We can always create an issue on Issues · QubesOS/qubes-issues · GitHub to discuss this with Qubes upstream.

Could you send a friendly ping please?
I cannot open that link currently.

Not sure what could be done. fasttrack might not be too important yet. Non-VirtualBox users could probably disable it. APT onionizer could mention it (always) and suggest to disable it for Non-VirtualBox users (both VM and host).

How about user custom added repositories? I was wondering if an APT onionizer would have a set of clearnet repositories mapped to onion repositories and then with one command line option / wizard could upgrade/downgrade from one version to another.

What about extrepo?

Would this feature fit there? Could be contributed to upstream?

Yeah. Quite complex, right? That’s why I never before wrote this up.

nyxnor · February 20, 2022, 3:42pm

Maybe you’d like to contribute?

Yes.

--clearnet (TLS only. For Kicksecure. Opt-in.)

--torified (I don’t really like that name --torified since --onion is also torified. But other names such as --torplus also seems misleading. Sounds like a fork of Tor. And --torplustls would be correct but a bit long and weird to type.

--onion

--clearnet
--tor-clearnet
--tor-onion

qubes-r4.list

Lets leave this one alone from now, first need to get derivative.list working.

Could you send a friendly ping please?
I cannot open that link currently.

I tried before, I couldn’t create an account. Can not pass captcha even with standard security on TBB.

Not sure what could be done. fasttrack might not be too important yet. Non-VirtualBox users could probably disable it.

K, good to know.

How about user custom added repositories? I was wondering if an APT onionizer would have a set of clearnet repositories mapped to onion repositories and then with one command line option / wizard could upgrade/downgrade from one version to another.

That is the best idea and exactly what I was thinking, mapping addresses to be switched.

How about user custom added repositories?

A mapping database they could insert their onion alternatives, with a comment for them to contribute to this new whonix package so more users would know. Not that non related repositories will be included, but if they even have an onion, that is scarce.

extrepo

testing it, but couldn’t make it work properly. I would be more comfortable with sed.

Patrick · February 20, 2022, 3:47pm

Sounds good!

I am not a big fan of sed for such simple string replacements. (string replacements are simple but the overall tool I wouldn’t call simple. Not meant to say this is overall an easy project.) syntax looks more difficult that possible. str_replace is great imo but it’s a dependency on helpers-scripts and I don’t know any other tools that are everywher pre-installed.

Not a big deal either way.

nyxnor · February 21, 2022, 8:45pm

Just to see later if possible to automate Qubes repos to onion http://www.dds6qkxpwdeubwucdiaord2xgbbeyds25rbsgr73tbfpqpt4a6vjwsyd.onion/wiki/Onionizing_Repositories

nyxnor · February 21, 2022, 9:11pm

This can be later s/sed/str_replace/

Haven’t tested yet, just a sketch for debian.list and derivative.list.

This still feels to manual, better think of a way to make this more easy to read, or a for loop.

This are the domains:

"deb.debian.org/debian "="2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian "
"deb.debian.org/debian-security "="5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/debian-security "
"fasttrack.debian.net/debian "=""

"deb.whonix.org"="deb.dds6qkxpwdeubwucdiaord2xgbbeyds25rbsgr73tbfpqpt4a6vjwsyd.onion"

protocolos are:

https:// ## plain
tor+https:// ## tor+plain
tor+http:// ## onion

## onion
sed -i'' "s| https://deb.debian.org/debian | tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| tor+https://deb.debian.org/debian | tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| https://deb.debian.org/debian-security | tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/debian-security |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| tor+https://deb.debian.org/debian-security | tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/debian-security |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| https://deb.whonix.org| tor+http://deb.dds6qkxpwdeubwucdiaord2xgbbeyds25rbsgr73tbfpqpt4a6vjwsyd.onion|" /etc/apt/sources.list.d/derivative.list
sed -i'' "s| tor+https://deb.whonix.org| tor+http://deb.dds6qkxpwdeubwucdiaord2xgbbeyds25rbsgr73tbfpqpt4a6vjwsyd.onion|" /etc/apt/sources.list.d/derivative.list

## tor+plain
sed -i'' "s| https://deb.debian.org/debian | tor+https://deb.debian.org/debian |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian | tor+https://deb.debian.org/debian |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| https://deb.debian.org/debian-security | tor+https://deb.debian.org/debian-security |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/debian-security | tor+https://deb.debian.org/debian-security |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| https://deb.whonix.org| tor+https://deb.whonix.org|" /etc/apt/sources.list.d/derivative.list
sed -i'' "s| tor+http://deb.dds6qkxpwdeubwucdiaord2xgbbeyds25rbsgr73tbfpqpt4a6vjwsyd.onion| tor+https://deb.whonix.org|" /etc/apt/sources.list.d/derivative.list

## plain
sed -i'' "s| tor+https://deb.debian.org/debian | https://deb.debian.org/debian |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian | https://deb.debian.org/debian |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| tor+https://deb.debian.org/debian-security | https://deb.debian.org/debian-security |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/debian-security | https://deb.debian.org/debian-security |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| tor+https://deb.whonix.org| https://deb.whonix.org|" /etc/apt/sources.list.d/derivative.list
sed -i'' "s| tor+http://deb.dds6qkxpwdeubwucdiaord2xgbbeyds25rbsgr73tbfpqpt4a6vjwsyd.onion| https://deb.whonix.org|" /etc/apt/sources.list.d/derivative.list

Patrick · February 22, 2022, 10:34am

Such an implementation would suffer from dpkg interactive conflict resolution dialog. Non-ideal but not a blocker. dpkg interactive conflict resolution dialog would probably mean “advanced users only”.

Also quiet hacky still… Idea maybe it would lead somewhere…
If some config is set (by creating a file or setting in /etc) the repository onionizer (working name until a better is invented) a dpkg trigger could redo any onionizing if file /etc/sources.list or any file in folder /etc/sources.list.d is modified by APT?

dpkg triggers are only working for files which are managed by any packages.

Managed by APT (because shipped by a package):

/etc/apt/debian.list
/etc/apt/qubes-r4.list

Not managed by APT (created by script):

/etc/apt/sources.list - maybe OK since empty by default in Kicksecure / Whonix
/etc/apt/sources.list.d/derivative.list - maybe OK because repository-dist could have native --tor-onion support

An even more hacky way would be to use inotifywait (some examples in Whonix source code). Whenever an APT sources list file would be modified (by package, script or user), any supported (known by repository onionizer) clearnet domain names would be changed back to clearnet.

In any case, for transparency and usability, if modifications are done by repository onionizer, then repository onionizer should add a comment (non-duplicating, not adding same comment over and over again)) to the file stating that it was done, link to documentation so the user can figure out what happened and how to disable this feature?

The repository onionizer might fit into the already existing repository-dist package?

I don’t know real world examples yet but it is conceivable to get onion combined with https repositories in the future. I.e. tor+https://deb.onion.onion. If you don’t mind, while scripting, could you please consider this future possibility?

That style to use sed looks easy to read, nice and clean to me.

nyxnor · February 22, 2022, 6:50pm

Also quiet hacky still… Idea maybe it would lead somewhere…
If some config is set (by creating a file or setting in /etc) the repository onionizer (working name until a better is invented) a dpkg trigger could redo any onionizing if file /etc/sources.list or any file in folder /etc/sources.list.d is modified by APT?

That is the objetive, manage the list. There will be no clearnet and onion at the same time on a list, there will be only one possible line and the script/package will manage the protocols, domains etc.

In any case, for transparency and usability, if modifications are done by repository onionizer, then repository onionizer should add a comment (non-duplicating, not adding same comment over and over again)) to the file stating that it was done, link to documentation so the user can figure out what happened and how to disable this feature?

Yes.

The repository onionizer might fit into the already existing repository-dist package?

Yes

I don’t know real world examples yet but it is conceivable to get onion combined with https repositories in the future. I.e. tor+https://deb.onion.onion. If you don’t mind, while scripting, could you please consider this future possibility?

Yes.

That style to use sed looks easy to read, nice and clean to me.

Great.

nyxnor · March 24, 2022, 9:17pm

Coding this now.
It would be easier to replace the whole file than using sed or str_replace. Or do you see any use case to replace per pattern if it is just gonna edit derivative.list anyway?

Preserve codename and stable/testers etc, I see.

nyxnor · March 24, 2022, 9:57pm

This code only modifies/replaces, it does not insert repo is no pattern is present.

getopts

           -t | --transport)
                transport="$2"
                if [ -z "${transport}" ]; then
                    echo "ERROR: transport must not be empty. Choose between:
 plain-tls
 plain-tls-tor
 onion
 onion-tls"
                    exit 1
                fi
                shift 2
                ;;

modifier

derivative_repo_onion="dds6qkxpwdeubwucdiaord2xgbbeyds25rbsgr73tbfpqpt4a6vjwsyd.onion"
derivative_repo_plain="whonix.org"

plain_tls="https://deb.${derivative_repo_plain}"
plain_tls_tor="tor+https://deb.${derivative_repo_plain}"
onion="tor+http://deb.${derivative_repo_onion}"
onion_tls="tor+https://deb.${derivative_repo_onion}"

transport_lines="
${plain_tls}
${plain_tls_tor}
${onion}
${onion_tls}
"

case "$transport" in
    plain-tls) repo="${plain_tls}";;
    plain-tls-tor) repo="${plain_tls_tor}";;
    onion) repo="${onion}";;
    onion-tls) repo="${onion_tls}";;
    *) echo "ERROR: Invalid transport type!"; exit 1;;
esac


for line in ${transport_lines}; do
    sed -i'' "s| ${line}| ${repo}|" "${sources_list_file_derivative}"
done

Patrick · March 25, 2022, 10:46am

Soon there will be separate repositories. Whonix and Kicksecure.

deb.kicksecure.com
deb.w5j6stm77zs6652pgsij4awcjeel3eco7kvipheu6mtr623eyyehj4yd.onion

Kicksecure will use Kicksecure repository only.

Whonix will use Kicksecure and Whonix repository.

nyxnor · March 25, 2022, 12:04pm

plan to use repository-dist in general (whonix and kicksecure)?

Seems possible. If you see the code, only 2 domains lines have to be altered, or even, --domain [whonix|kicksecure|debian|qubes] in the future, and the domains clear and onion will be chosen by this option. But lets start with kicksecure first to avoid dpkg conflicts as you said.

Patrick · March 25, 2022, 12:27pm

Yes.

repository-dist will manage both repository lines.

nyxnor · March 26, 2022, 7:47pm

Patrick · March 28, 2022, 6:29am

Thank you for working on this!

Pull request won’t work as is because:

In other words there will be two deb [...] lines in the derivative.list file for Whonix.
(And only 1 for Kicksecure.)

Autodetection of Whonix vs Kicksecure could be implemented by checking existence of these marker files:

/usr/share/kicksecure/marker
/usr/share/whonix/marker

Could you fix this please?

nyxnor · March 28, 2022, 9:11am

So whonix option to be enabled on kicksecure will be disabled?

The way I did on local branch now is:

if --url is not provided then
- if whonix is detected, use it
- if kicksecure is detected, use it
- if none are detected, error out and ask to provide url.

but the above will be supersed if on whonix for example, you run --url kicksecure to manage kicksecure repo in whonix.
Should this be disabled on kicksecure?

Patrick · March 28, 2022, 11:14am

Whonix enable:

Kicksecure, and
Whonix

Kicksecure enable:

Kicksecure only

The design is:
Whonix is based on Kicksecure.
Not historically but that’s the technical design end-goal how to organize things.

nyxnor · March 28, 2022, 11:40am

tool to onionize all APT sources

nyxnor commented 3 hours ago