tool to onionize all APT sources

This can be later s/sed/str_replace/

Haven’t tested yet, just a sketch for debian.list and derivative.list.

This still feels to manual, better think of a way to make this more easy to read, or a for loop.

This are the domains:

"deb.debian.org/debian "="2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian "
"deb.debian.org/debian-security "="5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/debian-security "
"fasttrack.debian.net/debian "=""

"deb.whonix.org"="deb.dds6qkxpwdeubwucdiaord2xgbbeyds25rbsgr73tbfpqpt4a6vjwsyd.onion"

protocolos are:

https:// ## plain
tor+https:// ## tor+plain
tor+http:// ## onion
## onion
sed -i'' "s| https://deb.debian.org/debian | tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| tor+https://deb.debian.org/debian | tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| https://deb.debian.org/debian-security | tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/debian-security |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| tor+https://deb.debian.org/debian-security | tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/debian-security |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| https://deb.whonix.org| tor+http://deb.dds6qkxpwdeubwucdiaord2xgbbeyds25rbsgr73tbfpqpt4a6vjwsyd.onion|" /etc/apt/sources.list.d/derivative.list
sed -i'' "s| tor+https://deb.whonix.org| tor+http://deb.dds6qkxpwdeubwucdiaord2xgbbeyds25rbsgr73tbfpqpt4a6vjwsyd.onion|" /etc/apt/sources.list.d/derivative.list

## tor+plain
sed -i'' "s| https://deb.debian.org/debian | tor+https://deb.debian.org/debian |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian | tor+https://deb.debian.org/debian |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| https://deb.debian.org/debian-security | tor+https://deb.debian.org/debian-security |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/debian-security | tor+https://deb.debian.org/debian-security |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| https://deb.whonix.org| tor+https://deb.whonix.org|" /etc/apt/sources.list.d/derivative.list
sed -i'' "s| tor+http://deb.dds6qkxpwdeubwucdiaord2xgbbeyds25rbsgr73tbfpqpt4a6vjwsyd.onion| tor+https://deb.whonix.org|" /etc/apt/sources.list.d/derivative.list

## plain
sed -i'' "s| tor+https://deb.debian.org/debian | https://deb.debian.org/debian |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian | https://deb.debian.org/debian |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| tor+https://deb.debian.org/debian-security | https://deb.debian.org/debian-security |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/debian-security | https://deb.debian.org/debian-security |" /etc/apt/sources.list.d/debian.list
sed -i'' "s| tor+https://deb.whonix.org| https://deb.whonix.org|" /etc/apt/sources.list.d/derivative.list
sed -i'' "s| tor+http://deb.dds6qkxpwdeubwucdiaord2xgbbeyds25rbsgr73tbfpqpt4a6vjwsyd.onion| https://deb.whonix.org|" /etc/apt/sources.list.d/derivative.list

Such an implementation would suffer from dpkg interactive conflict resolution dialog. Non-ideal but not a blocker. dpkg interactive conflict resolution dialog would probably mean “advanced users only”.


Also quiet hacky still… Idea maybe it would lead somewhere…
If some config is set (by creating a file or setting in /etc) the repository onionizer (working name until a better is invented) a dpkg trigger could redo any onionizing if file /etc/sources.list or any file in folder /etc/sources.list.d is modified by APT?

dpkg triggers are only working for files which are managed by any packages.

Managed by APT (because shipped by a package):

  • /etc/apt/debian.list
  • /etc/apt/qubes-r4.list

Not managed by APT (created by script):

  • /etc/apt/sources.list - maybe OK since empty by default in Kicksecure / Whonix
  • /etc/apt/sources.list.d/derivative.list - maybe OK because repository-dist could have native --tor-onion support

An even more hacky way would be to use inotifywait (some examples in Whonix source code). Whenever an APT sources list file would be modified (by package, script or user), any supported (known by repository onionizer) clearnet domain names would be changed back to clearnet.


In any case, for transparency and usability, if modifications are done by repository onionizer, then repository onionizer should add a comment (non-duplicating, not adding same comment over and over again)) to the file stating that it was done, link to documentation so the user can figure out what happened and how to disable this feature?

The repository onionizer might fit into the already existing repository-dist package?

I don’t know real world examples yet but it is conceivable to get onion combined with https repositories in the future. I.e. tor+https://deb.onion.onion. If you don’t mind, while scripting, could you please consider this future possibility?

That style to use sed looks easy to read, nice and clean to me.

1 Like

Also quiet hacky still… Idea maybe it would lead somewhere…
If some config is set (by creating a file or setting in /etc) the repository onionizer (working name until a better is invented) a dpkg trigger could redo any onionizing if file /etc/sources.list or any file in folder /etc/sources.list.d is modified by APT?

That is the objetive, manage the list. There will be no clearnet and onion at the same time on a list, there will be only one possible line and the script/package will manage the protocols, domains etc.

In any case, for transparency and usability, if modifications are done by repository onionizer, then repository onionizer should add a comment (non-duplicating, not adding same comment over and over again)) to the file stating that it was done, link to documentation so the user can figure out what happened and how to disable this feature?

Yes.

The repository onionizer might fit into the already existing repository-dist package?

Yes

I don’t know real world examples yet but it is conceivable to get onion combined with https repositories in the future. I.e. tor+https://deb.onion.onion. If you don’t mind, while scripting, could you please consider this future possibility?

Yes.

That style to use sed looks easy to read, nice and clean to me.

Great.

1 Like

Coding this now.
It would be easier to replace the whole file than using sed or str_replace. Or do you see any use case to replace per pattern if it is just gonna edit derivative.list anyway?

Preserve codename and stable/testers etc, I see.

1 Like

This code only modifies/replaces, it does not insert repo is no pattern is present.

getopts

           -t | --transport)
                transport="$2"
                if [ -z "${transport}" ]; then
                    echo "ERROR: transport must not be empty. Choose between:
 plain-tls
 plain-tls-tor
 onion
 onion-tls"
                    exit 1
                fi
                shift 2
                ;;

modifier

derivative_repo_onion="dds6qkxpwdeubwucdiaord2xgbbeyds25rbsgr73tbfpqpt4a6vjwsyd.onion"
derivative_repo_plain="whonix.org"

plain_tls="https://deb.${derivative_repo_plain}"
plain_tls_tor="tor+https://deb.${derivative_repo_plain}"
onion="tor+http://deb.${derivative_repo_onion}"
onion_tls="tor+https://deb.${derivative_repo_onion}"

transport_lines="
${plain_tls}
${plain_tls_tor}
${onion}
${onion_tls}
"

case "$transport" in
    plain-tls) repo="${plain_tls}";;
    plain-tls-tor) repo="${plain_tls_tor}";;
    onion) repo="${onion}";;
    onion-tls) repo="${onion_tls}";;
    *) echo "ERROR: Invalid transport type!"; exit 1;;
esac


for line in ${transport_lines}; do
    sed -i'' "s| ${line}| ${repo}|" "${sources_list_file_derivative}"
done
1 Like

Soon there will be separate repositories. Whonix and Kicksecure.

deb.kicksecure.com
deb.w5j6stm77zs6652pgsij4awcjeel3eco7kvipheu6mtr623eyyehj4yd.onion

Kicksecure will use Kicksecure repository only.

Whonix will use Kicksecure and Whonix repository.

plan to use repository-dist in general (whonix and kicksecure)?

Seems possible. If you see the code, only 2 domains lines have to be altered, or even, --domain [whonix|kicksecure|debian|qubes] in the future, and the domains clear and onion will be chosen by this option. But lets start with kicksecure first to avoid dpkg conflicts as you said.

Yes.

repository-dist will manage both repository lines.

1 Like
1 Like

Thank you for working on this!

Pull request won’t work as is because:

In other words there will be two deb [...] lines in the derivative.list file for Whonix.
(And only 1 for Kicksecure.)

Autodetection of Whonix vs Kicksecure could be implemented by checking existence of these marker files:

  • /usr/share/kicksecure/marker
  • /usr/share/whonix/marker

Could you fix this please?

So whonix option to be enabled on kicksecure will be disabled?

The way I did on local branch now is:

  • if --url is not provided then
    • if whonix is detected, use it
    • if kicksecure is detected, use it
    • if none are detected, error out and ask to provide url.

but the above will be supersed if on whonix for example, you run --url kicksecure to manage kicksecure repo in whonix.
Should this be disabled on kicksecure?

1 Like

Whonix enable:

  • Kicksecure, and
  • Whonix

Kicksecure enable:

  • Kicksecure only

The design is:
Whonix is based on Kicksecure.
Not historically but that’s the technical design end-goal how to organize things.

1 Like

In case of Whonix, both repository lines, Whonix and Kicksecure need to be enabled.

1 Like

Sorry, maybe I didn’t express this very clearly.

The plan is…

Kicksecure will use:

deb [signed-by=/usr/share/keyrings/derivative.asc] tor+https://deb.kicksecure.com bullseye main contrib non-free

Whonix will use:

deb [signed-by=/usr/share/keyrings/derivative.asc] tor+https://deb.kicksecure.com bullseye main contrib non-free
deb [signed-by=/usr/share/keyrings/derivative.asc] tor+https://deb.whonix.org bullseye main contrib non-free

on the same file?

I am using two files to test.

On whonix, running --transport onion will convert sources to onion on /etc/apt/sources.list.d/{derivative,kickescure}.list

Yes. I guess that is easiest. derivative.list

1 Like

link to github updated: