maintain and upload a known good version of tor from deb.torproject.org in Whonix repository

[phabricator-migrator]

Information

ID: 472
PHID: PHID-TASK-b3zcjxa4t4nidw5ftvae
Author: Patrick
Status at Migration Time: resolved
Priority at Migration Time: Normal

Description

Installing the tor from deb.torproject.org is great because it gives us more recent versions of Tor. Which has versions closer to the one included in TBB.

At one point this was even required:
https://blog.torproject.org/blog/how-to-handle-millions-new-tor-clients

The problem with deb.torproject.org is that it is a too fast moving target, i.e. a new release of Tor might break Whonix for all users. Could happen when they change something related to the systemd service or so.

The responsible way would be to download Tor from deb.torproject.org and to upload it to the Whonix repository.

I could not think of yet how this gets together with builds from source code. (You cannot run apt-get update from postinst scripts.)

TODO:

• automate download (verification) and adding to Whonix local / remote apt repository (done)
• disable TPO sources.list on existing installations using #whonix-legacy
• remove TPO signing key on existing installations using #whonix-legacy (happens during deb.torproject.org-keyring package removal Debian maintainer prerm script)
• actually upload TPO packages to Whonix repository
• more comments in /etc/apt/sources.list.d/torproject.list #anon-shared-build-apt-sources-tpo to more easily add experimental versions of Tor
• usability feature for testers
• output torproject repo in use:
• cat /etc/apt/sources.list.d/torproject.list | grep -v '#' | grep deb
• output of Tor version

Comments

---

HulaHoop

2016-02-22 20:43:02 UTC

I prefer we not touch binary packages unless there are solid security guarantees - similar to verification of binaries hosted on a mirror.

How complicated is it for using apt to check a package against some signing key in the keychain?

---

Patrick

2016-02-23 19:47:19 UTC

HulaHoop (HulaHoop):

I prefer we not touch binary packages unless there are solid security guarantees - similar to verification of binaries hosted on a mirror.

I don’t think there is such verification at the moment let alone anyone
on the internet keeping on auditing.

It won’t change a lot. Rather than building a Whonix source package from
source and then comparing it with the binary one from the Whonix
repository, an auditor would download the binary tor package from
deb.torproject.org and then compare it with the one in the Whonix
repository which should have matching hashes.

How complicated is it for using apt to check a package against some signing key in the keychain?

Is this in scope of this ticket? If you are asking if someone who
downloaded the tor package from Whonix repository could verify it
against the deb.torproject.org signing key, no that is not supported by
apt-get. This is (also) because in apt repositories, the repository
metadata is signed. Packages are verified against that metadata. The
metadata is verified against all available apt keys. Individual packages
are not signed. Debian unfortunately does not have an end-to-end
authentication mechanism for binary packages signed by the package
maintainer up the user who downloaded them through apt-get. This is
(also) because Debian packages are usually build on Debian build
servers. [off-topic, FYI: Some of these build machines are just located
in ‘some Debian Developer’s home’. Which is not great for security.
That’s why reproducible builds really matter.]

---

HulaHoop

2016-02-25 16:36:06 UTC

an auditor would download the binary tor package from
deb.torproject.org and then compare it with the one in the Whonix
repository which should have matching hashes.

Will this be done automatically by the build script? I don’t want this to be a manual verification process that depends on the interest of someone doing it sometimes.

If you are asking if someone whodownloaded the tor package from Whonix repository could verify it against the deb.torproject.org signing key, no that is not supported by apt-get.

Yes that’s what I was asking about. Very interesting as I wasn’t familiar with details about apt.

There are some problems I foresee with critical update logistics. At the moment if the Tor package needs an emergency update, the people tracking Debian stable only will get it as its backported and those tracking deb.torproject.org get it as soon as the new “testing” release is uploaded. If we arbitrarily freeze some “testing” Tor release then how is it decided to upgrade? Will you need to read the changelog of every minor release to decide? Is this sustainable?

---

Patrick

2016-02-25 18:07:18 UTC

! In T472#8166, @HulaHoop wrote:
an auditor would download the binary tor package from
deb.torproject.org and then compare it with the one in the Whonix
repository which should have matching hashes.

Will this be done automatically by the build script? I don’t want this to be a manual verification process that depends on the interest of someone doing it sometimes.

There are no verification scripts for any Debians repository at the moment at all. More info:

Verifiable Builds - Whonix

If you are asking if someone whodownloaded the tor package from Whonix repository could verify it against the deb.torproject.org signing key, no that is not supported by apt-get.

Yes that’s what I was asking about. Very interesting as I wasn’t familiar with details about apt.

There are some problems I foresee with critical update logistics. At the moment if the Tor package needs an emergency update, the people tracking Debian stable only will get it as its backported and those tracking deb.torproject.org get it as soon as the new “testing” release is uploaded.

If we arbitrarily freeze some “testing” Tor release then how is it decided to upgrade?

Upgrade every time. Unless the upgrade would break something that would require another Whonix package upgrade to make sure it does not break at the same time. Then upgrading would take longer.

Will you need to read the changelog of every minor release to decide?

Yes, I must do that anyhow.

Is this sustainable?

It has to be.

---

HulaHoop

2016-02-26 01:03:14 UTC

There are no verification scripts for any Debians repository at the moment at all. More info.

There is a semantics mix up. When I say verifiable I mean to check package signatures or hashes not build reproducibility.

Will the Whonix build script be able to download and compare hashes from the two different repos?

EDIT: I think Verifiable → Deterministic makes more sense in the documentation.

It has to be.

OK and I’ll help with keeping track too.

---

Patrick

2016-02-26 10:18:43 UTC

• The build using Whonix build script should not require using Whonix
  binary repository. So there is nothing to compare.

• Automated downloading of the tor package from deb.torproject.org
  certainly must use usual apt-get verification.

I could not think of yet how this gets together with builds from
source code. (You cannot run apt-get update from postinst scripts.)

Conflicts with goal T461 (getting rid of chroot scripts).

---

HulaHoop

2016-02-26 17:23:34 UTC

I’m confused. Source builds will have nothing to with these changes? This only applies for official builds?

I don’t think placing such a critical package in a third party repo is a good idea if we don’t have a reliable and automated way to trace the chain of trust back to the original binaries.

Conflicts with goal T461 (getting rid of chroot scripts).

Blocking this ticket is not really worth the gain. Why not switch to the older version in Debian’s repos if testing is moving too fast?

---

Patrick

2016-02-27 11:08:35 UTC

! In T472#8174, @HulaHoop wrote:
I’m confused. Source builds will have nothing to with these changes? This only applies for official builds?

• The chroot script GitHub - Kicksecure/anon-shared-build-apt-sources-tpo: Adds The Tor Projects's (TPO) APT repository to Derivative Linux Distributions during build. - Useful, because TPO's repository sometimes contains better/more recent versions of Tor, obfsproxy, torsocks, etc. This package is produced independently of, and carries no guarantee from, The Tor Project. should be abolished. (T461) Therefore it is related to builds from source code.
• deb.torproject.org should be removed (this ticket).
• Builds from source code should still download, verify and install tor from deb.torproject.org.
• You cannot run apt-get update from Debian packaging maintainer (postinst …) scripts. (Because apt-get and dpkg is already running, therefore locked.)
  – (Running apt-get update would be required within the process of temporarily adding deb.torproject.org, downloading and verifying tor.)

Why not switch to the older version in Debian’s repos if testing is moving too fast?

That would block lots of other stuff. Such as onionshare. (T448 etc.) And latest Tor security enhancements such as to the guard selection algorithm. Also in case they need to deploy a fix to cope up with another botnet attack on the Tor network, we need to react also.

---

marmarek

2016-02-27 11:35:46 UTC

• Builds from source code should still download, verify and install tor from deb.torproject.org.
• You cannot run apt-get update from Debian packaging maintainer (postinst …) scripts. (Because apt-get and dpkg is already running, therefore locked.)
  • (Running apt-get update would be required within the process of temporarily adding deb.torproject.org, downloading and verifying tor.)

It should be quite easy to download and verify it manually, but there should
be also utilities to do that, without modifying /etc/apt/sources.list*.
Take a look at https://wiki.debian.org/AptMedium for example - basically
it calls apt-get with alternative configuration, pointing at different
directory to download packages without interacting with system database.

---

Patrick

2016-02-27 12:23:06 UTC

How to work around the dpkg lock?

---

marmarek

2016-02-27 12:30:33 UTC

I understand apt-medium uses a separate dpkg database.

---

Patrick

2016-02-27 13:11:34 UTC

So we could install another deb package using dpkg from the postinst
while dpkg is already running and installing a package?

That is the circle I am trying to square here which will likely not work.

---

marmarek

2016-02-27 13:26:05 UTC

No, we do not want to install it there, we want to download it only,
don’t we? Then put it into local repository (same as other packages
build there), or upload to whonix apt repo.

Am I missing something?

---

Patrick

2016-04-05 17:01:05 UTC

It depends on how far Installation from Repository / proverbial "sudo apt-get install whonix" (T461) should go.

---

a)

1. start a (Qubes) Debian (template)
2. add Whonix binary apt repository [*]
3. sudo apt-get install whonix-gateway

---

b)

1 start a (Qubes) Debian (template)
2 have a script that
2a adds Whonix binary apt repository [*]
2b runs sudo apt-get install whonix-gateway
2c downloads and installs the tor package from deb.torproject.org

---

If it is supposed to become as simple as the following 3 steps in a)… Then we cannot download and install the tor package from deb.torproject.org. This is because while the package manager is running, we cannot run dpkg install package, since the package database would be already locked.

Which one do we require/want? a) or b)?

---

[*] Or create a local Whonix apt repository from source code.

---

marmarek

2016-04-05 18:38:34 UTC

If tor package is going to be uploaded to Whonix binary apt repository, it doesn’t matter. Downloading tor package from deb.torproject.org is the part of build process, not installation process.

---

Patrick

2016-04-05 21:55:34 UTC

Update: I am trying to avoid uploading the tor package from deb.torproject.org to the Whonix binary apt repository. After thinking more about, it significantly adds up a maintenance burden. Instead I am considering to implement some usability enhancements.

• a script run from .bashrc thats shows the current Tor version / Tor Project repository once some file is created (touch ~/tester or so)
• more comments in /etc/apt/sources.list.d/torproject.list to more easily allow enabling tor-experimental-0.2.7.x-jessie etc.

These should simplify/remind to keep track of newer Tor versions and see how they work with Whonix before they flow into Tor Project’s stable apt repository.

---

FYI, a hack (will probably not use it)…
install apt packages from deb postinst:

• install apt packages from deb postinst | javier.io
• https://gist.github.com/chilicuil/74fddda2a31bba09b973

---

Here is my work in progress attempt for downloading the tor package during Debian maintainer postinst script. It uses a separate apt database. It works.

anon-gw-anonymizer-config/debian/anon-gw-anonymizer-config.postinst at 5fdf627e7e4286cdab418e8e634df3c07e209133 · adrelanos/anon-gw-anonymizer-config · GitHub

Installation from there is more tricky. The above hack of merging dpkg status file changes should probably be avoided. The least awful hack might be to have a blocking startup script that runs early and installs the package at next reboot.

That would move us closer to “sudo apt-get install whonix-gateway” etc. Getting rid of chroot scripts. (T461) Then one could create a Qubes Debain ProxyVM by using method a) to morph it into a Whonix-Gateway. That would greatly simplify Whonix installation process. Make it more robust, easier/quicker testing overall, easier porting. I hope I don’t have to bend things too much to get there. (We are overwriting files such as /etc/resolv.conf, load firewall rules, etc. I hope none of this is a deal breaker during installation. I don’t think there is any other Debian derivative yet that can be installed using apt-get alone.)

That doesn’t give us a set of packages that could be installed from Qubes installer inside a Debian template to morph it into Whonix. (The purpose is to save space on the installer disc here.) But getting pure T461 done somewhat simplifies the development of that.

---

Patrick

2016-04-06 14:51:21 UTC

I succeeded to implement the download of

• tor
• tor-geoipdb
• deb.torproject.org-keyring

from deb.torproject.org using #anon-gw-anonymizer-config postinst script and installation during next reboot. [*]

• anon-gw-anonymizer-config/debian/anon-gw-anonymizer-config.postinst at T461 · adrelanos/anon-gw-anonymizer-config · GitHub
• https://github.com/adrelanos/anon-gw-anonymizer-config/blob/T461/lib/systemd/system/anon-gw-anonymizer-config.service
• https://github.com/adrelanos/anon-gw-anonymizer-config/blob/T461/usr/lib/anon-gw-anonymizer-config/startup
• https://github.com/adrelanos/anon-gw-anonymizer-config/blob/T461/usr/lib/anon-gw-anonymizer-config/startup_helper

---

#anon-gw-anonymizer-config now depends on #anon-shared-build-apt-sources-tpo.

Should the deb.torproject.org-keyring package be rather installed by #anon-shared-build-apt-sources-tpo? That certainly would be cleaner. But then #anon-shared-build-apt-sources-tpo would need a similar mechanism to download that package during postinst and to install it on next reboot. [*]

If yes, it would certainly be cleaner to a separate package for the download packages during postinst code being a standalone shell library and a generic install on next reboot mechanism [*] that gets used by both packages, #anon-shared-build-apt-sources-tpo and #anon-gw-anonymizer-config.

All doable but quite some work.

So I am wondering, do we still require an independent package add torproject apt repository (#anon-shared-build-apt-sources-tpo)?

Do we still want to enable torproject’s apt repository within Whonix-Workstation? We no longer install any packages from torproject’s apt repository in Whonix-Workstation. We used to install torsocks from there, but since Debian jessie this is no longer necessary. We might need that in future though, if Tor Browser gets added to torproject’s apt repository - if that ever happens.

---

[*] Or manually running sudo service anon-gw-anonymizer-config start after installation of the #anon-gw-anonymizer-config package. (I wouldn’t know how to automate this during apt-get install whonix-gateway etc. This is as long as dpkg is running, there is no (sane) way to dpkg install another package since the package database is locked. And there are no post apt-get hooks. So there is no sane way to not require manually running sudo service anon-gw-anonymizer-config start or reboot after apt-get install whonix-gateway.)

---

Patrick

2016-04-06 20:17:55 UTC

Ended up implementing exactly that:
T461#8622

---

Patrick

2016-04-08 00:15:59 UTC

The download during postinst as part of apt-get install whonix-... does not work. /etc/resolv.conf gets modified during distribution morphing, therefore breaks networking, so the download will fail. I wonder how this could be solved.

---

marmarek

2016-04-08 00:41:02 UTC

How exactly it gets modified? Maybe it is possible to make both old and new version working at the same time?

---

Patrick

2016-04-08 14:19:37 UTC

Small specification so we are on the same page. What I am not talking about here, is the installation without network access for morphing a Debian template into Whonix during Qubes firstboot. This is an issue with apt-get install whonix-... from a usual Debian system with network access. I.e. if subgraph was to ask “how to install Whonix” (for porting), it would be as simple as a). /etc/resolv.conf gets modified by package anon-gw-dns-conf. apt-get install whonix-gateway leads to installation for that package first. After that networking is broken so any other packages requiring networking won’t work.

apt-get install whonix-... is different from the way packages are currently installed when using the build script. Then the script installs them in the right order, the script can also create a backup of such files and read-only mount them in chroot. Also currently: we do not start services during initial install (think: whonix-gw-firewall) vs new: start services during install. This is somewhat an attempt to dumb down build-steps.d/1700_install-packages to apt-get install whonix-....

It’s great to do that. I just now need to solve issues I could not think of earlier how to solve.

Somehow I would have to express in Debian packaging terms to have the packages that require internet access (anon-gw-anonymizer-config) to be installed first. So other packages that do not require internet access and break internet access are installed after that.

---

Patrick

2016-04-08 19:04:06 UTC

After all the complication for software outside of the Debian repository and the workarounds being complex, I guessed that adding the TPO packages to the Whonix local and binary repository would be the most canonical and best understandable solution.

I’ve signed up for tor-announce which seems not to be super busy with stable release updates and seems to be on par with deb.torproject.org.

I’ve started working on scripts to automate the process of downloading (and verifying) packages from deb.torproject.org for reupload to whonix.org repository.

---

download binary and source packages from torproject repo and add to Whonix repo

The following packages:
tor tor-geoipdb deb.torproject.org-keyring

Are being added to Whonix local / remote repository.

https://phabricator.whonix.org/T472

https://github.com/Whonix/Whonix/commit/61397b057290f245c4ce0bebf16b64494b0d2e27

enabled deb-src so we can download source pages during build

https://phabricator.whonix.org/T472

enabled deb-src so we can download source pages during build · Kicksecure/anon-shared-build-apt-sources-tpo@822ca71 · GitHub

---

Patrick

2016-04-09 15:32:33 UTC

added anon-info - shows Tor apt repository status and Tor package version

https://phabricator.whonix.org/T472

added anon-info - shows Tor apt repository status and Tor package ver… · Whonix/anon-gw-anonymizer-config@950b9c4 · GitHub

---

Patrick

2016-04-13 16:58:24 UTC

more out commented by default repositories to ease testing

https://phabricator.whonix.org/T472

more out commented by default repositories to ease testing · Kicksecure/anon-shared-build-apt-sources-tpo@f621537 · GitHub

---

Patrick

2016-04-13 17:05:35 UTC

no longer download from deb.torproject.org because it gets uploaded to Whonix repo
    
https://phabricator.whonix.org/T472

no longer download from deb.torproject.org because it gets uploaded t… · Whonix/anon-gw-anonymizer-config@93b31ef · GitHub

---

Patrick

2016-04-13 17:26:38 UTC

have deb.torproject.org and its signing key removed during apt-get autoremove

for existing installations

https://phabricator.whonix.org/T472

have deb.torproject.org and its signing key removed during apt-get au… · Kicksecure/legacy-dist@e10915b · GitHub

---

Patrick

2016-04-14 19:31:06 UTC

simplify just refreshing deb.torproject.org packages in Whonix repository mirror

https://phabricator.whonix.org/T472

https://github.com/Whonix/Whonix/commit/8365ee40f6d3bc857b311404ce19f8a972273f5a

---

Patrick

2016-04-14 19:53:58 UTC

Adding anon-info to ~/.bashrc should be simple enough.

---

Patrick

2016-04-14 23:50:52 UTC

removed download of packages from deb.torproject.org

because using different implementation (uploading these packages to Whonix
local / remote repository.

Not installing apt-during-apt and not installing these packages also ensures a
smoother transition for existing Whonix versions where deb.torproject.org is to
be disabled.

https://phabricator.whonix.org/T472

removed download of packages from deb.torproject.org · Kicksecure/anon-shared-build-apt-sources-tpo@b416da5 · GitHub

removed apt-during-apt because decided not using this approach

https://phabricator.whonix.org/T472

https://github.com/Whonix/Whonix/commit/c19b1c7a9b2d021c37a26078e60b1f3eead04498

---

Patrick

2016-04-15 00:39:41 UTC

removed anon-gw-build-upgrade-tor from whonix-gateway-shared-packages-shared-meta

because no longer required / existing

https://phabricator.whonix.org/T472

removed anon-gw-build-upgrade-tor from whonix-gateway-shared-packages… · Whonix/anon-meta-packages@5a440f5 · GitHub

---

Patrick

2016-04-26 15:17:33 UTC

new builds: ok
upgraded builds: todo

---

Patrick

2016-04-28 00:03:12 UTC

upgraded builds: ok

---

Patrick

2016-08-04 23:38:23 UTC

Good users are not getting updates unfiltered from deb.torproject.org anymore since Whonix 13.

Users who did not upgrade and stayed on Whonix 12 did now run into an issue that was caused by a newer Tor version from deb.torproject.org that required anon-gw-anonymizer-config being upgraded beforehand:

https://forums.whonix.org/t/error-tor-bootstrap-result