Policy for Inclusion of Compiled Software

What is the policy for including non-packaged programs by default, that are distributed as Appimages or Jar files? Can you please document your policy on this on the wiki? Is Tor Browser the only exception?

Interesting software we might want to consider in this form is:

  • Freenet
  • Cryptocat

Edit by Patrick:

1 Like

Updated wiki page to answer a part of this:

Added a bit here Whonix Default Application Policy

Installing software other then Tor Browser (as documented above) from non-APT package sources by installed one by default would be nightmare, because:

  • Users wouldn’t know how it was installed and not update it.
  • Updating would be left to the user.
  • It is better if the whole process of download, verification, install, upgrade notification and upgrade is up to the user.

Experiences made with tb-updater and Tor Browser (see above chapter) further discourage this.

Wouldn’t call that a policy though. Rather for reasons of practicality / lowering maintenance effort / lack of developer manpower.

That doesn’t mean development help including Freenet / Cryptocat. That could in long run rather increase maintenance effort. Development help with something that would lower development in long run such as more whonixcheck tests or automated test suite.

2 Likes

But these two programs are designed with secure auto-update in mind.

Verification is a hurdle for many beginners who would then risk running the binaries without checking at all. Including them in Whonix while updating in sync with newer releases upstream, is a better solution.

Implementation:

Making them a part of an independent package that drops the binary at some location in the home folder. In Freenet’s case, all deps are self-contained except openjdk which we can include the small headless version of in the base packages. I can check/download/verify new releases and update the jar in the Whonix package. That shouldn’t be too much overhead since the releases are infrequent.

As for cryptocat, my main goal was to provide something with offline messaging for now until Gajim or CoyIM are ready, but it may be unnecessary.

Tor Browser i think we gonna have it from Debian as well through TBB-Downloader from debian repo. So i dont think its an exception anymore no?

although i like to have more support to other secure net philosophy support BUT if their developers doesnt help themselves to support and upload their stuff to debian then its better to avoid them until then. (we cant help lazy , or very low power community which cant upload stuff to debian repos)

@nurmagoz:

Tor Browser i think we gonna have it from Debian as well through TBB-Downloader from debian repo.

There’s no Tor Browser in packages.debian.org.

Assuming secure auto-update by these applications for the sake of discussion for now without having that checked yet.

HulaHoop:

Verification is a hurdle for many beginners who would then risk running the binaries without checking at all. Including them in Whonix while updating in sync with newer releases upstream, is a better solution.

Good point.

Implementation:

Making them a part of an independent package that drops the binary at some location in the home folder.
ought to avoid writing into linux user /home folder as per Dev/About Debian Packaging - Kicksecure

But this would be a detail that can probably be sorted somehow.

openjdk which we can include the small headless version of in the base packages.

Ok.

I can check/download/verify new releases and update the jar in the Whonix package.
Undecided. Various disadvantages come to my mind. Not sure if any or the totality of these are bad enough to avoid this?

I’d still have to recheck, otherwise we’d implement the first package which doesn’t get scrutinized before I merge or I just trust you’ll do it?

It’s not only me who should be easily and quickly capable of verifying the backdoor-freeness of the package. Also independent auditors should be capable to do so. At minimum securely check the gpg signature matches the binary. Doing so by script?

Is this contrary to verifiable builds / deterministic builds? I mean, by using this style, do we have to undo such a package containing binaries if one day verifiable builds / deterministic builds are in reach?

Freenet is ~ 18 MB. The way git handles binary files, with each release the repository would get bigger since git keeps old files. With each release that git repository would get bigger and bigger, therefore slower to fetch over Tor. If we delete old binary files (reset git / start fresh / git push force), we’d open new issues. https://github.com/Whonix/Whonix couldn’t be updated anymore by standard git commands. It would complain and abort due the force pushed versions. If git repository size alone isn’t a problem enough, would we get more and more contributions/pull requests for addition of more programs in binary form?

Unclean. We’d produce a “source package” that in fact contains binaries.

Needless to say that such a package is unfit for inclusion into Debian.

Why tb-updater then? Could a simplified a lot without download by using a package containing the binaries? (I actually worked on that implementation once but then threw it away due to the big size of the git repository.)

Why didn’t I develop “Whonix freeware” style? What I mean by “freeware” style: Not a “proper” Open Source project with instructions on how to build it from source code. Why bother spending hours and hours with the development of a build script if I could just install Debian inside VirtualBox, manually make all changes by keyboard and mice, run a cleanup script, export ova, upload? Could even post instructions how I did it so anyone can manually replicate. With “freeware style” development would be actually a lot easier, faster. What’s the difference between “freeware style” development and a package that contains binaries? I am trying to demonstrate the slippery line between a clean Open Source project and “freeware style”.

Excellent reasons I hadn’t considered. Bending the rules leads to dev hell. Please add these examples to Whonix ™ Default Application Policy

1 Like

Not yet clear on the way forward. I need to scrutinize my own arguments from my previous post Policy for Inclusion of Compiled Software - #6 by Patrick in this forum thread or someone else needs to scrutinize them.

I really would like to at least ship the electrum appimage by default somehow.
( ⚓ T215 install electrum bitcoin thin client by default? )
( How-to: Use Electrum Bitcoin Wallet in Whonix ™ )

Soon I will come up with some implementation ideas which need to be scrutinized. Maybe a downloader similar to tb-updater but much simpler without user interaction. Everything hardcoded.

1 Like

The lack of proper support for binary files in a way that doesn’t ballon the repo is a git limitation rather than an intrinsic drawback of this technique.

Alternatives like git-annex are widely supported

https://manpages.debian.org/stretch/git-annex/git-annex.1.en.html

git-annex allows managing files with git, without checking the file contents into git. While that may seem paradoxical, it is useful when dealing with files larger than git can currently easily handle, whether due to limitations in memory, checksumming time, or disk space.

With git-annex the step below is already done by the too itself instead of having to reinvent the wheel:

Here we can have a a source package that has a hardcoded mechanism for safely fetching the blob from a remote server other than the one hosting the repo like a folder on the Whonix package server. Bonus points if that mechanism is generic enough to support arbitrary software binaries in the future.


Offtopic: There’s talk by Freenet devs to eventually roll out a proper repo with .debs.

1 Like

To add yet more reasoning.

This forum thread is for example about adding the electrum appimage, which is a binary planned to be added to a Whonix “source package” which then would be compiled into a binary deb package.

The ideological issue here is, that it is not compiled from source during the process for creation of a binary deb package. (Which is difficult and could even have security issues unfortunately due to unverified third party library downloads.)

Electrum is fully Freedom Software. Electrum is not available as an up-to-date deb package from any source at time of writing.

On the other hand, it is the same quantity of “bad” as using binary Debian packages during build of Whonix. All of Tails, Qubes, Whonix and many derivatives does download binary deb packages from packages.debian.org during their iso/ova build process. Electrum is just another source of binary Freedom Software similar to packages.debian.org.

1 Like

install electrum appimage by default:

electrum appimage is shipped by this package:

2 Likes

A post was merged into an existing topic: wire for desktop - encrypted messenger

HulaHoop via Whonix Forum:

The lack of proper support for binary files in a way that doesn’t ballon the repo is a git limitation rather than an intrinsic drawback of this technique.

Alternatives like git-annex are widely supported

git-annex(1) — git-annex — Debian stretch — Debian Manpages

git-annex allows managing files with git, without checking the file contents into git. While that may seem paradoxical, it is useful when dealing with files larger than git can currently easily handle, whether due to limitations in memory, checksumming time, or disk space.

With git-annex the step below is already done by the too itself instead of having to reinvent the wheel:

That did not work. The actual binary really needs to be inside the
actual repository. Otherwise people who build Whonix from source code,
would build it without the actual binary.

Here we can have a a source package that has a hardcoded mechanism for safely fetching the blob from a remote server other than the one hosting the repo like a folder on the Whonix package server.

That is rather hard, and therefore likely of breaking the build.
Experiences with Tor Browser: kept changing links, signature schemes.
Experiences with electrum: was under DDOS. No download possible.

Assuming git-annex is added as a Whonix build dep, does this apply? It would be in charge of managing extraneous binaries

Two solutions: 1. Place and fetch the binaries from our repos 2. Make the build script skip fetching the binaries if it fails after a number of attempts.

Yes.

That would work in theory but when downloading them, these need to be reliably OpenPGP/gpg verified which ends up being a similarly complex solution as tb-updater.

That would result in inconsistent build results. Some have the binary, others do not. And then the build after being booted would claim binaries-freedom package but the actual binaries are missing.

For tb-updater there are build parameters:

–tb none|closed|open
none: Do not install Tor Browser.
closed: Fail closed if Tor Browser cannot be installed.
open: Fail open if Tor Browser cannot and installed.
Defaults to closed.
Requires the tb-updater package being installed. Be careful if you modify
the default package selection as well.

Also rather complex.

I am considering to deprecate tb-updater and to place the archive files and signatures into binaries-freedom package for simplification.

1 Like

Since electrum is in Debian bullseye and recent enough, it can be dropped from package binaries-freedom to reduce maintenance load.

#milestone_whonix_16

CoyIM 0.4 recently came up but cant be included in debian (at least for the coming years), Would be nice if it can be checked and considered for inclusion.

https://coy.im/coyim/release/2022/02/16/coyim-v04-released.html

1 Like

Despite not being available in Debian repos, it still deserves an entry for securely installing it from outside sources. I will look into this at some point.

1 Like