Whonix for arm64 / Raspberry Pi (RPi)

Patrick · August 29, 2018, 4:00pm

Merged. Some commits on top. Please check.

Algernon · August 30, 2018, 10:34am

Looks good the cli gateway is building and working fine. Could you please also merge the rpi-patches package?

Patrick · August 30, 2018, 10:52am

I.e. add GitHub - Algernon-01/rpi-patches to https://github.com/Whonix/Whonix/tree/master/packages?

GitHub - Algernon-01/rpi-patches has unaddressed bugs related to copyright and others. Will try to find my comment.

Patrick · August 30, 2018, 11:07am

Don’t add to package:

rpi-patches/usr/share/doc/raspi3-firmware/changelog.Debian.gz at master · Algernon-01/rpi-patches · GitHub (unless there is a good reason but usually no auto generated files added to source)

What’s the copyright of…:

This file is wrong then rpi-patches/debian/copyright at master · Algernon-01/rpi-patches · GitHub

Files created by you: please add copyright on top

Files not created by you: let’s see but rpi-patches/debian/copyright at master · Algernon-01/rpi-patches · GitHub needs to be very correct. Cannot break any copyright laws.

License: Proprietary is a problem. For now there are no non-free packages in Whonix repository. Nothing comparable.

github.com

Algernon-01/rpi-patches/blob/master/usr/share/doc/raspi3-firmware/copyright#L8


      
          Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
          Upstream-Name: raspberrypi / firmware
          Source: https://github.com/raspberrypi/firmware
          
          Files: boot/*
          Copyright: 2006, Broadcom Corporation
                     2015, Raspberry Pi (Trading) Ltd
          License: Proprietary
           All rights reserved.
           .
           Redistribution.  Redistribution and use in binary form, without
           modification, are permitted provided that the following conditions are
           met:
           .
           * This software may only be used for the purposes of developing for, 
             running or using a Raspberry Pi device.
           * Redistributions must reproduce the above copyright notice and the
             following disclaimer in the documentation and/or other materials

License really is Proprietary? It looks like a BSD-style license. The same license text (except for copyright holder) should already be a Debian accepted license in available from Debian.

Comment: downloaded from https://github.com/RPi-Distro/firmware-nonfree/blob/master/brcm80211/brcm/brcmfmac43430-sdio.txt

That links is broken.

I need to be able to check the file from the origin with the copy even if binary file. Any changes made to any of these files (firmware files) not originally created by you?

Why aren’t these binary files available from Debian repository? Is there at least a RFP (request for packaging) already?

https://github.com/RPi-Distro/firmware-nonfree/blob/master/debian/copyright.meta mentions Ben Hutchings who’s a Debian Developer so I wonder what the state of this package in Debian is.

Algernon · August 30, 2018, 7:59pm

All files including the binary ones were taken from this package (arm64 version): Debian -- Details of package raspi3-firmware in buster

I did not do any modifications to the binary files.

Good question. In the original files there is none.

What about modified ones?

Some were adapted to the Whonix build process. Like

github.com

Algernon-01/rpi-patches/blob/master/etc/kernel/postinst.d/50raspi3-firmware

#!/bin/sh
# vim:ts=2:sw=2:et
# see also:
# https://kernel-handbook.alioth.debian.org/ch-update-hooks.html#s-kernel-hooks

set -e

# Play nice when run under debconf.
exec </dev/null >&2

eval set -- "$DEB_MAINT_PARAMS"

# Only run on configure and remove to avoid unnecessary work.
case "$1" in
  configure|remove)
    ;;
  *)
    exit 0
    ;;
esac

This file has been truncated. show original

github.com

Algernon-01/rpi-patches/blob/master/debian/postinst

#!/bin/sh
# vim:ts=2:sw=2:et

set -e

case "$1" in
  configure)

    for file in /usr/lib/raspi3-firmware/*
    do
      file=$( basename "$file" )
      cp "/usr/lib/raspi3-firmware/$file" "/boot/$file"
      # sync might fail when running under qemu, which, as of version 2.7,
      # has not implemented the syncfs syscall.
      sync -f "/boot/$file" || true
    done

    # Manually trigger the kernel postinst hook when raspi3-firmware is first
    # installed (or upgraded), as the kernel package might already be installed
    # (or not upgraded) and hence the hook would not get triggered otherwise.

This file has been truncated. show original

Would this be an exclusion criteria? I think the official Debian images would also include these?
We can maybe also wait for buster and then try to include the official Debian package into the build process.

Patrick · August 30, 2018, 8:20pm

Commit the unmodified file fist. Or revert to unmodified version.
Then make sure it’s properly listed here: rpi-patches/debian/copyright at master · Algernon-01/rpi-patches · GitHub
If the format of the file allows:

add copyright on top
submit pull request to upstream so they fix their copyright

Commit the modifications on top. (So it can be seen what was the original, compared with upstream for verification, and seen/tracked in which file the file was modified.

What’s the copyright of…:
In the original files there is none.

Undefined copyright (probably not the case here since the package is in Debian) would be an exclusion criteria. This is because undefined copyright leads to the legal defaults which is fully copyrighted which is forbidden to redistribute.

Not necessarily. If licensing is sorted out, even if nonfree, if legal could be uploaded to Whonix nonfree. But we can do much easier here, I think. More at the very bottom of my post.

Just would have to make sure we’ll have Whonix nonfree repository. That’s currently not the case. /etc/apt/sources.list.d/whonix.list. This needs a fix in package whonix-repository (to fix new /etc/apt/sources.list.d/whonix.list creations, as well as whonix-legacy (to fix existing users /etc/apt/sources.list.d/whonix.list). Do you think you could work on that?

We can also do it in stretch if you like and if that works. Why don’t we download the whole Debian -- Details of package raspi3-firmware in buster package from Debian buster and upload to Whonix stretch as is? It has very few dependencies and might just work?

(The download → upload has no licensing issues since Debian is already strict about it so I am very comfortable from a licensing point of view.)

build-steps.d/1200_create-debian-packages has code for tor from TPO project repository already, for virtualbox guest addition… So why not add it for:

Debian -- Details of package raspi3-firmware in buster
and linux kernel - being discussed Kernel versions and security / Debian backports - #5 by Patrick

Do you think you could work on that?

Algernon · August 31, 2018, 10:02am

Using the official package would certainly be the best option. Though I could not get it working when I tested it I’m going to take a look at it again.

Algernon · August 31, 2018, 6:11pm

I tested it again with the official package from buster and it looks like only minor changes to the current build scripts would be required. The only thing that would be missing in the package would be the apparmor stuff from the kernel command line which could be added manually as an intermediate solution. But I’ll also open an issue for that on rpi firmware github page. Apparmor should be enabled by default in Buster anyways. For some final build test I need the official package from buster in the Whonix repo. Either in the official one or maybe there is also some way to put it in the local repo while building. The one from stretch definitely does not work, when I manually install the one from buster it works fine, however.
There is maybe also an issue with the package then being present in both the debian and in the whonix repo. In the latter one the version should be higher so I guess apt would install that but I’m not 100% sure. Otherwise we can’t use it in Depends and maybe have to install it manually in the build script.

Edit: btw, the deb.torproject.org signing key is expired and needs to be updated as soon as a new one is out.

Algernon · September 4, 2018, 6:20pm

@Patrick

Can you upload the package from buster to the Whonix repo? afaik the repo is not used during the build but only the local one and debian repos. So we would need to add the online repo during the build and then install only this package from there. Alternatives would be to download it manually during the build and then install it with dpkg or use apt pinning and install it from buster, though we would need to add buster then to the sources.list.

Patrick · September 5, 2018, 8:37am

Correct.

Can do… But… Could you please add the code for doing that here https://github.com/Whonix/Whonix/blob/master/build-steps.d/1200_create-debian-packages?

This would require a new file here as well: https://github.com/Whonix/Whonix/tree/master/build_sources

Or could we add buster just here https://github.com/Whonix/Whonix/blob/master/build_sources/debian_stable_current_clearnet.list without screwing the build process by installing buster packages in stretch (Whonix 14) Non-Qubes-Whonix GUI builds? In theory should work. Would you mind comparing the build log for Non-Qubes-Whonix GUI builds?

(If that is working in theory also we could abolish https://github.com/Whonix/Whonix/blob/master/build_sources/debian_testing_current_clearnet.list etc and simplify the code.)

For redistributable builds doing that manually would be a quality regression. Any opinion? @HulaHoop

HulaHoop · September 5, 2018, 1:47pm

It would complicate the process and can harm the security of the build environment.

Algernon · September 5, 2018, 9:22pm

I’m going to give apt-pinning a try.

Is this even used during the build?

Patrick · September 6, 2018, 6:40am

Yes, used.

mygrep -r build_sources

build-steps.d/1300_create-raw-image:   cp "$whonix_build_sources_list_primary" "/etc/debootstrap/etc/apt/sources.list"
build-steps.d/1100_prepare-build-machine:   cp "$whonix_build_sources_list_primary" "$cow_folder/etc/apt/sources.list"
build-steps.d/1100_prepare-build-machine:            true "INFO: Installing VirtualBox using $whonix_build_sources_list_primary..."
build-steps.d/1100_prepare-build-machine:               -o Dir::Etc::sourcelist="$whonix_build_sources_list_primary" \
build-steps.d/1100_prepare-build-machine:               -o Dir::Etc::sourcelist="$whonix_build_sources_list_primary" \
build-steps.d/1100_prepare-build-machine:            ## Get rid of $whonix_build_sources_list_primary package list, while keeping
build-steps.d/1200_create-debian-packages:   ## Update $whonix_build_sources_list_primary package list while keeping
build-steps.d/1200_create-debian-packages:      -o Dir::Etc::sourcelist="$whonix_build_sources_list_primary" \
build-steps.d/1200_create-debian-packages:   ## Download VirtualBox guest additions from $whonix_build_sources_list_primary.
build-steps.d/1200_create-debian-packages:         -o Dir::Etc::sourcelist="$whonix_build_sources_list_primary" \
build-steps.d/1200_create-debian-packages:         -o Dir::Etc::sourcelist="$whonix_build_sources_list_primary" \
build-steps.d/1200_create-debian-packages:               -o Dir::Etc::sourcelist="$whonix_build_sources_list_primary" \
build-steps.d/1200_create-debian-packages:   ## Get rid of $whonix_build_sources_list_primary package list, while keeping
buildconfig.d/30_apt_opts.conf:## Does not play well with onion build_sources.
buildconfig.d/30_dependencies.conf:## required for onion build_sources
buildconfig.d/30_apt_sources.conf:if [ "$whonix_build_sources_list_newer" = "" ]; then
buildconfig.d/30_apt_sources.conf:   if [ "$whonix_build_sources_clearnet_or_onion" = "clearnet" ]; then
buildconfig.d/30_apt_sources.conf:      whonix_build_sources_list_newer="$WHONIX_SOURCE_FOLDER/build_sources/debian_testing_current_clearnet.list"
buildconfig.d/30_apt_sources.conf:      whonix_build_sources_list_newer="$WHONIX_SOURCE_FOLDER/build_sources/debian_testing_current_onion.list"
buildconfig.d/30_apt_sources.conf:if [ "$whonix_build_sources_list_primary" = "" ]; then
buildconfig.d/30_apt_sources.conf:   if [ "$whonix_build_sources_clearnet_or_onion" = "clearnet" ]; then
buildconfig.d/30_apt_sources.conf:      whonix_build_sources_list_primary="$WHONIX_SOURCE_FOLDER/build_sources/debian_stable_current_clearnet.list"
buildconfig.d/30_apt_sources.conf:      whonix_build_sources_list_primary="$WHONIX_SOURCE_FOLDER/build_sources/debian_stable_current_onion.list"
buildconfig.d/30_apt_sources.conf:true "whonix_build_sources_list_primary      : $whonix_build_sources_list_primary"
buildconfig.d/30_apt_sources.conf:true "whonix_build_sources_list_newer        : $whonix_build_sources_list_newer"
buildconfig.d/30_apt_sources.conf:   temp_="$(grep --invert-match "#" "$whonix_build_sources_list_primary")"
buildconfig.d/30_apt_sources.conf:   temp_="$(grep --invert-match "#" "$whonix_build_sources_list_primary")"
buildconfig.d/30_apt_sources.conf:   temp_="$(grep --invert-match "#" "$whonix_build_sources_list_newer")"
help-steps/parse-cmd:                  whonix_build_sources_clearnet_or_onion="clearnet"
help-steps/parse-cmd:                  whonix_build_sources_clearnet_or_onion="onion"
help-steps/parse-cmd:               export whonix_build_sources_list_primary="build_sources/debian_testing_frozen.list"
help-steps/parse-cmd:   if [ "$whonix_build_sources_clearnet_or_onion" = "" ]; then
help-steps/parse-cmd:Defaulting whonix_build_sources_clearnet_or_onion to ${under}clearnet${eunder}.
help-steps/parse-cmd:      export whonix_build_sources_clearnet_or_onion="clearnet"                                                                                                           
help-steps/create-local-temp-apt-repo:   cp "$whonix_build_sources_list_primary" "$CHROOT_FOLDER/$WHONIX_SOURCES_LIST_TEMP_BUILD_FOLDER/build_sources.list"                                   
help-steps/create-local-temp-apt-repo:   cat "$CHROOT_FOLDER/$WHONIX_SOURCES_LIST_TEMP_BUILD_FOLDER/build_sources.list"                                                                       
help-steps/remove-local-temp-apt-repo:   rm --force "$CHROOT_FOLDER/$WHONIX_SOURCES_LIST_TEMP_BUILD_FOLDER/build_sources.list"

Algernon · September 6, 2018, 12:08pm

I made a test with apt-pinning and it just installed only the raspi3-firmware package from buster and the rest from stretch. Going to do some more tests, though. We could maybe also just enable the apt-pinning for the rpi-gateway so it won’t get used for the other builds.

Patrick · September 7, 2018, 5:45am

We could maybe also just enable the apt-pinning for the rpi-gateway so it won’t get used for the other builds.

Maybe.

As per Re: Can a package pull a another package from another repository to be added? and Re: Can a package pull a another package from another repository to be added? APT pinning can break in very bad ways.

And offend Debian.

(And don’t forget all the hate you will get from me and other deity@l.d.o inhabitants because your users end up reporting bugs about APT)

Our past APT pinning instructions lead to breaking the system. ( Templates incorrectly think they're not connected to a Whonix gateway. - #25 by safasfasv )

Advantages:

less code for download / reupload RPI firmware package
less maintenance work for me for download / reupload RPI firmware package

Disadvantages:

quality regression
offending Debian
possible issues on release upgrade
other possible issues due to APT pinning (no specific example, hard to foresee)

I am inclined to say yes to APT pinning.

Any opinion @HulaHoop?

How would that technically work? Shipping file /etc/apt/sources.list.d/rpi.list as well as /etc/apt/preferences.d/ in some package?

deb http://http.debian.net/debian stretch-backports main contrib non-free
deb tor+http://vwakviie2ienjx6t.onion/debian stretch-backports main contrib non-free

(Plus comments.)

And then include instructions to remove that file before release upgrade of Whonix when upgrading to Debian 10 / buster in Whonix release upgrade instructions?

Algernon · September 7, 2018, 9:17am

imo, the reason for this was that each package got upgraded. Not just the desired one.

Disadvantages for building from source are: currently does not work due to being an architecture specific package, while the rest of the packages is not. Lots of lintian errors which I think are also due to being architecture specific.

Currently it is implemented via copying the build_sources_testing.list and preferences file to the build folder during the create-local-temp-apt-repo step. They can also be removed later on.
The firmware package currently depends only on dosfstools, though no specific version. Therefore, currently only raspi3-firmware is installed from buster and nothing else. Since the sources.list in the final version also does not contain anything from buster I would not expect any breakage from updates. The package should be more or less fixed at that version until Whonix gets updated to buster. To admit, the last part is more or less theory.

HulaHoop · September 7, 2018, 10:52pm

apt-pinning done properly is safe enough. I’ve been running my host with it enabled before upgrading to Stretch without any problems and never had to remove packages before the upgrade. Everything updated seamlessly enough.

It would benefit it us in other cases besides RPi builds too.

The most important thing is to find and apply accurate and updated documentation about pinning priority so things don’t unexpectedly start pulling in more deps and wreaking havoc. I think the important thing is to advise users to keep their hands off and not add more programs if they don’t know what they are doing and not doing so voids our “support”.

Patrick · September 8, 2018, 5:59pm

https://github.com/Whonix/anon-meta-packages/pull/12

https://github.com/Whonix/Whonix/pull/420

Merged. Minor commit on top.

https://github.com/Whonix/Whonix/commit/653d97664f45fdbd21fab3cd162c19f5d22607c1

Algernon · September 8, 2018, 6:23pm

That was quick
Currently the network and date still has to be set manually. You likely also need to use a serial console. At least I could not yet get it working with HDMI. Either the reason is my hardware or something else during early boot like firmware or kernel. I’m going to make a test with a newer kernel and see if it helps. I’ll maybe also remove the wifi packages and make them Recommends instead of Depends. I think also a network config package specific to the cli versions would be required which sets eth0 to dhcp and the MAC to e.g. the one used for the gateway on Virtualbox.

Algernon · September 10, 2018, 6:14pm

I added the backports kernel as Depends for the rpi-gw metapackage. With this there is also HDMI output.