Derivative-Maker | Installing kernel from backports?

Support
1

Hello, I’m building kvm images with derivative-maker and need kernel version 6.12.
Installing the packages linux-image and linux-headers from backports seems like the simplest solution.

Dirty Solution:

#Adding to buildstep 3500_install-packages
chroot_run apt-get-noninteractive --yes --purge remove linux-image-6.1.0-28-amd64 linux-headers-6.1.0-28-amd64 && \
chroot_run apt-get-noninteractive "${APTGETOPT[@]}" $apt_sourcelist_empty $apt_sourceparts $apt_unattended_opts --yes linux-image-6.12.22+bpo-amd64 linux-headers-6.12.22+bpo-amd64

I would like the developer intended clean way of doing this, though.

For example, could I replace the following in build-steps.d/2800_create-lb-iso?

Before:

 sed -i "s/XXX_ARCHITECTURE_XXX/${dist_build_target_arch}/" package-lists/z-kicksecure.list.chroot

After:

KERNEL_VER="6.12.22+bpo-amd64"
sed -i "s/XXX_ARCHITECTURE_XXX/${KERNEL_VER}/" package-lists/z-kicksecure.list.chroot

Assuming, that live-build-data/live-build-config/live_build_package_list_kicksecure is where kernel packages are read in from and backports repo is initialized.

Edit: If I’m not mistaken, editing derivative-maker-master/buildconfig.d/25_apt_sources.conf may also be necessary to make the backports repo available in chroot @ buildstep 3500_install-packages.

Before:

         if echo $derivative_name_list | grep -q -i Whonix ; then
            [ -n "$sources_list_build_remote_derivative" ] || sources_list_build_remote_derivative="
            deb http://HTTPS///deb.kicksecure.com $codename_build_remote_derivative main contrib non-free
            deb http://HTTPS///deb.whonix.org $codename_build_remote_derivative main contrib non-free
            "
         else
            [ -n "$sources_list_build_remote_derivative" ] || sources_list_build_remote_derivative="
            deb http://HTTPS///deb.kicksecure.com $codename_build_remote_derivative main contrib non-free
            "

After:

         if echo $derivative_name_list | grep -q -i Whonix ; then
            [ -n "$sources_list_build_remote_derivative" ] || sources_list_build_remote_derivative="
            deb http://HTTPS///deb.kicksecure.com $codename_build_remote_derivative main contrib non-free
            deb http://HTTPS///deb.whonix.org $codename_build_remote_derivative main contrib non-free
            deb http://HTTPS///deb.debian.org/debian bookworm-backports main contrib non-free non-free-firmware
            "
         else
            [ -n "$sources_list_build_remote_derivative" ] || sources_list_build_remote_derivative="
            deb http://HTTPS///deb.kicksecure.com $codename_build_remote_derivative main contrib non-free
            deb http://HTTPS///deb.debian.org/debian bookworm-backports main contrib non-free non-free-firmware 
            "
1 Like
2

Hmm, that doesn’t seem to work. I’ll figure out today where the /etc/apt/sources.list.d/ files are configured in chroot @ buildstep 3500_install-packages and report back.

Derivative-Maker is awesome with tons of helpful comments, but sometimes I wish there was a bit more direction/documentation. I find myself constantly looking for variables and functions or config files all over the place. It’s almost impossible to make minor changes without deeply understanding the entire code structure.

1 Like
3

This might work:

–kernel [packages] : Specify kernel packages (e.g., ‘linux-image-amd64’ or ‘none’).
–headers [packages] : Specify kernel header packages.

Folder build_sources already comes with Debian backports folder. So no enabling of Debian backports is required.

Whonix-Host (ISO) does not exist at time of writing. Status and updates are here: Whonix-Host Operating System Live ISO, Whonix-Host Installer

Therefore this option file is irrelevant for Whonix at this time.

I don’t like many things about this part of derivative-maker source code.

  • ISO specific
  • not applicable to other build targets (VMs)
  • dynamically created file during the build process
  • dynamically written during the build process

But building a bootable (Debian) ISO with GRUB + BIOS + EFI + Secure Boot + dracut + multiple architecture support very difficult. (Development of System Image Creation and Bootstrapping Tools) So we rely on Debian live-build for that.

Variable sources_list_build_remote_derivative is only relevant in context of Whonix build script now optionally supports installing packages from Whonix remote repository rather than building packages locally. I don’t think it should be re-used for other purposes.

To install additional packages, expanding the flavor_meta_packages_to_install variable using Build Variables Changes might be an option.

But maybe that’s a bit inconvenient. If other custom, non-kernel/header packages shall be installable, we could add an additional variable user_custom_packages_to_install.

The full path to a complete build sources file can probably be set using variable dist_build_sources_list_primary.

Other options: Add your own custom build-step.

This amount of features and complexity is difficult to manage. Due to dependency on third-party projects, not something that could be centrally planned and improved.

Related: Linux User Experience versus Commercial Operating Systems

There are a ton of SystemBuildTools - Debian Wiki, but probably none can replace Derivative-Maker - Debian based Linux Bootable Image Builder or are simple to customize. It’s a hard problem to solve.

derivative-maker advanced configuration has very low priority. There are much more important tasks on the Whonix ™ Security Roadmap.

1 Like
4

–kernel [packages] : Specify kernel packages (e.g., ‘linux-image-amd64’ or ‘none’).
–headers [packages] : Specify kernel header packages.

Hey man, thanks for checking in.

NIce., that seems like the perfect solution, of course.
Since the backports repo is already available I should then be able to just directly enter the full package names like so:

–kernel linux-image-6.12.22+bpo-amd64
–headers linux-headers-6.12.22+bpo-amd64

If the metapackages linux-image-amd64 and linux-headers-amd64 are passed with these options, I assume the release repo is chosen by default. (Gonna check that, though.)

Folder build_sources already comes with Debian backports folder. So no enabling of Debian backports is required.

Ok, that’s what you meant by referencing dist_build_sources_list_primary

So one could just create a custom sources list config, put it in build_sources and assign dist_build_sources_list_primary to it in buildconfig.d/25_apt_sources.conf

For custom repos I guess the key can be placed in build_sources like you guys do. When I use live-build the list and key always needed to be in config/archives/repo.list.chroot and config/archives/repo.key.chroot.

But building a bootable (Debian) ISO with GRUB + BIOS + EFI + Secure Boot + dracut + multiple architecture support very difficult. (Development of System Image Creation and Bootstrapping Tools) So we rely on Debian live-build for that.

I built something to that effect involving Whonix a while ago and considered sharing it here, but didn’t think it was practical enough.

Basically, I wrote a script that incorporates live-build to build the ISO, with all the libvirt, iptables and dnsmasq configuration already set up, some other minor security improvements like hardened-kernel etc. and put the compressed qcow2 gateway/workstation images in includes.chroot, so they’ll be included in the root filesystem and usable later on. ~ 2.5 GB total if I remember correctly. (Could probably make it smaller)

Then you boot the ISO with the toram flag, and use libvirt snapshots to make VM writes in the overlayfs. (Since the images will be read-only in squash upon boot).

That was my dirty solution for achieving an amnesiac Whonix ISO.
If you’re interested I can put together a repo on my github. It worked quite well, but definitely not an optimal solution.

To install additional packages, expanding the flavor_meta_packages_to_install variable using Build Variables Changes might be an option.

That reminds me, I was gonna do some stuff with the kicksecure-meta-packages. Totally forgot, but thankfully trixie hasn’t been moved to stable yet. I promise I’ll have that done before you guys make the jump.

There are a ton of SystemBuildTools - Debian Wiki, but probably none can replace Derivative-Maker - Debian based Linux Bootable Image Builder or are simple to customize. It’s a hard problem to solve.

The one I’ve spent the most time with is the Debian Live Manual.
Uuff, I remember going through some real pain with that thing, while learning live-build. Derivative-Maker combines the best of many of those tools, which is why I respect it so much, because I can imagine the amount of hours necessary to just nail down documentation, let alone setting all this up.

Tbh, it’s almost more impressive than the Whonix Project itself to me. The versatility of such a tool is quite limitless.

1 Like
5

That might work. No need to change sources list files - backports already enabled in default build sources.

But there’s a disadvantage you should be aware of: You probably won’t get kernel upgrades. Because you’re not properly installing linux-image-amd64 / linux-headers-amd64 - the unversioned (in package name) meta packages that are responsible to pull latest available kernel packages.

As a sysadmin, the correct APT commands would be:

sudo apt-get install -t bookworm-backports linux-image-amd64
sudo apt-get install -t bookworm-backports linux-headers-amd64

Translated to derivative-maker command line options that would be, in theory - but won’t work:

--kernel "-t bookworm-backports linux-image-amd64"
--headers "-t bookworm-backports linux-headers-amd64"

Would be unclean due to quoting and unquoting.

For VMs: Might not work.

build-steps.d/3500_install-packages

         for build_kernel in $BUILD_KERNEL_PKGS; do
            pkg-add-to-install-list "$build_kernel"
         done

For ISO: bug: --kernel and --headers are currently silently ignored at time of writing.

These command line options were not designed with backports in mind. Instead, what I had in mind, was when porting to new architectures --kernel linux-image-arm64 / --headers linux-headers-amd64. (But not needed, because actually auto detected.)

Can be anywhere.

Note:

grep -r backports build_sources

Possibly. Environment variable might work too. Or use your own buildconfig.d folder outside the derivative-maker repository as documented.

Yet another option: Add your own Chroot Script.

  - install_package_list: Specify additional custom packages for installation.
    Examples:
      install_package_list='gparted'
      install_package_list='gparted gedit'

This also won’t work to acquire kernel from backports, because:

      for install_package_item in $install_package_list ; do
         true "${cyan}INFO: install_package_item: '$install_package_item'${reset}"
         pkg-add-to-install-list "$install_package_item"
      done

I tried to provide a lot flexibility but to install kernels from backports, there is currently no great solution.

I am wondering how advanced users can be provided with the biggest potential flexibility that includes running injecting arbitrary commands (including installation from backports). Ideally, without needing to touch the derivative-maker folder. So ../buildconfig.d folder can be used.

Your own build-step is too difficult, too heavyweight?

Inside build-steps.d/3500_install-packages after:

   pkg-add-to-install-list "$meta_package_firmware"

   pkg-list-install

Maybe the ability to define to define a whole shell function that gets executed at that point? If the shell function exists, execute it. Otherwise, do nothing, proceed as usual.

If implemented, something like this might be possible:

install_packages_middle_custom_function() {
   chroot_run apt-get-noninteractive "${APTGETOPT[@]}" $apt_sourcelist_empty $apt_sourceparts $apt_unattended_opts --yes -t bookworm-backports linux-image-amd64 linux-headers-amd64
}

(You could then combine this with --kernel none and --headers none - because you’re installing kernel/header using custom shell function.)

1 Like
6

Translated to derivative-maker command line options that would be, in theory - but won’t work. Would be unclean due to quoting and unquoting.

Hmm you could maybe escape those or single quotes? Looks terrible either way though, lol.

These command line options were not designed with backports in mind. Instead, what I had in mind, was when porting to new architectures --kernel linux-image-arm64 / --headers linux-headers-amd64. (But not needed, because actually auto detected.)
Yeah that makes sense anyway. Backports kernel is out of scope in that regard, and should be added via custom option.

         for build_kernel in $BUILD_KERNEL_PKGS; do
            pkg-add-to-install-list "$build_kernel"
         done

I have a question about functions pkg-add-to-install-list and pkg-install.

Particularly, the first part of pkg-install:

   exception_handler_setup "exception_handler_unchroot_unmount" ERR INT TERM

   local pkg_install_item
   pkg_install_item="$1"
   local skip_package
   for skip_package in $dist_build_script_skip_package_install; do
      if [ "$skip_package" = "$pkg_install_item" ]; then
         unset skip_package
         true "${bold}${cyan}INFO: Skipping installation of '$pkg_install_item', because variable dist_build_script_skip_package_install includes it.${reset}"
         return 0
      fi
   done
   unset skip_package

   true "${cyan}INFO: Installing of '$@', because variable dist_build_script_skip_package_install does not include it... \
This may take a while...${reset}"

The variable $pkg_install_list contains a space separated list of packages which is/can be assembled with the function pkg-add-to-install-list

Executing the following,

pkg-install $pkg_install_list

doesn’t this only pass the first package in $pkg_install_list as an argument?

And without shifting arguments or looping, how are all items in the list checked against the skip variable via if [ "$skip_package" = "$pkg_install_item" ]?
(Especially if return is used upon a match)

Could $pkg_install_list be turned into an array to unset individual packages from the list if it matches an item in $dist_build_script_skip_package_install?:

pkg-add-to-install-list() {
.
.
.
.
.
   if [ "$pkg_install_debug" = "true" ]; then
      pkg-install "$pkg_install_item"
   else
      pkg_install_list+=("$pkg_install_item")
   fi
}

pkg-add-to-install-list() {
   exception_handler_setup "exception_handler_unchroot_unmount" ERR INT TERM

for i in ${!pkg_install_list[@]}; do

	pkg_install_item="${pkg_install_list[i]}"

	if [[ "$dist_build_script_skip_package_install" =~ "$pkg_install_item" ]]; then
		true "${bold}${cyan}INFO: Skipping installation of '$pkg_install_item', because variable dist_build_script_skip_package_install includes it.${reset}"
		unset pkg_install_list[$i]
		pkg_install_list=( "${pkg_install_list[@]}" )
	fi

done
1 Like
7

The right way to do this is Bash Proper Whitespace Handling but the syntax also looks harder. Would be more difficult for users to use the environment variable.

But in any case, I don’t see how --kernel option could support backports. Allowing writing -t backports there or an argument that contains spaces seems hard to parse and wrong.

No. It’s obviously functional and installing all the packages.

You could test this with a simple standalone test script.

The more correct way to do this would also be Bash Proper Whitespace Handling.

Yes, an array would be cleaner.

${!pkg_install_list[@]} needs to be quoted.

[[ unnecessary. Matching with =~ seems wrong. A simple if with = would suffice.

Should be quoted.

2 Likes
8

But in any case, I don’t see how --kernel option could support backports. Allowing writing -t backports there or an argument that contains spaces seems hard to parse and wrong.

No, you’re right. What I did yesterday was grabbing the package(s) with apt show and putting it in a variable with --kernel ${IMAGE} and --headers ${HEADERS} . Worked fine. It’s already working the way it should, I agree.

No. It’s obviously functional and installing all the packages.
You could test this with a simple standalone test script.
The more correct way to do this would also be Bash Proper Whitespace Handling.

Yup, wrong again. :smile: Just ran it. Every time we have an exchange I learn a bunch of new stuff. (At the expense of your time)

Maybe the ability to define to define a whole shell function that gets executed at that point? If the shell function exists, execute it. Otherwise, do nothing, proceed as usual.

install_packages_middle_custom_function() {
  chroot_run apt-get-noninteractive "${APTGETOPT[@]}" $apt_sourcelist_empty $apt_sourceparts $apt_unattended_opts --yes -t bookworm-backports linux-image-amd64 linux-headers-amd64
}

Right, that makes sense.

Maybe as an additional option?

--custom-repo bookworm-backports

[ -z "$REPO" ] || { install_packages_middle_custom_function() { chroot_run apt-get-noninteractive "${APTGETOPT[@]}" $apt_sourcelist_empty $apt_sourceparts $apt_unattended_opts --yes -t "$REPO" linux-image-amd64 linux-headers-amd64; }; install_packages_middle_custom_function; }

needs to be quoted. Should be quoted

You’re making me extremely self-conscious about quotes now. :smile:

[[ unnecessary. Matching with =~ seems wrong. A simple if with = would suffice.

Exposing my ignorance (further), but I always get a synthax error when matching =~ with single brackets.

1 Like