Derivative-Maker | Installing kernel from backports?

Support
1

Hello, I’m building kvm images with derivative-maker and need kernel version 6.12.
Installing the packages linux-image and linux-headers from backports seems like the simplest solution.

Dirty Solution:

#Adding to buildstep 3500_install-packages
chroot_run apt-get-noninteractive --yes --purge remove linux-image-6.1.0-28-amd64 linux-headers-6.1.0-28-amd64 && \
chroot_run apt-get-noninteractive "${APTGETOPT[@]}" $apt_sourcelist_empty $apt_sourceparts $apt_unattended_opts --yes linux-image-6.12.22+bpo-amd64 linux-headers-6.12.22+bpo-amd64

I would like the developer intended clean way of doing this, though.

For example, could I replace the following in build-steps.d/2800_create-lb-iso?

Before:

 sed -i "s/XXX_ARCHITECTURE_XXX/${dist_build_target_arch}/" package-lists/z-kicksecure.list.chroot

After:

KERNEL_VER="6.12.22+bpo-amd64"
sed -i "s/XXX_ARCHITECTURE_XXX/${KERNEL_VER}/" package-lists/z-kicksecure.list.chroot

Assuming, that live-build-data/live-build-config/live_build_package_list_kicksecure is where kernel packages are read in from and backports repo is initialized.

Edit: If I’m not mistaken, editing derivative-maker-master/buildconfig.d/25_apt_sources.conf may also be necessary to make the backports repo available in chroot @ buildstep 3500_install-packages.

Before:

         if echo $derivative_name_list | grep -q -i Whonix ; then
            [ -n "$sources_list_build_remote_derivative" ] || sources_list_build_remote_derivative="
            deb http://HTTPS///deb.kicksecure.com $codename_build_remote_derivative main contrib non-free
            deb http://HTTPS///deb.whonix.org $codename_build_remote_derivative main contrib non-free
            "
         else
            [ -n "$sources_list_build_remote_derivative" ] || sources_list_build_remote_derivative="
            deb http://HTTPS///deb.kicksecure.com $codename_build_remote_derivative main contrib non-free
            "

After:

         if echo $derivative_name_list | grep -q -i Whonix ; then
            [ -n "$sources_list_build_remote_derivative" ] || sources_list_build_remote_derivative="
            deb http://HTTPS///deb.kicksecure.com $codename_build_remote_derivative main contrib non-free
            deb http://HTTPS///deb.whonix.org $codename_build_remote_derivative main contrib non-free
            deb http://HTTPS///deb.debian.org/debian bookworm-backports main contrib non-free non-free-firmware
            "
         else
            [ -n "$sources_list_build_remote_derivative" ] || sources_list_build_remote_derivative="
            deb http://HTTPS///deb.kicksecure.com $codename_build_remote_derivative main contrib non-free
            deb http://HTTPS///deb.debian.org/debian bookworm-backports main contrib non-free non-free-firmware 
            "
1 Like
2

Hmm, that doesn’t seem to work. I’ll figure out today where the /etc/apt/sources.list.d/ files are configured in chroot @ buildstep 3500_install-packages and report back.

Derivative-Maker is awesome with tons of helpful comments, but sometimes I wish there was a bit more direction/documentation. I find myself constantly looking for variables and functions or config files all over the place. It’s almost impossible to make minor changes without deeply understanding the entire code structure.

1 Like
3

This might work:

–kernel [packages] : Specify kernel packages (e.g., ‘linux-image-amd64’ or ‘none’).
–headers [packages] : Specify kernel header packages.

Folder build_sources already comes with Debian backports folder. So no enabling of Debian backports is required.

Whonix-Host (ISO) does not exist at time of writing. Status and updates are here: Whonix-Host Operating System Live ISO, Whonix-Host Installer

Therefore this option file is irrelevant for Whonix at this time.

I don’t like many things about this part of derivative-maker source code.

  • ISO specific
  • not applicable to other build targets (VMs)
  • dynamically created file during the build process
  • dynamically written during the build process

But building a bootable (Debian) ISO with GRUB + BIOS + EFI + Secure Boot + dracut + multiple architecture support very difficult. (Development of System Image Creation and Bootstrapping Tools) So we rely on Debian live-build for that.

Variable sources_list_build_remote_derivative is only relevant in context of Whonix build script now optionally supports installing packages from Whonix remote repository rather than building packages locally. I don’t think it should be re-used for other purposes.

To install additional packages, expanding the flavor_meta_packages_to_install variable using Build Variables Changes might be an option.

But maybe that’s a bit inconvenient. If other custom, non-kernel/header packages shall be installable, we could add an additional variable user_custom_packages_to_install.

The full path to a complete build sources file can probably be set using variable dist_build_sources_list_primary.

Other options: Add your own custom build-step.

This amount of features and complexity is difficult to manage. Due to dependency on third-party projects, not something that could be centrally planned and improved.

Related: Linux User Experience versus Commercial Operating Systems

There are a ton of SystemBuildTools - Debian Wiki, but probably none can replace Derivative-Maker - Debian based Linux Bootable Image Builder or are simple to customize. It’s a hard problem to solve.

derivative-maker advanced configuration has very low priority. There are much more important tasks on the Whonix ™ Security Roadmap.

1 Like
4

–kernel [packages] : Specify kernel packages (e.g., ‘linux-image-amd64’ or ‘none’).
–headers [packages] : Specify kernel header packages.

Hey man, thanks for checking in.

NIce., that seems like the perfect solution, of course.
Since the backports repo is already available I should then be able to just directly enter the full package names like so:

–kernel linux-image-6.12.22+bpo-amd64
–headers linux-headers-6.12.22+bpo-amd64

If the metapackages linux-image-amd64 and linux-headers-amd64 are passed with these options, I assume the release repo is chosen by default. (Gonna check that, though.)

Folder build_sources already comes with Debian backports folder. So no enabling of Debian backports is required.

Ok, that’s what you meant by referencing dist_build_sources_list_primary

So one could just create a custom sources list config, put it in build_sources and assign dist_build_sources_list_primary to it in buildconfig.d/25_apt_sources.conf

For custom repos I guess the key can be placed in build_sources like you guys do. When I use live-build the list and key always needed to be in config/archives/repo.list.chroot and config/archives/repo.key.chroot.

But building a bootable (Debian) ISO with GRUB + BIOS + EFI + Secure Boot + dracut + multiple architecture support very difficult. (Development of System Image Creation and Bootstrapping Tools) So we rely on Debian live-build for that.

I built something to that effect involving Whonix a while ago and considered sharing it here, but didn’t think it was practical enough.

Basically, I wrote a script that incorporates live-build to build the ISO, with all the libvirt, iptables and dnsmasq configuration already set up, some other minor security improvements like hardened-kernel etc. and put the compressed qcow2 gateway/workstation images in includes.chroot, so they’ll be included in the root filesystem and usable later on. ~ 2.5 GB total if I remember correctly. (Could probably make it smaller)

Then you boot the ISO with the toram flag, and use libvirt snapshots to make VM writes in the overlayfs. (Since the images will be read-only in squash upon boot).

That was my dirty solution for achieving an amnesiac Whonix ISO.
If you’re interested I can put together a repo on my github. It worked quite well, but definitely not an optimal solution.

To install additional packages, expanding the flavor_meta_packages_to_install variable using Build Variables Changes might be an option.

That reminds me, I was gonna do some stuff with the kicksecure-meta-packages. Totally forgot, but thankfully trixie hasn’t been moved to stable yet. I promise I’ll have that done before you guys make the jump.

There are a ton of SystemBuildTools - Debian Wiki, but probably none can replace Derivative-Maker - Debian based Linux Bootable Image Builder or are simple to customize. It’s a hard problem to solve.

The one I’ve spent the most time with is the Debian Live Manual.
Uuff, I remember going through some real pain with that thing, while learning live-build. Derivative-Maker combines the best of many of those tools, which is why I respect it so much, because I can imagine the amount of hours necessary to just nail down documentation, let alone setting all this up.

Tbh, it’s almost more impressive than the Whonix Project itself to me. The versatility of such a tool is quite limitless.

1 Like
5

That might work. No need to change sources list files - backports already enabled in default build sources.

But there’s a disadvantage you should be aware of: You probably won’t get kernel upgrades. Because you’re not properly installing linux-image-amd64 / linux-headers-amd64 - the unversioned (in package name) meta packages that are responsible to pull latest available kernel packages.

As a sysadmin, the correct APT commands would be:

sudo apt-get install -t bookworm-backports linux-image-amd64
sudo apt-get install -t bookworm-backports linux-headers-amd64

Translated to derivative-maker command line options that would be, in theory - but won’t work:

--kernel "-t bookworm-backports linux-image-amd64"
--headers "-t bookworm-backports linux-headers-amd64"

Would be unclean due to quoting and unquoting.

For VMs: Might not work.

build-steps.d/3500_install-packages

         for build_kernel in $BUILD_KERNEL_PKGS; do
            pkg-add-to-install-list "$build_kernel"
         done

For ISO: bug: --kernel and --headers are currently silently ignored at time of writing.

These command line options were not designed with backports in mind. Instead, what I had in mind, was when porting to new architectures --kernel linux-image-arm64 / --headers linux-headers-amd64. (But not needed, because actually auto detected.)

Can be anywhere.

Note:

grep -r backports build_sources

Possibly. Environment variable might work too. Or use your own buildconfig.d folder outside the derivative-maker repository as documented.

Yet another option: Add your own Chroot Script.

  - install_package_list: Specify additional custom packages for installation.
    Examples:
      install_package_list='gparted'
      install_package_list='gparted gedit'

This also won’t work to acquire kernel from backports, because:

      for install_package_item in $install_package_list ; do
         true "${cyan}INFO: install_package_item: '$install_package_item'${reset}"
         pkg-add-to-install-list "$install_package_item"
      done

I tried to provide a lot flexibility but to install kernels from backports, there is currently no great solution.

I am wondering how advanced users can be provided with the biggest potential flexibility that includes running injecting arbitrary commands (including installation from backports). Ideally, without needing to touch the derivative-maker folder. So ../buildconfig.d folder can be used.

Your own build-step is too difficult, too heavyweight?

Inside build-steps.d/3500_install-packages after:

   pkg-add-to-install-list "$meta_package_firmware"

   pkg-list-install

Maybe the ability to define to define a whole shell function that gets executed at that point? If the shell function exists, execute it. Otherwise, do nothing, proceed as usual.

If implemented, something like this might be possible:

install_packages_middle_custom_function() {
   chroot_run apt-get-noninteractive "${APTGETOPT[@]}" $apt_sourcelist_empty $apt_sourceparts $apt_unattended_opts --yes -t bookworm-backports linux-image-amd64 linux-headers-amd64
}

(You could then combine this with --kernel none and --headers none - because you’re installing kernel/header using custom shell function.)

1 Like
6

Translated to derivative-maker command line options that would be, in theory - but won’t work. Would be unclean due to quoting and unquoting.

Hmm you could maybe escape those or single quotes? Looks terrible either way though, lol.

These command line options were not designed with backports in mind. Instead, what I had in mind, was when porting to new architectures --kernel linux-image-arm64 / --headers linux-headers-amd64. (But not needed, because actually auto detected.)
Yeah that makes sense anyway. Backports kernel is out of scope in that regard, and should be added via custom option.

         for build_kernel in $BUILD_KERNEL_PKGS; do
            pkg-add-to-install-list "$build_kernel"
         done

I have a question about functions pkg-add-to-install-list and pkg-install.

Particularly, the first part of pkg-install:

   exception_handler_setup "exception_handler_unchroot_unmount" ERR INT TERM

   local pkg_install_item
   pkg_install_item="$1"
   local skip_package
   for skip_package in $dist_build_script_skip_package_install; do
      if [ "$skip_package" = "$pkg_install_item" ]; then
         unset skip_package
         true "${bold}${cyan}INFO: Skipping installation of '$pkg_install_item', because variable dist_build_script_skip_package_install includes it.${reset}"
         return 0
      fi
   done
   unset skip_package

   true "${cyan}INFO: Installing of '$@', because variable dist_build_script_skip_package_install does not include it... \
This may take a while...${reset}"

The variable $pkg_install_list contains a space separated list of packages which is/can be assembled with the function pkg-add-to-install-list

Executing the following,

pkg-install $pkg_install_list

doesn’t this only pass the first package in $pkg_install_list as an argument?

And without shifting arguments or looping, how are all items in the list checked against the skip variable via if [ "$skip_package" = "$pkg_install_item" ]?
(Especially if return is used upon a match)

Could $pkg_install_list be turned into an array to unset individual packages from the list if it matches an item in $dist_build_script_skip_package_install?:

pkg-add-to-install-list() {
.
.
.
.
.
   if [ "$pkg_install_debug" = "true" ]; then
      pkg-install "$pkg_install_item"
   else
      pkg_install_list+=("$pkg_install_item")
   fi
}

pkg-add-to-install-list() {
   exception_handler_setup "exception_handler_unchroot_unmount" ERR INT TERM

for i in ${!pkg_install_list[@]}; do

	pkg_install_item="${pkg_install_list[i]}"

	if [[ "$dist_build_script_skip_package_install" =~ "$pkg_install_item" ]]; then
		true "${bold}${cyan}INFO: Skipping installation of '$pkg_install_item', because variable dist_build_script_skip_package_install includes it.${reset}"
		unset pkg_install_list[$i]
		pkg_install_list=( "${pkg_install_list[@]}" )
	fi

done
1 Like
7

The right way to do this is Bash Proper Whitespace Handling but the syntax also looks harder. Would be more difficult for users to use the environment variable.

But in any case, I don’t see how --kernel option could support backports. Allowing writing -t backports there or an argument that contains spaces seems hard to parse and wrong.

No. It’s obviously functional and installing all the packages.

You could test this with a simple standalone test script.

The more correct way to do this would also be Bash Proper Whitespace Handling.

Yes, an array would be cleaner.

${!pkg_install_list[@]} needs to be quoted.

[[ unnecessary. Matching with =~ seems wrong. A simple if with = would suffice.

Should be quoted.

2 Likes
8

But in any case, I don’t see how --kernel option could support backports. Allowing writing -t backports there or an argument that contains spaces seems hard to parse and wrong.

No, you’re right. What I did yesterday was grabbing the package(s) with apt show and putting it in a variable with --kernel ${IMAGE} and --headers ${HEADERS} . Worked fine. It’s already working the way it should, I agree.

No. It’s obviously functional and installing all the packages.
You could test this with a simple standalone test script.
The more correct way to do this would also be Bash Proper Whitespace Handling.

Yup, wrong again. :smile: Just ran it. Every time we have an exchange I learn a bunch of new stuff. (At the expense of your time)

Maybe the ability to define to define a whole shell function that gets executed at that point? If the shell function exists, execute it. Otherwise, do nothing, proceed as usual.

install_packages_middle_custom_function() {
  chroot_run apt-get-noninteractive "${APTGETOPT[@]}" $apt_sourcelist_empty $apt_sourceparts $apt_unattended_opts --yes -t bookworm-backports linux-image-amd64 linux-headers-amd64
}

Right, that makes sense.

Maybe as an additional option?

--custom-repo bookworm-backports

[ -z "$REPO" ] || { install_packages_middle_custom_function() { chroot_run apt-get-noninteractive "${APTGETOPT[@]}" $apt_sourcelist_empty $apt_sourceparts $apt_unattended_opts --yes -t "$REPO" linux-image-amd64 linux-headers-amd64; }; install_packages_middle_custom_function; }

needs to be quoted. Should be quoted

You’re making me extremely self-conscious about quotes now. :smile:

[[ unnecessary. Matching with =~ seems wrong. A simple if with = would suffice.

Exposing my ignorance (further), but I always get a synthax error when matching =~ with single brackets.

1 Like
9

Maybe as an additional option?

--custom-repo bookworm-backports

What custom repo? Only Debian suites? We enable all Debian suites during
build anyhow.

1 Like
10

I meant an option to install packages from specific repos, like for example linux-image from backports.

Maybe --custom-packages makes more sense to describe that?

Something like this for example (could be cleaner).

[ -z "$CUSTOM_PACKS" ] || { install_packages_middle_custom_function()
{ mapfile -t CUSTOM_LIST < <(echo "${CUSTOM_PACKS}" | grep -Po "((?<=\:)[a-z0-9-_,]+|[a-z0-9-_]+:)" | sed -e "s|,| |; s|:||")

for i in $(seq 1 $((${#CUSTOM_LIST[@]}/2))); do

	chroot_run apt-get-noninteractive "${APTGETOPT[@]}" $apt_sourcelist_empty $apt_sourceparts $apt_unattended_opts --yes -t ${CUSTOM_LIST[0]} ${CUSTOM_LIST[1]};
	mapfile -t CUSTOM_LIST < <(printf '%s\n' "${CUSTOM_LIST[@]:2}");

done; }
install_packages_middle_custom_function; }

Here a test example that just echos the command.

Scheme:
<repo>:<package1>,<package2> <repo>:<package1>,<package2>
Option:
--custom-packages bookworm-backports:linux-image-amd64,linux-headers-amd64 sid:tmux,mpv 

CUSTOM_PACKS="bookworm-backports:linux-image-amd64,linux-headers-amd64 sid:tmux,mpv"

[ -z "$CUSTOM_PACKS" ] || { install_packages_middle_custom_function()
{ mapfile -t CUSTOM_LIST < <(echo "${CUSTOM_PACKS}" | grep -Po "((?<=\:)[a-z0-9-_,]+|[a-z0-9-_]+:)" | sed -e "s|,| |; s|:||")

for i in $(seq 1 $((${#CUSTOM_LIST[@]}/2))); do

	echo "chroot_run apt-get-noninteractive "${APTGETOPT[@]}" $apt_sourcelist_empty $apt_sourceparts $apt_unattended_opts --yes -t ${CUSTOM_LIST[0]} ${CUSTOM_LIST[1]}";
	mapfile -t CUSTOM_LIST < <(printf '%s\n' "${CUSTOM_LIST[@]:2}");

done; }
install_packages_middle_custom_function; }
1 Like
11

Looks too complex for the few people going to ever use this.

install_packages_middle_custom_function() seems simpler to implement and has best customization freedom.

2 Likes
12

Yeah sure, just an idea.

I think I’ll fork derivative-maker and go deep with it. I want to completely understand how it works and make some proper suggestions.

Hoping it’s cool to drop in if I have another question.

Thanks for your help, Patrick.

2 Likes
13

@Patrick hey man quick question if you don’t mind.

I’ve been updating my docker project that uses derivative-maker and noticed a weird build error with 17.3.9.4-developers-only saying that sgdisk is not installed. (full log)

Line: 74396

+ printf ' %s*%s Finished execution of grml-debootstrap. Enjoy your Debian system.\n' 'e[32;01m' 'e[0m'
 e[32;01m*e[0m Finished execution of grml-debootstrap. Enjoy your Debian system.
+ LAST_E_CMD=einfon
+ return 0
+ return 0
+ type_code=
+ case "$dist_build_target_arch" in
+ sudo --non-interactive --preserve-env=APTGETOPT_SERIALIZED,tbb_version,tb_onion,tpo_downloader_debug,tb_disable_anon_ws_dnf_conf,anon_shared_inst_tb,SKIP_SCRIPTS,SOURCE_DATE_EPOCH,dist_aptgetopt_file,dist_build_sources_list_primary,dist_mmdebstrap_build_sources_list_primary,dist_build_sources_list_primary_contents,dist_build_apt_sources_mirror,dist_build_apt_stable_release,dist_build_target_arch,dist_grml_mount_point,dist_source_help_steps_folder,dist_build_multiarch_package_item,dist_build_unsafe_io,dist_build_version,derivative_maker,user_name,LD_PRELOAD,LC_ALL,TZ,DEBDEBUG,XZ_OPT,REPO_PROXY,apt_unattended_opts,DERIVATIVE_APT_REPOSITORY_OPTS,DEBOOTSTRAP,http_proxy,https_proxy,ALL_PROXY,DEBIAN_FRONTEND,DEBIAN_PRIORITY,DEBCONF_NOWARNINGS,APT_LISTCHANGES_FRONTEND,INITRD,HOMEVAR_VBOX_TEMP sgdisk --typecode=3:4F68BCE3-E8CD-4DB1-96E7-FBCAF984B709 /home/user/derivative-binary/17.3.9.4/Whonix-Gateway-CLI-17.3.9.4.Intel_AMD64.raw
sudo: sgdisk: command not found
++ exception_handler_general ERR
++ last_failed_exit_code=1
++ last_failed_bash_command='$SUDO_TO_ROOT sgdisk --typecode='\''3:4F68BCE3-E8CD-4DB1-96E7-FBCAF984B709'\'' "$binary_image_raw_file"'

I also noticed that some kernel modules failed to load, but that’s probably unrelated.

Line: 69611

+ modprobe loop
modprobe: FATAL: Module loop not found in directory /lib/modules/6.12.22-amd64
+ true
+ losetup -f
+ modprobe dm-mod
modprobe: FATAL: Module dm-mod not found in directory /lib/modules/6.12.22-amd64

I’m gonna check the code and find where sgdisk is installed, but already suspecting that the error is really caused by something else.

1 Like
14

Missing build dependency.

sudo apt install gdisk

Fixed in git just now.

This is to be expected when using docker because there’s no kernel and no kernel modules running. As long as nothing is broken, this can be safely ignored.

Added to Expected Build Warnings just now.

2 Likes
15

This is to be expected when using docker because there’s no kernel and no kernel modules running. As long as nothing is broken, this can be safely ignored.

Thanks for the super quick reply and fixes!

From what I’ve read, device-mapper is a docker related issue.
What’s odd is that another build succeeded with the same settings and environment, so maybe it’s something else. I’m diff-ing build logs right now, but I think this particular error only occurs with the gateway.

+ stage prepare_vm
+ '[' -n '' ']'
+ grep -q done /var/cache/grml-debootstrap/stages_Whonix-Gateway-CLI-17.3.9.4.Intel_AMD64.raw/prepare_vm
+ prepare_vm
+ '[' -z 1 ']'
+ '[' -b /home/user/derivative-binary/17.3.9.4/Whonix-Gateway-CLI-17.3.9.4.Intel_AMD64.raw ']'
+ '[' '!' -b /home/user/derivative-binary/17.3.9.4/Whonix-Gateway-CLI-17.3.9.4.Intel_AMD64.raw ']'
+ '[' -z 1 ']'
+ modprobe loop
modprobe: FATAL: Module loop not found in directory /lib/modules/6.12.22-amd64
+ true
+ losetup -f
+ modprobe dm-mod
modprobe: FATAL: Module dm-mod not found in directory /lib/modules/6.12.22-amd64
+ true
+ grep -q device-mapper /proc/misc
+ eerror 'Device-mapper support missing in kernel.'
+ '[' '' '!=' yes ']'
+ '[' einfon = ebegin ']'
+ printf ' %s*%s Device-mapper support missing in kernel.\n' '' ''
 * Device-mapper support missing in kernel.
+ LAST_E_CMD=eerror
+ return 0
+ bailout 1
+ eend 1
+ local retval=1
+ shift
+ '[' 1 -gt 0 ']'
+ printf ' %s-> Failed (rc=%s)%s\n' '' 1 '' -> Failed (rc=1)

+ return 1
1 Like
16

That’s code inside grml-debootstrap.

2 Likes
17

Thanks man, got through a successful run now.

1.) sudo modprobe loop dm-mod @ host cures the device-mapper error. (Warnings to be expected nontheless, as you explained)

2.) Looks like /home/user/derivative-maker is expected. I was previously cloning derivative-maker into a folder named by the tag ~/17.3.9.4-developers-only

/usr/bin/dm-prepare-release: line 14: /home/user/derivative-maker/help-steps/pre: No such file or directory

Fixed with the following:
git clone --depth=1 --branch ${TAG} --jobs=4 --recurse-submodules --shallow-submodules https://github.com/Whonix/derivative-maker.git ~/derivative-maker

3.) This was a weird one, but I’ve always suspected tee to do funny things in the past.

Previous build command (that failed):

tbb_version=${TBB_VERSION}; build_cmd ${#FLAVOR[@]} ${BUILD_LOG} '/home/user/derivative-maker/derivative-maker --flavor ${FLAVOR[i]} 
--target ${TARGET} --arch ${ARCH} --repo ${REPO} --type ${TYPE} ${ONION} ${OPTS}' | tee -a ${BUILD_LOG}"

Fixed by directly piping stdout instead:

tbb_version=${TBB_VERSION}; build_cmd ${#FLAVOR[@]} ${BUILD_LOG} '/home/user/derivative-maker/derivative-maker --flavor ${FLAVOR[i]} 
--target ${TARGET} --arch ${ARCH} --repo ${REPO} --type ${TYPE} ${ONION} ${OPTS}' &>> ${BUILD_LOG}"
1 Like
18

3 posts were merged into an existing topic: derivative-maker fails with onion sources

19

Hi Patrick, I noticed something that I wanted to run by you (if you have the time).

Environments without systemd (docker for example (I’m building one without systemd now)), require that apt-cacher-ng is started/restarted directly, not via systemctl.

This seems to be handled in build-steps.d/1200_prepare-build-machine

   if [ "$APPLE_SILICON_BUILDER" = "true" ]; then
     ## Find the process and kill it if running.
     PID="$(pgrep -f "apt-cacher-ng")"

     if [ -n "$PID" ]; then
       $SUDO_TO_ROOT kill "$PID"
     fi

     ## There is no systemd inside docker. Hence, manually restarting apt-cacher-ng.
     $SUDO_TO_ROOT /usr/sbin/apt-cacher-ng -c /etc/apt-cacher-ng &

The problem is that in docker, with a USER + ENTRYPOINT configuration, apt-cacher has to be started with sudo & which launches multiple processes:

user@user:~$ ps -aux | grep apt-cacher-ng
root         855  0.0  0.0   5260  3532 pts/0    S    14:06   0:00 sudo /usr/sbin/apt-cacher-ng -c /etc/apt-cacher-ng
root         856  0.0  0.0   5260  1632 pts/1    Ss+  14:06   0:00 sudo /usr/sbin/apt-cacher-ng -c /etc/apt-cacher-ng
root         857  0.0  0.2  93404 11904 pts/1    Sl   14:06   0:00 /usr/sbin/apt-cacher-ng -c /etc/apt-cacher-ng

This causes the following command to fail, because there are multiple PIDs

$SUDO_TO_ROOT kill "$PID"

Could something like this be added to cover that case?

   if [ "$APPLE_SILICON_BUILDER" = "true" ]; then
     ## Find the process and kill it if running.
     PID="$(pgrep -f "apt-cacher-ng")"

     if [ -n "$PID" ]; then
       echo "$PID" | while read pid; do $SUDO_TO_ROOT kill "$pid" || true; done
     fi

     ## There is no systemd inside docker. Hence, manually restarting apt-cacher-ng.
     $SUDO_TO_ROOT /usr/sbin/apt-cacher-ng -c /etc/apt-cacher-ng &

Or maybe as a one-liner

pgrep -f "apt-cacher-ng"  | while read PID; do $SUDO_TO_ROOT kill "$PID" || true; done

I think with psmisc installed one could also just do $SUDO_TO_ROOT killall apt-cacher-ng

Tested with APPLE_SILICON_BUILDER=true ./derivative-maker .... and either worked.

Lastly, there is also /help-steps/apt-cacher-ng-reset where systemctl is handling apt-cacher-ng, but I don’t think it’s used anywhere so possibly irrelevant anyway.

1 Like
20

Nevermind :slight_smile:

If AllowUserPorts: 0 is simply added to /etc/apt-cacher-ng/acng.conf beforehand, then apt-cacher-ng_config_check firstcheck returns because there is no need to restart apt-cacher-ng.

Should’ve seen that earlier, thanks anyway!

1 Like