I left the || out during the key check. Would it make sense to return 1 or do smth if [ -f ${KEY} ] fails because there is no key present? (this part actually has to run during derivative-maker execution, or at least after gpg-agent is installed ā maybe during prepare-build-machine ?)
git verify checks the tag if env TAG is set, otherwise it assumes master
no more loops or functions needed; arguments passed directly to build command
CMD only executes /bin/bash now ā this is overwritten by the docker run command, meaning if the user does not pass a command with docker run, /bin/bash is automatically executed instead.
However, there are no errors, because this now successfully dismounts the loop devices.
+ true 'INFO: kpartx -d -s -v failed to unmount all loop devices, attempting to unmount...'
+ sudo --non-interactive --preserve-env=APTGETOPT kpartx -d -s -v /dev/loop4
sudo dmsetup info
No devices found
Effectively, what happens is that with each build, /dev/loop[0-9]+ will simply be incremented by +1, potentially creating a load of loop devices, but there wonāt be any errors at least.
Thanks, man!
I updated my dirty fix with your --associated flag, thatās very nice.
I also fixed some minor things with the docker stuff.
For example this was necessary instead of -c to prevent another subshell, or gpg would error with /dev/tty0 No such file or directory:
Does derivative-maker only runs kpartx -d -s -v /dev/loop4 but not for /dev/loop3 etc.? If so, that would probably be a bash code level bug that Iād be very much interested to fix
Could you provide that part of the debug output of unmount-raw step please?
Please remove any hacks. Consider leftover loop devices my task. Post merge Iāll unmount them in unmount-raw.
/home/user/derivative-maker ${@:1:$(($#-1))} 2>&1 | tee -a ${BUILD_LOG}; set ā ${@: -1}; exec ā$@ā
This seems unnecessarily complex.
Full command should just be: home/user/derivative-maker "$@"
${@:1:$(($#-1))}
Whatās that for? Simply "$@" should do?
| tee -a ${BUILD_LOG};
Not a job for docker?
For users, if someone wanted redirect output to the terminal, that is easy:
your-command &>/path/to/log/file
Bash feature. Redirects both stdout and stderr.
If log handling should be modified such as always creating a log file, then please post a derivative-maker feature request.
git checkout --recurse-submodules ${TAG};
Is this still needed? Should docker handle git tag checkout?
loop_devices_output="$($SUDO_TO_ROOT losetup --associated "$img" --noheadings --output NAME)"
mapfile -t loop_devices_list <<< "$loop_devices_output"
for loop_devices_item in "${loop_devices_list[@]}" ; do
## Check if the block device still exists.
if test -b "$loop_devices_item" ; then
true "INFO: kpartx -d -s -v failed to unmount all loop devices, attempting to unmount..."
## Inside docker (and maybe other corner cases) 'kpartx -d -s -v "$img"' might fail to unmount loop devices.
## https://forums.whonix.org/t/docker-container-that-builds-whonix-images/17494/43
$SUDO_TO_ROOT kpartx -d -s -v "$loop_devices_item"
$SUDO_TO_ROOT losetup -nl -O name | grep -w "$loop_devices_item" && \
$SUDO_TO_ROOT losetup -d "$loop_devices_item" || true
fi
done
No it catches every single one, thatās working fine.
Apparently, both kpartx -d and losetup -d are necessary in this scenario.
Iām gonna read up on some documentation to understand why.
+ true 'INFO: Start of unmount_raw. Running losetup --all for debugging before running kpartx.'
+ sudo --non-interactive --preserve-env=APTGETOPT_SERIALIZED,tbb_version,tb_onion,tpo_downloader_debug,tb_disable_anon_ws_dnf_conf,anon_shared_inst_tb,SKIP_SCRIPTS,SOURCE_DATE_EPOCH,dist_aptgetopt_file,dist_build_sources_list_primary,dist_mmdebstrap_build_sources_list_primary,dist_build_sources_list_primary_contents,dist_build_apt_sources_mirror,dist_build_apt_stable_release,dist_build_target_arch,dist_grml_mount_point,dist_source_help_steps_folder,dist_build_multiarch_package_item,dist_build_unsafe_io,dist_build_version,derivative_maker,user_name,LD_PRELOAD,LC_ALL,TZ,DEBDEBUG,XZ_OPT,REPO_PROXY,apt_unattended_opts,DERIVATIVE_APT_REPOSITORY_OPTS,DEBOOTSTRAP,http_proxy,https_proxy,ALL_PROXY,DEBIAN_FRONTEND,DEBIAN_PRIORITY,DEBCONF_NOWARNINGS,APT_LISTCHANGES_FRONTEND,INITRD,HOMEVAR_VBOX_TEMP losetup --all
/dev/loop1: [65027]:1217898 (/home/user/derivative-binary/17.4.0.3/Whonix-Gateway-CLI-17.4.0.3.Intel_AMD64.raw)
/dev/loop0: [65027]:1217898 (/home/user/derivative-binary/17.4.0.3/Whonix-Gateway-CLI-17.4.0.3.Intel_AMD64.raw)
+ '[' '' = true ']'
+ /home/user/17.4.0.3-developers-only/help-steps/unmount-helper /home/user/derivative-binary/Whonix-Gateway-CLI_image
umount: /home/user/derivative-binary/Whonix-Gateway-CLI_image (/dev/mapper/loop1p3) unmounted
INFO: Script /home/user/17.4.0.3-developers-only/help-steps/unmount-helper completed. Exit Code: 0. Errors Detected: 0. Execution Time: 00:00:01
+ '[' '' = '' ']'
+ local img=/home/user/derivative-binary/17.4.0.3/Whonix-Gateway-CLI-17.4.0.3.Intel_AMD64.raw
+ wait 289323
+ sleep 2
+ sync
+ sudo --non-interactive --preserve-env=APTGETOPT_SERIALIZED,tbb_version,tb_onion,tpo_downloader_debug,tb_disable_anon_ws_dnf_conf,anon_shared_inst_tb,SKIP_SCRIPTS,SOURCE_DATE_EPOCH,dist_aptgetopt_file,dist_build_sources_list_primary,dist_mmdebstrap_build_sources_list_primary,dist_build_sources_list_primary_contents,dist_build_apt_sources_mirror,dist_build_apt_stable_release,dist_build_target_arch,dist_grml_mount_point,dist_source_help_steps_folder,dist_build_multiarch_package_item,dist_build_unsafe_io,dist_build_version,derivative_maker,user_name,LD_PRELOAD,LC_ALL,TZ,DEBDEBUG,XZ_OPT,REPO_PROXY,apt_unattended_opts,DERIVATIVE_APT_REPOSITORY_OPTS,DEBOOTSTRAP,http_proxy,https_proxy,ALL_PROXY,DEBIAN_FRONTEND,DEBIAN_PRIORITY,DEBCONF_NOWARNINGS,APT_LISTCHANGES_FRONTEND,INITRD,HOMEVAR_VBOX_TEMP kpartx -d -s -v /home/user/derivative-binary/17.4.0.3/Whonix-Gateway-CLI-17.4.0.3.Intel_AMD64.raw
+ sync
+ local loop_devices_output loop_devices_list loop_devices_item
++ sudo --non-interactive --preserve-env=APTGETOPT_SERIALIZED,tbb_version,tb_onion,tpo_downloader_debug,tb_disable_anon_ws_dnf_conf,anon_shared_inst_tb,SKIP_SCRIPTS,SOURCE_DATE_EPOCH,dist_aptgetopt_file,dist_build_sources_list_primary,dist_mmdebstrap_build_sources_list_primary,dist_build_sources_list_primary_contents,dist_build_apt_sources_mirror,dist_build_apt_stable_release,dist_build_target_arch,dist_grml_mount_point,dist_source_help_steps_folder,dist_build_multiarch_package_item,dist_build_unsafe_io,dist_build_version,derivative_maker,user_name,LD_PRELOAD,LC_ALL,TZ,DEBDEBUG,XZ_OPT,REPO_PROXY,apt_unattended_opts,DERIVATIVE_APT_REPOSITORY_OPTS,DEBOOTSTRAP,http_proxy,https_proxy,ALL_PROXY,DEBIAN_FRONTEND,DEBIAN_PRIORITY,DEBCONF_NOWARNINGS,APT_LISTCHANGES_FRONTEND,INITRD,HOMEVAR_VBOX_TEMP losetup --associated /home/user/derivative-binary/17.4.0.3/Whonix-Gateway-CLI-17.4.0.3.Intel_AMD64.raw --noheadings --output NAME
+ loop_devices_output='/dev/loop1
/dev/loop0'
+ mapfile -t loop_devices_list
+ for loop_devices_item in "${loop_devices_list[@]}"
+ test -b /dev/loop1
+ true 'INFO: kpartx -d -s -v failed to unmount all loop devices, attempting to unmount...'
+ sudo --non-interactive --preserve-env=APTGETOPT_SERIALIZED,tbb_version,tb_onion,tpo_downloader_debug,tb_disable_anon_ws_dnf_conf,anon_shared_inst_tb,SKIP_SCRIPTS,SOURCE_DATE_EPOCH,dist_aptgetopt_file,dist_build_sources_list_primary,dist_mmdebstrap_build_sources_list_primary,dist_build_sources_list_primary_contents,dist_build_apt_sources_mirror,dist_build_apt_stable_release,dist_build_target_arch,dist_grml_mount_point,dist_source_help_steps_folder,dist_build_multiarch_package_item,dist_build_unsafe_io,dist_build_version,derivative_maker,user_name,LD_PRELOAD,LC_ALL,TZ,DEBDEBUG,XZ_OPT,REPO_PROXY,apt_unattended_opts,DERIVATIVE_APT_REPOSITORY_OPTS,DEBOOTSTRAP,http_proxy,https_proxy,ALL_PROXY,DEBIAN_FRONTEND,DEBIAN_PRIORITY,DEBCONF_NOWARNINGS,APT_LISTCHANGES_FRONTEND,INITRD,HOMEVAR_VBOX_TEMP kpartx -d -s -v /dev/loop1
del devmap : loop1p1
del devmap : loop1p2
del devmap : loop1p3
+ for loop_devices_item in "${loop_devices_list[@]}"
+ test -b /dev/loop0
+ true 'INFO: kpartx -d -s -v failed to unmount all loop devices, attempting to unmount...'
+ sudo --non-interactive --preserve-env=APTGETOPT_SERIALIZED,tbb_version,tb_onion,tpo_downloader_debug,tb_disable_anon_ws_dnf_conf,anon_shared_inst_tb,SKIP_SCRIPTS,SOURCE_DATE_EPOCH,dist_aptgetopt_file,dist_build_sources_list_primary,dist_mmdebstrap_build_sources_list_primary,dist_build_sources_list_primary_contents,dist_build_apt_sources_mirror,dist_build_apt_stable_release,dist_build_target_arch,dist_grml_mount_point,dist_source_help_steps_folder,dist_build_multiarch_package_item,dist_build_unsafe_io,dist_build_version,derivative_maker,user_name,LD_PRELOAD,LC_ALL,TZ,DEBDEBUG,XZ_OPT,REPO_PROXY,apt_unattended_opts,DERIVATIVE_APT_REPOSITORY_OPTS,DEBOOTSTRAP,http_proxy,https_proxy,ALL_PROXY,DEBIAN_FRONTEND,DEBIAN_PRIORITY,DEBCONF_NOWARNINGS,APT_LISTCHANGES_FRONTEND,INITRD,HOMEVAR_VBOX_TEMP kpartx -d -s -v /dev/loop0
del devmap : loop0p1
del devmap : loop0p2
del devmap : loop0p3
+ true 'INFO: Middle of unmount_raw. Running losetup --all for debugging after kpartx -d (Delete partition mappings.)'
+ sudo --non-interactive --preserve-env=APTGETOPT_SERIALIZED,tbb_version,tb_onion,tpo_downloader_debug,tb_disable_anon_ws_dnf_conf,anon_shared_inst_tb,SKIP_SCRIPTS,SOURCE_DATE_EPOCH,dist_aptgetopt_file,dist_build_sources_list_primary,dist_mmdebstrap_build_sources_list_primary,dist_build_sources_list_primary_contents,dist_build_apt_sources_mirror,dist_build_apt_stable_release,dist_build_target_arch,dist_grml_mount_point,dist_source_help_steps_folder,dist_build_multiarch_package_item,dist_build_unsafe_io,dist_build_version,derivative_maker,user_name,LD_PRELOAD,LC_ALL,TZ,DEBDEBUG,XZ_OPT,REPO_PROXY,apt_unattended_opts,DERIVATIVE_APT_REPOSITORY_OPTS,DEBOOTSTRAP,http_proxy,https_proxy,ALL_PROXY,DEBIAN_FRONTEND,DEBIAN_PRIORITY,DEBCONF_NOWARNINGS,APT_LISTCHANGES_FRONTEND,INITRD,HOMEVAR_VBOX_TEMP losetup --all
/dev/loop1: [65027]:1217898 (/home/user/derivative-binary/17.4.0.3/Whonix-Gateway-CLI-17.4.0.3.Intel_AMD64.raw)
/dev/loop0: [65027]:1217898 (/home/user/derivative-binary/17.4.0.3/Whonix-Gateway-CLI-17.4.0.3.Intel_AMD64.raw)
+ sync
+ '[' '' = true ']'
+ test -d /home/user/derivative-binary/Whonix-Gateway-CLI_image
+ ls -la /home/user/derivative-binary/Whonix-Gateway-CLI_image
total 8
drwxr-xr-x 2 user user 4096 May 28 13:03 .
drwxr-xr-x 8 user user 4096 May 28 13:09 ..
+ '[' '' = true ']'
+ sudo --non-interactive --preserve-env=APTGETOPT_SERIALIZED,tbb_version,tb_onion,tpo_downloader_debug,tb_disable_anon_ws_dnf_conf,anon_shared_inst_tb,SKIP_SCRIPTS,SOURCE_DATE_EPOCH,dist_aptgetopt_file,dist_build_sources_list_primary,dist_mmdebstrap_build_sources_list_primary,dist_build_sources_list_primary_contents,dist_build_apt_sources_mirror,dist_build_apt_stable_release,dist_build_target_arch,dist_grml_mount_point,dist_source_help_steps_folder,dist_build_multiarch_package_item,dist_build_unsafe_io,dist_build_version,derivative_maker,user_name,LD_PRELOAD,LC_ALL,TZ,DEBDEBUG,XZ_OPT,REPO_PROXY,apt_unattended_opts,DERIVATIVE_APT_REPOSITORY_OPTS,DEBOOTSTRAP,http_proxy,https_proxy,ALL_PROXY,DEBIAN_FRONTEND,DEBIAN_PRIORITY,DEBCONF_NOWARNINGS,APT_LISTCHANGES_FRONTEND,INITRD,HOMEVAR_VBOX_TEMP rmdir /home/user/derivative-binary/Whonix-Gateway-CLI_image
+ sync
+ exithandler
+ local exit_code=0
+ '[' '!' 0 = 0 ']'
++ benchmarktimeend 1748437779
+++ date +%s
++ benchmarktimeend=1748437782
++ benchmark_took_seconds=3
+++ convertsecs 3
+++ local h m s
+++ (( h=3/3600 ))
+++ true
+++ (( m=(3%3600)/60 ))
+++ true
+++ (( s=3%60 ))
+++ printf '%02d:%02d:%02d\n' 0 0 3
++ echo 00:00:03
+ benchmark_took_time=00:00:03
+ output_cmd_set
+ '[' -o xtrace ']'
+ output_cmd=true
+ '[' 0 = 0 ']'
+ true 'INFO: Script /home/user/17.4.0.3-developers-only/help-steps/unmount-raw completed. Exit Code: 0. Errors Detected: 0. Execution Time: 00:00:03'
+ exit 0
Each round with stray from previous:
true 'INFO: Start of unmount_raw. Running losetup --all for debugging before running kpartx.'
+ sudo --non-interactive --preserve-env=APTGETOPT_SERIALIZED,tbb_version,tb_onion,tpo_downloader_debug,tb_disable_anon_ws_dnf_conf,anon_shared_inst_tb,SKIP_SCRIPTS,SOURCE_DATE_EPOCH,dist_aptgetopt_file,dist_build_sources_list_primary,dist_mmdebstrap_build_sources_list_primary,dist_build_sources_list_primary_contents,dist_build_apt_sources_mirror,dist_build_apt_stable_release,dist_build_target_arch,dist_grml_mount_point,dist_source_help_steps_folder,dist_build_multiarch_package_item,dist_build_unsafe_io,dist_build_version,derivative_maker,user_name,LD_PRELOAD,LC_ALL,TZ,DEBDEBUG,XZ_OPT,REPO_PROXY,apt_unattended_opts,DERIVATIVE_APT_REPOSITORY_OPTS,DEBOOTSTRAP,http_proxy,https_proxy,ALL_PROXY,DEBIAN_FRONTEND,DEBIAN_PRIORITY,DEBCONF_NOWARNINGS,APT_LISTCHANGES_FRONTEND,INITRD,HOMEVAR_VBOX_TEMP losetup --all
/dev/loop1: [65027]:1217898 (/home/user/derivative-binary/17.4.0.3/Whonix-Gateway-CLI-17.4.0.3.Intel_AMD64.raw)
/dev/loop2: [65027]:1217898 (/home/user/derivative-binary/17.4.0.3/Whonix-Gateway-CLI-17.4.0.3.Intel_AMD64.raw)
/dev/loop0: [65027]:1217898 (/home/user/derivative-binary/17.4.0.3/Whonix-Gateway-CLI-17.4.0.3.Intel_AMD64.raw)
Yeah sure, thatās a remnant from the other stuff.
Scrapping.
Nah thatās fine, just my own nonsense.
I wanted terminal output and simultaneous pipe to a log with stderr, because of constant errors (that I was causing). your-command &>/path/to/log/file leaves empty terminal output I believe.
Yeah youāre right. Most of the stuff like this that I am doing, is already done on derivative-maker level anyway, but way superior and more sophisticated.
Your code on key checking for example, is already perfect anyway but you donāt even mention it.
I like how you use silence as a way to educate, because when I find out after having to look myself the learning experience is twice as effective.
Yeah, my thinking was that (lets assume /dev/loop0) if kpartx -d -s -v /dev/loop0 fails to remove only then losetup -d /dev/loop0 is executed.
#attempts to remove loop device /dev/loop0
$SUDO_TO_ROOT kpartx -d -s -v "$loop_devices_item"
#checks if /dev/loop0 is still there
$SUDO_TO_ROOT losetup -nl -O name | grep -w "$loop_devices_item" && \
#executes losetup -d only if grep exit 0 or true so no exit 1 on fail
$SUDO_TO_ROOT losetup -d "$loop_devices_item" || true
Awesome, thanks man!
Thatās totally it, nice find.
You werenāt experiencing this at all though, right?
Thatās why I thought it was exclusive to docker, but apparently not I guess.
Not really, I just thought it might be cool because those are some of the more useful env variables used by derivative-maker. Could be handy, no?
Yeah sure, was really just a placeholder anyway to check if it works.
But apart from that, should the user be expected to type in every single option? (including tag)
How would this be automated?
The pre-existing options_list files could be interesting. So for example a āKVMā preset with default ramsize,vmsize,qcow2 etc. and same for the rest.
Yeah, leftovers.
sudo dmsetup info -c
Name Maj Min Stat Open Targ Event UUID
loop0p1 252 0 L--w 0 1 0 part1-devnode_7:0_Wh5pYvM
loop0p2 252 1 L--w 0 1 0 part2-devnode_7:0_Wh5pYvM
loop0p3 252 2 L--w 0 1 0 part3-devnode_7:0_Wh5pYvM
Yes, of course. I know debootsrap from live-build but this looks like a different animal.
Iāll first read some docs, because Iām genuinely interested.
Will report back!
Iām only imagining how many bugs youāve discovered and worked around or fixed lol. I love this project, thereās just an insane amount of work and the coolest parts of some many tools stuffed into it.
There is a check after kdpartx -d which greps for the corresponding loop device/part, verifying whether itās been removed and executing kpartx -d /dev/loop if it hasnāt.
Looking at grml-deboostrap, this is confirmed. (Lines 326 - 334)
if [ -n "${ORIG_TARGET}" ] ; then
einfo "Removing loopback mount of file ${ORIG_TARGET}."
kpartx -d "${ORIG_TARGET}" || eend $?
# Workaround for a bug in kpartx which doesn't clean up properly,
# see Debian Bug #891077 and Github-PR grml/grml-debootstrap#112
if dmsetup ls | grep -q "^${LOOP_PART} "; then
kpartx -d "/dev/${LOOP_DISK}" >/dev/null || eend $?
fi
fi
Yes, after removing the space grep exits 0 and the if works.
if dmsetup ls | grep -q "^${LOOP_PART}"; then
kpartx -d "/dev/${LOOP_DISK}" >/dev/null || eend $?
fi
Sure, Iāll report it. Your changes in derivative-maker work btw. No stray loop devices whatsoever.
Iāll finish with the derivative-maker-docker (new name) stuff today or tomorrow and report back.
@arraybolt3 Thanks for the great suggestions. I added you as a collaborator in case you wanted to modify things directly. Itās pretty much done now, only minor things left.
