derivative-maker fails with onion sources

Support
1

Good evening,

I recently forked and cleaned up this github project that builds whonix images in a docker container.
@ tabletseeker/whonix_docker

Everything works except onion sources with apt-cacher-ng.
Looks like an issue with apt-transport-tor because apt-cacher-ng works fine otherwise.

+ sudo --non-interactive --preserve-env=tbb_version,tb_onion,tpo_downloader_debug,tb_disable_anon_ws_dnf_conf,anon_shared_inst_tb,SKIP_SCRIPTS,SOURCE_DATE_EPOCH,dist_aptgetopt_file,dist_build_sources_list_primary,dist_mmdebstrap_build_sources_list_primary,dist_build_sources_list_primary_contents,dist_build_apt_sources_mirror,dist_build_apt_stable_release,dist_build_target_arch,dist_grml_mount_point,dist_source_help_steps_folder,dist_build_multiarch_package_item,dist_build_unsafe_io,dist_build_version,derivative_maker,user_name,LD_PRELOAD,LANG,LC_ALL,TZ,DEBDEBUG,XZ_OPT,REPO_PROXY,APTGETOPT,apt_unattended_opts,DERIVATIVE_APT_REPOSITORY_OPTS,DEBOOTSTRAP,http_proxy,https_proxy,ALL_PROXY,DEBIAN_FRONTEND,DEBIAN_PRIORITY,DEBCONF_NOWARNINGS,APT_LISTCHANGES_FRONTEND,INITRD apt-get -o Acquire::http::Proxy=http://127.0.0.1:3142 -o Acquire::https::Proxy=http://127.0.0.1:3142 -o Acquire::tor::Proxy=http://127.0.0.1:3142 -o APT::Update::Error-Mode=any -o Acquire::Languages=none -o Acquire::IndexTargets::deb::Contents-deb::DefaultEnabled=false -o Apt::Install-Recommends=false -o Acquire::Retries=5 -o Dpkg::Options::=--force-confnew -o Dir::Etc::sourcelist=/home/user/17.2.0.7-stable/build_sources/debian_stable_current_onion.list -o Dir::Etc::sourceparts=- update
Ign:1 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/debian-security bookworm-security InRelease
Ign:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-updates InRelease
Ign:3 tor+http://5phjdr2nmprmhdhw4fdqfxvpvt363jyoeppewju2oqllec7ymnolieyd.onion/debian bookworm-fasttrack InRelease
Ign:4 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Ign:5 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Ign:1 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/debian-security bookworm-security InRelease
Ign:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-updates InRelease
Ign:3 tor+http://5phjdr2nmprmhdhw4fdqfxvpvt363jyoeppewju2oqllec7ymnolieyd.onion/debian bookworm-fasttrack InRelease
Ign:4 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Ign:5 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Ign:1 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/debian-security bookworm-security InRelease
Ign:3 tor+http://5phjdr2nmprmhdhw4fdqfxvpvt363jyoeppewju2oqllec7ymnolieyd.onion/debian bookworm-fasttrack InRelease
Ign:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-updates InRelease
Ign:4 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Ign:5 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Ign:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-updates InRelease
Ign:1 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/debian-security bookworm-security InRelease
Ign:3 tor+http://5phjdr2nmprmhdhw4fdqfxvpvt363jyoeppewju2oqllec7ymnolieyd.onion/debian bookworm-fasttrack InRelease
Ign:4 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Ign:5 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Ign:1 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/debian-security bookworm-security InRelease
Ign:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-updates InRelease
Ign:3 tor+http://5phjdr2nmprmhdhw4fdqfxvpvt363jyoeppewju2oqllec7ymnolieyd.onion/debian bookworm-fasttrack InRelease
Ign:4 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Ign:5 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Err:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-updates InRelease
  503  Operation not permitted [IP: 127.0.0.1 3142]
Err:1 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/debian-security bookworm-security InRelease
  503  Operation not permitted [IP: 127.0.0.1 3142]
Err:3 tor+http://5phjdr2nmprmhdhw4fdqfxvpvt363jyoeppewju2oqllec7ymnolieyd.onion/debian bookworm-fasttrack InRelease
  503  Operation not permitted [IP: 127.0.0.1 3142]
Err:4 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
  503  Operation not permitted [IP: 127.0.0.1 3142]
Err:5 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
  503  Operation not permitted [IP: 127.0.0.1 3142]
Reading package lists...
E: Failed to fetch tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/debian-security/dists/bookworm-security/InRelease  503  Operation not permitted [IP: 127.0.0.1 3142]
E: Failed to fetch tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian/dists/bookworm-updates/InRelease  503  Operation not permitted [IP: 127.0.0.1 3142]
E: Failed to fetch tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian/dists/bookworm-backports/InRelease  503  Operation not permitted [IP: 127.0.0.1 3142]
E: Failed to fetch tor+http://5phjdr2nmprmhdhw4fdqfxvpvt363jyoeppewju2oqllec7ymnolieyd.onion/debian/dists/bookworm-fasttrack/InRelease  503  Operation not permitted [IP: 127.0.0.1 3142]
E: Failed to fetch tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian/dists/bookworm/InRelease  503  Operation not permitted [IP: 127.0.0.1 3142]
E: Some index files failed to download. They have been ignored, or old ones used instead.
++ exception_handler_general ERR
++ last_failed_exit_code=100
++ last_failed_bash_command='$SUDO_TO_ROOT apt-get ${APTGETOPT[@]} -o Dir::Etc::sourcelist="$dist_build_sources_list_primary" -o Dir::Etc::sourceparts="-" update'
++ output_cmd_set
++ '[' -o xtrace ']'
++ output_cmd=true
++ true 'INFO: Middle of function exception_handler_general of ././build-steps.d/1200_prepare-build-machine.'
++ exception_handler_process_shared ERR
++ last_script=././build-steps.d/1200_prepare-build-machine
++ trap_signal_type_previous=
++ '[' '' = '' ']'
++ trap_signal_type_previous=unset
++ trap_signal_type_last=ERR
++ dist_build_error_counter=1
+++ benchmarktimeend 1723038846
++++ date +%s
+++ benchmarktimeend=1723038877
+++ benchmark_took_seconds=31
++++ convertsecs 31
++++ local h m s
++++ (( h=31/3600 ))
++++ true
++++ (( m=(31%3600)/60 ))
++++ true
++++ (( s=31%60 ))
++++ printf '%02d:%02d:%02d\n' 0 0 31
+++ echo 00:00:31
++ benchmark_took_time=00:00:31
++ local first
++ read -r first _
++ process_backtrace_function
++ true 'INFO: BEGIN: process_backtrace_function'
++ '[' -o xtrace ']'
++ set +x
++ true 'INFO: END  : process_backtrace_function'
++ function_trace_function
++ true 'INFO: BEGIN: function_trace_function'
++ '[' -o xtrace ']'
++ set +x
++ true 'INFO: END  : function_trace_function'
++ output_cmd_set
++ '[' -o xtrace ']'
++ output_cmd=true
++ true '
############################################################
ERROR detected in script!: ././build-steps.d/1200_prepare-build-machine

#####
User Help Message 2/2:

Please READ this message carefully.

Copying/pasting/screenshotting this box alone will not be insightful, and no help can be provided with it alone as it may not contain sufficient information by itself.

In many instances, providing a longer segment above this box or the entire log may be necessary for an effective diagnosis.
#####

dist_build_version: 17.2.0.7
dist_build_error_counter: 1
benchmark: 00:00:31
last_failed_exit_code: 100
trap_signal_type_previous: unset
trap_signal_type_last    : ERR

process_backtrace_result:
1: : init
2: : /bin/bash -exc source /etc/docker-entrypoint-cmd 
3: : /bin/bash /starter.sh 
7:timestamp '\''Git Start'\'' ~/logs/git.log; [ -d ~/17.2.0.7-stable ] || { cd ~/ && git clone --depth=1 --branch 17.2.0.7-stable --jobs=4 --recurse-submodules --shallow-submodules https://github.com/Whonix/derivative-maker.git 17.2.0.7-stable &>> ~/logs/git.log; }; 
8:}; [ -f ~/derivative.asc ] || { wget https://www.whonix.org/keys/derivative.asc -O ~/derivative.asc && gpg --keyid-format long --import --import-options show-only --with-fingerprint ~/derivative.asc && gpg --import ~/derivative.asc && gpg --check-sigs 916B8D99C38EAF5E8ADC7A2A8D66066A2EEACCDA; } &> ~/logs/key.log; 
17:${1} Time: $(date +'\''%H:%M:%S'\'')
20: : sudo -u user /bin/bash -c timestamp () 
24:timestamp '\''Git Start'\'' ~/logs/git.log; [ -d ~/17.2.0.7-stable ] || { cd ~/ && git clone --depth=1 --branch 17.2.0.7-stable --jobs=4 --recurse-submodules --shallow-submodules https://github.com/Whonix/derivative-maker.git 17.2.0.7-stable &>> ~/logs/git.log; }; 
25:}; [ -f ~/derivative.asc ] || { wget https://www.whonix.org/keys/derivative.asc -O ~/derivative.asc && gpg --keyid-format long --import --import-options show-only --with-fingerprint ~/derivative.asc && gpg --import ~/derivative.asc && gpg --check-sigs 916B8D99C38EAF5E8ADC7A2A8D66066A2EEACCDA; } &> ~/logs/key.log; 
34:${1} Time: $(date +'\''%H:%M:%S'\'')
37: : sudo -u user /bin/bash -c timestamp () 
41:timestamp '\''Git Start'\'' ~/logs/git.log; [ -d ~/17.2.0.7-stable ] || { cd ~/ && git clone --depth=1 --branch 17.2.0.7-stable --jobs=4 --recurse-submodules --shallow-submodules https://github.com/Whonix/derivative-maker.git 17.2.0.7-stable &>> ~/logs/git.log; }; 
42:}; [ -f ~/derivative.asc ] || { wget https://www.whonix.org/keys/derivative.asc -O ~/derivative.asc && gpg --keyid-format long --import --import-options show-only --with-fingerprint ~/derivative.asc && gpg --import ~/derivative.asc && gpg --check-sigs 916B8D99C38EAF5E8ADC7A2A8D66066A2EEACCDA; } &> ~/logs/key.log; 
51:${1} Time: $(date +'\''%H:%M:%S'\'')
54: : /bin/bash -c timestamp () 
55: : /bin/bash /home/user/17.2.0.7-stable/derivative-maker --flavor whonix-workstation-cli --target raw --arch amd64 --repo true --type vm --connection onion --report true --verifiable true --freshness current --retry-max 5 
56: : /bin/bash ././build-steps.d/1200_prepare-build-machine --flavor whonix-workstation-cli --target raw --arch amd64 --repo true --type vm --connection onion --report true --verifiable true --freshness current --retry-max 5 

function_trace_result:
main (line number: 500)
main (line number: 488)
build_machine_setup (line number: 85)
exception_handler_general (line number: 85)
exception_handler_process_shared (line number: 85)

last_failed_bash_command: $SUDO_TO_ROOT apt-get ${APTGETOPT[@]} -o Dir::Etc::sourcelist="$dist_build_sources_list_primary" -o Dir::Etc::sourceparts="-" update
############################################################

Building on the host yields the same error.
Any idea what might be causing this?

2

Here are the setup commands for apt-cache-ng:

#50_user.conf
cat > /lib/systemd/system/apt-cacher-ng.service.d/50_user.conf << EOF
[Service]
ExecStart=
ExecStart=torsocks /usr/sbin/apt-cacher-ng SocketPath=/run/apt-cacher-ng/socket -c /etc/apt-cacher-ng ForeGround=1
EOF

#installs supporting packages
apt-get install -y torsocks tor apt-transport-tor

#installs bookworm version 3.7.4-1 
echo no | apt-get install -y apt-cacher-ng && \	
chmod 777 /var/cache/apt-cacher-ng

#apt-cacher-ng config
cat >> /etc/apt-cacher-ng/acng.conf << EOF
PassThroughPattern: .*
BindAddress: localhost
SocketPath: /run/apt-cacher-ng/socket
Port:3142
Proxy: http://127.0.0.1:3142
AllowUserPorts: 0
EOF

#30user config
echo "Acquire::BlockDotOnion \"false\";" > /etc/apt/apt.conf.d/30user && \

#starting services
systemctl daemon-reload && systemctl start tor.service && \
systemctl restart apt-cacher-ng.service
1 Like
3

This needs some fixes in documentation.

4

Hi I will update the wiki no problem but those settings and commands don’t seem to be enough for onion sources to work. Is something missing?

5

I assumed the commands provided would fix it. If that is not the case, back to square one.

It is broken and remains broken until someone finds out how to fix it.

Makes no sense because then nothing is actually cached?

6

I got that from resonat0r git. It’s supposed to ensure the proxy accepts any source. Could make sense with onion I thought.

In order for SSL/TLS to work, apt-cacher-ng has to be told beforehand which domains it can CONNECT to via the PassThroughPattern: option in /etc/apt-cacher-ng/acng.conf. For instance, to allow apt-cacher-ng to proxy anything, you can do: PassThroughPattern: .*

Ok so I will try to figure this out then and come back. I’m gonna compile it now and try every option.

1 Like
7

The first thing to figure out how to use Debian + onion repositories + apt-cacher-ng.

That is, while excluding derivative-maker for now. Once that has been figured out, we can look into fixing derivative-maker.