Docker Container that builds Whonix Images

Patrick · May 22, 2025, 5:56pm

Alternative test:

./build-steps.d/1100_sanity-tests --target virtualbox --flavor whonix-gateway-cli
./build-steps.d/1200_prepare-build-machine --target virtualbox --flavor whonix-gateway-cli
./build-steps.d/3200_create-raw-image --target virtualbox --flavor whonix-gateway-cli
./help-steps/mount-raw --target virtualbox --flavor whonix-gateway-cli
./help-steps/unmount-raw --target virtualbox --flavor whonix-gateway-cli

I’ve also completed a full build and checked sudo losetup --all. No stray loop devices.

It’s really important to cleanly mount and unmount. But I cannot find any bugs.

e-coin · May 23, 2025, 12:05pm

Hey Patrick, yes this.

I’ve run some more tests in the container and can confirm that is the problem.

./build-steps.d/3200_create-raw-image --target virtualbox --flavor whonix-gateway-cli
./build-steps.d/4400_zerofree-raw --target virtualbox --flavor whonix-gateway-cli

losetup --all
/dev/loop1: [65027]:1228234 (/home/user/derivative-binary/17.4.0.2/Whonix-Gateway-CLI-17.4.0.2.Intel_AMD64.raw)
/dev/loop0: [65027]:1228234 (/home/user/derivative-binary/17.4.0.2/Whonix-Gateway-CLI-17.4.0.2.Intel_AMD64.raw)

It’s kpartx that fails to remove the loop device.

kpartx -asv /home/user/derivative-binary/17.4.0.2/Whonix-Gateway-CLI-17.4.0.2.Intel_AMD64.raw
add map loop3p1 (252:7): 0 204800 linear 7:3 2048
add map loop3p2 (252:8): 0 2048 linear 7:3 206848
add map loop3p3 (252:9): 0 209504256 linear 7:3 208896

losetup --all
/dev/loop1: [65027]:1228234 (/home/user/derivative-binary/17.4.0.2/Whonix-Gateway-CLI-17.4.0.2.Intel_AMD64.raw)

kpartx -d -s -v /home/user/derivative-binary/17.4.0.2/Whonix-Gateway-CLI-17.4.0.2.Intel_AMD64.raw; echo $?
0

losetup --all
/dev/loop1: [65027]:1228234 (/home/user/derivative-binary/17.4.0.2/Whonix-Gateway-CLI-17.4.0.2.Intel_AMD64.raw)

The loop device needs to be targeted directly to remove the mount.

kpartx -dsv /dev/loop1
del devmap : loop1p1
del devmap : loop1p2
del devmap : loop1p3

This is accomplishes the same as dmsetup remove

But that’s the only way to completely remove it, after the dismount.

losetup -d /dev/loop1
losetup --all

kpartx -d against the raw seems to do nothing. Whether that’s docker related or maybe a kernel specific issue, I’m not sure.

uname -a 
Linux 3a5a4cah2c01 6.12.22-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.22-1 (2025-04-10) x86_64 GNU/Linux

If you have the time, could you maybe run the container once?

I would like to confirm if I’m going crazy or not.

Anyone would be welcome to share a test run, actually. There’ve been a bunch of clones but no feedback

Just one tiny thing if you do, comment line 20 in entrypoint.sh/

#lo_check; \

Patrick · May 23, 2025, 1:21pm

Latest git tag now includes code to attempt to clean up stray loop devices. Untested.

The best way to test this would be to contribute a minimal dockerfile to derivative-maker/derivative-maker.

It should do as little as possible even if not yet functional.

And then this could hopefully be tested on the CI.

For example, all of the following either should not be needed or if needed, it needs to be done inside the build-steps to keep any docker specific code to a minimum for maintainability.

|  |/run/apt-cacher-ng ${APT_CACHER_NG_CACHE_DIR} ${APT_CACHER_NG_LOG_DIR} && \|
|---|---|
||### setup permissions ###|
||chown -R ${APT_CACHER_USER}:${APT_CACHER_USER} /run/apt-cacher-ng \|
||${APT_CACHER_NG_LOG_DIR} && \|
||chown -R ${APT_CACHER_USER}:0 ${APT_CACHER_NG_CACHE_DIR} && \|
||chmod -R 0755 /run/apt-cacher-ng ${APT_CACHER_NG_CACHE_DIR} \|
||${APT_CACHER_NG_LOG_DIR} && \|

Speaking of maintainability and simplify… I think it would be better if the docker image had systemd so all of the apt-cacher-ng docker specific code can be avoided (and we stay agile and can move to another APT cacher at some point in the future, if one exists). Or for any other systemd units that might be needed.

Docker systemd would be most helpful if you could figure that out.

Anything else such as:

sed -i "s|http|https|g" /etc/apt/sources.list.d/debian.sources && \

A feature request could be posted to do this in derivative-maker always for all builds, not only docker.

e-coin · May 23, 2025, 9:14pm

Sure, I created a new branch for you.
Everything works, including systemd and minimal setup, but it’s almost 300MB.

Should I make a pull request?

Maybe in derivative-maker/docker and then derivative-maker/docker/apple?

Patrick · May 24, 2025, 11:52am

Pull request is optional. It would allow maybe easier line by line discussion.

It needs to become more minimal. What I mean by that…

This:

Patrick:

Anything else such as:
sed -i "s|http|https|g" /etc/apt/sources.list.d/debian.sources && \
A feature request could be posted to do this in derivative-maker always for all builds, not only docker.

APT_CACHER_NG_CACHE_DIR=/var/cache/apt-cacher-ng

Removable now thanks to systemd usage?

||apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y -t bookworm \|
|---|---|
||systemd systemd-sysv dbus dbus-user-session git time curl lsb-release fakeroot dpkg-dev \|
||fasttrack-archive-keyring apt-utils wget procps gpg gpg-agent debian-keyring sudo adduser \|
||apt-transport-https ca-certificates torsocks tor apt-transport-tor dmsetup apt-cacher-ng && \|

Could you reduce please to the same packages as listed here? Host Preparation Steps

At time of writing:

sudo apt install git time curl apt-cacher-ng lsb-release fakeroot dpkg-dev fasttrack-archive-keyring safe-rm

Plus of course any packages absolutely required by docker.

Any packages you think are missing, should either be added to the wiki or preferably to derivative-maker.

Rationale: I want to keep the delta (difference) between docker and non-docker as small as possible for simplicity. So any improvements should be upstreamed to derivative-maker generally (unspecific to docker).

For instance: apt-utils procps gpg gpg-agent debian-keyring

These packages should either:

A) not be needed
B) already a dependency of derivative-maker
C) if missing and needed, that’s a bug that should be fixed in derivative-maker directly (not inside docker)

DNS related changes should not be done without defining the way forward for the upstream ticket: Use DNSCrypt by default in Kicksecure? (not Whonix!)
(related wiki page: DNS Security)

||rm -rf /var/lib/apt/lists/* /var/cache/apt/* /tmp/* /var/tmp/* && \|
|---|---|
||rm -f /lib/systemd/system/multi-user.target.wants/* && \|
||rm -f /etc/systemd/system/*.wants/* && \|
||rm -f /lib/systemd/system/local-fs.target.wants/* && \|
||rm -f /lib/systemd/system/sockets.target.wants/*udev* && \|
||rm -f /lib/systemd/system/sockets.target.wants/*initctl* && \|
||rm -f /lib/systemd/system/basic.target.wants/* && \|
||rm -f /lib/systemd/system/anaconda.target.wants/* && \|
||rm -f /lib/systemd/system/plymouth* && \|
||rm -f /lib/systemd/system/systemd-update-utmp*|

I don’t understand why that would be needed.

e-coin · May 24, 2025, 4:39pm

It needs to become more minimal. What I mean by that…

Yeah, that sed has been removed. Obsolete anyway.

sed -i "s|http|https|g" /etc/apt/sources.list.d/debian.sources &&

This one is necessary to implement trixie sources for dnscrypt-proxy. It’s been omitted from bookworm. (But if you don’t want dnscrypt-proxy this can be removed as well)

sed -i '0,/bookworm/ s/bookworm/bookworm trixie/' /etc/apt/sources.list.d/debian.sources

Removable now thanks to systemd usage?

Sure, can be removed. I read something about docker unionfs VOLUME > --volume and /var/cache with double instructions being helpful, but that’s probably nonsense anyway.

Could you reduce please to the same packages as listed here? Host Preparation Steps

Yeah, will do.

Plus of course any packages absolutely required by docker.
Any packages you think are missing, should either be added to the wiki or preferably to derivative-maker.
Plus of course any packages absolutely required by docker.

Ok sure, that makes sense. Couple packages can also be scrapped bc of reverse dependency.

Absolutely crucial for docker:

systemd: dbus dbus-user-session

Convenient to be installed/initialized before/with entrypoint:

user: adduser
apt-cacher: apt-cacher-ng
dnscrypt: ca-certificates dnscrypt-proxy
if onion: tor

Whenever:

residual: apt-utils procps gpg gpg-agent debian-keyring torsocks apt-transport-tor …etc

Most likely obsolete

dmsetup: (my lo_check nonsense - which you obviously don’t want lol)
Haven’t tested your kpartx commit yet, but it’s probably gonna work anyway.

Rationale: I want to keep the delta (difference) between docker and non-docker as small as possible for simplicity. So any improvements should be upstreamed to derivative-maker generally (unspecific to docker).

Yeah, makes sense. As raw as humanly possible at the docker level, got it.

DNS related changes should not be done without defining the way forward for the upstream ticket: Use DNSCrypt by default in Kicksecure? (not Whonix!)
(related wiki page: DNS Security)

Your position was that dnscrypt should be scrapped due to non-selfvalidating dnssec. For Kicksecure I think it’s still pretty nice, though.

I personally like having a tail of the query.log and iftop running during a build, it just feels nice. Is it really necessary? Probably, not. It sounds cool in a readme and I like the flare, but in terms of actual functionality or privacy it’s meh. I’ll accept your judgement in that regard.

I don’t understand why that would be needed.

This can save a bit. It’s all in the same RUN, i.e layer, thus will take effect. (as far as I know)

/var/lib/apt/lists/* /var/cache/apt/*

Preemptive. For potential builds I always choose WORKDIR /tmp, but can be scrapped, sure.

/tmp/*

Honestly, that’s a copy/paste. Never checked, but I’ll figure out which ones are reasonable.

||rm -f /lib/systemd/system/multi-user.target.wants/* && \|
||rm -f /etc/systemd/system/*.wants/* && \|
||rm -f /lib/systemd/system/local-fs.target.wants/* && \|
||rm -f /lib/systemd/system/sockets.target.wants/*udev* && \|
||rm -f /lib/systemd/system/sockets.target.wants/*initctl* && \|
||rm -f /lib/systemd/system/basic.target.wants/* && \|
||rm -f /lib/systemd/system/anaconda.target.wants/* && \|
||rm -f /lib/systemd/system/plymouth* && \|
||rm -f /lib/systemd/system/systemd-update-utmp*|

Patrick · May 24, 2025, 5:02pm

Once ready the dockerfile should be added to derivative-maker? No need to have it in a separate repository?

In that case, please fork derivative-maker and add minimal, essential files to derivative-maker. A sub folder docker would probably be good if multiple files are required. However, if convention and suitable, the dockerfile itself could be in the root of the derivative-maker source code folder.

Replied here:

Please drop all of that. Can later be re-introduced at derivative-maker level.

e-coin · May 25, 2025, 10:15am

Yeah, I called it derivative-docker and removed all my labels. You can determine the naming and that stuff later, I just put that in as a placeholder.

Yes, I put everything in for now, but most will obviously get scrapped.

systemd should probably be initialized with ENTRYPOINT instead of how I do it right now.

CMD ["/bin/bash", "-c", "/usr/bin/entrypoint.sh /usr/bin/start_services.sh /usr/bin/su ${USER} --command '/usr/bin/start_build.sh'"]

Since services will be started at derivative-maker level, start_services.sh is obsolete. Same goes for the rest.

start_services.sh

SERVICES=("apt-cacher-ng" "dnscrypt-proxy")
LOG_DIR="${HOME}/logs"

[ ! ${CONNECTION} = "onion" ] || SERVICES+=("tor")

[ -d ${LOG_DIR} ] || { mkdir -p ${LOG_DIR}; chown -R ${USER}:${USER} ${LOG_DIR}; }

systemctl restart ${SERVICES[@]}

echo 'Waiting for services to start...'
sleep 5

systemctl status ${SERVICES[@]}

exec "$@"

I can integrate some of the code from start_build.sh and run.sh for another pull, but it’ll probably be more convenient on github now.

Patrick · May 25, 2025, 1:34pm

Any reason to call add /apple/ into the file path? It’s not really limited to Apple?

e-coin:

systemd should probably be initialized with ENTRYPOINT instead of how I do it right now.
CMD ["/bin/bash", "-c", "/usr/bin/entrypoint.sh /usr/bin/start_services.sh /usr/bin/su ${USER} --command '/usr/bin/start_build.sh'"]

Yes, please. Please do as that is usually done with docker.

Please kindly reduce further

remove DNSCrypt
/etc/apt-cacher-ng/acng.conf - unless absolutely unavoidable and not being able to implement at derivative-maker level
/etc/tor/torrc - same as above

e-coin · May 25, 2025, 3:45pm

Done.

With ENTRYPOINT entrypoint.sh will be run regardless of the docker run appended command, which normally overwrites aCMD instruction. Following CMD are passed in $@

I’ll have to adjust ExecStopPost accordingly so that Ctrl+C doesn’t freeze.

ExecStopPost=/bin/bash -ec "if echo \${EXIT_STATUS} | grep [A-Z] > /dev/null; then echo >&2 \"got signal \${EXIT_STATUS}\"; systemctl exit \$(( 128 + \$( kill -l \${EXIT_STATUS} ) )); else systemctl exit \${EXIT_STATUS}; fi"

Personally, I like when /bin/bash is executed after the build finishes and the container doesn’t just exit.

Do you want Ctrl + C to exit or restart?

Patrick · May 25, 2025, 4:26pm

What’s the convention usually for docker?

Patrick · May 25, 2025, 4:34pm

Why is git clone still needed? Full source code is already available?

e-coin · May 25, 2025, 5:17pm

Exit, I guess.

In the event of an error, it’s nice to enter the shell after aborting derivative-maker to make changes, instead of exiting completely. Although, I believe to have seen an Enter shell option in the error dialog once. (Haven’t tested it, though) If that works then exit is fine, though.

I’m assuming a terminal editor is probably already installed during prepare-build-machine or smth.

Yeah, I’ll start trimming all the redundant stuff now. Just wanted to have everything up so you could have a full picture before I do and commit history is complete.

Do you prefer we discuss things here, so that others can follow, or should I post comments over there from now on?

Patrick · May 25, 2025, 5:25pm

Looks promising.

I am not aware of that.

Appreciated.

Probably best to stay away from the Microsoft platform (GitHub) where comments can vanish.

e-coin · May 26, 2025, 8:23am

I’ll start rewriting some stuff today.

run.sh

Most can be scrapped apart from volume assignment.

Edit: ../ doesn’t work, must be absolute. BUILDER_VOLUME=$(dirname $PWD) for example.

BUILDER_VOLUME="../"
CACHER_VOLUME="/var/cache/apt-cacher-ng"

Basically, it’ll work like so:

user clones derivative-maker
enters derivative-maker/docker
executes run.sh (which will also build the image if not docker images | grep -q "derivative-docker")
BUILDER_VOLUME mounts clone location
Assuming that $PWD is $HOME/derivative-maker/docker which would make it ../ or maybe use find
docker run with start_build.sh

Do you have a preference for environment variable defaults and the amount of total variables that is offered?

sudo docker run --name derivative-docker -it --rm --privileged \
	--env "tbb_version=${TOR}" \
	--env 'FLAVOR=whonix-gateway-cli whonix-workstation-cli' \
	--env 'TARGET=qcow2' \
	--env 'ARCH=amd64' \
	--env 'TYPE=vm' \
	--env 'CONNECTION=clearnet' \
	--env 'CLEAN=false' \
	--env 'REPO=false' \
 	--env 'OPTS=' \
	--env 'REPO_PROXY=http://127.0.0.1:3142' \
	--env 'APT_CACHER_ARGS=' \
	--volume ${BUILDER_VOLUME}:/home/user \
	--volume ${CACHER_VOLUME}:/var/cache/apt-cacher-ng \
	--dns 127.0.2.1 ${IMG}

That should work. I do like the idea of just docker pulling and the clone occurring automatically based on tag choice too, though. Maybe derivative-maker could have a dockerhub profile for such an image?

Patrick · May 26, 2025, 12:27pm

Your way of starting systemd, is that the docker conventional way to do that?

derivative-binary folder seems more suitable for non-source code, binary files, temporary files, mounts, build results?

	--env 'FLAVOR=whonix-gateway-cli whonix-workstation-cli' \
	--env 'TARGET=qcow2' \
	--env 'ARCH=amd64' \
	--env 'TYPE=vm' \
	--env 'CONNECTION=clearnet' \
	--env 'CLEAN=false' \
	--env 'REPO=false' \
 	--env 'OPTS=' \
	--env 'REPO_PROXY=http://127.0.0.1:3142' \
	--env 'APT_CACHER_ARGS=' \

Why do we need those anyhow?

Better to use command line options than environment variables. These are much less likely to silently break if renamed.

Can we keep it unspecific to Whonix vs Kicksecure as well as unspecific to CLI vs GUI or do we need to hardcode?

	--env "tbb_version=${TOR}" \

Should not be auto-detected. If there was a safe way to autodetect it, we’d need to implement that in derivative-maker / tb-udpater.

[ -f ~/derivative.asc ] || { wget https://www.whonix.org/keys/derivative.asc -O ~/derivative.asc; \

Can use ./packages/kicksecure/repository-dist/usr/share/keyrings/derivative.asc?

Can be considered. I haven’t used that yet.

What is its purpose?

So I’ll create an account and then?

e-coin · May 26, 2025, 1:56pm

I don’t think there is one. systemd in docker is considered counterintuitive and thus quite frowned upon. The script I use was sourced from an older repo, I’ll check if I can find it.

There are different ways, though.
I could look into alternative methods, if you’re not happy with the current one.
Like this for example. Not as efficient as a simple entry script and may require additional packages, stages etc, but maybe looks a bit cleaner?

FROM debian:bookworm

ENV container=docker \
    DEBIAN_FRONTEND=noninteractive

RUN INSTALL_PKGS='findutils iproute2 python3 python3-apt sudo systemd' \
    && apt-get update && apt-get install $INSTALL_PKGS -y --no-install-recommends \
    && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

RUN find /etc/systemd/system \
    /lib/systemd/system \
    -path '*.wants/*' \
    -not -name '*journald*' \
    -not -name '*systemd-tmpfiles*' \
    -not -name '*systemd-user-sessions*' \
    -print0 | xargs -0 rm -vf

VOLUME [ "/sys/fs/cgroup" ]

ENTRYPOINT [ "/lib/systemd/systemd" ]

Originally, I used this method because I wanted to play around with arrays.
For example read -a FLAVOR <<< "$FLAVOR" FLAVOR is transformed into an array in start_build.sh, and the build command loops based on how many arguments you pass to ${FLAVOR}. So, the number of indexes ${#FLAVOR[@]}

build_cmd() { for ((i=0;i<${1};i++)); do timestamp 'Build Start' ${2}; \
/home/user/derivative-maker \
--flavor ${FLAVOR[i]} \
--target ${TARGET} \
--arch ${ARCH} \
--type ${TYPE} \
--connection ${CONNECTION} \
--repo ${REPO} \
${OPTS}; timestamp 'Build End' ${2}; done; }

Sure, the arguments can be passed to the container in a variety of different ways.
How do you like having an options file in derivative-maker/docker, or wherever you’d prefer, that contains a list of arguments with default values? Then we simply invoke that file in the build command. Seems a lot cleaner and no need for variables.

read -a OPTIONS <<< $(cat derivative-maker/docker/options_list)

build_cmd() { for ((i=0;i<${1};i++)); do timestamp 'Build Start' ${2}; \
/home/user/derivative-maker ${OPTIONS[@]}; timestamp 'Build End' ${2}; done; }

Yeah sure. For example, if you like the options_list idea, there could be different kind of options files for specific builds, or an ambiguous one to keep it unspecific.
A simple if in start_build.sh could select which file, based on a single boolean --env variable or whatever.

Sure, that’s just a remnant of testing anyway. I used to do this, but I’m sure that’s inadequate.

tbb_version=$(curl -s https://aus1.torproject.org/torbrowser/update_3/release/download-linux-x86_64.json | jq -r '.version')

Gonna scrap it.

Done, thanks for pointing that out.

Then you can upload your images with docker push and others can download them with docker pull. Or docker run with your image name.

I’m doing this with whonix_builder for example:

docker pull tabletseeker/whonix_builder:latest

Patrick · May 26, 2025, 2:11pm

No need.

Since I am not much into docker, I actually given a choice will prefer the version without docker specific commands. RUN etc.

Keeping that layer minimal will help me later to move things from docker to derivative-maker, if feasiable.

That is one option. Can we just pass all using command line as if using ./derivative-maker --...?

Because otherwise it seems we’ll hardcode a certain default and then need to tell users to change the default config file or provide their own?

Ok, something like that.

e-coin:

Then you can upload your images with docker push and others can download them with docker pull. Or docker run with your image name.

I’m doing this with whonix_builder for example:
docker pull tabletseeker/whonix_builder:latest

That sounds good.

What changes would a derivative-maker image have compared to its base, Debian image?

Or do you mean providing a Kicksecure docker image on dockerhub, a Whonix-Gateway image and a Whonix-Workstaiton image? CLI and seaprate Xfce versions?

e-coin · May 26, 2025, 3:36pm

OK, I see what you mean.

You can almost do it that way, as far as I know.

With docker run you can pass a command like this for example:

docker run debian:bookworm -it /bin/bash

In our case, ENTRYPOINT will execute entrypoint.sh first and every following CMD is appended in $@.

You could pass the derivative-maker arguments to start_build.sh through a CMD instruction:

CMD ["/bin/bash", "-c", "/usr/bin/su ${USER} --command '/usr/bin/start_build.sh --flavor whonix-gateway-cli --type vm --connection onion'"]

Or with a docker run appended command which overwrites CMD instructions anyway (but we use entrypoint now)

sudo docker run --name derivative-docker -it --rm --privileged \
	--volume ${BUILDER_VOLUME}:/home/user \
	--volume ${CACHER_VOLUME}:/var/cache/apt-cacher-ng ${IMG} /bin/bash -c "/usr/bin/su ${USER} --command '/usr/bin/start_build.sh --flavor whonix-gateway-cli --type vm --connection onion'"

Then in start_build.sh you just execute like so

build_cmd() { for ((i=0;i<${1};i++)); do timestamp 'Build Start' ${2}; \
/home/user/derivative-maker ${@}; timestamp 'Build End' ${2}; done; }

Do you want that loop in there btw?
The basic idea was to have sequential builds, for example if you want to build gateway and workstation, you just pass both flavors.

That’s a no then.
I get what you mean by unwittingly locking defaults down, though. Makes sense.

In this case the derivative-maker image which is built via docker build comes with the packages, start_build.sh, entrypoint.sh and labels. This docker image, ~200MB can be uploaded to dockerhub with docker push

So if you were to docker pull derivative-maker/derivative-docker:latest, you could just use the customized docker run command that passes specific arguments to start_build.sh and produce whatever image you like (Whonix-Workstaiton image, Whonix-Gateway image …etc)

Patrick · May 26, 2025, 3:56pm

Sounds good!

No. If we want loops, these should be implemented in derivative-maker.