Docker Container that builds Whonix Images

Development
82

These git tags are a mistake. Deleted just now.

1 Like
83

Merged just now.

Some lower priority cleanup will be done later by @arraybolt3.

1 Like
84

Hey, cool!
Iā€™ll keep the fork to offer suggestions down the line, if needed.

I can see my accuracy and general coding style improving with projects like these.

If there is anything else I can help with in terms of derivative-maker or otherwise, please let me know. :+1:

1 Like
85
1 Like
86

Thank you! Merged.

1 Like
87

Sure, thank you! :+1:

Iā€™ll spend some time testing this now, and file more shortly.

1 Like
88

Hey Patrick, found one minor thing. Iā€™ll make a proper pull request including everything after testing is done.

run.sh

1,)

-*)
printf '%s\n' "$0: ERROR: unknown option: $1" >&2
exit 1
;;
  • -*) matches regular -- options like --flavor since not previously caught ā†’ removing and letting derivative-maker parse-cmd handle wrong options probably best solution.
while true; do
  case "$1" in
    -t|--tag)
      TAG="${2}"
      shift 2
    ;;
    --)
      shift
      break
      ;;
    *)
      break
      ;;

2.)

USER="user"
  • USER must be defined, because run.sh is executed on host and Docker ENV does not apply. (mixup before with start_build.sh)

3.)

  • The first sudo was supposed to give privileges to docker. If docker should be run non-sudo then this can be removed and user will be instructed to add to group docker.
  • Second sudo runs the CMD as user inside the container - fine with ā€œ$@ā€ but still needs sudo instruction
    docker \
      run \
      --name derivative-maker-docker \
      --interactive \
      --tty \
      --rm \
      --privileged \
      --env "TAG=${TAG}" \
      --env 'flavor_meta_packages_to_install=' \
      --env 'install_package_list=' \
      --env 'DERIVATIVE_APT_REPOSITORY_OPTS=' \
      --volume "${BUILDER_VOLUME}:/home/user/derivative-maker" \
      --volume "${CACHER_VOLUME}:/var/cache/apt-cacher-ng" "${IMG}" \
      sudo \
      --preserve-env \
      -u "${USER}" \
      -- \
      "${@}"
1 Like
89

Yes. Fixes are very much welcome. I did rather theoretical code security and best practices improvements and probably introduced new functionality issues while doing so.

Iā€™ve also added some TODO items in case you like to tackle these.

Should a new wiki page be created for derivative-maker docker support or any suggestions on how to document this?

1 Like
90

Ok, basically just fine tuning now. Iā€™ll also add the changes in derivative-maker then, like packages etc.

Yeah I saw that and added to the list.

Iā€™ll do all of that, no problem.

1 Like
91

https://github.com/derivative-maker/derivative-maker/pull/21#issuecomment-2950998344

tabletseeker:

tabletseeker left a comment (derivative-maker/derivative-maker#21)

If removal of /usr/bin/start_build.sh in docker run is intended, CMD ["/bin/bash"] in Dockerfile should probably switch to user and have WORKDIR /home/user

I cannot find this comment on github. Please re-post elsewhere.

1 Like
92

dockerifle: is is nice and minimal. I was wondering if the long RUN should be split into multiple commands or outsourced to a different script?

build.sh: self-explanatory.

run.sh vs start_build.sh: Dunno what the different use cases would be.

If removal of /usr/bin/start_build.sh in docker run is intended,

Probably not intended.

Dockerfile should probably switch to user and have WORKDIR /home/user

Not sure what is best.

1 Like
93

I also could not find this comment on github:

Maybe an additional argument -s|--shell in run.sh could determine whether the user wants to immediately execute start_build.sh on entry or run /bin/bash inside the container first?

Might be useful.

1 Like
94

Hey there, yes I just filed a new one here. Everything works. Iā€™ll note some minor stuff that I didnā€™t include later on.

Every RUN instruction creates a new layer, so itā€™s usually considered good practice to sum them up.

Youā€™re right though, putting that stuff in a script to be executed on the top layer will look much cleaner. Iā€™ll do that.

Your cleanup of run.sh already implements the correct logic.

You can simply run the build like so:

./run.sh --tag 17.4.0.9-developers-only /usr/bin/start_build.sh --flavor kicksecure-xfce --target iso --repo true --arch amd64

Or in case youā€™d like to execute a different command:

./run.sh /bin/bash

The way arguments are parsed --tag must be $1 or empty, though.
I did some stuff with for i; do so that the loop traverses all arguments once, allowing random order, but I didnā€™t include it in the pull. Yours is the cleanest possible way.

However, with --tag as the only option, the current while loop could be summed up with this:

[[ "$1" =~ ^(--tag|-t)$ ]] && \
{ TAG="$2"
  shift 2
}

while true; do
  case "$1" in
    -t|--tag)
      TAG="${2}"
      shift 2
      ;;
    --)
      shift
      break
      ;;
    *)
      break
      ;;
  esac
done

sudo \
  -- \
    docker \
      run \
      --name derivative-maker-docker \
      --interactive \
      --tty \
      --rm \
      --privileged \
      --env "TAG=${TAG}" \
      --env 'flavor_meta_packages_to_install=' \
      --env 'install_package_list=' \
      --env 'DERIVATIVE_APT_REPOSITORY_OPTS=' \
      --volume "${SOURCE_VOLUME}:/home/${DOCKER_USER}/derivative-maker" \
      --volume "${BINARY_VOLUME}:/home/${DOCKER_USER}/derivative-binary" \
      --volume "${CACHER_VOLUME}:/var/cache/apt-cacher-ng" "${IMG}" \
      sudo \
      --preserve-env \
      -u "${DOCKER_USER}" \
      -- \
      "${@}"

This would execute any command given to run.sh arguments as the user, so switching user is not needed.

      sudo \
      --preserve-env \
      -u "${DOCKER_USER}" \
      -- \
      "${@}"

What I meant was that the Dockerfile CMD instruction is overwritten by any command added to docker run.

So if docker run is executed without an additional command, then the container automatically executes /bin/bash in our case bc of CMD ["/bin/bash"]. But as root, since there is no USER instruction due to ENTRYPOINT

To circumvent this various solutions using sudo gosu etc. are used.

WORKDIR @ $HOME or $HOME/derivative-maker could be nice though. Iā€™ll add that too.

1 Like
95

removing git verify-commit because obsolete and requires gpg before derivative-maker

Would actually be cool if that was functional. Could we install gpg at the dockerfile level?

1 Like
96

Done. Just added it now.
Hereā€™s the key handling before git verify-commit "${TAG}^{commit}"

start_build.sh

FINGERPRINT="916B8D99C38EAF5E8ADC7A2A8D66066A2EEACCDA"
KEY="${SOURCE_DIR}/packages/kicksecure/repository-dist/usr/share/keyrings/derivative.asc"

  gpg --quiet --list-keys -- "${FINGERPRINT}" &>/dev/null || {
  gpg --keyid-format long --import --import-options show-only --with-fingerprint -- "${KEY}"
  gpg --import -- "${KEY}"
  gpg --check-sigs -- "${FINGERPRINT}"
} 2>&1 | tee -a -- "${KEY_LOG}"

As to use cases for start_build.sh and run.sh, the transition could be cleaner.

Default container behavior could be to launch start_build.sh via CMD and make use of command override via docker run. Thus if the user wants to execute something else, it can simply be appended to docker run

Otherwise

CMD ["/usr/bin/sudo", "--preserve-env", "-u ${USER}", "/usr/bin/start_build.sh"]

Arguments would have to be passed through a single --env variable in this case, though.

1 Like
97

Regarding wiki edit Wiki History - I donā€™t think we should discontinue support for non-docker builds.

98

This is getting really good and super clean.

Iā€™ve added more strict shell options. Untested.

other wishlist:

  • avoid /tmp:
    • use /usr/bin Instead (potentially future noexex, convention, maybe security)
  • rename scripts:
    • more descriptive, less generic script names, for example:
    • setup.sh ā†’ derivative-maker-docker-setup
  • add short 1-2 line summary comment below script copyright to explain what the script does

That might certainly be useful.

Thatā€™s quite a lengthy command to type. Perhaps something likeā€¦

derivative-maker-docker-wrapper --flavor kicksecure-xfce --target iso --repo true

Would be more easy and handy?

1 Like
99

You can already see that?
I was just messing around a bit, grabbing certain parts as templates.
(Forgot my pass on the other acc, now e-coin1)

Iā€™m thinking it could be placed right after Get Source Code @ 7.) Docker

Basic idea:

  • Intro:
    The use of a docker image to containerize the build process is also an available option.
    derivative-maker-docker already ships with the current source code, allowing for quick and simple deployment with pre-defined user scripts.

  • Script Overview

  • Possibly Dockerhub instructions with docker pull

  • Usage

  • Hints

Sound okay?
Edit: Any chance I can write this on Github and transfer it afterwards?

Yeah, thatā€™s fine. Iā€™ll test it after Iā€™m done with the new changes later.

Done. But the setup.sh will be inside the image then. In case you donā€™t want that, it could also just be removed after itā€™s done.

Roger that.

How about just putting this part in a function docker_run() and only passing /usr/bin/start_build.sh with args when "$#" -gt "1"?

sudo \
  -- \
    docker \
      run \
      --name derivative-maker-docker \
      --interactive \
      --tty \
      --rm \
      --privileged \
      --env "TAG=${TAG}" \
      --env 'flavor_meta_packages_to_install=' \
      --env 'install_package_list=' \
      --env 'DERIVATIVE_APT_REPOSITORY_OPTS=' \
      --volume "${SOURCE_VOLUME}:/home/${DOCKER_USER}/derivative-maker" \
      --volume "${BINARY_VOLUME}:/home/${DOCKER_USER}/derivative-binary" \
      --volume "${CACHER_VOLUME}:/var/cache/apt-cacher-ng" "${IMG}" \
      sudo \
      --preserve-env \
      -u "${DOCKER_USER}" \
      -- \
      "${@}"
1 Like
100

As for the wiki, I am not sure yet. Was wondering to use a dedicated wiki page but could result in maintenance issue as a lot content would be duplicated.

On the same wiki page, could make an existing complicated wiki page even more complicated.

Perhaps document it as you suggested and then Iā€™ll try to square the circle using tab controller(s).

Reference: Tab Template / Tab Content Controller

/wiki/Special:RecentChanges

Inside the docker image? Very much ok. Not an issue at all.

Might also be an option. Dunno what other commands one might want to inside the docker image. Perhaps specific build-steps.

Perhaps some shortcut command such as dm / derivative-maker?

1 Like
101

On a second thought it might not be possible to simplify this much more.

Just /usr/bin/start_build.sh will be difficult to remember and is somewhat surprising.

How aboutā€¦ To run derivative-makerā€¦

./run.sh --tag 17.4.0.9-developers-only -- ./derivative-maker --flavor kicksecure-xfce --target iso --repo true --arch amd64

Or to run a build-stepā€¦

./run.sh --tag 17.4.0.9-developers-only -- ./build-steps.d/1100_sanity-tests --flavor kicksecure-xfce --target iso --repo true --arch amd64

This would assume the working folder to be the root of the derivative-maker source code folder.

My goal is to adhere to the principle of the least astonishment, follow common convention, good usability, etc.

1 Like