Thanks kinda redid the script a bit now so it does verify the gpg key, But still want some comment if this is okay?
so same as last time it work by just passing the tag as a argument. And instead of being a bash function for bashrc its now i script. So you can either make a bash alias to it and run it that way or put it into /usr/bin whatever you prefer.
#!/bin/bash
# check if argument is not missing
if [ -z "$1" ]; then
echo "Error: Missing whonix version. Please provide an whonix version."
exit 1
fi
# set options
set -o nounset
set -e
set -x
# variables
WHONIX_VERSION="$1"
VERSION_NUMBER="${WHONIX_VERSION%%-*}"
# Functions
function cleanup() {
# clean up setup
rm -rf "$HOME"/derivative-maker
rm -rf "$HOME"/derivative-binary
rm -rf "$HOME"/whonix-binary
rm -rf "$HOME"/build-log
touch "$HOME"/build-log
}
function update_install() {
# updates/upgrades and autoremove packages
echo "Starting whonix build for version $WHONIX_VERSION"
sudo apt update -y
sudo apt upgrade -y
sudo apt autoremove -y
# install dependencies
local packages=("git" "time" "curl" "apt-cacher-ng" "lsb-release" "fakeroot" "fasttrack-archive-keyring")
# Loop through each package
for package in "${packages[@]}"; do
if ! dpkg -l "$package" &> /dev/null; then
echo "$package is not installed. Installing..."
sudo apt install -y "$package" 2>&1 | tee -a ~/build-log
else
echo "$package is already installed."
fi
done
}
function repo_down() {
# download git repository
echo "Git clone whonix version $WHONIX_VERSION"
git clone --depth=1 --branch "$WHONIX_VERSION" --jobs=4 --recurse-submodules --shallow-submodules https://github.com/Whonix/derivative-maker.git
cd "$HOME"/derivative-maker
# verifies correct version is downloaded
GIT_VERSION="$(git describe --tags --abbrev=0)"
if [ "$GIT_VERSION" == "$WHONIX_VERSION" ]; then
echo "Git version matches the desired version ($WHONIX_VERSION)."
else
echo "Git version does not match the desired version ($WHONIX_VERSION)."
exit 2
fi
}
function verify_tag() {
# verifies git tag with gpg keyring
cd "$HOME"/derivative-maker
# Verify if the current directory is a git repository
if git rev-parse --git-dir > /dev/null 2>&1; then
if git verify-tag "$WHONIX_VERSION"; then
echo "Commit verification succeeded."
else
echo "Commit verification failed."
# Handle the failure case, e.g., exit the script or perform some recovery actions
exit 4
fi
else
echo "Not a git repository. Please run this script within a git repository."
exit 3
fi
}
function build_whonix() {
# build the project
cd "$HOME"/derivative-maker
echo "Building gateway"
~/derivative-maker/derivative-maker --flavor whonix-gateway-xfce --target utm --arch arm64 --tb open --repo true --vmsize 15G
echo "Building workstation"
~/derivative-maker/derivative-maker --flavor whonix-workstation-xfce --target utm --arch arm64 --tb open --repo true --vmsize 25G
}
function pack_whonix() {
# cp binary tar.gz files into another directory for easy access
mkdir "$HOME"/whonix-binary
cp -vr "$HOME"/derivative-binary/"$VERSION_NUMBER"/*.utm.tar.gz "$HOME"/whonix-binary
echo "Build of whonix version $WHONIX_VERSION is finished"
}
# starts scripts
echo "Starting whonix build function"
cd "$HOME"
cleanup
# everything from here on will be included in build-log
update_install 2>&1 | tee -a ~/build-log
repo_down 2>&1 | tee -a ~/build-log
verify_tag 2>&1 | tee -a ~/build-log
build_whonix 2>&1 | tee -a ~/build-log
pack_whonix 2>&1 | tee -a ~/build-log
# cp build log to whonix folder and completing the script
cp "$HOME"/build-log "$HOME"/whonix-binary/
echo "Build function finished, please check log"
cd "$HOME"
exit 0
So this is what i have made so far. This basically make it so you only need to download and install a current debian wm. Make a user and add it with sudo privileges with no password needed. Then add the gpg keys manually (as that is safest). Then all you need to do to build whonix for mac with apple silicon is to call on this script with the tag as a argument.
$ whonix 17.1.1.8-developers-only
Do you find this script good enough as a user/developer script? and is it okay to publish this on my own github as a open sourced script? Personally i just want to make process of building this for myself and others easier.
And if anything goes wrong you got the build log to share? (even tough it might be massive doing it this way). But do wanna know what people thinks? as i am still learning how to do bash scripting and so on.
PS: I also recommend to run this script while running a session of tmux or screen. Just so you do not loose your progress in case of ssh connection loss or anything.
Automating any kind of gpg verification, git verification is very difficult to withstand actual attacks. I am not sure if there can be cases where you think you checked out a specific git tag but actually you’re on a different git tag or git branch.
One can run git verify-tag some-tag, which would succeed, while git currently being at some different commit, branch, tag.
Therefore no steps/commands from the build documentation on git digital software verification can be considered optional.
Automating any kind of gpg verification isn’t simple. So I don’t want to bless it stable or secure.
Here I would prefer to keep less levels of if.
if git rev-parse --git-dir > /dev/null 2>&1; then
# ...
exit 3
The less nested ifs the better. The earlier you can close a level of if the better.
derivative-maker isn’t Whonix-only. So I also don’t like any hardcoded Whonix only strings in any scripts.
I find this unclean, painful to watch. It’s like you don’t really know how to use git and use a big hammer instead.
Not the kind of quality that I want to endorse. Rather I would prefer to see: fetch new commits from git, view the diff, then locally merge.
I see, as i did say this is something i want to improve. So of course i did not expect this to be perfect at the first try. As i am still a beginner at this. So i see this as a learning experience. I will think a bit how to do gpg vertification on my script for a bit, as making it verified with the imported gpg key would be best. Maybe find the latest git commit or tag, find the signing key used then compare that with the imported key and check it that way? idk. Again still learning.
When it comes to the very harsh way i dealt with the whole repository. Well i know its unclean. But to implement a cleaner way would be a lot more work. First i would need to make a check if its even is there, if it has the right tag installed. Then i would need to implement a way to fetch/pull it all down including submodules and so on. Would be a lot more of what is a simple script that just build whonix for mac would be for a newbie like myself. But i can take on the challenge to implement that. If you are interested in me doing that? again i am still learning. But hey thanks for the comments. At least i am learning about this now.
Well can start with some of the simpler stuff as the hardcoded whonix strings and put them up as a variable. And some of the other stuff. Hope its okay to post about this here?
oh its a new thread? well thats cool. I will continue with posts about the script here. Again by this week i will have made some updates on it.
Also not sure if i will do the automatic verification, but rather have a output with a text directing to the whonix wiki Signing_Key page. So people can do that after? That means also its not required to have imported the key but a option. This can save me some time and headache. But i can include
as a output on the build-log so that its included there at least? This is a user-script so do kinda wanna hear what people wanna know.
what i will fix when i got the time is how i deal with the git repository as a whole and using variables so no whonix string is hard coded in the script.
Hopefully this script can be considered a simpler way of building the project for beginners/intermediate linux users. Advance users should still be able to do this manually.
Another thing i was thinking about, but here i wanna hear what people think. For now i have downloaded the repository with a depth of 1 and with shallow submodules. Basically a smaller shallow repository. This is well and all when building from one tag. But i was considering having it all unshallow. So i can make it so that when passing no arguments. It will build from the latest tag available? this will make it all a bit bigger. But then you can just run the script and you get the latest tag without checking (if it will build is another question). Or i can still make a argument required and it will just fetch the selected tag you have passed. What do you think?
#!/bin/bash
# variables and configuration
NAME="whonix"
DM="derivative-maker"
DB="derivative-binary"
WB="whonix-binary"
BL="build-log"
VERSION="$1"
VERSION_MASTER="master"
VERSION_NUMBER="${VERSION%%-*}"
REPOSITORY_EXIST="false"
REPOSITORY_URL="https://github.com/Whonix/derivative-maker.git"
FLAVOR_GATEWAY="whonix-gateway-xfce"
FLAVOR_WORKSTATION="whonix-workstation-xfce"
TARGET="utm"
ARCH="arm64"
TB="open"
REPO="true"
SIZE_GATEWAY="15G"
SIZE_WORKSTATION="25G"
# do not run this as root
if [ "$(id -u)" = "0" ]; then
error "Do not run this as root!"
fi
# check if argument is not missing
if [ -z "$1" ]; then
echo "Error: Missing tag argument. Please provide an tag argument."
exit 1
fi
if [ -f "$HOME"/"$DM" ]; then
REPOSITORY_EXIST="true"
fi
# set options
set -o nounset
set -e
set -x
# Functions
function cleanup() {
# clean up setup
rm -rf "$HOME"/"$WB"
rm -rf "$HOME"/"$BL"
touch "$HOME"/"$BL"
}
function update() {
# updates/upgrades and autoremove packages
sudo apt update -y
sudo apt upgrade -y
sudo apt autoremove -y
}
function install() {
# install dependencies
local PACKAGES=("git" "time" "curl" "apt-cacher-ng" "lsb-release" "fakeroot" "fasttrack-archive-keyring")
# Loop through each package
for PACKAGE in "${PACKAGES[@]}"; do
if ! dpkg -l "$PACKAGE" &> /dev/null; then
echo "$PACKAGE is not installed. Installing..."
sudo apt install -y "$PACKAGE"
else
echo "$PACKAGE is already installed."
fi
done
}
function repo_down() {
# download git repository
echo "Git clone $NAME repository"
git clone --depth=1 --branch "$VERSION_MASTER" --jobs=4 --recurse-submodules --shallow-submodules $REPOSITORY_URL
cd "$HOME"/derivative-maker
}
function fetch_tag() {
# verifies git tag with gpg keyring
cd "$HOME"/"$DM"
# Verify if the current directory is a git repository
if ! git rev-parse --git-dir > /dev/null 2>&1; then
echo "Not a git repository. Please run this script within a git repository."
exit 2
fi
if [ ! "$(git status --porcelain)" = "" ]; then
error "Command git status --porcelain failed at the beginning!"
fi
git fetch --depth=1 --recurse-submodules --jobs=4 origin tag "$VERSION"
git checkout --recurse-submodules "$VERSION"
git verify-tag "$VERSION"
git verify-commit "${VERSION}^{commit}"
git verify-commit HEAD
git submodule sync --recursive >/dev/null
git submodule update --init --recursive --jobs=200 >/dev/null
if [ ! "$(git status --porcelain)" = "" ]; then
error "Command git status --porcelain failed at the end!"
fi
# verifies correct version is downloaded
GIT_VERSION="$(git describe --tags --abbrev=0)"
if [ "$GIT_VERSION" == "$VERSION" ]; then
git describe
echo "Git version matches the desired version ($VERSION)."
else
git describe
echo "Git version does not match the desired version ($VERSION)."
exit 3
fi
}
function build() {
# build the project
cd "$HOME"/"$DM"
echo "Building gateway"
"$HOME"/"$DM"/"$DM" --flavor "$FLAVOR_GATEWAY" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_GATEWAY"
echo "Building workstation"
"$HOME"/"$DM"/"$DM" --flavor "$FLAVOR_WORKSTATION" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_WORKSTATION"
}
function pack() {
# cp binary tar.gz files into another directory for easy access
mkdir "$HOME"/"$WB"
cp -vr "$HOME"/"$DB"/"$VERSION_NUMBER"/*.utm.tar.gz "$HOME"/"$WB"
echo "Build of $NAME version $VERSION is finished"
}
function main() {
echo "Starting $NAME build for version $VERSION"
update
install
if [ "REPOSITORY_EXIST" == "false" ]; then
repo_down
fi
fetch_tag
build
pack
}
# starts scripts
echo "Starting $NAME build function"
cd "$HOME"
cleanup
# everything from here on will be included in build-log
main 2>&1 | tee -a "$HOME"/"$BL"
# cp build log to binary easy access folder and completing the script
cp "$HOME"/build-log "$HOME"/"$WB"/
echo "Build function finished, please check log"
cd "$HOME"
exit 0
This code is still very much under development and not working. And i did this while really tired. And not fully tested. But what i was wondering about is how should i do the fetching? i get fault as it cannot update all submodules? should i just do git fetch origin tag <tag>instead? and any other comments?
will look at this after work tomorrow. And improve the code.
I highly recommend using shellcheck. Available from packages.debian.org.
Also shell options.
set -e
set -o errexit
set -o nounset
set -o errtrace
set -o pipefail
And trapERR.
That’s how I would start nowadays with the knowledge I didn’t have back then.
ChatGPT is capable of explaining all of these. I’d look them up in ChatGPT and then confirm with authoritative sources.
Yes.
Yes.
Well, but the problem is, instead of verifying the real thing (derivative-maker), users would now have to download, trust and use a script to do that.
Even as the script is now in its early stages. Users are unable to audit it. So users would have to verify the script itself. Back to square one. Trying to square a circle so to speak.
If it was possible to have such a script, then that would also raise the issue why isn’t that provided by upstream?
Check out Whonix Linux Installer for VirtualBox. Not for the purpose of using VirtualBox. No need to actually run it. Just to show the usability as simplified as we managed to make it. Related:
Just 1 script that users can download (and verify) which takes care of downloading, verification, installing Whonix on Linux distributions.
With verify, verification referring to digital software signature verification.
When you do a normal clone you end up with git head, which is most likely not a (stable) git tag.
A normal clone is fine but to build a git tag, a git tag needs to get checked out.
unanswered questions: Dunno. I would have to test this myself to know. It’s a bit similar to driving a car. I can use git but not really explain “after 30 seconds, move the steering wheel by 35 degree”.
Okay so i have something functional now. But will wait with tagging it yet.
#!/bin/bash
# do not run this as root
if [ "$(id -u)" = "0" ]; then
error "Do not run this as root!"
fi
# check if argument is not missing
if [ -z "$1" ]; then
echo "Error: Missing tag argument. Please provide an tag argument."
exit 1
fi
# set options
set -e
set -o errexit
set -o nounset
set -o errtrace
set -o pipefail
set -x
# variables and configuration
NAME="whonix"
DM="derivative-maker"
DB="derivative-binary"
BL="build-log"
VERSION="$1"
VERSION_MASTER="master"
VERSION_NUMBER="${VERSION%%-*}"
REPOSITORY_EXIST="false"
REPOSITORY_URL="https://github.com/Whonix/derivative-maker.git"
FLAVOR_GATEWAY="whonix-gateway-xfce"
FLAVOR_WORKSTATION="whonix-workstation-xfce"
TARGET="utm"
ARCH="arm64"
TB="open"
REPO="true"
SIZE_GATEWAY="15G"
SIZE_WORKSTATION="25G"
WB="whonix-binary-$VERSION"
# check if the repository is in the home folder
if [ -d "$HOME"/"$DM" ]; then
REPOSITORY_EXIST="true"
fi
# set up trap
trap 'catch $? $LINENO' EXIT
# Functions
function catch() {
echo "Error Report"
if [ "$1" != "0" ]; then
# error handling goes here
echo "Error $1 occurred on line $2"
fi
}
function update() {
# updates/upgrades and autoremove packages
sudo apt update -y
sudo apt upgrade -y
sudo apt autoremove -y
}
function install() {
# install dependencies
local PACKAGES=("git" "time" "curl" "apt-cacher-ng" "lsb-release" "fakeroot" "fasttrack-archive-keyring")
# Loop through each package
for PACKAGE in "${PACKAGES[@]}"; do
if ! dpkg -l "$PACKAGE" &> /dev/null; then
echo "$PACKAGE is not installed. Installing..."
sudo apt install -y "$PACKAGE"
else
echo "$PACKAGE is already installed."
fi
done
}
function repo_down() {
# download git repository
echo "Git clone $NAME repository"
git clone --depth=1 --branch "$VERSION_MASTER" --jobs=4 --recurse-submodules --shallow-submodules $REPOSITORY_URL
cd "$HOME"/derivative-maker
}
function fetch_tag() {
# verifies git tag with gpg keyring
cd "$HOME"/"$DM"
# Verify if the current directory is a git repository
if ! git rev-parse --git-dir > /dev/null 2>&1; then
echo "Not a git repository. Please run this script within a git repository."
exit 2
fi
if [ ! "$(git status --porcelain)" = "" ]; then
error "Command git status --porcelain failed at the beginning!"
fi
git fetch origin tag "$VERSION"
git checkout "$VERSION"
git verify-tag "$VERSION"
git verify-commit "${VERSION}^{commit}"
git verify-commit HEAD
git submodule sync --recursive >/dev/null
git submodule update --init --recursive --jobs=200 >/dev/null
if [ ! "$(git status --porcelain)" = "" ]; then
error "Command git status --porcelain failed at the end!"
fi
# verifies correct version is downloaded
GIT_VERSION="$(git describe --tags --abbrev=0)"
if [ "$GIT_VERSION" == "$VERSION" ]; then
git describe
echo "Git version matches the desired version ($VERSION)."
else
git describe
echo "Git version does not match the desired version ($VERSION)."
exit 3
fi
}
function build() {
# build the project
cd "$HOME"/"$DM"
echo "Building gateway"
"$HOME"/"$DM"/"$DM" --flavor "$FLAVOR_GATEWAY" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_GATEWAY"
echo "Building workstation"
"$HOME"/"$DM"/"$DM" --flavor "$FLAVOR_WORKSTATION" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_WORKSTATION"
}
function pack() {
# cp binary tar.gz files into another directory for easy access
mkdir "$HOME"/"$WB"
cp -vr "$HOME"/"$DB"/"$VERSION_NUMBER"/*.utm.tar.gz "$HOME"/"$WB"
echo "Build of $NAME version $VERSION is finished"
}
function main() {
echo "Starting $NAME build for version $VERSION"
update
install
if [ "$REPOSITORY_EXIST" == "false" ]; then
repo_down
fi
fetch_tag
build
pack
}
# starts scripts
echo "Starting $NAME build function"
cd "$HOME"
# everything from here on will be included in build-log
main 2>&1 | tee -a "$HOME"/"$BL"
# cp build log to binary easy access folder and completing the script
mv "$HOME"/build-log "$HOME"/"$WB"/
echo "Build function finished, please check log"
cd "$HOME"
echo "Its recommended that you verify the signing key on the repository and build, more info here:"
echo "https://www.whonix.org/wiki/Signing_Key"
exit 0
added all set commands and -x for debug now. But might remove x when done.
added a simple trap ERR i found online so i can see where it fails.
Signing is just displayed not required. Echo note at the end where to find more info on how to check signing self.
Cannot run as root or without arguments with the tag you wanna build
6, everything is now variables, no mentioning of whonix elsewhere in the code apart from var.
if there is no repo, the master branch will be downloaded first. If there is a repo then it will fetch everything needed for the spesified tag.
no cleanup function anymore
did also do some smaller changes on how the script behaves.
everything is tested and it can download multiple tags now. There are some times fetching fails tough? like when it sync for submodules it kinda get errors? but so far i have found out it just rerun the code then it works. If you got any tips there do please tell me. And yeah comments on how to improve this in anyways? or anything special you guys want from this?. If you find this satisfactory then i will tag it and be done for it for now.
Made even more improvements tonight for the script. At this point i am not sure what else to do. Asked chatgpt how i could improve my code even more until there was nothing else to do. Added more absolute path to git and apt. Made more traps and error checks. Also removed set -o errexit as its a redundancy of set -e and tried my best to improve this as much as i could.
#!/bin/bash
# do not run this as root
if [ "$(id -u)" = "0" ]; then
echo "Error: Do not run this as root!"
exit 1
fi
# check if argument is not missing
if [ -z "$1" ]; then
echo "Error: Missing tag argument. Please provide an tag argument."
exit 2
fi
# set options
set -e
set -o nounset
set -o errtrace
set -o pipefail
# variables and configuration
NAME="whonix"
DM="derivative-maker"
DB="derivative-binary"
BL="build-log"
VERSION="$1"
VERSION_MASTER="master"
VERSION_NUMBER="${VERSION%%-*}"
REPOSITORY_URL="https://github.com/Whonix/derivative-maker.git"
FLAVOR_GATEWAY="whonix-gateway-xfce"
FLAVOR_WORKSTATION="whonix-workstation-xfce"
TARGET="utm"
ARCH="arm64"
TB="open"
REPO="true"
SIZE_GATEWAY="15G"
SIZE_WORKSTATION="25G"
WB="whonix-binary-$VERSION"
APT="/usr/bin/apt"
GIT="/usr/bin/git"
# catching errors as they occurre
function catch() {
echo "Error Report"
if [ "$1" != "0" ]; then
echo "Error $1 occurred on line $2, command: '$3'"
# Additional error handling can go here
fi
}
# SIGINT or SIGTERM
function handle_signal() {
echo "Error: received SIGINT or SIGTERM exiting"
exit 3
}
# start text
function beginning() {
echo "Starting $NAME build for version $VERSION"
}
# finishing text
function bye() {
echo "Exiting without errors."
echo "Build function finished, please check log"
echo "Its recommended that you verify the signing key on the repository and build, more info here:"
echo "https://www.whonix.org/wiki/Signing_Key"
}
# safe exit of the script
function safe_exit() {
trap bye EXIT
exit 0
}
function check_apt_sources() {
echo "Checking APT sources availability..."
if ! sudo "$APT" update &> /dev/null; then
echo "Error: APT sources are not reachable. Please check your sources.list and internet connection."
exit 4
fi
}
function check_connectivity() {
local HOST="github.com"
echo "Checking connectivity to $HOST..."
if ! ping -c 1 -W 2 "$HOST" &> /dev/null; then
echo "Error: Unable to reach $HOST. Please check your internet connection."
exit 5
fi
}
function repository_not_found() {
echo "Error: $HOME/$DM not found"
exit 6
}
# Set up trap
trap 'catch $? $LINENO "$BASH_COMMAND"' EXIT
trap handle_signal SIGINT SIGTERM
function update() {
# updates/upgrades and autoremove packages
check_apt_sources
sudo "$APT" update -y
sudo "$APT" upgrade -y
sudo "$APT" autoremove -y
}
function install() {
# install dependencies
local PACKAGES=("git" "time" "curl" "apt-cacher-ng" "lsb-release" "fakeroot" "fasttrack-archive-keyring")
# Loop through each package
for PACKAGE in "${PACKAGES[@]}"; do
if ! dpkg -l "$PACKAGE" &> /dev/null; then
echo "$PACKAGE is not installed. Installing..."
sudo "$APT" install -y "$PACKAGE"
else
echo "$PACKAGE is already installed."
fi
done
}
function repo_down() {
if [ ! -d "$HOME"/"$DM" ]; then
# download git repository
check_connectivity
echo "Git clone $NAME repository"
"$GIT" clone --depth=1 --branch "$VERSION_MASTER" --jobs=4 --recurse-submodules --shallow-submodules "$REPOSITORY_URL"
cd "$HOME"/"$DM" || repository_not_found
fi
}
function fetch_tag() {
# verifies git tag with gpg keyring
cd "$HOME"/"$DM" || repository_not_found
# Verify if the current directory is a git repository
if ! "$GIT" rev-parse --git-dir > /dev/null 2>&1; then
echo "Error: Not a git repository. Please run this script within a git repository."
exit 7
fi
if [ ! "$($GIT status --porcelain)" = "" ]; then
echo "Error: Command git status --porcelain failed at the beginning!"
exit 8
fi
"$GIT" fetch origin tag "$VERSION"
"$GIT" checkout "$VERSION"
"$GIT" verify-tag "$VERSION"
"$GIT" verify-commit "${VERSION}^{commit}"
"$GIT" verify-commit HEAD
"$GIT" submodule sync --recursive >/dev/null
"$GIT" submodule update --init --recursive --jobs=200 >/dev/null
if [ ! "$($GIT status --porcelain)" = "" ]; then
echo "Error: Command git status --porcelain failed at the end!"
exit 9
fi
# verifies correct version is downloaded
GIT_VERSION="$($GIT describe --tags --abbrev=0)"
if [ "$GIT_VERSION" == "$VERSION" ]; then
"$GIT" describe
echo "Git version matches the desired version ($VERSION)."
else
"$GIT" describe
echo "Error: Git version does not match the desired version ($VERSION)."
exit 10
fi
}
function build() {
# build the project
cd "$HOME"/"$DM" || repository_not_found
echo "Building gateway"
"$HOME"/"$DM"/"$DM" --flavor "$FLAVOR_GATEWAY" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_GATEWAY"
echo "Building workstation"
"$HOME"/"$DM"/"$DM" --flavor "$FLAVOR_WORKSTATION" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_WORKSTATION"
}
function pack() {
# cp binary tar.gz files into another directory for easy access
mkdir "$HOME"/"$WB"
cp -vr "$HOME"/"$DB"/"$VERSION_NUMBER"/*.utm.tar.gz "$HOME"/"$WB"
echo "Build of $NAME version $VERSION is finished"
}
function main() {
beginning
update
install
repo_down
fetch_tag
build
pack
}
# starts scripts
echo "Starting $NAME build function"
cd "$HOME"
# everything from here on will be included in build-log
main 2>&1 | tee -a "$HOME"/"$BL"
# cp build log to binary easy access folder and completing the script
mv "$HOME"/"$BL" "$HOME"/"$WB"/
cd "$HOME"
# exit the script
safe_exit
if i do not hear from anybody after testing the code tomorrow or by the weekend i guess i am done with this for now. Its been a learning experience. Also do tell me if there is a way to improve this or if i have done to much? still wanna hear what else to do here.
Do not run this yet tough as i have not tested this yet! Its 3:20 am now so gonna do it when i wake up.
will also check if there is anything i missed then or improve the readme if i need to?
Done now i have done everything, hopefully this is a more acceptable script this time?
#!/bin/bash
# do not run this as root
if [ "$(id -u)" = "0" ]; then
echo "Error: Do not run this as root!"
exit 1
fi
# check if argument is not missing
if [ -z "$1" ]; then
echo "Error: Missing tag argument. Please provide a tag argument."
exit 2
fi
# set options
set -e
set -o nounset
set -o errtrace
set -o pipefail
# variables and configuration
NAME="whonix"
DM="derivative-maker"
DB="derivative-binary"
BL="build-log"
VERSION="$1"
VERSION_MASTER="master"
VERSION_NUMBER="${VERSION%%-*}"
REPOSITORY_URL="https://github.com/Whonix/derivative-maker.git"
FLAVOR_GATEWAY="whonix-gateway-xfce"
FLAVOR_WORKSTATION="whonix-workstation-xfce"
TARGET="utm"
ARCH="arm64"
TB="open"
REPO="true"
SIZE_GATEWAY="15G"
SIZE_WORKSTATION="25G"
WB="whonix-binary-$VERSION"
APT="/usr/bin/apt"
GIT="/usr/bin/git"
# catching errors as they occur
function catch() {
echo "Error Report"
if [ "$1" != "0" ]; then
echo "Error $1 occurred on line $2, command: '$3'"
exit 11 # Exit on any error caught by this function
fi
}
# SIGINT or SIGTERM
function handle_signal() {
echo "Error: received SIGINT or SIGTERM exiting"
exit 3
}
# start text
function beginning() {
echo "Starting $NAME build for version $VERSION"
}
# finishing text
function bye() {
echo "Exiting without errors."
echo "Build function finished, please check log"
echo "Its recommended that you verify the signing key on the repository and build, more info here:"
echo "https://www.whonix.org/wiki/Signing_Key"
}
# safe exit of the script
function safe_exit() {
trap bye EXIT
exit 0
}
function check_apt_sources() {
echo "Checking APT sources availability..."
if ! sudo "$APT" update &> /dev/null; then
echo "Error: APT sources are not reachable. Please check your sources.list and internet connection."
exit 4
fi
}
function check_connectivity() {
local HOST="github.com"
echo "Checking connectivity to $HOST..."
if ! ping -c 1 -W 2 "$HOST" &> /dev/null; then
echo "Error: Unable to reach $HOST. Please check your internet connection."
exit 5
fi
}
function repository_not_found() {
echo "Error: ${HOME}/${DM} not found"
exit 6
}
# Set up trap
trap 'catch $? $LINENO "$BASH_COMMAND"' EXIT
trap handle_signal SIGINT SIGTERM
function update() {
# updates/upgrades and autoremove packages
check_apt_sources
sudo "$APT" update -y
sudo "$APT" upgrade -y
sudo "$APT" autoremove -y
}
function install() {
# install dependencies
local PACKAGES=("git" "time" "curl" "apt-cacher-ng" "lsb-release" "fakeroot" "fasttrack-archive-keyring")
# Loop through each package
for PACKAGE in "${PACKAGES[@]}"; do
if ! dpkg -l "$PACKAGE" &> /dev/null; then
echo "$PACKAGE is not installed. Installing..."
sudo "$APT" install -y "$PACKAGE"
else
echo "$PACKAGE is already installed."
fi
done
}
function repo_down() {
if [ ! -d "${HOME}/${DM}" ]; then
# download git repository
check_connectivity
echo "Git clone $NAME repository"
"$GIT" clone --depth=1 --branch "$VERSION_MASTER" --jobs=4 --recurse-submodules --shallow-submodules "$REPOSITORY_URL"
cd "${HOME}/${DM}" || repository_not_found
fi
}
function fetch_tag() {
cd "${HOME}/${DM}" || repository_not_found
# Verify if the current directory is a git repository
if ! "$GIT" rev-parse --git-dir > /dev/null 2>&1; then
echo "Error: Not a git repository. Please run this script within a git repository."
exit 7
fi
if [ ! "$($GIT status --porcelain)" = "" ]; then
echo "Error: Command git status --porcelain failed at the beginning!"
exit 8
fi
# fetch the selected git tag version
"$GIT" fetch origin tag "$VERSION"
"$GIT" checkout "$VERSION"
"$GIT" verify-tag "$VERSION"
"$GIT" verify-commit "${VERSION}^{commit}"
"$GIT" verify-commit HEAD
"$GIT" submodule sync --recursive >/dev/null
"$GIT" submodule update --init --recursive --jobs=200 >/dev/null
if [ ! "$($GIT status --porcelain)" = "" ]; then
echo "Error: Command git status --porcelain failed at the end!"
exit 9
fi
# verifies correct version is downloaded
GIT_VERSION="$($GIT describe --tags --abbrev=0)"
if [ "$GIT_VERSION" == "$VERSION" ]; then
"$GIT" describe
echo "Git version matches the desired version ($VERSION)."
else
"$GIT" describe
echo "Error: Git version does not match the desired version ($VERSION)."
exit 10
fi
}
function build() {
cd "${HOME}/${DM}" || repository_not_found
echo "Building gateway"
"${HOME}/${DM}/${DM}" --flavor "$FLAVOR_GATEWAY" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_GATEWAY"
echo "Building workstation"
"${HOME}/${DM}/${DM}" --flavor "$FLAVOR_WORKSTATION" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_WORKSTATION"
}
function pack() {
mkdir -p "${HOME}/${WB}"
cp -vr "${HOME}/${DB}/${VERSION_NUMBER}"/*.utm.tar.gz "${HOME}/${WB}"
echo "Build of $NAME version $VERSION is finished"
}
function main() {
beginning
update
install
repo_down
fetch_tag
build
pack
}
# Script execution starts here
echo "Starting $NAME build function"
cd "$HOME" || { echo "Error: Failed to change directory to $HOME"; exit 12; }
main 2>&1 | tee -a "${HOME}/${BL}"
# Copy the build log to the binary easy access folder and complete the script
mv "${HOME}/${BL}" "${HOME}/${WB}/" || { echo "Error: Failed to move build log"; exit 13; }
cd "$HOME" || { echo "Error: Failed to change directory to $HOME at script end"; exit 14; }
# Exit the script safely
safe_exit
I have tested this and it works. And should meet expectation of script quality. Learned a lot while doing this script. So glad i took the challenge. So this is a simple script that can build whonix fairly simple for mac with apple silicon.
you can see it all there. So now you only need to setup a debian wm, make a user there with sudo priv with nopasswd enabled. Then just import this script somewhere and run it. Also recommend to have the public keys for signing verification. Hopefully it make some peoples life a bit easier when dealing with building the project.
Do please comment if there is anything you guys want me to add or change now. Hope this code is better now Patrick? and that it is something you would consider acceptable.
great idea on the script just have a couple of issues.
Tested with Debian 12 from UTM Gallery
getutm.app/gallery/debian-12
1.) Had to manually install apt-catcher-ng since it asks you a question in ncurses which causes the script to hang. Maybe just have that as a requirement prior to running script.
2.) On multiple branches I am encountering submodule fetch failures. Here is the ouput from the console log.
Errors during submodule fetch:
packages/tor-ctrl
packages/serial-console-enable
packages/qubes-whonix
packages/kloak
packages/hardened_malloc
packages/hardened-kernel
packages/developer-meta-files
packages/anon-meta-packages
kloak
bindp
Error Report
Error 1 occurred on line 1, command: 'tee -a "${HOME}/${BL}"'
I uploaded full build log to a issue ticket on the github repo.
Hey been on a vacation and will look at these the comming days and try to replicate this. Also had the submodule failing on me. Not to sure how to fix it. Idk if adding --remote would help as then it would track things a bit different. But i do see some people doing
Do deal with submodules, but not sure what is the best course of action yet. But will look into this.
Also have not had any experience with the apt-catcher-ng stopping on install. But will check it out on a fresh install on debian and see what i can do.
There’s a build script. It’s called derivative-maker. It’s in a git repository. Why do we need another build script in another git repository to run the build script? Why can’t derivative-maker itself handle this?
apt-cacher-ng:
The reason why 1200_prepare-build-machine doesn’t handle installation of apt-cacher-ng is that derivative-maker always runs APT through apt-cacher-ng. That isn’t possible while apt-cacher-ng isn’t installed.
Full cycle. whonix-script running into, not solving the same issues that derivative-maker couldn’t solve.
So to solve this for real, simplify user documentation, require less prerequisites, automate more, derivative-maker needs to handle this.
Another thing which it doesn’t handle yet at time of writing is sudo installation and sudo setup (see build documentation). Also missing is VirtualBox installation (unrelated in this forum thread but also doable thanks to the Linux Installer).
So how to use it? The whonix-script could re-invent, duplicate that code, which I would consider unclean.
issues:
That seems like somehow a very old git revision has been fetched. I don’t know where textual string packages/tor-ctrl is coming from. Nowadays the folder structure is:
