Well can start with some of the simpler stuff as the hardcoded whonix strings and put them up as a variable. And some of the other stuff. Hope its okay to post about this here?
Thanks for your work on this
No problem, well again this is really early stuff. It works but its not good design yet. But i will work on this week when i have time.
git verify-tag 17.1.1.5-stable
git verify-commit 17.1.1.5-stable^{commit}
git checkout --recurse-submodules 17.1.1.5-stable
git verify-commit HEAD
git describe
git status --porcelain
See also:
./packages/kicksecure/developer-meta-files/usr/bin/dm-git-commit-checkout-latest
./packages/kicksecure/developer-meta-files/usr/bin/dm-git-tag-checkout-latest
oh its a new thread? well thats cool. I will continue with posts about the script here. Again by this week i will have made some updates on it.
Also not sure if i will do the automatic verification, but rather have a output with a text directing to the whonix wiki Signing_Key page. So people can do that after? That means also its not required to have imported the key but a option. This can save me some time and headache. But i can include
git verify-tag <tag>
git verify-commit <tag>
git verify-commit HEAD
git describe
as a output on the build-log so that its included there at least? This is a user-script so do kinda wanna hear what people wanna know.
what i will fix when i got the time is how i deal with the git repository as a whole and using variables so no whonix string is hard coded in the script.
Hopefully this script can be considered a simpler way of building the project for beginners/intermediate linux users. Advance users should still be able to do this manually.
Another thing i was thinking about, but here i wanna hear what people think. For now i have downloaded the repository with a depth of 1 and with shallow submodules. Basically a smaller shallow repository. This is well and all when building from one tag. But i was considering having it all unshallow. So i can make it so that when passing no arguments. It will build from the latest tag available? this will make it all a bit bigger. But then you can just run the script and you get the latest tag without checking (if it will build is another question). Or i can still make a argument required and it will just fetch the selected tag you have passed. What do you think?
Okay so i have came this far with the new code:
#!/bin/bash
# variables and configuration
NAME="whonix"
DM="derivative-maker"
DB="derivative-binary"
WB="whonix-binary"
BL="build-log"
VERSION="$1"
VERSION_MASTER="master"
VERSION_NUMBER="${VERSION%%-*}"
REPOSITORY_EXIST="false"
REPOSITORY_URL="https://github.com/Whonix/derivative-maker.git"
FLAVOR_GATEWAY="whonix-gateway-xfce"
FLAVOR_WORKSTATION="whonix-workstation-xfce"
TARGET="utm"
ARCH="arm64"
TB="open"
REPO="true"
SIZE_GATEWAY="15G"
SIZE_WORKSTATION="25G"
# do not run this as root
if [ "$(id -u)" = "0" ]; then
error "Do not run this as root!"
fi
# check if argument is not missing
if [ -z "$1" ]; then
echo "Error: Missing tag argument. Please provide an tag argument."
exit 1
fi
if [ -f "$HOME"/"$DM" ]; then
REPOSITORY_EXIST="true"
fi
# set options
set -o nounset
set -e
set -x
# Functions
function cleanup() {
# clean up setup
rm -rf "$HOME"/"$WB"
rm -rf "$HOME"/"$BL"
touch "$HOME"/"$BL"
}
function update() {
# updates/upgrades and autoremove packages
sudo apt update -y
sudo apt upgrade -y
sudo apt autoremove -y
}
function install() {
# install dependencies
local PACKAGES=("git" "time" "curl" "apt-cacher-ng" "lsb-release" "fakeroot" "fasttrack-archive-keyring")
# Loop through each package
for PACKAGE in "${PACKAGES[@]}"; do
if ! dpkg -l "$PACKAGE" &> /dev/null; then
echo "$PACKAGE is not installed. Installing..."
sudo apt install -y "$PACKAGE"
else
echo "$PACKAGE is already installed."
fi
done
}
function repo_down() {
# download git repository
echo "Git clone $NAME repository"
git clone --depth=1 --branch "$VERSION_MASTER" --jobs=4 --recurse-submodules --shallow-submodules $REPOSITORY_URL
cd "$HOME"/derivative-maker
}
function fetch_tag() {
# verifies git tag with gpg keyring
cd "$HOME"/"$DM"
# Verify if the current directory is a git repository
if ! git rev-parse --git-dir > /dev/null 2>&1; then
echo "Not a git repository. Please run this script within a git repository."
exit 2
fi
if [ ! "$(git status --porcelain)" = "" ]; then
error "Command git status --porcelain failed at the beginning!"
fi
git fetch --depth=1 --recurse-submodules --jobs=4 origin tag "$VERSION"
git checkout --recurse-submodules "$VERSION"
git verify-tag "$VERSION"
git verify-commit "${VERSION}^{commit}"
git verify-commit HEAD
git submodule sync --recursive >/dev/null
git submodule update --init --recursive --jobs=200 >/dev/null
if [ ! "$(git status --porcelain)" = "" ]; then
error "Command git status --porcelain failed at the end!"
fi
# verifies correct version is downloaded
GIT_VERSION="$(git describe --tags --abbrev=0)"
if [ "$GIT_VERSION" == "$VERSION" ]; then
git describe
echo "Git version matches the desired version ($VERSION)."
else
git describe
echo "Git version does not match the desired version ($VERSION)."
exit 3
fi
}
function build() {
# build the project
cd "$HOME"/"$DM"
echo "Building gateway"
"$HOME"/"$DM"/"$DM" --flavor "$FLAVOR_GATEWAY" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_GATEWAY"
echo "Building workstation"
"$HOME"/"$DM"/"$DM" --flavor "$FLAVOR_WORKSTATION" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_WORKSTATION"
}
function pack() {
# cp binary tar.gz files into another directory for easy access
mkdir "$HOME"/"$WB"
cp -vr "$HOME"/"$DB"/"$VERSION_NUMBER"/*.utm.tar.gz "$HOME"/"$WB"
echo "Build of $NAME version $VERSION is finished"
}
function main() {
echo "Starting $NAME build for version $VERSION"
update
install
if [ "REPOSITORY_EXIST" == "false" ]; then
repo_down
fi
fetch_tag
build
pack
}
# starts scripts
echo "Starting $NAME build function"
cd "$HOME"
cleanup
# everything from here on will be included in build-log
main 2>&1 | tee -a "$HOME"/"$BL"
# cp build log to binary easy access folder and completing the script
cp "$HOME"/build-log "$HOME"/"$WB"/
echo "Build function finished, please check log"
cd "$HOME"
exit 0
This code is still very much under development and not working. And i did this while really tired. And not fully tested. But what i was wondering about is how should i do the fetching? i get fault as it cannot update all submodules? should i just do git fetch origin tag <tag>
instead? and any other comments?
will look at this after work tomorrow. And improve the code.
-
- I highly recommend using
shellcheck
. Available frompackages.debian.org
.
- I highly recommend using
-
- Also shell options.
set -e
set -o errexit
set -o nounset
set -o errtrace
set -o pipefail
-
- And
trap
ERR
.
- And
- That’s how I would start nowadays with the knowledge I didn’t have back then.
ChatGPT is capable of explaining all of these. I’d look them up in ChatGPT and then confirm with authoritative sources.
Yes.
Yes.
Well, but the problem is, instead of verifying the real thing (derivative-maker), users would now have to download, trust and use a script to do that.
Even as the script is now in its early stages. Users are unable to audit it. So users would have to verify the script itself. Back to square one. Trying to square a circle so to speak.
If it was possible to have such a script, then that would also raise the issue why isn’t that provided by upstream?
Check out Whonix Linux Installer for VirtualBox. Not for the purpose of using VirtualBox. No need to actually run it. Just to show the usability as simplified as we managed to make it. Related:
Just 1 script that users can download (and verify) which takes care of downloading, verification, installing Whonix on Linux distributions.
With verify, verification referring to digital software signature verification.
When you do a normal clone you end up with git head, which is most likely not a (stable) git tag.
A normal clone is fine but to build a git tag, a git tag needs to get checked out.
unanswered questions: Dunno. I would have to test this myself to know. It’s a bit similar to driving a car. I can use git but not really explain “after 30 seconds, move the steering wheel by 35 degree”.
Okay so i have something functional now. But will wait with tagging it yet.
#!/bin/bash
# do not run this as root
if [ "$(id -u)" = "0" ]; then
error "Do not run this as root!"
fi
# check if argument is not missing
if [ -z "$1" ]; then
echo "Error: Missing tag argument. Please provide an tag argument."
exit 1
fi
# set options
set -e
set -o errexit
set -o nounset
set -o errtrace
set -o pipefail
set -x
# variables and configuration
NAME="whonix"
DM="derivative-maker"
DB="derivative-binary"
BL="build-log"
VERSION="$1"
VERSION_MASTER="master"
VERSION_NUMBER="${VERSION%%-*}"
REPOSITORY_EXIST="false"
REPOSITORY_URL="https://github.com/Whonix/derivative-maker.git"
FLAVOR_GATEWAY="whonix-gateway-xfce"
FLAVOR_WORKSTATION="whonix-workstation-xfce"
TARGET="utm"
ARCH="arm64"
TB="open"
REPO="true"
SIZE_GATEWAY="15G"
SIZE_WORKSTATION="25G"
WB="whonix-binary-$VERSION"
# check if the repository is in the home folder
if [ -d "$HOME"/"$DM" ]; then
REPOSITORY_EXIST="true"
fi
# set up trap
trap 'catch $? $LINENO' EXIT
# Functions
function catch() {
echo "Error Report"
if [ "$1" != "0" ]; then
# error handling goes here
echo "Error $1 occurred on line $2"
fi
}
function update() {
# updates/upgrades and autoremove packages
sudo apt update -y
sudo apt upgrade -y
sudo apt autoremove -y
}
function install() {
# install dependencies
local PACKAGES=("git" "time" "curl" "apt-cacher-ng" "lsb-release" "fakeroot" "fasttrack-archive-keyring")
# Loop through each package
for PACKAGE in "${PACKAGES[@]}"; do
if ! dpkg -l "$PACKAGE" &> /dev/null; then
echo "$PACKAGE is not installed. Installing..."
sudo apt install -y "$PACKAGE"
else
echo "$PACKAGE is already installed."
fi
done
}
function repo_down() {
# download git repository
echo "Git clone $NAME repository"
git clone --depth=1 --branch "$VERSION_MASTER" --jobs=4 --recurse-submodules --shallow-submodules $REPOSITORY_URL
cd "$HOME"/derivative-maker
}
function fetch_tag() {
# verifies git tag with gpg keyring
cd "$HOME"/"$DM"
# Verify if the current directory is a git repository
if ! git rev-parse --git-dir > /dev/null 2>&1; then
echo "Not a git repository. Please run this script within a git repository."
exit 2
fi
if [ ! "$(git status --porcelain)" = "" ]; then
error "Command git status --porcelain failed at the beginning!"
fi
git fetch origin tag "$VERSION"
git checkout "$VERSION"
git verify-tag "$VERSION"
git verify-commit "${VERSION}^{commit}"
git verify-commit HEAD
git submodule sync --recursive >/dev/null
git submodule update --init --recursive --jobs=200 >/dev/null
if [ ! "$(git status --porcelain)" = "" ]; then
error "Command git status --porcelain failed at the end!"
fi
# verifies correct version is downloaded
GIT_VERSION="$(git describe --tags --abbrev=0)"
if [ "$GIT_VERSION" == "$VERSION" ]; then
git describe
echo "Git version matches the desired version ($VERSION)."
else
git describe
echo "Git version does not match the desired version ($VERSION)."
exit 3
fi
}
function build() {
# build the project
cd "$HOME"/"$DM"
echo "Building gateway"
"$HOME"/"$DM"/"$DM" --flavor "$FLAVOR_GATEWAY" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_GATEWAY"
echo "Building workstation"
"$HOME"/"$DM"/"$DM" --flavor "$FLAVOR_WORKSTATION" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_WORKSTATION"
}
function pack() {
# cp binary tar.gz files into another directory for easy access
mkdir "$HOME"/"$WB"
cp -vr "$HOME"/"$DB"/"$VERSION_NUMBER"/*.utm.tar.gz "$HOME"/"$WB"
echo "Build of $NAME version $VERSION is finished"
}
function main() {
echo "Starting $NAME build for version $VERSION"
update
install
if [ "$REPOSITORY_EXIST" == "false" ]; then
repo_down
fi
fetch_tag
build
pack
}
# starts scripts
echo "Starting $NAME build function"
cd "$HOME"
# everything from here on will be included in build-log
main 2>&1 | tee -a "$HOME"/"$BL"
# cp build log to binary easy access folder and completing the script
mv "$HOME"/build-log "$HOME"/"$WB"/
echo "Build function finished, please check log"
cd "$HOME"
echo "Its recommended that you verify the signing key on the repository and build, more info here:"
echo "https://www.whonix.org/wiki/Signing_Key"
exit 0
So this is what i have done so far in my script at https://github.com/Goldeneye128/whonix-script
- i am using the shellcheck
- added all set commands and -x for debug now. But might remove x when done.
- added a simple trap ERR i found online so i can see where it fails.
- Signing is just displayed not required. Echo note at the end where to find more info on how to check signing self.
- Cannot run as root or without arguments with the tag you wanna build
6, everything is now variables, no mentioning of whonix elsewhere in the code apart from var. - if there is no repo, the master branch will be downloaded first. If there is a repo then it will fetch everything needed for the spesified tag.
- no cleanup function anymore
- did also do some smaller changes on how the script behaves.
everything is tested and it can download multiple tags now. There are some times fetching fails tough? like when it sync for submodules it kinda get errors? but so far i have found out it just rerun the code then it works. If you got any tips there do please tell me. And yeah comments on how to improve this in anyways? or anything special you guys want from this?. If you find this satisfactory then i will tag it and be done for it for now.
Made even more improvements tonight for the script. At this point i am not sure what else to do. Asked chatgpt how i could improve my code even more until there was nothing else to do. Added more absolute path to git and apt. Made more traps and error checks. Also removed set -o errexit
as its a redundancy of set -e
and tried my best to improve this as much as i could.
#!/bin/bash
# do not run this as root
if [ "$(id -u)" = "0" ]; then
echo "Error: Do not run this as root!"
exit 1
fi
# check if argument is not missing
if [ -z "$1" ]; then
echo "Error: Missing tag argument. Please provide an tag argument."
exit 2
fi
# set options
set -e
set -o nounset
set -o errtrace
set -o pipefail
# variables and configuration
NAME="whonix"
DM="derivative-maker"
DB="derivative-binary"
BL="build-log"
VERSION="$1"
VERSION_MASTER="master"
VERSION_NUMBER="${VERSION%%-*}"
REPOSITORY_URL="https://github.com/Whonix/derivative-maker.git"
FLAVOR_GATEWAY="whonix-gateway-xfce"
FLAVOR_WORKSTATION="whonix-workstation-xfce"
TARGET="utm"
ARCH="arm64"
TB="open"
REPO="true"
SIZE_GATEWAY="15G"
SIZE_WORKSTATION="25G"
WB="whonix-binary-$VERSION"
APT="/usr/bin/apt"
GIT="/usr/bin/git"
# catching errors as they occurre
function catch() {
echo "Error Report"
if [ "$1" != "0" ]; then
echo "Error $1 occurred on line $2, command: '$3'"
# Additional error handling can go here
fi
}
# SIGINT or SIGTERM
function handle_signal() {
echo "Error: received SIGINT or SIGTERM exiting"
exit 3
}
# start text
function beginning() {
echo "Starting $NAME build for version $VERSION"
}
# finishing text
function bye() {
echo "Exiting without errors."
echo "Build function finished, please check log"
echo "Its recommended that you verify the signing key on the repository and build, more info here:"
echo "https://www.whonix.org/wiki/Signing_Key"
}
# safe exit of the script
function safe_exit() {
trap bye EXIT
exit 0
}
function check_apt_sources() {
echo "Checking APT sources availability..."
if ! sudo "$APT" update &> /dev/null; then
echo "Error: APT sources are not reachable. Please check your sources.list and internet connection."
exit 4
fi
}
function check_connectivity() {
local HOST="github.com"
echo "Checking connectivity to $HOST..."
if ! ping -c 1 -W 2 "$HOST" &> /dev/null; then
echo "Error: Unable to reach $HOST. Please check your internet connection."
exit 5
fi
}
function repository_not_found() {
echo "Error: $HOME/$DM not found"
exit 6
}
# Set up trap
trap 'catch $? $LINENO "$BASH_COMMAND"' EXIT
trap handle_signal SIGINT SIGTERM
function update() {
# updates/upgrades and autoremove packages
check_apt_sources
sudo "$APT" update -y
sudo "$APT" upgrade -y
sudo "$APT" autoremove -y
}
function install() {
# install dependencies
local PACKAGES=("git" "time" "curl" "apt-cacher-ng" "lsb-release" "fakeroot" "fasttrack-archive-keyring")
# Loop through each package
for PACKAGE in "${PACKAGES[@]}"; do
if ! dpkg -l "$PACKAGE" &> /dev/null; then
echo "$PACKAGE is not installed. Installing..."
sudo "$APT" install -y "$PACKAGE"
else
echo "$PACKAGE is already installed."
fi
done
}
function repo_down() {
if [ ! -d "$HOME"/"$DM" ]; then
# download git repository
check_connectivity
echo "Git clone $NAME repository"
"$GIT" clone --depth=1 --branch "$VERSION_MASTER" --jobs=4 --recurse-submodules --shallow-submodules "$REPOSITORY_URL"
cd "$HOME"/"$DM" || repository_not_found
fi
}
function fetch_tag() {
# verifies git tag with gpg keyring
cd "$HOME"/"$DM" || repository_not_found
# Verify if the current directory is a git repository
if ! "$GIT" rev-parse --git-dir > /dev/null 2>&1; then
echo "Error: Not a git repository. Please run this script within a git repository."
exit 7
fi
if [ ! "$($GIT status --porcelain)" = "" ]; then
echo "Error: Command git status --porcelain failed at the beginning!"
exit 8
fi
"$GIT" fetch origin tag "$VERSION"
"$GIT" checkout "$VERSION"
"$GIT" verify-tag "$VERSION"
"$GIT" verify-commit "${VERSION}^{commit}"
"$GIT" verify-commit HEAD
"$GIT" submodule sync --recursive >/dev/null
"$GIT" submodule update --init --recursive --jobs=200 >/dev/null
if [ ! "$($GIT status --porcelain)" = "" ]; then
echo "Error: Command git status --porcelain failed at the end!"
exit 9
fi
# verifies correct version is downloaded
GIT_VERSION="$($GIT describe --tags --abbrev=0)"
if [ "$GIT_VERSION" == "$VERSION" ]; then
"$GIT" describe
echo "Git version matches the desired version ($VERSION)."
else
"$GIT" describe
echo "Error: Git version does not match the desired version ($VERSION)."
exit 10
fi
}
function build() {
# build the project
cd "$HOME"/"$DM" || repository_not_found
echo "Building gateway"
"$HOME"/"$DM"/"$DM" --flavor "$FLAVOR_GATEWAY" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_GATEWAY"
echo "Building workstation"
"$HOME"/"$DM"/"$DM" --flavor "$FLAVOR_WORKSTATION" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_WORKSTATION"
}
function pack() {
# cp binary tar.gz files into another directory for easy access
mkdir "$HOME"/"$WB"
cp -vr "$HOME"/"$DB"/"$VERSION_NUMBER"/*.utm.tar.gz "$HOME"/"$WB"
echo "Build of $NAME version $VERSION is finished"
}
function main() {
beginning
update
install
repo_down
fetch_tag
build
pack
}
# starts scripts
echo "Starting $NAME build function"
cd "$HOME"
# everything from here on will be included in build-log
main 2>&1 | tee -a "$HOME"/"$BL"
# cp build log to binary easy access folder and completing the script
mv "$HOME"/"$BL" "$HOME"/"$WB"/
cd "$HOME"
# exit the script
safe_exit
if i do not hear from anybody after testing the code tomorrow or by the weekend i guess i am done with this for now. Its been a learning experience. Also do tell me if there is a way to improve this or if i have done to much? still wanna hear what else to do here.
Do not run this yet tough as i have not tested this yet! Its 3:20 am now so gonna do it when i wake up.
will also check if there is anything i missed then or improve the readme if i need to?
- 2 spaces indention would be preferable
-
This quoting style seems a bit weird. Preferable would be:
“${HOME}/${DM}”
If you do not exit on error, then also digital signature verification issues would get silently ignored.
Done now i have done everything, hopefully this is a more acceptable script this time?
#!/bin/bash
# do not run this as root
if [ "$(id -u)" = "0" ]; then
echo "Error: Do not run this as root!"
exit 1
fi
# check if argument is not missing
if [ -z "$1" ]; then
echo "Error: Missing tag argument. Please provide a tag argument."
exit 2
fi
# set options
set -e
set -o nounset
set -o errtrace
set -o pipefail
# variables and configuration
NAME="whonix"
DM="derivative-maker"
DB="derivative-binary"
BL="build-log"
VERSION="$1"
VERSION_MASTER="master"
VERSION_NUMBER="${VERSION%%-*}"
REPOSITORY_URL="https://github.com/Whonix/derivative-maker.git"
FLAVOR_GATEWAY="whonix-gateway-xfce"
FLAVOR_WORKSTATION="whonix-workstation-xfce"
TARGET="utm"
ARCH="arm64"
TB="open"
REPO="true"
SIZE_GATEWAY="15G"
SIZE_WORKSTATION="25G"
WB="whonix-binary-$VERSION"
APT="/usr/bin/apt"
GIT="/usr/bin/git"
# catching errors as they occur
function catch() {
echo "Error Report"
if [ "$1" != "0" ]; then
echo "Error $1 occurred on line $2, command: '$3'"
exit 11 # Exit on any error caught by this function
fi
}
# SIGINT or SIGTERM
function handle_signal() {
echo "Error: received SIGINT or SIGTERM exiting"
exit 3
}
# start text
function beginning() {
echo "Starting $NAME build for version $VERSION"
}
# finishing text
function bye() {
echo "Exiting without errors."
echo "Build function finished, please check log"
echo "Its recommended that you verify the signing key on the repository and build, more info here:"
echo "https://www.whonix.org/wiki/Signing_Key"
}
# safe exit of the script
function safe_exit() {
trap bye EXIT
exit 0
}
function check_apt_sources() {
echo "Checking APT sources availability..."
if ! sudo "$APT" update &> /dev/null; then
echo "Error: APT sources are not reachable. Please check your sources.list and internet connection."
exit 4
fi
}
function check_connectivity() {
local HOST="github.com"
echo "Checking connectivity to $HOST..."
if ! ping -c 1 -W 2 "$HOST" &> /dev/null; then
echo "Error: Unable to reach $HOST. Please check your internet connection."
exit 5
fi
}
function repository_not_found() {
echo "Error: ${HOME}/${DM} not found"
exit 6
}
# Set up trap
trap 'catch $? $LINENO "$BASH_COMMAND"' EXIT
trap handle_signal SIGINT SIGTERM
function update() {
# updates/upgrades and autoremove packages
check_apt_sources
sudo "$APT" update -y
sudo "$APT" upgrade -y
sudo "$APT" autoremove -y
}
function install() {
# install dependencies
local PACKAGES=("git" "time" "curl" "apt-cacher-ng" "lsb-release" "fakeroot" "fasttrack-archive-keyring")
# Loop through each package
for PACKAGE in "${PACKAGES[@]}"; do
if ! dpkg -l "$PACKAGE" &> /dev/null; then
echo "$PACKAGE is not installed. Installing..."
sudo "$APT" install -y "$PACKAGE"
else
echo "$PACKAGE is already installed."
fi
done
}
function repo_down() {
if [ ! -d "${HOME}/${DM}" ]; then
# download git repository
check_connectivity
echo "Git clone $NAME repository"
"$GIT" clone --depth=1 --branch "$VERSION_MASTER" --jobs=4 --recurse-submodules --shallow-submodules "$REPOSITORY_URL"
cd "${HOME}/${DM}" || repository_not_found
fi
}
function fetch_tag() {
cd "${HOME}/${DM}" || repository_not_found
# Verify if the current directory is a git repository
if ! "$GIT" rev-parse --git-dir > /dev/null 2>&1; then
echo "Error: Not a git repository. Please run this script within a git repository."
exit 7
fi
if [ ! "$($GIT status --porcelain)" = "" ]; then
echo "Error: Command git status --porcelain failed at the beginning!"
exit 8
fi
# fetch the selected git tag version
"$GIT" fetch origin tag "$VERSION"
"$GIT" checkout "$VERSION"
"$GIT" verify-tag "$VERSION"
"$GIT" verify-commit "${VERSION}^{commit}"
"$GIT" verify-commit HEAD
"$GIT" submodule sync --recursive >/dev/null
"$GIT" submodule update --init --recursive --jobs=200 >/dev/null
if [ ! "$($GIT status --porcelain)" = "" ]; then
echo "Error: Command git status --porcelain failed at the end!"
exit 9
fi
# verifies correct version is downloaded
GIT_VERSION="$($GIT describe --tags --abbrev=0)"
if [ "$GIT_VERSION" == "$VERSION" ]; then
"$GIT" describe
echo "Git version matches the desired version ($VERSION)."
else
"$GIT" describe
echo "Error: Git version does not match the desired version ($VERSION)."
exit 10
fi
}
function build() {
cd "${HOME}/${DM}" || repository_not_found
echo "Building gateway"
"${HOME}/${DM}/${DM}" --flavor "$FLAVOR_GATEWAY" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_GATEWAY"
echo "Building workstation"
"${HOME}/${DM}/${DM}" --flavor "$FLAVOR_WORKSTATION" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_WORKSTATION"
}
function pack() {
mkdir -p "${HOME}/${WB}"
cp -vr "${HOME}/${DB}/${VERSION_NUMBER}"/*.utm.tar.gz "${HOME}/${WB}"
echo "Build of $NAME version $VERSION is finished"
}
function main() {
beginning
update
install
repo_down
fetch_tag
build
pack
}
# Script execution starts here
echo "Starting $NAME build function"
cd "$HOME" || { echo "Error: Failed to change directory to $HOME"; exit 12; }
main 2>&1 | tee -a "${HOME}/${BL}"
# Copy the build log to the binary easy access folder and complete the script
mv "${HOME}/${BL}" "${HOME}/${WB}/" || { echo "Error: Failed to move build log"; exit 13; }
cd "$HOME" || { echo "Error: Failed to change directory to $HOME at script end"; exit 14; }
# Exit the script safely
safe_exit
I have tested this and it works. And should meet expectation of script quality. Learned a lot while doing this script. So glad i took the challenge. So this is a simple script that can build whonix fairly simple for mac with apple silicon.
you can see it all there. So now you only need to setup a debian wm, make a user there with sudo priv with nopasswd enabled. Then just import this script somewhere and run it. Also recommend to have the public keys for signing verification. Hopefully it make some peoples life a bit easier when dealing with building the project.
Do please comment if there is anything you guys want me to add or change now. Hope this code is better now Patrick? and that it is something you would consider acceptable.
great idea on the script just have a couple of issues.
Tested with Debian 12 from UTM Gallery
getutm.app/gallery/debian-12
1.) Had to manually install apt-catcher-ng since it asks you a question in ncurses which causes the script to hang. Maybe just have that as a requirement prior to running script.
2.) On multiple branches I am encountering submodule fetch failures. Here is the ouput from the console log.
Errors during submodule fetch:
packages/tor-ctrl
packages/serial-console-enable
packages/qubes-whonix
packages/kloak
packages/hardened_malloc
packages/hardened-kernel
packages/developer-meta-files
packages/anon-meta-packages
kloak
bindp
Error Report
Error 1 occurred on line 1, command: 'tee -a "${HOME}/${BL}"'
I uploaded full build log to a issue ticket on the github repo.
Hey been on a vacation and will look at these the comming days and try to replicate this. Also had the submodule failing on me. Not to sure how to fix it. Idk if adding --remote
would help as then it would track things a bit different. But i do see some people doing
$ git submodule update --force --recursive --init --remote
Do deal with submodules, but not sure what is the best course of action yet. But will look into this.
Also have not had any experience with the apt-catcher-ng
stopping on install. But will check it out on a fresh install on debian and see what i can do.
General purpose:
There’s a build script. It’s called derivative-maker. It’s in a git repository. Why do we need another build script in another git repository to run the build script? Why can’t derivative-maker itself handle this?
apt-cacher-ng:
The reason why 1200_prepare-build-machine doesn’t handle installation of apt-cacher-ng is that derivative-maker always runs APT through apt-cacher-ng. That isn’t possible while apt-cacher-ng isn’t installed.
Related:
Repository Caching
Yes. That is what Whonix ™ VM Build Documentation currently does.
Full cycle. whonix-script running into, not solving the same issues that derivative-maker couldn’t solve.
So to solve this for real, simplify user documentation, require less prerequisites, automate more, derivative-maker needs to handle this.
Another thing which it doesn’t handle yet at time of writing is sudo installation and sudo setup (see build documentation). Also missing is VirtualBox installation (unrelated in this forum thread but also doable thanks to the Linux Installer).
Interactive APT questions:
That is why apt-get-noninteractive (and dpkg-noninteractive) have been invented.
So how to use it? The whonix-script could re-invent, duplicate that code, which I would consider unclean.
issues:
That seems like somehow a very old git revision has been fetched. I don’t know where textual string packages/tor-ctrl
is coming from. Nowadays the folder structure is:
packages/kicksecure/...
packages/whonix/...
Hey i have just fixed most of the problem you have experienced, i apologized for the inconvenience.
To explain a bit the apt-catcher-ng problem have been solved by including a noninteractive variable before running the install, i also saw how dpkg query worked and was not optimal as not all dependencies was installed when doing this, also added some new and extra dependencies as required. The portion of the code looks like this now:
# install dependencies
local PACKAGES=("git" "time" "curl" "apt-cacher-ng" "lsb-release" "dpkg-dev" "fakeroot" "fasttrack-archive-keyring" "safe-rm" "extrepo-offline-data")
# Loop through each package
for PACKAGE in "${PACKAGES[@]}"; do
if ! dpkg-query -W -f='${Status}' "$PACKAGE" 2>/dev/null | grep -q "install ok installed"; then
echo "$PACKAGE is not installed. Installing..."
sudo DEBIAN_FRONTEND=noninteractive "$APT" install -y "$PACKAGE"
else
echo "$PACKAGE is already installed."
fi
done
also added importing the gpg keys as it would fail if not done so, Still check the gpg key manually after the build is done please for your own safety.
when it comes to the submodule fetch failure it is my fault for not being careful with the fetching process. But the problem was that i forgot to add --depth=1
when fetching so it fetched every tag ever instead of just the one this is now fixed on the latest git commit.
# Fetch only the specific tag
"$GIT" fetch --depth=1 origin tag "$VERSION"
is how it looks now.
I have also added a convenience function before the install so if you are using a freshly installed/downloaded debian 12 UTM gallery image you do not have to add any no password line on your sudoers file, it will automatically add this if using the UTM image.
Do please check if everything works, should be the latest github commit.
Just to answer this question Patrick is that my script is in NO WAY a replacement for the derivate-maker script. This is just a simplified user script for lazy people like me who do not wanna type in a bunch of commands and install dependencies all the time when building for whonix for mac m1. So it is what is it is. A user script to do the build process fast and simple without a lot of configurations.
For anyone who are a developers i also recommend doing this process manually and not use my script. But if you are lazy and just want this to work. Then i do not see a reason for not having this user script available for others if they wish to use this.
I believe in making whonix available to as many people as possible, not just devs who understand all of this. And hopefully at least as a temporary solution until something more permanent is made for the mac m1. I hope that this will help a bit on that part. I wish i could help more on the main project, but idk what i could do.
Dependency installation has been automated in 17.1.3.4-developers-only and above.
It’s not yet tested but thanks to automated CI testing, I’ll notice soon if something is broken and that would certainly be fixed before the next stable release tag.
Well thats news for me, also 17.1.3.4? thats fairly new feature then. Might need to test that in the future.
Well if the script will download dependencies, then all it needs is some sort of config file so i do not need to type all the flags all the time then my script is useless. Guess its better in the long term.
local PACKAGES=(“git” “time” “curl” “apt-cacher-ng” “lsb-release” “dpkg-dev” “fakeroot” “fasttrack-archive-keyring” “safe-rm” “extrepo-offline-data”)
This is now automated since Whonix 17.2.0.1 - All Platforms - Point Release!. Build documentation has been updated accordingly.