Goldeneye128 - Simple script to build whonix on a mac with apple silicon

Unsupported Platforms
5

I see, as i did say this is something i want to improve. So of course i did not expect this to be perfect at the first try. As i am still a beginner at this. So i see this as a learning experience. I will think a bit how to do gpg vertification on my script for a bit, as making it verified with the imported gpg key would be best. Maybe find the latest git commit or tag, find the signing key used then compare that with the imported key and check it that way? idk. Again still learning.

When it comes to the very harsh way i dealt with the whole repository. Well i know its unclean. But to implement a cleaner way would be a lot more work. First i would need to make a check if its even is there, if it has the right tag installed. Then i would need to implement a way to fetch/pull it all down including submodules and so on. Would be a lot more of what is a simple script that just build whonix for mac would be for a newbie like myself. But i can take on the challenge to implement that. If you are interested in me doing that? again i am still learning. But hey thanks for the comments. At least i am learning about this now.

3 Likes
6

Well can start with some of the simpler stuff as the hardcoded whonix strings and put them up as a variable. And some of the other stuff. Hope its okay to post about this here?

3 Likes
7

Thanks for your work on this :smile:

1 Like
8

No problem, well again this is really early stuff. It works but its not good design yet. But i will work on this week when i have time.

2 Likes
9 
git verify-tag 17.1.1.5-stable
git verify-commit 17.1.1.5-stable^{commit}
git checkout --recurse-submodules 17.1.1.5-stable
git verify-commit HEAD
git describe
git status --porcelain

See also:

./packages/kicksecure/developer-meta-files/usr/bin/dm-git-commit-checkout-latest
./packages/kicksecure/developer-meta-files/usr/bin/dm-git-tag-checkout-latest
1 Like
10

oh its a new thread? well thats cool. I will continue with posts about the script here. Again by this week i will have made some updates on it.

Also not sure if i will do the automatic verification, but rather have a output with a text directing to the whonix wiki Signing_Key page. So people can do that after? That means also its not required to have imported the key but a option. This can save me some time and headache. But i can include

git verify-tag <tag>
git verify-commit <tag>
git verify-commit HEAD
git describe

as a output on the build-log so that its included there at least? This is a user-script so do kinda wanna hear what people wanna know.

what i will fix when i got the time is how i deal with the git repository as a whole and using variables so no whonix string is hard coded in the script.

Hopefully this script can be considered a simpler way of building the project for beginners/intermediate linux users. Advance users should still be able to do this manually.

1 Like
11

Another thing i was thinking about, but here i wanna hear what people think. For now i have downloaded the repository with a depth of 1 and with shallow submodules. Basically a smaller shallow repository. This is well and all when building from one tag. But i was considering having it all unshallow. So i can make it so that when passing no arguments. It will build from the latest tag available? this will make it all a bit bigger. But then you can just run the script and you get the latest tag without checking (if it will build is another question). Or i can still make a argument required and it will just fetch the selected tag you have passed. What do you think?

1 Like
12

Okay so i have came this far with the new code:

#!/bin/bash

# variables and configuration
NAME="whonix"
DM="derivative-maker"
DB="derivative-binary"
WB="whonix-binary"
BL="build-log"
VERSION="$1"
VERSION_MASTER="master"
VERSION_NUMBER="${VERSION%%-*}"
REPOSITORY_EXIST="false"
REPOSITORY_URL="https://github.com/Whonix/derivative-maker.git"
FLAVOR_GATEWAY="whonix-gateway-xfce"
FLAVOR_WORKSTATION="whonix-workstation-xfce"
TARGET="utm"
ARCH="arm64"
TB="open"
REPO="true"
SIZE_GATEWAY="15G"
SIZE_WORKSTATION="25G"

# do not run this as root
if [ "$(id -u)" = "0" ]; then
   error "Do not run this as root!"
fi

# check if argument is not missing
if [ -z "$1" ]; then
	echo "Error: Missing tag argument. Please provide an tag argument."
	exit 1
fi

if [ -f "$HOME"/"$DM" ]; then
  REPOSITORY_EXIST="true"
fi

# set options
set -o nounset
set -e
set -x

# Functions
function cleanup() {
  # clean up setup
  rm -rf "$HOME"/"$WB"
  rm -rf "$HOME"/"$BL"
  touch "$HOME"/"$BL"
}

function update() {
  # updates/upgrades and autoremove packages
  sudo apt update -y
  sudo apt upgrade -y
  sudo apt autoremove -y
}

function install() {
  # install dependencies
  local PACKAGES=("git" "time" "curl" "apt-cacher-ng" "lsb-release" "fakeroot" "fasttrack-archive-keyring")
  # Loop through each package
  for PACKAGE in "${PACKAGES[@]}"; do
      if ! dpkg -l "$PACKAGE" &> /dev/null; then
          echo "$PACKAGE is not installed. Installing..."
          sudo apt install -y "$PACKAGE"
      else
          echo "$PACKAGE is already installed."
      fi
  done
}

function repo_down() {
  # download git repository
  echo "Git clone $NAME repository"
  git clone --depth=1 --branch "$VERSION_MASTER" --jobs=4 --recurse-submodules --shallow-submodules $REPOSITORY_URL
  cd "$HOME"/derivative-maker

}

function fetch_tag() {
  # verifies git tag with gpg keyring
  cd "$HOME"/"$DM"

  # Verify if the current directory is a git repository
  if ! git rev-parse --git-dir > /dev/null 2>&1; then
    echo "Not a git repository. Please run this script within a git repository."
    exit 2
  fi

  if [ ! "$(git status --porcelain)" = "" ]; then
   error "Command git status --porcelain failed at the beginning!"
  fi

  git fetch --depth=1 --recurse-submodules --jobs=4 origin tag "$VERSION"

  git checkout --recurse-submodules "$VERSION"
  
  git verify-tag "$VERSION"
  
  git verify-commit "${VERSION}^{commit}"

  git verify-commit HEAD
  
  git submodule sync --recursive >/dev/null
  
  git submodule update --init --recursive --jobs=200 >/dev/null
  
  if [ ! "$(git status --porcelain)" = "" ]; then
     error "Command git status --porcelain failed at the end!"
  fi

  # verifies correct version is downloaded
  GIT_VERSION="$(git describe --tags --abbrev=0)"
  if [ "$GIT_VERSION" == "$VERSION" ]; then
    git describe
    echo "Git version matches the desired version ($VERSION)."
  else
    git describe
    echo "Git version does not match the desired version ($VERSION)."
    exit 3
  fi
}

function build() {
  # build the project
  cd "$HOME"/"$DM"

  echo "Building gateway"
  "$HOME"/"$DM"/"$DM" --flavor "$FLAVOR_GATEWAY" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_GATEWAY"
  
  echo "Building workstation"
  "$HOME"/"$DM"/"$DM" --flavor "$FLAVOR_WORKSTATION" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_WORKSTATION"
}

function pack() {
  # cp binary tar.gz files into another directory for easy access
  mkdir "$HOME"/"$WB"
  cp -vr "$HOME"/"$DB"/"$VERSION_NUMBER"/*.utm.tar.gz "$HOME"/"$WB"
  echo "Build of $NAME version $VERSION is finished"
}

function main() {
  echo "Starting $NAME build for version $VERSION" 
  update
  install
  if [ "REPOSITORY_EXIST" == "false" ]; then
    repo_down
  fi
  fetch_tag
  build
  pack
} 

# starts scripts
echo "Starting $NAME build function"
cd "$HOME"
cleanup

# everything from here on will be included in build-log
main 2>&1 | tee -a "$HOME"/"$BL"

# cp build log to binary easy access folder and completing the script
cp "$HOME"/build-log "$HOME"/"$WB"/
echo "Build function finished, please check log"
cd "$HOME"
exit 0

This code is still very much under development and not working. And i did this while really tired. And not fully tested. But what i was wondering about is how should i do the fetching? i get fault as it cannot update all submodules? should i just do git fetch origin tag <tag>instead? and any other comments?
will look at this after work tomorrow. And improve the code.

1 Like
13
    1. I highly recommend using shellcheck. Available from packages.debian.org.
    1. Also shell options.
set -e
set -o errexit
set -o nounset
set -o errtrace
set -o pipefail
    1. And trap ERR.
  • That’s how I would start nowadays with the knowledge I didn’t have back then.

ChatGPT is capable of explaining all of these. I’d look them up in ChatGPT and then confirm with authoritative sources.

Yes.

Yes.

Well, but the problem is, instead of verifying the real thing (derivative-maker), users would now have to download, trust and use a script to do that.

Even as the script is now in its early stages. Users are unable to audit it. So users would have to verify the script itself. Back to square one. Trying to square a circle so to speak.

If it was possible to have such a script, then that would also raise the issue why isn’t that provided by upstream?

Check out Whonix Linux Installer for VirtualBox. Not for the purpose of using VirtualBox. No need to actually run it. Just to show the usability as simplified as we managed to make it. Related:

Just 1 script that users can download (and verify) which takes care of downloading, verification, installing Whonix on Linux distributions.

With verify, verification referring to digital software signature verification.

When you do a normal clone you end up with git head, which is most likely not a (stable) git tag.

A normal clone is fine but to build a git tag, a git tag needs to get checked out.

unanswered questions: Dunno. I would have to test this myself to know. It’s a bit similar to driving a car. I can use git but not really explain “after 30 seconds, move the steering wheel by 35 degree”.

1 Like
14

Okay so i have something functional now. But will wait with tagging it yet.

#!/bin/bash

# do not run this as root
if [ "$(id -u)" = "0" ]; then
   error "Do not run this as root!"
fi

# check if argument is not missing
if [ -z "$1" ]; then
	echo "Error: Missing tag argument. Please provide an tag argument."
	exit 1
fi

# set options
set -e
set -o errexit
set -o nounset
set -o errtrace
set -o pipefail
set -x

# variables and configuration
NAME="whonix"
DM="derivative-maker"
DB="derivative-binary"
BL="build-log"
VERSION="$1"
VERSION_MASTER="master"
VERSION_NUMBER="${VERSION%%-*}"
REPOSITORY_EXIST="false"
REPOSITORY_URL="https://github.com/Whonix/derivative-maker.git"
FLAVOR_GATEWAY="whonix-gateway-xfce"
FLAVOR_WORKSTATION="whonix-workstation-xfce"
TARGET="utm"
ARCH="arm64"
TB="open"
REPO="true"
SIZE_GATEWAY="15G"
SIZE_WORKSTATION="25G"
WB="whonix-binary-$VERSION"

# check if the repository is in the home folder
if [ -d "$HOME"/"$DM" ]; then
  REPOSITORY_EXIST="true"
fi

# set up trap
trap 'catch $? $LINENO' EXIT

# Functions
function catch() {
  echo "Error Report"
  if [ "$1" != "0" ]; then
    # error handling goes here
    echo "Error $1 occurred on line $2"
  fi
}

function update() {
  # updates/upgrades and autoremove packages
  sudo apt update -y
  sudo apt upgrade -y
  sudo apt autoremove -y
}

function install() {
  # install dependencies
  local PACKAGES=("git" "time" "curl" "apt-cacher-ng" "lsb-release" "fakeroot" "fasttrack-archive-keyring")
  # Loop through each package
  for PACKAGE in "${PACKAGES[@]}"; do
      if ! dpkg -l "$PACKAGE" &> /dev/null; then
          echo "$PACKAGE is not installed. Installing..."
          sudo apt install -y "$PACKAGE"
      else
          echo "$PACKAGE is already installed."
      fi
  done
}

function repo_down() {
  # download git repository
  echo "Git clone $NAME repository"
  git clone --depth=1 --branch "$VERSION_MASTER" --jobs=4 --recurse-submodules --shallow-submodules $REPOSITORY_URL
  cd "$HOME"/derivative-maker

}

function fetch_tag() {
  # verifies git tag with gpg keyring
  cd "$HOME"/"$DM"

  # Verify if the current directory is a git repository
  if ! git rev-parse --git-dir > /dev/null 2>&1; then
    echo "Not a git repository. Please run this script within a git repository."
    exit 2
  fi

  if [ ! "$(git status --porcelain)" = "" ]; then
   error "Command git status --porcelain failed at the beginning!"
  fi

  git fetch origin tag "$VERSION"

  git checkout "$VERSION"
  
  git verify-tag "$VERSION"
  
  git verify-commit "${VERSION}^{commit}"

  git verify-commit HEAD
  
  git submodule sync --recursive >/dev/null
  
  git submodule update --init --recursive --jobs=200 >/dev/null
  
  if [ ! "$(git status --porcelain)" = "" ]; then
     error "Command git status --porcelain failed at the end!"
  fi

  # verifies correct version is downloaded
  GIT_VERSION="$(git describe --tags --abbrev=0)"
  if [ "$GIT_VERSION" == "$VERSION" ]; then
    git describe
    echo "Git version matches the desired version ($VERSION)."
  else
    git describe
    echo "Git version does not match the desired version ($VERSION)."
    exit 3
  fi
}

function build() {
  # build the project
  cd "$HOME"/"$DM"

  echo "Building gateway"
  "$HOME"/"$DM"/"$DM" --flavor "$FLAVOR_GATEWAY" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_GATEWAY"
  
  echo "Building workstation"
  "$HOME"/"$DM"/"$DM" --flavor "$FLAVOR_WORKSTATION" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_WORKSTATION"
}

function pack() {
  # cp binary tar.gz files into another directory for easy access
  mkdir "$HOME"/"$WB"
  cp -vr "$HOME"/"$DB"/"$VERSION_NUMBER"/*.utm.tar.gz "$HOME"/"$WB"
  echo "Build of $NAME version $VERSION is finished"
}

function main() {
  echo "Starting $NAME build for version $VERSION" 
  update
  install
  if [ "$REPOSITORY_EXIST" == "false" ]; then
    repo_down
  fi
  fetch_tag
  build
  pack
} 

# starts scripts
echo "Starting $NAME build function"
cd "$HOME"

# everything from here on will be included in build-log
main 2>&1 | tee -a "$HOME"/"$BL"

# cp build log to binary easy access folder and completing the script
mv "$HOME"/build-log "$HOME"/"$WB"/
echo "Build function finished, please check log"
cd "$HOME"
echo "Its recommended that you verify the signing key on the repository and build, more info here:"
echo "https://www.whonix.org/wiki/Signing_Key"
exit 0

So this is what i have done so far in my script at https://github.com/Goldeneye128/whonix-script

  1. i am using the shellcheck
  2. added all set commands and -x for debug now. But might remove x when done.
  3. added a simple trap ERR i found online so i can see where it fails.
  4. Signing is just displayed not required. Echo note at the end where to find more info on how to check signing self.
  5. Cannot run as root or without arguments with the tag you wanna build
    6, everything is now variables, no mentioning of whonix elsewhere in the code apart from var.
  6. if there is no repo, the master branch will be downloaded first. If there is a repo then it will fetch everything needed for the spesified tag.
  7. no cleanup function anymore
  8. did also do some smaller changes on how the script behaves.

everything is tested and it can download multiple tags now. There are some times fetching fails tough? like when it sync for submodules it kinda get errors? but so far i have found out it just rerun the code then it works. If you got any tips there do please tell me. And yeah comments on how to improve this in anyways? or anything special you guys want from this?. If you find this satisfactory then i will tag it and be done for it for now.

1 Like
15

Made even more improvements tonight for the script. At this point i am not sure what else to do. Asked chatgpt how i could improve my code even more until there was nothing else to do. Added more absolute path to git and apt. Made more traps and error checks. Also removed set -o errexit as its a redundancy of set -e and tried my best to improve this as much as i could.

#!/bin/bash

# do not run this as root
if [ "$(id -u)" = "0" ]; then
   echo "Error: Do not run this as root!"
   exit 1
fi

# check if argument is not missing
if [ -z "$1" ]; then
	echo "Error: Missing tag argument. Please provide an tag argument."
	exit 2
fi

# set options
set -e
set -o nounset
set -o errtrace
set -o pipefail

# variables and configuration
NAME="whonix"
DM="derivative-maker"
DB="derivative-binary"
BL="build-log"
VERSION="$1"
VERSION_MASTER="master"
VERSION_NUMBER="${VERSION%%-*}"
REPOSITORY_URL="https://github.com/Whonix/derivative-maker.git"
FLAVOR_GATEWAY="whonix-gateway-xfce"
FLAVOR_WORKSTATION="whonix-workstation-xfce"
TARGET="utm"
ARCH="arm64"
TB="open"
REPO="true"
SIZE_GATEWAY="15G"
SIZE_WORKSTATION="25G"
WB="whonix-binary-$VERSION"
APT="/usr/bin/apt"
GIT="/usr/bin/git"

# catching errors as they occurre
function catch() {
  echo "Error Report"
  if [ "$1" != "0" ]; then
    echo "Error $1 occurred on line $2, command: '$3'"
    # Additional error handling can go here
  fi
}

# SIGINT or SIGTERM
function handle_signal() {
  echo "Error: received SIGINT or SIGTERM exiting"
  exit 3
}

# start text
function beginning() {
  echo "Starting $NAME build for version $VERSION" 
}

# finishing text
function bye() {
  echo "Exiting without errors."
  echo "Build function finished, please check log"
  echo "Its recommended that you verify the signing key on the repository and build, more info here:"
  echo "https://www.whonix.org/wiki/Signing_Key"
}

# safe exit of the script
function safe_exit() {
  trap bye EXIT
  exit 0
}

function check_apt_sources() {
  echo "Checking APT sources availability..."
  if ! sudo "$APT" update &> /dev/null; then
      echo "Error: APT sources are not reachable. Please check your sources.list and internet connection."
      exit 4
  fi
}

function check_connectivity() {
  local HOST="github.com"
  echo "Checking connectivity to $HOST..."
  if ! ping -c 1 -W 2 "$HOST" &> /dev/null; then
      echo "Error: Unable to reach $HOST. Please check your internet connection."
      exit 5
  fi
}

function repository_not_found() {
  echo "Error: $HOME/$DM not found"
  exit 6
}

# Set up trap
trap 'catch $? $LINENO "$BASH_COMMAND"' EXIT
trap handle_signal SIGINT SIGTERM

function update() {
  # updates/upgrades and autoremove packages
  check_apt_sources
  sudo "$APT" update -y
  sudo "$APT" upgrade -y
  sudo "$APT" autoremove -y
}

function install() {
  # install dependencies
  local PACKAGES=("git" "time" "curl" "apt-cacher-ng" "lsb-release" "fakeroot" "fasttrack-archive-keyring")
  # Loop through each package
  for PACKAGE in "${PACKAGES[@]}"; do
      if ! dpkg -l "$PACKAGE" &> /dev/null; then
          echo "$PACKAGE is not installed. Installing..."
          sudo "$APT" install -y "$PACKAGE"
      else
          echo "$PACKAGE is already installed."
      fi
  done
}

function repo_down() {
  if [ ! -d "$HOME"/"$DM" ]; then
    # download git repository
    check_connectivity
    echo "Git clone $NAME repository"
    "$GIT" clone --depth=1 --branch "$VERSION_MASTER" --jobs=4 --recurse-submodules --shallow-submodules "$REPOSITORY_URL"
    cd "$HOME"/"$DM" || repository_not_found
  fi
}

function fetch_tag() {
  # verifies git tag with gpg keyring
  cd "$HOME"/"$DM" || repository_not_found

  # Verify if the current directory is a git repository
  if ! "$GIT" rev-parse --git-dir > /dev/null 2>&1; then
    echo "Error: Not a git repository. Please run this script within a git repository."
    exit 7
  fi

  if [ ! "$($GIT status --porcelain)" = "" ]; then
   echo "Error: Command git status --porcelain failed at the beginning!"
   exit 8
  fi

  "$GIT" fetch origin tag "$VERSION"

  "$GIT" checkout "$VERSION"

  "$GIT" verify-tag "$VERSION"

  "$GIT" verify-commit "${VERSION}^{commit}"

  "$GIT" verify-commit HEAD

  "$GIT" submodule sync --recursive >/dev/null

  "$GIT" submodule update --init --recursive --jobs=200 >/dev/null
  
  if [ ! "$($GIT status --porcelain)" = "" ]; then
     echo "Error: Command git status --porcelain failed at the end!"
     exit 9
  fi

  # verifies correct version is downloaded
  GIT_VERSION="$($GIT describe --tags --abbrev=0)"
  if [ "$GIT_VERSION" == "$VERSION" ]; then
    "$GIT" describe
    echo "Git version matches the desired version ($VERSION)."
  else
    "$GIT" describe
    echo "Error: Git version does not match the desired version ($VERSION)."
    exit 10
  fi
}

function build() {
  # build the project
  cd "$HOME"/"$DM" || repository_not_found

  echo "Building gateway"
  "$HOME"/"$DM"/"$DM" --flavor "$FLAVOR_GATEWAY" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_GATEWAY"
  
  echo "Building workstation"
  "$HOME"/"$DM"/"$DM" --flavor "$FLAVOR_WORKSTATION" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_WORKSTATION"
}

function pack() {
  # cp binary tar.gz files into another directory for easy access
  mkdir "$HOME"/"$WB"
  cp -vr "$HOME"/"$DB"/"$VERSION_NUMBER"/*.utm.tar.gz "$HOME"/"$WB"
  echo "Build of $NAME version $VERSION is finished"
}

function main() {
  beginning
  update
  install
  repo_down
  fetch_tag
  build
  pack
} 

# starts scripts
echo "Starting $NAME build function"
cd "$HOME"

# everything from here on will be included in build-log
main 2>&1 | tee -a "$HOME"/"$BL"

# cp build log to binary easy access folder and completing the script
mv "$HOME"/"$BL" "$HOME"/"$WB"/
cd "$HOME"

# exit the script
safe_exit

if i do not hear from anybody after testing the code tomorrow or by the weekend i guess i am done with this for now. Its been a learning experience. Also do tell me if there is a way to improve this or if i have done to much? still wanna hear what else to do here.

Do not run this yet tough as i have not tested this yet! Its 3:20 am now so gonna do it when i wake up.
will also check if there is anything i missed then or improve the readme if i need to?

1 Like
16

  • This quoting style seems a bit weird. Preferable would be:

    “${HOME}/${DM}”

If you do not exit on error, then also digital signature verification issues would get silently ignored.

1 Like
17

Done now i have done everything, hopefully this is a more acceptable script this time?

#!/bin/bash

# do not run this as root
if [ "$(id -u)" = "0" ]; then
  echo "Error: Do not run this as root!"
  exit 1
fi

# check if argument is not missing
if [ -z "$1" ]; then
  echo "Error: Missing tag argument. Please provide a tag argument."
  exit 2
fi

# set options
set -e
set -o nounset
set -o errtrace
set -o pipefail

# variables and configuration
NAME="whonix"
DM="derivative-maker"
DB="derivative-binary"
BL="build-log"
VERSION="$1"
VERSION_MASTER="master"
VERSION_NUMBER="${VERSION%%-*}"
REPOSITORY_URL="https://github.com/Whonix/derivative-maker.git"
FLAVOR_GATEWAY="whonix-gateway-xfce"
FLAVOR_WORKSTATION="whonix-workstation-xfce"
TARGET="utm"
ARCH="arm64"
TB="open"
REPO="true"
SIZE_GATEWAY="15G"
SIZE_WORKSTATION="25G"
WB="whonix-binary-$VERSION"
APT="/usr/bin/apt"
GIT="/usr/bin/git"

# catching errors as they occur
function catch() {
  echo "Error Report"
  if [ "$1" != "0" ]; then
    echo "Error $1 occurred on line $2, command: '$3'"
    exit 11  # Exit on any error caught by this function
  fi
}

# SIGINT or SIGTERM
function handle_signal() {
  echo "Error: received SIGINT or SIGTERM exiting"
  exit 3
}

# start text
function beginning() {
  echo "Starting $NAME build for version $VERSION"
}

# finishing text
function bye() {
  echo "Exiting without errors."
  echo "Build function finished, please check log"
  echo "Its recommended that you verify the signing key on the repository and build, more info here:"
  echo "https://www.whonix.org/wiki/Signing_Key"
}

# safe exit of the script
function safe_exit() {
  trap bye EXIT
  exit 0
}

function check_apt_sources() {
  echo "Checking APT sources availability..."
  if ! sudo "$APT" update &> /dev/null; then
    echo "Error: APT sources are not reachable. Please check your sources.list and internet connection."
    exit 4
  fi
}

function check_connectivity() {
  local HOST="github.com"
  echo "Checking connectivity to $HOST..."
  if ! ping -c 1 -W 2 "$HOST" &> /dev/null; then
    echo "Error: Unable to reach $HOST. Please check your internet connection."
    exit 5
  fi
}

function repository_not_found() {
  echo "Error: ${HOME}/${DM} not found"
  exit 6
}

# Set up trap
trap 'catch $? $LINENO "$BASH_COMMAND"' EXIT
trap handle_signal SIGINT SIGTERM

function update() {
  # updates/upgrades and autoremove packages
  check_apt_sources
  sudo "$APT" update -y
  sudo "$APT" upgrade -y
  sudo "$APT" autoremove -y
}

function install() {
  # install dependencies
  local PACKAGES=("git" "time" "curl" "apt-cacher-ng" "lsb-release" "fakeroot" "fasttrack-archive-keyring")
  # Loop through each package
  for PACKAGE in "${PACKAGES[@]}"; do
    if ! dpkg -l "$PACKAGE" &> /dev/null; then
      echo "$PACKAGE is not installed. Installing..."
      sudo "$APT" install -y "$PACKAGE"
    else
      echo "$PACKAGE is already installed."
    fi
  done
}

function repo_down() {
  if [ ! -d "${HOME}/${DM}" ]; then
    # download git repository
    check_connectivity
    echo "Git clone $NAME repository"
    "$GIT" clone --depth=1 --branch "$VERSION_MASTER" --jobs=4 --recurse-submodules --shallow-submodules "$REPOSITORY_URL"
    cd "${HOME}/${DM}" || repository_not_found
  fi
}

function fetch_tag() {
  cd "${HOME}/${DM}" || repository_not_found

  # Verify if the current directory is a git repository
  if ! "$GIT" rev-parse --git-dir > /dev/null 2>&1; then
    echo "Error: Not a git repository. Please run this script within a git repository."
    exit 7
  fi

  if [ ! "$($GIT status --porcelain)" = "" ]; then
    echo "Error: Command git status --porcelain failed at the beginning!"
    exit 8
  fi

  # fetch the selected git tag version
  "$GIT" fetch origin tag "$VERSION"
  "$GIT" checkout "$VERSION"
  "$GIT" verify-tag "$VERSION"
  "$GIT" verify-commit "${VERSION}^{commit}"
  "$GIT" verify-commit HEAD
  "$GIT" submodule sync --recursive >/dev/null
  "$GIT" submodule update --init --recursive --jobs=200 >/dev/null
  
  if [ ! "$($GIT status --porcelain)" = "" ]; then
    echo "Error: Command git status --porcelain failed at the end!"
    exit 9
  fi

  # verifies correct version is downloaded
  GIT_VERSION="$($GIT describe --tags --abbrev=0)"
  if [ "$GIT_VERSION" == "$VERSION" ]; then
    "$GIT" describe
    echo "Git version matches the desired version ($VERSION)."
  else
    "$GIT" describe
    echo "Error: Git version does not match the desired version ($VERSION)."
    exit 10
  fi
}

function build() {
  cd "${HOME}/${DM}" || repository_not_found

  echo "Building gateway"
  "${HOME}/${DM}/${DM}" --flavor "$FLAVOR_GATEWAY" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_GATEWAY"
  
  echo "Building workstation"
  "${HOME}/${DM}/${DM}" --flavor "$FLAVOR_WORKSTATION" --target "$TARGET" --arch "$ARCH" --tb "$TB" --repo "$REPO" --vmsize "$SIZE_WORKSTATION"
}

function pack() {
  mkdir -p "${HOME}/${WB}"
  cp -vr "${HOME}/${DB}/${VERSION_NUMBER}"/*.utm.tar.gz "${HOME}/${WB}"
  echo "Build of $NAME version $VERSION is finished"
}

function main() {
  beginning
  update
  install
  repo_down
  fetch_tag
  build
  pack
} 

# Script execution starts here
echo "Starting $NAME build function"
cd "$HOME" || { echo "Error: Failed to change directory to $HOME"; exit 12; }

main 2>&1 | tee -a "${HOME}/${BL}"

# Copy the build log to the binary easy access folder and complete the script
mv "${HOME}/${BL}" "${HOME}/${WB}/" || { echo "Error: Failed to move build log"; exit 13; }

cd "$HOME" || { echo "Error: Failed to change directory to $HOME at script end"; exit 14; }

# Exit the script safely
safe_exit

I have tested this and it works. And should meet expectation of script quality. Learned a lot while doing this script. So glad i took the challenge. So this is a simple script that can build whonix fairly simple for mac with apple silicon.

you can see it all there. So now you only need to setup a debian wm, make a user there with sudo priv with nopasswd enabled. Then just import this script somewhere and run it. Also recommend to have the public keys for signing verification. Hopefully it make some peoples life a bit easier when dealing with building the project.

Do please comment if there is anything you guys want me to add or change now. Hope this code is better now Patrick? and that it is something you would consider acceptable.

1 Like
18

great idea on the script just have a couple of issues.

Tested with Debian 12 from UTM Gallery
getutm.app/gallery/debian-12

1.) Had to manually install apt-catcher-ng since it asks you a question in ncurses which causes the script to hang. Maybe just have that as a requirement prior to running script.

2.) On multiple branches I am encountering submodule fetch failures. Here is the ouput from the console log.

Errors during submodule fetch:
	packages/tor-ctrl
	packages/serial-console-enable
	packages/qubes-whonix
	packages/kloak
	packages/hardened_malloc
	packages/hardened-kernel
	packages/developer-meta-files
	packages/anon-meta-packages
	kloak
	bindp
Error Report
Error 1 occurred on line 1, command: 'tee -a "${HOME}/${BL}"'

I uploaded full build log to a issue ticket on the github repo.

19

Hey been on a vacation and will look at these the comming days and try to replicate this. Also had the submodule failing on me. Not to sure how to fix it. Idk if adding --remote would help as then it would track things a bit different. But i do see some people doing

$ git submodule update --force --recursive --init --remote

Do deal with submodules, but not sure what is the best course of action yet. But will look into this.
Also have not had any experience with the apt-catcher-ng stopping on install. But will check it out on a fresh install on debian and see what i can do.

20

General purpose:

There’s a build script. It’s called derivative-maker. It’s in a git repository. Why do we need another build script in another git repository to run the build script? Why can’t derivative-maker itself handle this?

apt-cacher-ng:

The reason why 1200_prepare-build-machine doesn’t handle installation of apt-cacher-ng is that derivative-maker always runs APT through apt-cacher-ng. That isn’t possible while apt-cacher-ng isn’t installed.

Related:
Repository Caching

Yes. That is what Whonix ™ VM Build Documentation currently does.

Full cycle. whonix-script running into, not solving the same issues that derivative-maker couldn’t solve.

So to solve this for real, simplify user documentation, require less prerequisites, automate more, derivative-maker needs to handle this.

Another thing which it doesn’t handle yet at time of writing is sudo installation and sudo setup (see build documentation). Also missing is VirtualBox installation (unrelated in this forum thread but also doable thanks to the Linux Installer).

Interactive APT questions:

That is why apt-get-noninteractive (and dpkg-noninteractive) have been invented.

So how to use it? The whonix-script could re-invent, duplicate that code, which I would consider unclean.

issues:

That seems like somehow a very old git revision has been fetched. I don’t know where textual string packages/tor-ctrl is coming from. Nowadays the folder structure is:

packages/kicksecure/...
packages/whonix/...
21

Hey i have just fixed most of the problem you have experienced, i apologized for the inconvenience.

To explain a bit the apt-catcher-ng problem have been solved by including a noninteractive variable before running the install, i also saw how dpkg query worked and was not optimal as not all dependencies was installed when doing this, also added some new and extra dependencies as required. The portion of the code looks like this now:

  # install dependencies
  local PACKAGES=("git" "time" "curl" "apt-cacher-ng" "lsb-release" "dpkg-dev" "fakeroot" "fasttrack-archive-keyring" "safe-rm" "extrepo-offline-data")
  # Loop through each package
  for PACKAGE in "${PACKAGES[@]}"; do
    if ! dpkg-query -W -f='${Status}' "$PACKAGE" 2>/dev/null | grep -q "install ok installed"; then
      echo "$PACKAGE is not installed. Installing..."
      sudo DEBIAN_FRONTEND=noninteractive "$APT" install -y "$PACKAGE"
    else
      echo "$PACKAGE is already installed."
    fi
  done

also added importing the gpg keys as it would fail if not done so, Still check the gpg key manually after the build is done please for your own safety.

when it comes to the submodule fetch failure it is my fault for not being careful with the fetching process. But the problem was that i forgot to add --depth=1 when fetching so it fetched every tag ever instead of just the one this is now fixed on the latest git commit.

  # Fetch only the specific tag
  "$GIT" fetch --depth=1 origin tag "$VERSION"

is how it looks now.
I have also added a convenience function before the install so if you are using a freshly installed/downloaded debian 12 UTM gallery image you do not have to add any no password line on your sudoers file, it will automatically add this if using the UTM image.

Do please check if everything works, should be the latest github commit.

Just to answer this question Patrick is that my script is in NO WAY a replacement for the derivate-maker script. This is just a simplified user script for lazy people like me who do not wanna type in a bunch of commands and install dependencies all the time when building for whonix for mac m1. So it is what is it is. A user script to do the build process fast and simple without a lot of configurations.

For anyone who are a developers i also recommend doing this process manually and not use my script. But if you are lazy and just want this to work. Then i do not see a reason for not having this user script available for others if they wish to use this.

I believe in making whonix available to as many people as possible, not just devs who understand all of this. And hopefully at least as a temporary solution until something more permanent is made for the mac m1. I hope that this will help a bit on that part. I wish i could help more on the main project, but idk what i could do.

22

Dependency installation has been automated in 17.1.3.4-developers-only and above.

It’s not yet tested but thanks to automated CI testing, I’ll notice soon if something is broken and that would certainly be fixed before the next stable release tag.

23

Well thats news for me, also 17.1.3.4? thats fairly new feature then. Might need to test that in the future.

24

Well if the script will download dependencies, then all it needs is some sort of config file so i do not need to type all the flags all the time then my script is useless. Guess its better in the long term.