Derivative Maker Automated CI Builder

Patrick · August 22, 2022, 10:27am

Is there some sort of protection from too many commits? I keep committing whenever useful. Preferring many small change over a big change. That could result in overwhelming the server with too many builds which “never” finish. Perhaps would be sufficient to build newer git tags only? Because also nowadays when I create a tag I either do so because there wasn’t a tag in a while and it seems worthwhile to “summarize” git with a new tag or because I intend to build that version (manually, not great).

(Sometimes I quickly create another tag because something else that can be improved comes to mind but there’s less “git tag spam” than “git commit spam”.)

Patrick · August 22, 2022, 11:34am

Build succeeded.

Mycobee · August 22, 2022, 2:14pm

I can switch it to build whenever a new tag is released. I am going to keep it as commits for now on my fork, so I can troubleshoot as I work. But once we get it in to the upstream main, I will switch it to build new tags only.

In the meanwhile, building on commits is fine because they don’t run in parrallel. If you were to make a whole lot of commits, like 20 an hour, it would create a long line of jobs but the server shouldnt crash or anything.

Mycobee · August 22, 2022, 2:16pm

The upside I see to building it on commits is when we get WATS integrated or if you are troubleshooting a build

something like

push commit
CI triggers build on VPS
See if build is successful
Run automated test suite
VNC in to the whonix VMs and poke around if necessary
Create tag if all above criteria pass

My thought is that it could be useful for you to know everything is groovy before you tag, without having to check it all manually.

IMO it should create builds for both new tags and new commits

Mycobee · August 22, 2022, 2:21pm

I will have an S3 bucket that the OVAs will be pushed to, in organized folders

Something like

builds/
├── commits
│   └── 3d69f4b
│       ├── 3d69f4b-gateway-vm.ova
│       └── 3d69f4b-workstation-vm.ova
└── tags
    └── 16.0.6.7-developers-only
        ├── 16.0.6.7-developers-only-gateway-vm.ova
        └── 16.0.6.7-workstation-vm.ova

I will of course give you login credentials to the bucket. I can give you login credentials to the server too, although I would like to build this in a way where you dont even have to think about the server beyond knowing it exists and is working for you behind the scenes

I will likely set it up to clear the commits folder pretty regularly (once a week or something) so that it doesnt get too huge with images, but I will try to set it up to always keep the last 5 tags or something in storage

Mycobee · August 22, 2022, 2:56pm

The good news @Patrick is that the non interactive build works now with passwordless sudo

The bad news is new strange failures, which can be viewed in build.log

logs

Patrick · August 22, 2022, 7:13pm

As for the git commits and git tags that sounds really good!

That is really weird indeed.

DEBUG manual reproduction:
cd /home/ansible/derivative-maker/packages/kicksecure/monero-gui
DEBUG manual reproduction:
find ‘.’ -not -path ‘/.git/’ -not -path ‘/.git’ -not -name ‘.gitignore’ -not -path './debian’ -print0 | tar --null --no-recursion --create --owner=0 --group=0 --numeric-owner --mode=go=rX,u+rw,a-s --sort=name --mtime=‘2015-10-21 00:00Z’ --xz --file=‘/home/ansible/derivative-binary/genmkfile-packages-result/monero-gui_0.18.0.0.0.orig.tar.xz’ -T -

Killed
tar: /home/ansible/derivative-binary/genmkfile-packages-result/monero-gui_0.18.0.0.0.orig.tar.xz: Cannot write: Broken pipe

A standard Linux command if failing. Getting killed by something other than derivative-maker. Which then breaks the build. To manually reproduce…

mkdir -p /home/ansible/derivative-binary/genmkfile-packages-result

cd /home/ansible/derivative-maker/packages/kicksecure/monero-gui

find '.' -not -path '*/\.git/*' -not -path '*/\.git' -not -name '.gitignore' -not -path './debian*' -print0 | tar --null --no-recursion --create --owner=0 --group=0 --numeric-owner --mode=go=rX,u+rw,a-s --sort=name --mtime='2015-10-21 00:00Z' --xz --file='/home/ansible/derivative-binary/genmkfile-packages-result/monero-gui_0.18.0.0.0.orig.tar.xz' -T -

Does the issue still happen when trying this manually? What could be killing the process? Something that if there is too much system load, kills the process? A low RAM issue might be the most likely cause?

Mycobee · August 22, 2022, 9:17pm

I think you were right about the RAM. I added some resources and think I got past that error. Now I am seeing a new one.

Thank you for your patience. I am not very familiar with how these builds work. I think we are getting close

See build.log again for new issues
https://github.com/Mycobee/derivative-maker/suites/7923343518/artifacts/338014284

INFO: Script running as as non-root, ok.
INFO: Running 'sudo --non-interactive test -d /usr' to test if sudo password entry prompt is needed...
sudo: unable to resolve host host: No address associated with hostname
INFO: sudo password already previously cached (entered) or this system has passwordless sudo, ok.
INFO: root_check ok.
/home/ansible/derivative-maker/help-steps/pre INFO: set -x
++ true '/home/ansible/derivative-maker/help-steps/pre INFO: End of script, ok.'
+ source /home/ansible/derivative-maker/help-steps/colors
++ '[' '!' true = true ']'
+ source /home/ansible/derivative-maker/help-steps/variables
++ true '/home/ansible/derivative-maker/help-steps/variables INFO: begin'
++ '[' '!' '' = true ']'
++ true '/home/ansible/derivative-maker/help-steps/variables INFO: dist_build_no_unset_xtrace is not yet to true. Therefore set +x.'
++ set +x
sudo: unable to resolve host host: No address associated with hostname
INFO: Setting... export UWT_DEV_PASSTHROUGH="1"
INFO: Variable dist_build_version was already set to: 4edaa2e08373519cf5e24c0b76e76f04d2b69d93
sudo: unable to resolve host host: No address associated with hostname
/home/ansible/derivative-maker/help-steps/variables INFO: set -x
++ true '/home/ansible/derivative-maker/help-steps/variables INFO: End of script, ok.'
+ main
+ trap error_handler_unchroot-raw ERR INT TERM
+ sync
+ '[' 0 = 1 ']'
+ sudo --non-interactive umount /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/dev/pts
sudo: unable to resolve host host: No address associated with hostname
umount: /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/dev/pts: no mount point specified.
+ true
+ sudo --non-interactive umount /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/dev
sudo: unable to resolve host host: No address associated with hostname
umount: /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/dev: no mount point specified.
+ true
+ sudo --non-interactive umount /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/proc
sudo: unable to resolve host host: No address associated with hostname
umount: /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/proc: no mount point specified.
+ true
+ sudo --non-interactive umount /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/sys
sudo: unable to resolve host host: No address associated with hostname
umount: /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/sys: no mount point specified.
+ true
+ sudo --non-interactive umount /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/usr/bin/udevadm
sudo: unable to resolve host host: No address associated with hostname
umount: /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/usr/bin/udevadm: no mount point specified.
+ true
+ sudo --non-interactive rm --force /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/usr/bin/udevadm
sudo: unable to resolve host host: No address associated with hostname
+ sudo --non-interactive umount /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image//mnt/initialdeb
sudo: unable to resolve host host: No address associated with hostname
umount: /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image//mnt/initialdeb: no mount point specified.
+ true
+ sudo --non-interactive rmdir --ignore-fail-on-non-empty /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image//mnt/initialdeb
sudo: unable to resolve host host: No address associated with hostname
rmdir: failed to remove '/home/ansible/derivative-binary/Whonix-Gateway-XFCE_image//mnt/initialdeb': No such file or directory
+ true
+ sudo --non-interactive umount /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/etc/resolv.conf
sudo: unable to resolve host host: No address associated with hostname
umount: /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/etc/resolv.conf: no mount point specified.
+ true
+ sudo --non-interactive umount /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/etc/hosts
sudo: unable to resolve host host: No address associated with hostname
umount: /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/etc/hosts: no mount point specified.
+ true
+ sudo --non-interactive umount /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/etc/hostname
sudo: unable to resolve host host: No address associated with hostname
umount: /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/etc/hostname: no mount point specified.
+ true
+ sudo --non-interactive umount /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/etc/network/interfaces
sudo: unable to resolve host host: No address associated with hostname
umount: /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/etc/network/interfaces: no mount point specified.
+ true
+ '[' 0 = 1 ']'
+ sudo --non-interactive /home/ansible/derivative-maker/help-steps/umount_kill.sh /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/
sudo: unable to resolve host host: No address associated with hostname
-> Attempting to kill any processes still running in '/home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/' before un-mounting
+ sync
+ exithandler
+ local exit_code=0
+ '[' '!' 0 = 0 ']'
++ benchmarktimeend 1661202575
+++ date +%s
++ benchmarktimeend=1661202575
++ benchmark_took_seconds=0
+++ convertsecs 0
+++ local h m s
+++ (( h=0/3600 ))
+++ true
+++ (( m=(0%3600)/60 ))
+++ true
+++ (( s=0%60 ))
+++ true
+++ printf '%02d:%02d:%02d\n' 0 0 0
++ echo 00:00:00
+ benchmark_took_time=00:00:00
+ output_cmd_set
+ '[' -o xtrace ']'
+ output_cmd=true
+ '[' 0 = 0 ']'
+ true 'INFO: End of: /home/ansible/derivative-maker/help-steps/unchroot-raw | exit_code: 0 | error(s) detected: 0 | benchmark: 00:00:00'
+ exit 0
++ /home/ansible/derivative-maker/help-steps/unprevent-daemons-from-starting
+ set -e
+ true 'INFO: Currently running script: /home/ansible/derivative-maker/help-steps/unprevent-daemons-from-starting '
+++ dirname /home/ansible/derivative-maker/help-steps/unprevent-daemons-from-starting
++ cd /home/ansible/derivative-maker/help-steps
++ pwd
+ MYDIR=/home/ansible/derivative-maker/help-steps
+ source /home/ansible/derivative-maker/help-steps/pre
++ true '/home/ansible/derivative-maker/help-steps/pre INFO: begin'
++ '[' '!' '' = true ']'
++ true '/home/ansible/derivative-maker/help-steps/pre INFO: dist_build_no_unset_xtrace is not yet to true. Therefore set +x.'
++ set +x
INFO: Script running as as non-root, ok.
INFO: Running 'sudo --non-interactive test -d /usr' to test if sudo password entry prompt is needed...
sudo: unable to resolve host host: No address associated with hostname
INFO: sudo password already previously cached (entered) or this system has passwordless sudo, ok.
INFO: root_check ok.
/home/ansible/derivative-maker/help-steps/pre INFO: set -x
++ true '/home/ansible/derivative-maker/help-steps/pre INFO: End of script, ok.'
+ source /home/ansible/derivative-maker/help-steps/colors
++ '[' '!' true = true ']'
+ source /home/ansible/derivative-maker/help-steps/variables
++ true '/home/ansible/derivative-maker/help-steps/variables INFO: begin'
++ '[' '!' '' = true ']'
++ true '/home/ansible/derivative-maker/help-steps/variables INFO: dist_build_no_unset_xtrace is not yet to true. Therefore set +x.'
++ set +x
sudo: unable to resolve host host: No address associated with hostname
INFO: Setting... export UWT_DEV_PASSTHROUGH="1"
INFO: Variable dist_build_version was already set to: 4edaa2e08373519cf5e24c0b76e76f04d2b69d93
sudo: unable to resolve host host: No address associated with hostname
/home/ansible/derivative-maker/help-steps/variables INFO: set -x
++ true '/home/ansible/derivative-maker/help-steps/variables INFO: End of script, ok.'
+ main
+ sync
+ sudo --non-interactive rm --verbose --force /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/usr/sbin/policy-rc.d
sudo: unable to resolve host host: No address associated with hostname
+ sync
+ exithandler
+ local exit_code=0
+ '[' '!' 0 = 0 ']'
++ benchmarktimeend 1661202575
+++ date +%s
++ benchmarktimeend=1661202576
++ benchmark_took_seconds=1
+++ convertsecs 1
+++ local h m s
+++ (( h=1/3600 ))
+++ true
+++ (( m=(1%3600)/60 ))
+++ true
+++ (( s=1%60 ))
+++ printf '%02d:%02d:%02d\n' 0 0 1
++ echo 00:00:01
+ benchmark_took_time=00:00:01
+ output_cmd_set
+ '[' -o xtrace ']'
+ output_cmd=true
+ '[' 0 = 0 ']'
+ true 'INFO: End of: /home/ansible/derivative-maker/help-steps/unprevent-daemons-from-starting | exit_code: 0 | error(s) detected: 0 | benchmark: 00:00:01'
+ exit 0
++ /home/ansible/derivative-maker/help-steps/unmount-raw
+ set -e
+ true 'INFO: Currently running script: /home/ansible/derivative-maker/help-steps/unmount-raw '
+++ dirname /home/ansible/derivative-maker/help-steps/unmount-raw
++ cd /home/ansible/derivative-maker/help-steps
++ pwd
+ MYDIR=/home/ansible/derivative-maker/help-steps
+ source /home/ansible/derivative-maker/help-steps/pre
++ true '/home/ansible/derivative-maker/help-steps/pre INFO: begin'
++ '[' '!' '' = true ']'
++ true '/home/ansible/derivative-maker/help-steps/pre INFO: dist_build_no_unset_xtrace is not yet to true. Therefore set +x.'
++ set +x
INFO: Script running as as non-root, ok.
INFO: Running 'sudo --non-interactive test -d /usr' to test if sudo password entry prompt is needed...
sudo: unable to resolve host host: No address associated with hostname
INFO: sudo password already previously cached (entered) or this system has passwordless sudo, ok.
INFO: root_check ok.
/home/ansible/derivative-maker/help-steps/pre INFO: set -x
++ true '/home/ansible/derivative-maker/help-steps/pre INFO: End of script, ok.'
+ source /home/ansible/derivative-maker/help-steps/colors
++ '[' '!' true = true ']'
+ source /home/ansible/derivative-maker/help-steps/variables
++ true '/home/ansible/derivative-maker/help-steps/variables INFO: begin'
++ '[' '!' '' = true ']'
++ true '/home/ansible/derivative-maker/help-steps/variables INFO: dist_build_no_unset_xtrace is not yet to true. Therefore set +x.'
++ set +x
sudo: unable to resolve host host: No address associated with hostname
INFO: Setting... export UWT_DEV_PASSTHROUGH="1"
INFO: Variable dist_build_version was already set to: 4edaa2e08373519cf5e24c0b76e76f04d2b69d93
sudo: unable to resolve host host: No address associated with hostname
/home/ansible/derivative-maker/help-steps/variables INFO: set -x
++ true '/home/ansible/derivative-maker/help-steps/variables INFO: End of script, ok.'
+ main
+ '[' 0 = 1 ']'
+ unmount_raw
+ trap errorhandlerunmount-raw ERR INT TERM
+ '[' '' = '' ']'
+ true
+ sync
+ sudo --non-interactive /home/ansible/derivative-maker/help-steps/umount_kill.sh /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/
sudo: unable to resolve host host: No address associated with hostname
-> Attempting to kill any processes still running in '/home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/' before un-mounting
+ sync
+ wait 205274
+ sleep 2
+ sync
+ '[' '' = true ']'
+ sudo --non-interactive umount /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image
sudo: unable to resolve host host: No address associated with hostname
umount: /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image: not mounted.
++ errorhandlerunmount-raw
+++ caller
++ true 'BASH_COMMAND: $SUDO_TO_ROOT umount "$CHROOT_FOLDER"
ERROR /home/ansible/derivative-maker/help-steps/unmount-raw: | caller: 52 /home/ansible/derivative-maker/help-steps/unmount-raw'
++ exit 1
+ exithandler
+ local exit_code=1
+ '[' '!' 1 = 0 ']'
+ exit 1
++ errorhandlermaybeexit ERR
++ '[' '' = true ']'
++ '[' false = true ']'
++ abort_or_continue=Aborted
++ rm --force /etc/apt/sources.list.d/whonix_temp.list
++ rm --force /run/package_manager_lock
++ '[' false = true ']'
++ '[' ERR = INT ']'
++ '[' ERR = TERM ']'
++ trap - EXIT
++ true 'INFO: Now exiting from ././build-steps.d/2300_run-chroot-scripts-post-d (because error was detected, see above) with exit code 1.'
++ exit 1
++ error_handler_dist_build_one
++ true 'ERROR in /home/ansible/derivative-maker/derivative-maker'
++ true 'BASH_COMMAND: "./$dist_build_one_build_step_current"'
++ true 'dist_build_one_build_step_current: ./build-steps.d/2300_run-chroot-scripts-post-d'
++ true 'args: --flavor whonix-gateway-xfce --target virtualbox --build'
++ true 'INFO: Now exiting from /home/ansible/derivative-maker/derivative-maker (because error was detected, see above).'
++ exit 1
+ exithandler
+ local exit_code=1
+ '[' '!' 1 = 0 ']'
+ exit 1

Mycobee · August 22, 2022, 9:18pm

I am pretty sure this is the issue, and can be resolved with /etc/hostname but not sure what it should be named for these whonix builds

Mycobee · August 22, 2022, 11:15pm

I got past the above errors by adding the following line to /etc/hosts

127.0.0.1 localhost host

New errors below. logs

INFO: Script running as as non-root, ok.
INFO: Running 'sudo --non-interactive test -d /usr' to test if sudo password entry prompt is needed...
INFO: sudo password already previously cached (entered) or this system has passwordless sudo, ok.
INFO: root_check ok.
/home/ansible/derivative-maker/help-steps/pre INFO: set -x
++ true '/home/ansible/derivative-maker/help-steps/pre INFO: End of script, ok.'
+ source /home/ansible/derivative-maker/help-steps/colors
++ '[' '!' true = true ']'
+ source /home/ansible/derivative-maker/help-steps/variables
++ true '/home/ansible/derivative-maker/help-steps/variables INFO: begin'
++ '[' '!' '' = true ']'
++ true '/home/ansible/derivative-maker/help-steps/variables INFO: dist_build_no_unset_xtrace is not yet to true. Therefore set +x.'
++ set +x
INFO: Setting... export UWT_DEV_PASSTHROUGH="1"
INFO: Variable dist_build_version was already set to: 4edaa2e08373519cf5e24c0b76e76f04d2b69d93
/home/ansible/derivative-maker/help-steps/variables INFO: set -x
++ true '/home/ansible/derivative-maker/help-steps/variables INFO: End of script, ok.'
+ main
+ '[' 0 = 1 ']'
+ unmount_raw
+ trap errorhandlerunmount-raw ERR INT TERM
+ '[' '' = '' ']'
+ true
+ sync
+ sudo --non-interactive /home/ansible/derivative-maker/help-steps/umount_kill.sh /home/ansible/derivative-binary/Whonix-Workstation-XFCE_image/
-> Attempting to kill any processes still running in '/home/ansible/derivative-binary/Whonix-Workstation-XFCE_image/' before un-mounting
+ sync
+ wait 612712
+ sleep 2
+ sync
+ '[' '' = true ']'
+ sudo --non-interactive umount /home/ansible/derivative-binary/Whonix-Workstation-XFCE_image
+ sync
+ '[' '' = '' ']'
+ local img=/home/ansible/derivative-binary/4edaa2e08373519cf5e24c0b76e76f04d2b69d93/Whonix-Workstation-XFCE-4edaa2e08373519cf5e24c0b76e76f04d2b69d93.Intel_AMD64.raw
+ wait 612717
+ sleep 2
+ sync
+ sudo --non-interactive losetup --all
/dev/loop0: [65025]:274308 (/home/ansible/derivative-binary/4edaa2e08373519cf5e24c0b76e76f04d2b69d93/Whonix-Workstation-XFCE-4edaa2e08373519cf5e24c0b76e76f04d2b69d93.Intel_AMD64.raw)
+ sync
+ sudo --non-interactive kpartx -d -s -v /home/ansible/derivative-binary/4edaa2e08373519cf5e24c0b76e76f04d2b69d93/Whonix-Workstation-XFCE-4edaa2e08373519cf5e24c0b76e76f04d2b69d93.Intel_AMD64.raw
loop deleted : /dev/loop0
del devmap : loop0p1
+ sync
+ sudo --non-interactive losetup --all
+ sync
+ '[' '' = true ']'
+ ls -la /home/ansible/derivative-binary/Whonix-Workstation-XFCE_image
total 8
drwxr-xr-x 2 ansible ansible 4096 Aug 22 22:57 .
drwxr-xr-x 8 ansible ansible 4096 Aug 22 23:06 ..
+ '[' '' = true ']'
+ sudo --non-interactive rmdir /home/ansible/derivative-binary/Whonix-Workstation-XFCE_image
+ sync
+ exithandler
+ local exit_code=0
+ '[' '!' 0 = 0 ']'
++ benchmarktimeend 1661209563
+++ date +%s
++ benchmarktimeend=1661209568
++ benchmark_took_seconds=5
+++ convertsecs 5
+++ local h m s
+++ (( h=5/3600 ))
+++ true
+++ (( m=(5%3600)/60 ))
+++ true
+++ (( s=5%60 ))
+++ printf '%02d:%02d:%02d\n' 0 0 5
++ echo 00:00:05
+ benchmark_took_time=00:00:05
+ output_cmd_set
+ '[' -o xtrace ']'
+ output_cmd=true
+ '[' 0 = 0 ']'
+ true 'INFO: End of: /home/ansible/derivative-maker/help-steps/unmount-raw | exit_code: 0 | error(s) detected: 0 | benchmark: 00:00:05'
+ exit 0
++ errorhandlermaybeexit ERR
++ '[' '' = true ']'
++ '[' false = true ']'
++ abort_or_continue=Aborted
++ rm --force /etc/apt/sources.list.d/whonix_temp.list
++ rm --force /run/package_manager_lock
++ '[' false = true ']'
++ '[' ERR = INT ']'
++ '[' ERR = TERM ']'
++ trap - EXIT
++ true 'INFO: Now exiting from ././build-steps.d/1700_install-packages (because error was detected, see above) with exit code 1.'
++ exit 1
++ error_handler_dist_build_one
++ true 'ERROR in /home/ansible/derivative-maker/derivative-maker'
++ true 'BASH_COMMAND: "./$dist_build_one_build_step_current"'
++ true 'dist_build_one_build_step_current: ./build-steps.d/1700_install-packages'
++ true 'args: --flavor whonix-workstation-xfce --target virtualbox --build'
++ true 'INFO: Now exiting from /home/ansible/derivative-maker/derivative-maker (because error was detected, see above).'
++ exit 1
+ exithandler
+ local exit_code=1
+ '[' '!' 1 = 0 ']'
+ exit 1

It feels like we are getting very close to success

Patrick · August 23, 2022, 10:14am

How much RAM is not enough and how much RAM was sufficient? Asking because, maybe I could

the build script could check if there is enough RAM and show a more useful error message and/or
perhaps exclude building that package from CI build somehow to safe RAM (and expenses).

Sure thing.

Broken link. (Not important since solved.)

Glad that links is functional. Here is the actual error.

INFO: Running connectivity check...  Downloading...: https://www.torproject.org
INFO: CURL_OUT_FILE: /var/cache/tb-binary/.cache/tb/temp/tbb_remote_folder

ERROR: https://www.torproject.org could not be reached.

Possible reasons:
- https://www.torproject.org/dist/torbrowser is down
- download location changed

Workaround: Skipping to download Tor Browser. Append to derivative-maker build command:

--tb open

It would then fail open, i.e. not break the build. We’d still see that Tor Browser download didn’t succeed in the logs and would unblock you from completing a full build and/or detecting other build issues until this one is fixed.

But that’s not the real fix of course.

Does the server block connections to torproject.org?

For debugging could you please try:

curl --head https://www.torproject.org

Also:

nslookup torproject.org

Another possibility is that it was a transient issue, though maybe not likely.

Yet another possibly is some DNS issue inside the chroot. If not easily resolved, I might need to add more debugging code to get to the bottom of this.

Mycobee · August 24, 2022, 4:30pm

I think it was a one off thing, because I got past that error

I have a new problem, and it has to do with the working directory structure.

I mounted a block storage volume for VM and things, and I put it a folder in it as a symlink in /home/ansible/automated_build

The issue I am running in to is that the script is using ~/ for some path things

cd /home/ansible/automated_build/derivative-maker/help-steps
cd ..
main
check-unicode
/home/ansible/automated_build/derivative-maker/packages/kicksecure/developer-meta-files/usr/bin/dm-check-unicode
set -e
true '/home/ansible/automated_build/derivative-maker/packages/kicksecure/developer-meta-files/usr/bin/dm-check-unicode: START.'
folder=/home/ansible/derivative-maker
wrapper=/home/ansible/derivative-maker/packages/kicksecure/helper-scripts/usr/bin/grep-find-unicode-wrapper
test -d /home/ansible/derivative-maker
errorhandlergeneral ERR
last_failed_exit_code=1
last_failed_bash_command='"${dist_developer_meta_files_folder}/usr/bin/dm-check-unicode"'
output_cmd_set
'[' -o xtrace ']'
output_cmd=true
true 'INFO: Middle of function errorhandlergeneral of ././build-steps.d/1100_sanity-tests.'
errorhandlerprocessshared ERR

/home/ansible/derivative-maker does not exist, and I want it to use /home/ansible/automated_build/derivative-maker instead

Is there a way to config this as the default working directory instead of ~/?

Mycobee · August 24, 2022, 4:34pm

basically i’d like these three lines

folder=/home/ansible/derivative-maker
wrapper=/home/ansible/derivative-maker/packages/kicksecure/helper-scripts/usr/bin/grep-find-unicode-wrapper
test -d /home/ansible/derivative-maker

to function like this

folder=/home/ansible/automated_build/derivative-maker
wrapper=/home/ansible/automated_build/derivative-maker/packages/kicksecure/helper-scripts/usr/bin/grep-find-unicode-wrapper
test -d /home/ansible/automated_build/derivative-maker

Mycobee · August 24, 2022, 4:35pm

On the upside, I am getting better at reading these build logs lol

Patrick · August 25, 2022, 12:13am

Done in 16.0.6.8-developers-only.

Mycobee · August 25, 2022, 3:54pm

It is working for the MYDIR variable, but I found another one that slipped by when calling source @Patrick

MYDIR=/home/ansible/automated_build/derivative-maker/packages/kicksecure/developer-meta-files/usr/bin
cd /home/ansible/automated_build/derivative-maker/packages/kicksecure/developer-meta-files/usr/bin
source /home/ansible/derivative-maker/help-steps/pre
/home/ansible/automated_build/derivative-maker/packages/kicksecure/developer-meta-files/usr/bin/dm-reprepro-wrapper: line 20: /home/ansible/derivative-maker/help-steps/pre: No such file or directory
errorhandlergeneral ERR
last_failed_exit_code=1
last_failed_bash_command='"$dist_developer_meta_files_folder/usr/bin/dm-reprepro-wrapper" includedeb "$dist_build_apt_codename" "$package_absolute_path"'
output_cmd_set
'[' -o xtrace ']'
output_cmd=true
true 'INFO: Middle of function errorhandlergeneral of ././build-steps.d/1200_create-debian-packages.'

logs

Patrick · August 26, 2022, 4:36pm

Fixed in 16.0.7.0.

Mycobee · August 28, 2022, 5:48pm

@Patrick, it looks like I am getting very close.

Questions:

When the build commands get run, it puts derivative-binary in ~/. Is there any way we could have it put it in ../ instead. Again, this is because of the way I am mounting the machines and what not.
My CI is successfully building the gateway, which is great news. When building the workstation however, I ran in to an error about being unable to umount. Relevant logs are at the bottom of post.
I am definitely thinking that only building these VMs when a tagged is pushed is the way to go, because it is taking CI over an hour to build the VMs for both the gateway and the workstation. So when developing derivative-maker, you can push a developers only tag for troubleshooting, or perhaps even create a debug tag in order to iterate. Is that okay?

Logs for question 2:

Full Logs

+ sudo --non-interactive umount /home/ansible/derivative-binary/Whonix-Workstation-XFCE_image
umount: /home/ansible/derivative-binary/Whonix-Workstation-XFCE_image: target is busy.
++ errorhandlerunmount-raw
+++ caller
++ true 'BASH_COMMAND: $SUDO_TO_ROOT umount "$CHROOT_FOLDER"
ERROR /home/ansible/automated_build/derivative-maker/help-steps/unmount-raw: | caller: 52 /home/ansible/automated_build/derivative-maker/help-steps/unmount-raw'
++ exit 1
+ exithandler
+ local exit_code=1
+ '[' '!' 1 = 0 ']'
+ exit 1
++ errorhandlermaybeexit ERR
++ '[' '' = true ']'
++ '[' false = true ']'
++ abort_or_continue=Aborted
++ rm --force /etc/apt/sources.list.d/whonix_temp.list
++ rm --force /run/package_manager_lock
++ '[' false = true ']'
++ '[' ERR = INT ']'
++ '[' ERR = TERM ']'
++ trap - EXIT
++ true 'INFO: Now exiting from ././build-steps.d/1700_install-packages (because error was detected, see above) with exit code 1.'

Mycobee · August 28, 2022, 5:51pm

My ideal goal would be for us to have this CI feature implemented by end of this week, if we can troubleshoot the build issues together.

Thanks for your patience with my challenges on the derivative-maker/derivative-maker build step. Still learning how all of it works

Mycobee · August 28, 2022, 8:11pm

When troubleshooting the “Target is busy” error, I see the following behavior

$ sudo --non-interactive umount /home/ansible/derivative-binary/Whonix-Workstation-XFCE_image
umount: /home/ansible/derivative-binary/Whonix-Workstation-XFCE_image: target is busy.

$ mount | grep derivative
/dev/mapper/loop0p1 on /home/ansible/derivative-binary/Whonix-Workstation-XFCE_image type ext4 (rw,relatime)

Not sure if helpful @Patrick, but I suspect this umount is not working because of whatever is mounting loop0p1