If the build script and/or the CI scripts check for stray mounts, I guess is best if you decide.
The build script will check for stray mounts anyhow because non-CI builds should get a build error early when there is a stray mount. That test can stay independent from any potential test by the CI scripts.
Usually a CI sets environment variable CI=true. And depending on that, the build script could perform some action it does not do outside of CI.
But if the build script is a clean place to say “please reboot the CI” that is highly questionable. Could be a rather unclean hack.
I guess -l, –lazy won’t help much here. The mount is probably staying forever until reboot.
Sure, if that works.
An additional test that would probably be useful in the CI scripts (similar to the test already in the build script):
losetup_output=$($SUDO_TO_ROOT losetup --all)
if [ "$losetup_output" = "" ]; then
true "INFO: Output of losetup_output is empty. No stray loop devices, OK."
return 0
else
error TODO bail out here and reboot CI server
fi
That is probably ideal and better than what I said above.
Does it work for you if you run that command manually? I mean, the command will exit without error but will the mount point be unmounted indeed? If yes, I’ll happily add it to the build script.
Actually, I might just try that now.
I am very surprised a normal umount doesn’t work but umount with -l / --lazy works. Yeah, then that would mean there’s some sort of bug in umount or some other low level tool or even the kernel.
Is there any way to speed up the builds to only run the relevant steps that have been affected in the code changes?
i.e. - Do we need to build ../monero-gui_0.18.1.0-1_all.deb when you only change a few things in the help steps or something?
Would be nice to have a “light” build or something for iterating more quickly. Currently it takes over 1.5 hours to build fresh workstation and gateway VMs. I’d love it if we could make it where you have the ability to get feedback more quickly
I guess though when troubleshooting you can always just SSH in to the VPS and run the troublesome build step manually and see what is causing the problem. Just a thought though, I want this CI feature to make your life easier
No packages should be built and all packages would be downloaded from the Whonix binary repository. It would skip all the lengthy package creation.
How does that sounds?
How often will we use --remote-derivative-packages true? Maybe for git commits, use it. For git tags, do it “proper” and drop it?
Though, when using --remote-derivative-packages true we would not notice when package builds fail. But that isn’t very likely since if packages are updated, I need to build them locally anyhow.
Absolutely makes sense. In my previous build, a rookie mistake for forgetting $SUDO_TO_ROOT has lead to a failed build. Another 40 minutes to wait for me now until I can see if that is fixed now - unless I do a local build with local hacks which with the CI we’re trying to avoid.
That’s also why I suggested Derivative Maker Automated CI Builder - #74 by Patrick - because then I would hack the build command to a much simpler various to a point where only a minimal raw image gets created with even nothing useful inside just to test various mount / umount to quickly get that fixed.
The below text is to outline longer term goals, and initially I think we can ship this stripped down but getting all this running shouldnt be too heavy of lift
Conditional logic for CI builds
# if ci_trigger == commit
# run build suite using --remote-derivative-packages
# send logs as artifacts to github actions and notify success or failure
# nuke excess VM data (OVAs, VDI, etc.) to save space on VPS
# elsif ci_trigger == tag
# run build suite without using remote derivative packages
# load and start VMs in to VBoxManage
# push OVAs to S3 storage bucket so Devs/Testers can download and experiment
# Allow VNC access to VPS and VMs for quicker testing
# Run the WATS suite on the Tagged VM
This one will probably not be used. Could as well as save storage / storage costs.
(Btw also the build logs can be trimmed. We probably don’t want to keep (all) build logs for each build forever once these take significant space. Also an occasionally manual wipe would probably be ok.)
Using S3 and cloud hosting for development-only purposes without a strong dependency and without introducing any risk that a compromised CI could break security for users is OK.
However, if any testers would have to download something form S3, that would probably be criticized. Not as bad, but kinda similar to Whonix adding Google Analytics to its website (not going to happen!). Might backfire.
Found it already.
++ realpath /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/etc/network/interfaces
realpath: /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/etc/network/interfaces: No such file or directory
Just a test that I added for debugging the mount issue which might be no longer needed caused a non-zero exit code.
But indeed. Finding the error from the log is non-trivial due to the log size. In the latest commit, I prefixed error: or ERROR: (best to search case-insensitive). To find this issue, I rather searched the log for “detected” than “error”.
To make the build log contain the word “error” less often, I’ll refactor, rename some functions (the error handlers). But I do that only once some builds are completely passing.
Already fixed in git.
Not sure it would be faster. I am using the CI now to debug this. Now waiting for the new git commit to show up under https://github.com/Mycobee/derivative-maker/actions to see if the last bug was squashed now. Feel free to contact me on telegram (as previously added). For actual call, many apps will work for me.
As of git commit bf65075d4cf4edc2f3c0291dee37c80c0207c5b7 same as git tag 16.0.7.6-developers-only if I make a local build with --remote-derivative-packages true there is no build issue and no stray mounts. Checked. Both gw and ws build were successful.
(I am building the tag, not commit. There is a slim chance this might make a difference due to perhaps triggering a bug due to too long file names.)
The umount bug avoidance strategy of avoiding to umount non-existing mount points as well as umount --lazy seems to be functional.
True. In the event you needed it for some reason you could scp it from the VPS to a machine.
yes, I was going to keep the bucket password protected, so that CI builds were only traded by people using it for situations where it is known not to be “secure” build. But since it is not needed, it doesn’t matter. Also I was using the digital ocean equivalent, not quite as bad as amazon data collection wise I’d imagine, but who knows what companies do behind the curtains
The logs delete each time a CI build run occurs. If you notice the first run in the create_vm.yml file, it redirects stdout and err to build.log, and additional build steps are append that file. The next time a build runs, the redirect with > overwrites the log. It is a lot of output but storage is no concern, as it heals itself.
If you want to trim the logs, I am open to suggestions on how best to do it.
I have to rebase my branch on your upstream master and force push my commit manually to trigger the build, but once my automated_builder it is merged in to your master and configured in the GH, when you push to derivative-maker it will automatically trigger the build. (I pushed the rebased changes btw)
