No files were found with the provided path: ./automated_builder/logs/. No artifacts will be uploaded.
Not sure about this. Would having a whitelist of github orgs (so at least in theory others can set up their own CI server, even though not likely to happen) be a way? Or simply hardcode this somewhere and run only in derivative-maker/derivative-maker?
Something is happening with the ansible vault decryption. It is not repo specific, so maybe similar to the PGP key issue something happened with the ENV var setup.
i gotta do some stuff for my day job but once I am done I will look further in to this and try to figure out what is wrong
I don’t understand why it wouldnt be able to decrypt the ansible vault, since that is unspecific to the repo itself. ./automated_builder/vars/main.yml is encrypted with that ENV password github uses, and should work
https://github.com/adrelanos/derivative-maker is non-functional, also as expected. I didn’t add the ANSIBLE_VAULT_PASSWORD there. That’s because I wound not want the build to be duplicated. There’s no need to make adrelanos/derivative-maker functional either.
(adrelanos/derivative-maker being just a mirror. Until now didn’t have unique contents.)
Just wondering how we could exclude adrelanos/derivative-maker and other private forks to avoid their github actions (and thereby their e-mail inbox) getting spammed with failures.
idk yet I will figure it out though. Maybe an environment variable RUN_AUTOMATED_BUILDER=true in the repo settings to “turn on” github actions or something like that…turned off by default
Working on automatic VPS provisioning and destruction now.
My thoughts are designing it something like this
# If ci_job is triggered by a new tagtag
# spin up large debian droplet meeting minimum system requirements for whonix VMs and GUI
# have automatic workflow scheduled to run in CI hourly that will only destroy the droplet if a VNC connection does not exist
# If ci_job is triggered by a commit
# spin up small headless debian VPS to build VMs with remote packages
# automatically destroy at the end of the CI build after exporting logs
This will allow @Patrick to VNC in to tag built VPS and test things, without having to manually destroy them. It will also allow for quicker iteration with commit build VMs
It will probably take me a few weeks. I will get it running with headless VM commits first, then work on conditionally building the large VM with GUI.
After all this is working, I think it is time to revisit the WATS repo and figure out a clever way to make it run with CI
First coming to mind: How can we reliably avoid spawning tons of droplets but forgetting to shut them down (due to bug), ending up with 100s of droplets that all have to be paid?
Hm. That seems like a huge ask. More than I’d dare asking for. Not sure it’s worth the effort then. Not sure how often I’d actually use the VNC.
The CI as now is very useful as I don’t build every commit / every tag locally.
Every now and then I am building and testing locally anyhow.
Write conditional logic that will not spawn new droplets if droplets already exist, be careful about changing source code in the CI. DigitalOcean has a max limit of 25 droplets set by default, this would prevent hundreds from being created
Alternatively we could simply have 1 large server, and skip automatic droplet creation. This server could run everything and a GUI, 40ish bucks a month. Save lots of time, easy to do.
If we are going to run WATS as it was built, we must have a GUI on the VPS. Alternatively, we could have a smaller server and simply create some sort of CLI testing of the builds, which would be a lot easier to set up than GUI testing of the new builds. But also not as good for verifying things work properly with new whonix builds
curl an address
download some packages
other misc things that can be verified with CLI?? not sure what all would be useful
I want to work to help the project. I am willing to spend time on whatever you feel is most important. I will do my best to stay engaged. You can pick what is most useful to you and the project, and I will build it if I am able.
I dont mind spending time if it is helpful. Building the automated builder CI took probably took total of 50 hours or so (I didn’t count). I do this work outside of my normal job, so thats why I say “weeks”…a few hours here, a few hours there sort of thing…not 100s of hours or anything like that.
I will test this out and make sure it works. If it does work, it will cost $24 a month. I am willing to configure WATS to run on it before asking to transfer VPS to a Whonix specific cloud VPS account.
or $20 a month if we were to switch to Linode. Might be worth it, idk. Is there any reaason to trust one cloud provider over another? Or should price be the main concern? Avoiding microsoft, google, and AWS seems good on principle bc fuckem
Good reasoning. Makes sense. Please do. Linode why not. Nothing security critical will be running on that server. It’s basically a developer usability tool.
@Patrick I hopefully will have a solution in the not too distant future for both tagged VMs and commit VMs…testing it now and fixing bugs
My next question is, do we want CI for building kicksecure VM? If so, when do we want that? Currently derivative-maker CI is set up to follow this logic
# if commit without tag is pushed,
# create Whonix Gateway and Workstation with --remote-derivative-packages
# if tag is pushed
# create full Whonix Gateway and Workstation builds without remote packages
Do we care about building kicksecure?? Or just focus on getting whonix + WATS first, then worry about kicksecure later?
I guess WATS is more important. It seems pretty unlikely (though possible) that Whonix-Gateway and Whonix-Workstation build succeeds but Kicksecure build fails. I guess adding Kicksecure builds would be a lot easier than WATS. Kicksecure builds just need slightly different build parameters.
Not sure it’s worth the effort then. Not sure how often I’d actually use the VNC.