Derivative Maker Automated CI Builder

Tiny issue. In the “wrong” repository that is adrelanos/derivative-maker.

There seems to be an issue:

No files were found with the provided path: ./automated_builder/logs/. No artifacts will be uploaded.

Not sure about this. Would having a whitelist of github orgs (so at least in theory others can set up their own CI server, even though not likely to happen) be a way? Or simply hardcode this somewhere and run only in derivative-maker/derivative-maker?

They are already documented here

Something is happening with the ansible vault decryption. It is not repo specific, so maybe similar to the PGP key issue something happened with the ENV var setup.

i gotta do some stuff for my day job but once I am done I will look further in to this and try to figure out what is wrong :man_shrugging:

I don’t understand why it wouldnt be able to decrypt the ansible vault, since that is unspecific to the repo itself. ./automated_builder/vars/main.yml is encrypted with that ENV password github uses, and should work :confused:

1 Like

Wait a minute. The issue is much, much smaller than you might think

https://github.com/derivative-maker/derivative-maker is functional, as expected.

https://github.com/adrelanos/derivative-maker is non-functional, also as expected. I didn’t add the ANSIBLE_VAULT_PASSWORD there. That’s because I wound not want the build to be duplicated. There’s no need to make adrelanos/derivative-maker functional either.

(adrelanos/derivative-maker being just a mirror. Until now didn’t have unique contents.)

Just wondering how we could exclude adrelanos/derivative-maker and other private forks to avoid their github actions (and thereby their e-mail inbox) getting spammed with failures.

ah I understand now

idk yet I will figure it out though. Maybe an environment variable RUN_AUTOMATED_BUILDER=true in the repo settings to “turn on” github actions or something like that…turned off by default

I will think more about it

1 Like

A simple workaround for now is to turn off github actions manually in your adrelanos/derivative-maker repo

  1. On GitHub.com, navigate to the main page of the repository.
  2. Under your repository name, click Settings.
  3. In the left sidebar, click Actions, then click General.
  4. Under “Actions permissions”, select “Disable Actions”

This will turn it off. Eventually we should set this by default to turned off, and manually turn on when a fork wants to run a CI server

1 Like

Working on automatic VPS provisioning and destruction now.

My thoughts are designing it something like this

# If ci_job is triggered by a new tagtag
# spin up large debian droplet meeting minimum system requirements for whonix VMs and GUI
# have automatic workflow scheduled to run in CI hourly that will only destroy the droplet if a VNC connection does not exist

# If ci_job is triggered by a commit
# spin up small headless debian VPS to build VMs with remote packages
# automatically destroy at the end of the CI build after exporting logs

This will allow @Patrick to VNC in to tag built VPS and test things, without having to manually destroy them. It will also allow for quicker iteration with commit build VMs

It will probably take me a few weeks. I will get it running with headless VM commits first, then work on conditionally building the large VM with GUI.

After all this is working, I think it is time to revisit the WATS repo and figure out a clever way to make it run with CI :thinking:

1 Like

Is everyone having fun?? Fun is the cornerstone !

1 Like

First coming to mind: How can we reliably avoid spawning tons of droplets but forgetting to shut them down (due to bug), ending up with 100s of droplets that all have to be paid?

Hm. That seems like a huge ask. More than I’d dare asking for. :slight_smile: Not sure it’s worth the effort then. Not sure how often I’d actually use the VNC.

The CI as now is very useful as I don’t build every commit / every tag locally.

Every now and then I am building and testing locally anyhow.

But running wats on CI seems highly useful.

Write conditional logic that will not spawn new droplets if droplets already exist, be careful about changing source code in the CI. DigitalOcean has a max limit of 25 droplets set by default, this would prevent hundreds from being created

Alternatively we could simply have 1 large server, and skip automatic droplet creation. This server could run everything and a GUI, 40ish bucks a month. Save lots of time, easy to do.

If we are going to run WATS as it was built, we must have a GUI on the VPS. Alternatively, we could have a smaller server and simply create some sort of CLI testing of the builds, which would be a lot easier to set up than GUI testing of the new builds. But also not as good for verifying things work properly with new whonix builds

  • curl an address
  • download some packages
  • other misc things that can be verified with CLI?? not sure what all would be useful

I want to work to help the project. I am willing to spend time on whatever you feel is most important. I will do my best to stay engaged. You can pick what is most useful to you and the project, and I will build it if I am able.

I dont mind spending time if it is helpful. Building the automated builder CI took probably took total of 50 hours or so (I didn’t count). I do this work outside of my normal job, so thats why I say “weeks”…a few hours here, a few hours there sort of thing…not 100s of hours or anything like that.

1 Like

Could we also use the existing server to run VMs and run wats?

Testing that includes GUI seems exponentially more useful.

Yes, but we will need to make it much larger. Currently it is 2gb of RAM and 50gb disk

How much resources would be required by the “smallest” machine that can effectively run host + GUI + Whonix VMs ??

1 Like

Quote Whonix ™ RAM and VRAM Defaults:

  • Whonix ™ Xfce:
    • Whonix-Gateway Xfce - 1280 MB RAM and 128 MB VRAM.
    • Whonix-Workstation ™ Xfce - 2048 MB RAM and 128 MB VRAM.

(Up to date at time of writing. This might be outdated in the future. Wiki takes precedence.)

Not sure Whonix-Workstation ™ Xfce RAM Saving Mode is worth it. Probably not.

So in conclusion, 4 GB should be sufficient for host + VMs with default settings?

On System Requirements - Whonix we say “10 GB free hard drive space.” (but there’s no automated monitoring should that change).

Seems like the 50 GB disk is sufficient and upgrade RAM from 2 GB to 4 GB?

Ill give it a whirl. Any rec for GUI on the Debian host @Patrick ?

Xfce for sure. Since that’s also what Kicksecure / Whonix are using atm.

1 Like

I will test this out and make sure it works. If it does work, it will cost $24 a month. I am willing to configure WATS to run on it before asking to transfer VPS to a Whonix specific cloud VPS account.

1 Like

or $20 a month if we were to switch to Linode. Might be worth it, idk. Is there any reaason to trust one cloud provider over another? Or should price be the main concern? Avoiding microsoft, google, and AWS seems good on principle bc fuckem

1 Like

Good reasoning. Makes sense. Please do. Linode why not. Nothing security critical will be running on that server. It’s basically a developer usability tool. :slight_smile:

@Patrick I hopefully will have a solution in the not too distant future for both tagged VMs and commit VMs…testing it now and fixing bugs

My next question is, do we want CI for building kicksecure VM? If so, when do we want that? Currently derivative-maker CI is set up to follow this logic

# if commit without tag is pushed, 
# create Whonix Gateway and Workstation with --remote-derivative-packages 
# if tag is pushed 
# create full Whonix Gateway and Workstation builds without remote packages

Do we care about building kicksecure?? Or just focus on getting whonix + WATS first, then worry about kicksecure later?

1 Like

I guess WATS is more important. It seems pretty unlikely (though possible) that Whonix-Gateway and Whonix-Workstation build succeeds but Kicksecure build fails. I guess adding Kicksecure builds would be a lot easier than WATS. Kicksecure builds just need slightly different build parameters.

  • –flavor whonix-gateway-xfce
  • –flavor whonix-workstation-xfce
  • –flavor kicksecure-xfce

But not important which one you pick first.

I am doing WATS then…more valuable. We will create kicksecure stuff later

Testing my code today. I will let you know when it is working, then you can merge my new branch for commit and tag builds

[Imprint] [Privacy Policy] [Cookie Policy] [Terms of Use] [E-Sign Consent] [DMCA] [Contributors] [Investors] [Priority Support] [Professional Support]