Derivative Maker Automated CI Builder

Development
83

The below text is to outline longer term goals, and initially I think we can ship this stripped down but getting all this running shouldnt be too heavy of lift

Conditional logic for CI builds

# if ci_trigger == commit
  # run build suite using --remote-derivative-packages
  # send logs as artifacts to github actions and notify success or failure
  # nuke excess VM data (OVAs, VDI, etc.) to save space on VPS
# elsif ci_trigger == tag
  # run build suite without using remote derivative packages
  # load and start VMs in to VBoxManage 
  # push OVAs to S3 storage bucket so Devs/Testers can download and experiment 
  # Allow VNC access to VPS and VMs for quicker testing
  # Run the WATS suite on the Tagged VM

Thoughts @Patrick?/

84

@Patrick it is unclear to me what happened with this build

https://github.com/Mycobee/derivative-maker/actions/runs/3023870954

Any chance we could hop on a Jitsi call at your convenience and debug together? Would love to speed up our iteration a bit

85

Perfect!

Exceptā€¦

This one will probably not be used. Could as well as save storage / storage costs.

(Btw also the build logs can be trimmed. We probably donā€™t want to keep (all) build logs for each build forever once these take significant space. Also an occasionally manual wipe would probably be ok.)

Using S3 and cloud hosting for development-only purposes without a strong dependency and without introducing any risk that a compromised CI could break security for users is OK.

However, if any testers would have to download something form S3, that would probably be criticized. Not as bad, but kinda similar to Whonix adding Google Analytics to its website (not going to happen!). Might backfire.

Found it already.

++ realpath /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/etc/network/interfaces
realpath: /home/ansible/derivative-binary/Whonix-Gateway-XFCE_image/etc/network/interfaces: No such file or directory

Just a test that I added for debugging the mount issue which might be no longer needed caused a non-zero exit code.

But indeed. Finding the error from the log is non-trivial due to the log size. In the latest commit, I prefixed error: or ERROR: (best to search case-insensitive). To find this issue, I rather searched the log for ā€œdetectedā€ than ā€œerrorā€.

To make the build log contain the word ā€œerrorā€ less often, Iā€™ll refactor, rename some functions (the error handlers). But I do that only once some builds are completely passing.

Already fixed in git.

Not sure it would be faster. I am using the CI now to debug this. Now waiting for the new git commit to show up under https://github.com/Mycobee/derivative-maker/actions to see if the last bug was squashed now. Feel free to contact me on telegram (as previously added). For actual call, many apps will work for me.

1 Like
86

Fixed for me locally.

As of git commit bf65075d4cf4edc2f3c0291dee37c80c0207c5b7 same as git tag 16.0.7.6-developers-only if I make a local build with --remote-derivative-packages true there is no build issue and no stray mounts. Checked. Both gw and ws build were successful.

(I am building the tag, not commit. There is a slim chance this might make a difference due to perhaps triggering a bug due to too long file names.)

The umount bug avoidance strategy of avoiding to umount non-existing mount points as well as umount --lazy seems to be functional.

1 Like
87

Git tag 16.0.7.7-developers-only has some build script cosmetic improvements only. (Less unnecessary lsof output.)

1 Like
88

True. In the event you needed it for some reason you could scp it from the VPS to a machine.

yes, I was going to keep the bucket password protected, so that CI builds were only traded by people using it for situations where it is known not to be ā€œsecureā€ build. But since it is not needed, it doesnā€™t matter. Also I was using the digital ocean equivalent, not quite as bad as amazon data collection wise Iā€™d imagine, but who knows what companies do behind the curtains :man_shrugging:

The logs delete each time a CI build run occurs. If you notice the first run in the create_vm.yml file, it redirects stdout and err to build.log, and additional build steps are append that file. The next time a build runs, the redirect with > overwrites the log. It is a lot of output but storage is no concern, as it heals itself.

If you want to trim the logs, I am open to suggestions on how best to do it.

I have to rebase my branch on your upstream master and force push my commit manually to trigger the build, but once my automated_builder it is merged in to your master and configured in the GH, when you push to derivative-maker it will automatically trigger the build. (I pushed the rebased changes btw)

1 Like
89

Yay, build issues fixed! Build https://github.com/Mycobee/derivative-maker/actions/runs/3026018094 completed without error.

Should I merge your branch? Seems very much ready. And the changes also zero risk and non-controversial as they stay contained in one folder that does not affect the build.

Small note:

  • I cannot hit the merge button for pull request Add automated_builder ansible suite to CI by Mycobee Ā· Pull Request #1 Ā· Mycobee/derivative-maker Ā· GitHub as this is a pull request for Mycobee github organisation, not a pull request for derivative-maker github organisation. To be able to hit the merge button, the pull request would have to be opened against the derivative-maker github organisation.
  • Thereā€™s no need for a pull request if that makes additional work on your side. Even if youā€™d say ā€œmy branch is now ready for mergeā€ or something like that, I could just add your branch, fetch and merge using git command line, that I am comfortable with for such cases. Pull request or not is a matter of choice.

As for git push --forceā€¦ Not sure why youā€™re using that, but I donā€™t mind multiple git commits. In other words, Iā€™ve never asked any contributors to squash any commits yet into a single commit or to rewrite git history or any such stuff. Thatā€™s because if I contribute to other projects and being asked to do that, thatā€™s always a bit cumbersome, a deterrent for me. Therefore keeping it simple here and not requesting git history beautification. For the future, I donā€™t mind merging many smaller commits either.

Ah, that sounds good. So not logs clogging the server.

The latest log being 11 MB I guess is still far from running into size issues. Even if it was 50 MB for a more verbose build, that wouldnā€™t be a space issue, I guess?
(But an issue for humans to check the log.)

90

I have a few more things Iā€™d like to do for tags, but I will get it done soon and let you know when it is 100% ready to go in to your repo. Also, we will have to work together to do something in the configuration settings of your repo to make this work. I will send those steps in a PGP message

No worries. For me git work is automatic, and it adds minimal extra work. I dont mind squashing things, rebasing, pull requests, etc. I appreciate clean git histories and commits that are well written, but also I am happy to do what works best for you.

I do think if the project were to grow, implementing some git best practices would be a good thing. But if only a few people are working who know the project well, I donā€™t think it is a big deal.

Itā€™s because I keep rebasing on top of your branch. imagine that 1-4 are the order of commits

my repo looks like:

  • patrick commit 1
  • patrick commit 2
  • rob commit 3

but you do an upstream commit (3) and I want to put my commit on top of it, when I rebase it looks like

  • patrick commit 1
  • patrick commit 2
  • patrick commit 3 (from upstream repo)
  • rob commit 4

Now when I push, git says the histories do not line up. This requires a force push. It is the way git works, basically warning that I have changed history and this is not something it allows you to do unless you know what you are doing (hence the -f flag)

Compared to the size of the VMs, 50mb is no big deal. The VPS has 100gb disk. The Gateway, Workstation, and codebases (binary and derivative maker) are a much larger percentage of storage than logs. But again, self healing non issue.

As for making readable logs, that is a subjective thing and if you feel it should be done in some kind of way I am happy to oblige. I just wanna help make your life more efficient. I will likely never have the security understanding and deep knowledge you do of the OS, so I am just trying to build tools that help you move faster with less time doing manual building/testing.

git used to drive me crazy, but every job Iā€™ve worked has strict git guidelines so I really dont even think much about it anymore. itā€™s mostly automatic. You can fetch my branch, cherry pick my commit, or have an upstream PR at any point you want. It is all easy peasy for me. Whatever is best for you is best for me, just ask !

Give me like a day or two to get everything tidy and we can merge it in to your master

Cheers :slight_smile:

1 Like
91

@Patrick I am doing a final build after some cleanup. If it succeeds, my changes can go in to the upstream master.

Currently, it only builds when you push a tag. I will eventually update to have stripped down builds for commits with --remote-derivative-packages true and conditional logic to check if it is a tagged push or a commit push without a tag

For now, when you push a tag, it will build the VMs and start them on the VPS. You can VNC in and test them if you like

May I have your PGP key to send instructions on config settings with your derivative-maker repo?

Thanks

1 Like
92

I will need to set you up with server access so you can VNC in and test the running VMs if you like after a successful build

1 Like
93

Sure. Always up to date here: Contact - Whonix

1 Like
94

Instructions sent, things have been double checked. Feel free to cherry pick or merge my branch whenever you are ready

CI builds are good to go :slight_smile:

1 Like
95

So now, for next steps I have some thoughts

As for VNC in to server, the VPS is a headless server and the Whonix VMs are headless as well. This is not great for long term testing and usefulness of CI.

I can add more resources and a GUI to the server, for testingā€¦but it will be more expensive. Currently I am paying $12 a month to run the VPS (since it needs a large disk for VDI and everything). If we could direct a few more funds towards the server we can make it larger. For now I am happy donating the 12 bucks a month + the time I put in for it to come together, but Iā€™d rather not spend too much more money.

For now, the CI serverā€™s purpose is to test that builds successfully work. You can also ssh in to the Whonix VMS and verify things work from the command line, just no GUI testing until we work that out.

I think a simple CLI testing suite on the Whonix VMs could be good to add, so at least you can verify things like "VM connects to internet, core functionality in tact, debian packages can be downloaded, etc.)

As for WATS that the students made, we will need a GUI. But even if the server costs $30 bucks a month or something like that, I think it could still be very useful for releasing new whonix versions and saving you time @Patrick

What are your thoughts?

1 Like
96

Yes, sounds great! Server costs can be paid in full by me.

1 Like
97

In time, we could set it up where the CI creates and provisions a digital ocean server, then destroys it after a build. That would save a lot of money

Basically something like

  1. Commit/Tag goes in to derivative-maker
  2. CI build is triggered, makes API call to digital ocean that spins up a provisioned server, passes IP address back to CI
  3. ansible suite configures VPS and builds the VMs,
  4. WATS suite runs on servers
  5. logs passed back to CI as artifacts
  6. CI destroys digitalocean server

Sounds rad to me :sunny:

1 Like
98

:eyes:
https://docs.ansible.com/ansible/latest/collections/community/digitalocean/digital_ocean_droplet_module.html#ansible-collections-community-digitalocean-digital-ocean-droplet-module

1 Like
99

Merged just now! :slight_smile:

1 Like
100

The would be ideal indeed.

101

I will build that next, then work on ā€œdietā€ builds for commits with remote packages

Then I will try to figure out how to make the non-diet builds run WATS, but that may take a little while :thinking:

Let me know if there is anything else more urgent youā€™d prefer I direct my efforts towards. I am flexible :slight_smile:

1 Like
102

Now having this merged for a few daysā€¦

Yeah, I got these instructions a few days ago already but then it kept hanging in my inbox. Sorry for the detail. Itā€™s set up on github now according to instructions, which were rather simple?

I suppose I could re-post the instructions here of course excluding the password?

ā€¦made a new git tag just now. Time for a new release soonish anyhow.

Seems like that worked. I can see it already under https://github.com/derivative-maker/derivative-maker/actions which is great!

Thereā€™s actually 1 thing which is hopefully small and easy.
Currently when downloading the log zip file, itā€™s an archive with several sub folders that I have to click through. Would it be possible to have a ā€œflat archiveā€, that is no sub folders (or just a flat level / very few sub folders)?

1 Like