Continuous Integration (CI) Whonix Testing - Automated Test Suite

Mycobee · November 20, 2022, 9:04pm

It is not quite ready for merge, but the PR can be found here:

github.com/Whonix/derivative-maker

Dynamic vps creation

Whonix:master ← Whonix:dynamic-vps-creation

opened 09:02PM - 20 Nov 22 UTC

Mycobee

+442 -217

## Description This pull request is a large refactor, that implements the fol…lowing things - Dynamically create and destroy digitalocean resources to save money, and use more powerful machines for quicker builds - Refactor tasks in to roles for GUI builds and headless. Reduce reliance on shell scripts and leverage ansible functionality. - Dynamically bootstrap VPS with all needed packages, for repeatability and full host automation - Ensure no lingering VPS resources exist before and after builds, to prevent ballooning cloud bills - Rotate SSH keys for security reasons after accidentally plaintext private key commit - Separate log files for workstation and gateway builds

Mycobee · November 26, 2022, 2:55am

@Patrick I almost have a working build again for commits, running in to problems cloning the source code modules. Thoughts?

 * [new tag]         5.1-1      -> 5.1-1
From https://github.com/Whonix/anon-meta-packages
 * branch            d7a64ffa81fa03c431382a00d55b0cb6c8161eb1 -> FETCH_HEAD
From https://github.com/Kicksecure/anon-apt-sources-list
 * branch            a4e0d9935a6c9ae3a816e7dea32fbc8dc92b4a6d -> FETCH_HEAD
fatal: failed to unpack tree object fbc486e0a1a0b24127180ef901cf643e4a204473
error: Submodule 'packages/kicksecure/sdwdate' could not be updated.
fatal: failed to unpack tree object 76720e379fc8c7f5f2e16ecc0d472b31279a393d
error: Submodule 'packages/kicksecure/systemcheck' could not be updated.
error: Cannot update submodule:
	packages/kicksecure/sdwdate
	packages/kicksecure/systemcheck

Aborting

Patrick · November 26, 2022, 8:09am

Hopefully fixed.

Mycobee · November 26, 2022, 5:15pm

Thank you. Pulling source code is working now. Very close to having working CI again that dynamically builds everything

New problem with gateway VM build, artifacts here:

https://github.com/Whonix/derivative-maker/suites/9518408326/artifacts/451981884

/home/ansible/derivative-maker/packages/kicksecure/genmkfile/usr/share/genmkfile/make-helper-one.bsh: INFO: debdist
/home/ansible/derivative-maker/packages/kicksecure/genmkfile/usr/share/genmkfile/make-helper-one.bsh: INFO: debdsc
dpkg-source: info: using source format '3.0 (custom)'
dpkg-source: info: building developer-meta-files in developer-meta-files_22.1-1.dsc
/home/ansible/derivative-maker/packages/kicksecure/genmkfile/usr/share/genmkfile/make-helper-one.bsh: INFO: deb-pkg-build
E: cannot canonicalize filename /var/cache/pbuilder/base.cow_amd64, does not exist
++ errorhandlergeneral ERR
++ last_failed_exit_code=1
++ last_failed_bash_command='"$source_code_folder_dist/packages/kicksecure/genmkfile/usr/bin/genmkfile" deb-pkg'
++ output_cmd_set
++ '[' -o xtrace ']'
++ output_cmd=true
++ true 'INFO: Middle of function errorhandlergeneral of ././build-steps.d/1140_local-dependencies.'
++ errorhandlerprocessshared ERR
++ last_script=././build-steps.d/1140_local-dependencies

Mycobee · November 27, 2022, 2:02am

@Patrick, CI is working again and this PR can be merged (resolving the previously mentioned errors should make the builds green)

github.com/Whonix/derivative-maker

Dynamic vps creation

Whonix:master ← Whonix:dynamic-vps-creation

opened 09:02PM - 20 Nov 22 UTC

Mycobee

+442 -217

## Description This pull request is a large refactor, that implements the fol…lowing things - Dynamically create and destroy digitalocean resources to save money, and use more powerful machines for quicker builds - Refactor tasks in to roles for GUI builds and headless. Reduce reliance on shell scripts and leverage ansible functionality. - Dynamically bootstrap VPS with all needed packages, for repeatability and full host automation - Ensure no lingering VPS resources exist before and after builds, to prevent ballooning cloud bills - Rotate SSH keys for security reasons after accidentally plaintext private key commit - Separate log files for workstation and gateway builds

Now servers are generated and destroyed automatically, and a large refactor has occured for maintainability.

There should be no more interruptions in CI, and it should be functional from here on out as you work on the core OS features.

Next Steps

WATS automation in CI (should be here shortly, proven to work manually)
Optimizations (Build Whonix VMs in parallel and generate VPS from snapshots instead of building from scratch)
Thorough documentation

Mycobee · November 27, 2022, 2:03am

This is a big and thorough PR. I am happy with this work

Mycobee · November 27, 2022, 3:42am

Looks like building from a tag got closer (no --remote-derivative-packages on tag builds)

Mycobee · November 27, 2022, 3:19pm

I see the new error. I forgot I need to add 127.0.0.1 host localhost in /etc/hosts to satisfy the scripts. Rebuilding on a new branch. Hopefully this should work

Mycobee · November 27, 2022, 3:33pm

--remote-derivative-packages build for commit is still failing. Testing a demo tag build now though to see if that one works

/home/ansible/derivative-maker/packages/kicksecure/genmkfile/usr/share/genmkfile/make-helper-one.bsh: INFO: debdist
/home/ansible/derivative-maker/packages/kicksecure/genmkfile/usr/share/genmkfile/make-helper-one.bsh: INFO: debdsc
dpkg-source: info: using source format '3.0 (custom)'
dpkg-source: info: building developer-meta-files in developer-meta-files_22.1-1.dsc
/home/ansible/derivative-maker/packages/kicksecure/genmkfile/usr/share/genmkfile/make-helper-one.bsh: INFO: deb-pkg-build
E: cannot canonicalize filename /var/cache/pbuilder/base.cow_amd64, does not exist
++ errorhandlergeneral ERR
++ last_failed_exit_code=1
++ last_failed_bash_command='"$source_code_folder_dist/packages/kicksecure/genmkfile/usr/bin/genmkfile" deb-pkg'
++ output_cmd_set
++ '[' -o xtrace ']'
++ output_cmd=true
++ true 'INFO: Middle of function errorhandlergeneral of ././build-steps.d/1140_local-dependencies.'
++ errorhandlerprocessshared ERR
++ last_script=././build-steps.d/1140_local-dependencies
++ trap_signal_type_previous=

Mycobee · November 27, 2022, 7:02pm

@Patrick this PR fixes the most recent issue and is ready to merge

Currently, building full VMs from tags without remote derivative packages is working, and builds are passing

But the issue mentioned above still persists for remote derivative paackage builds from commit

Patrick · November 27, 2022, 7:11pm

Merged. Thank you!

The build worked locally for me:

I didn’t know there’s such a requirement.

Will investigate.

Mycobee · November 27, 2022, 7:12pm

A look at these logs from your developer tag shows the error

No idea why, but adding that line to /etc/hosts/ fixes things

https://github.com/Whonix/derivative-maker/actions/runs/3558225207

Mycobee · November 27, 2022, 7:14pm

To clarify, the build only is breaking when I set --remote-derivative-packages true

Full build is fine

Patrick · November 27, 2022, 7:20pm

Ok, I understand the issue now.

true ‘INFO: build_remote_derivative_pkgs is set to true, skipping cowbuilder setup.’

But the 1140_local-dependencies step uses cowbuilder by default. Will fix ASP.

Mycobee · November 28, 2022, 4:35am

you must have fixed it. Tag builds and commit builds are both working now and green

Patrick · November 28, 2022, 10:26am

Yay!

Mycobee · February 9, 2023, 3:15pm

@Patrick I am getting very close. WATS runner script is executing, but I am running in to some weird failures with apt packages not downloading on a fresh machine. Do the User Agreements have to be completed before the machine will connect to the internet? If so, I am going to have to get orca working.

Would be nice if there was a way to build the OS without the user agreements or a flag to auto-accept them

Read error - read (104: Connection reset by peer) Reading the greet back from SOCKS proxy socks5h://127.0.0.1:9050 failed [IP: 127.0.0.1 9050]
Err:3 tor+https://deb.debian.org/debian bullseye-updates InRelease
  Read error - read (104: Connection reset by peer) Reading the greet back from SOCKS proxy socks5h://127.0.0.1:9050 failed [IP: 127.0.0.1 9050]
Err:4 tor+https://deb.debian.org/debian-security bullseye-security InRelease
  Read error - read (104: Connection reset by peer) Reading the greet back from SOCKS proxy socks5h://127.0.0.1:9050 failed [IP: 127.0.0.1 9050]
Err:5 tor+https://deb.debian.org/debian bullseye-backports InRelease
  Read error - read (104: Connection reset by peer) Reading the greet back from SOCKS proxy socks5h://127.0.0.1:9050 failed [IP: 127.0.0.1 9050]
Reading package lists...
E: Failed to fetch tor+https://deb.debian.org/debian/dists/bullseye/InRelease  Read error - read (104: Connection reset by peer) Reading the greet back from SOCKS proxy socks5h://127.0.0.1:9050 failed [IP: 127.0.0.1 9050]
E: Failed to fetch tor+https://deb.debian.org/debian/dists/bullseye-updates/InRelease  Read error - read (104: Connection reset by peer) Reading the greet back from SOCKS proxy socks5h://127.0.0.1:9050 failed [IP: 127.0.0.1 9050]
E: Failed to fetch tor+https://deb.debian.org/debian-security/dists/bullseye-security/InRelease  Read error - read (104: Connection reset by peer) Reading the greet back from SOCKS proxy socks5h://127.0.0.1:9050 failed [IP: 127.0.0.1 9050]
E: Failed to fetch tor+https://deb.debian.org/debian/dists/bullseye-backports/InRelease  Read error - read (104: Connection reset by peer) Reading the greet back from SOCKS proxy socks5h://127.0.0.1:9050 failed [IP: 127.0.0.1 9050]
E: Failed to fetch tor+https://fasttrack.debian.net/debian/dists/bullseye-fasttrack/InRelease  Read error - read (104: Connection reset by peer) Reading the greet back from SOCKS proxy socks5h://127.0.0.1:9050 failed [IP: 127.0.0.1 9050]
E: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package git-all
E: Unable to locate package python3-behave
E: Unable to locate package python3-pip
E: Unable to locate package python3-pyatspi
E: Unable to locate package python3-dogtail

Patrick · February 9, 2023, 4:22pm

The exact reply is “no”. Internet connectivity isn’t restricted depending on the reply to whonixsetup disclaimer wizard. But what’s required is enabling Tor connectivity. (Setting DisableNetwork 0.) Users usually do that using ACW. (Anon Connection Wizard - Whonix)

In the far future it would be good to have a test for whonixsetup and ACW but for now it will be easiest to complete whonixsetup / ACW using command line tools. [1]

orca?

Why orca?

[1] But indeed on that wiki page something is documented which isn’t documented elsewhere yet. That is:

easo

which automates whonixsetup and ACW acceptance. However, easo runs dsudo easyorca-root and that runs sudo setup-dist-noninteractive 1.

So all is hopefully nicely compartmentalized and there’s already support for headless usage of Whonix.

So in summary, either should work:

A)

easo

B)

dsudo setup-dist-noninteractive 1

Not sure which one is best? I guess easo isn’t bad so we get a free automated test for that feature however with a small overhead for orca. Or dsudo setup-dist-noninteractive 1 which should do the same but without orca.

Please let me know if there’s any issues with the automation / headless way to use Whonix. I’ll fix ASP.

Another thing coming to mind is, Tor might need a while until it’s really functional. And since packages are downloaded over Tor, apt-get update might fail. To avoid this, sleeping 3 to 6 minutes might be required for this to work reliable.

I am also happy to install the dependencies of wats by default so this step won’t be needed.

git-all → git
sufficient?

Installing git by default doesn’t seem great but I’d also be happy and eager to package wats and install it by default. Any issues with that such as how it’s updated? If so, I might have a solution for that in mind too.

Mycobee · February 9, 2023, 4:32pm

I changed to git-all after git didnt work, but am changing it back to the git package now that I realize it doesn’t have to do with package name.

I think not packaging dependencies on Whonix is best. A normal user does not need dogtail or behave testing packages by default. Also I think it is important to verify that the apt functionality is in tact, and downloading a package is a good way to see that.

Regarding orca vs dsudo, I will try the dsudo route first to get it functional. I think having running WATS is more important than getting the additional orca stuff for now.

Lastly, I think it is good not to package WaTS on the Whonix VMs. This has been a bit more labor intensive, but IMO a better long term solution. WATS connects to the internet and runs Tor browser tests, so grabbing the latest master branch of WATS via Tor is a small thing.

Thank you for your patience

Patrick · February 9, 2023, 4:33pm

Ooops. I committed directly because I didn’t know I had write access to your repository. And it’s untested. (Syntax was checked, was OK.)