Qubes-Whonix using openQA Instance by Qubes

marmarek · January 19, 2020, 12:56pm

In Qubes we have openQA instance, which is running various tests. Basically there are two types of them:

external: run some script, check output and exit status - this is the majority of the tests, using python unittest framework (Automated tests | Qubes OS)
native openQA: click somewhere, then compare (part of) screenshot - we have some tests for qubes manager and other GUI elements done this way; and also tests for installation itself

One of the tests run this way is whonixcheck in all Whonix-related VMs (sys-whonix, anon-whonix, whonix-* templates). Recent example: https://openqa.qubes-os.org/tests/5550

We run them in three modes:

new installation image - here only installation tests are run and only basic sanity checks on installed system
updates in current-testing repositories
proposed changes (open pull-requests)

Recently we also have reporting results back to github, like here: Update to 6.0.0 by marmarek · Pull Request #21 · QubesOS/qubes-core-libvirt · GitHub (pull request) or here: core-agent-linux v4.1.7 (r4.1) · Issue #1589 · QubesOS/updates-status · GitHub (update).

What is missing and hard to automate is correlating individual test failures with a specific change (for example which package in current-testing caused problem). We don’t have resources to run test with each package separately and it would also be undesirable, as in the final system all those updates are together.

For Whonix templates, the second point includes also proposed-updates repository.
I can easily add another Whonix repository to the test system. And also some more commands to run and check results. But if some more elaborate tests would be desired, someone would need to write them. I can help with technicalities.

Patrick · January 19, 2020, 1:04pm

Need to read this more carefully, but…

…immediately clear to me that I would like to see the buster-developers repository added there too. If resources allow also all the other repositories, buster-testers and buster (stable).

If I am somehow notified that a change in the developers repository breaks something - the challenge here is that there are many different version combinations (also TemplateBasedVM vs standalone vs DispVM etc.) - then of course I will fix that first. Upload a fixed package to the developers repository. And never migrate any broken package (combination) to any other repository.

marmarek · January 19, 2020, 2:11pm

Any preference? I already have a script commenting on github (existing issues/pull requests), but something else should be easy too (email to a mailing list? irc?)

marmarek · January 20, 2020, 1:09am

Added separate group on openqa.qubes-os.org for Qubes-Whonix tests. Example test run: Qubes OS openQA: Test summary
If you click on green dot on “system_tests_update” job, you’ll see:

Logs & Assets tab containing list of packages in each template
Settings tab with test settings, including WHONIX_REPO set here to developers

Failed “suspend” test is unrelated to Whonix (some issue in Xen).

I need from you:

info when should those tests be run (some schedule, like weekly? give you access to triggering them manually?)
how would you like be notified about results

Later we can expand the actual tests, but for now it at least verifies that packages successfully install and basic stuff still works (according to whonixcheck).

Patrick · January 20, 2020, 12:09pm

Sounds great!

I need from you:

info when should those tests be run (some schedule, like weekly? give you access to triggering them manually?)

Ideally when new package tags are created but I guess that has to wait
until ⚓ T709 port Whonix package build process to Qubes package build process exists.

Weekly is good. Manual would be super useful too. If it’s either/or,
then I need manually more. If both are possible, I take that.

Any easy (little time effort on your side) way to trigger these from
command line (such as curl POST)? Then I could automate that with my
package build scripts.

how would you like be notified about results

Only one strong preference: I cannot follow (github watch) all of
Issues · QubesOS/updates-status · GitHub since it generates way
too many e-mails. Or is there a way to github e-mail subscribe to 1
specific tag only?

A public place preferred over private - to not add needlessly
centralization into me.

Some dedicated github repository? Great!

I could create a new dedicated mailing list too if that is required.
(Would not want to mix with whonix-devel.)

E-mail me privately: least preferred way since not public.

marmarek · January 20, 2020, 3:32pm

Ok, I’ll setup both.

Yes, this is basically how it works. There is openqa-client tool for that (looks like it’s packaged in Fedora, but not in Debian), it pulls in ~250 perl packages…
Generally what is does it making a single POST request with fairly simple content, but authentication headers would need to be prepared manually (hmac of API path and timestamp using API secret). If using openqa-client directly would be an issue, I can probably write some ~10 lines of python doing the same.
Credentials request for openQA access sent in private email.

That can be the simplest option for now. Do you want to create one under Whonix org?

Patrick · January 20, 2020, 4:46pm

I guess I could use qrexec service or fedora chroot for that.

Created Whonix/updates-status · GitHub invited you with permission “maintain”, Invitation link: https://github.com/Whonix/updates-status/invitations

marmarek · January 20, 2020, 5:58pm

openqa-client configuration is in ~/.config/openqa/client.conf (you need to create it):

[openqa.qubes-os.org]
key = APIKEY
secret = APISECRET

(replace with actual values you get from the web interface)

Then you can call:

openqa-client --host https://openqa.qubes-os.org isos post DISTRI=qubesos VERSION=4.1 FLAVOR=qubes-whonix ARCH=x64_64 BUILD=4.1-$(date +%Y%m%d%H) UPDATE=1 WHONIX_REPO=developers

Interesting variables:

BUILD - arbitrary build number, but needs to be unique and monotonic; it is listed on the main page, so should be meaningful. Doesn’t need to be fully numeric - for example some meaningful word as a suffix is fine
VERSION - qubes version to test on - currently supported 4.0 and 4.1
WHONIX_REPO - which repository to install updates from (appended to buster- or whatever base distro is there)

Others should stay the same

Patrick · January 21, 2020, 4:05pm

Where to find the API key?

What I can see in web interface:

Logged in as adrelanos

Operators Menu

Medium types Machines
Test suites
Job groups
Scheduled products
Assets
Workers
API help
Changelog
Logout

API help shows:

API help

For developers, devops, and similar folks to get routes is easy. Call script/openqa routes . Even with matching regexes (with -v option).

Hint: Calling any non-existant route from your browser will show an error page with all available routes as help. E.g. try this link.

marmarek · January 21, 2020, 4:43pm

Try now, upgraded your user.

Patrick · January 22, 2020, 1:23pm

openqa-client --host https://openqa.qubes-os.org isos post DISTRI=qubesos VERSION=4.0 FLAVOR=qubes-whonix ARCH=x64_64 BUILD=4.1-$(date +%Y%m%d%H) UPDATE=1 WHONIX_REPO=buster-developers

{ count => 0, failed => [], ids => [], scheduled_product_id => 228 }

Does not show up under Qubes OS openQA: Test results running, scheduled or finished.

(Made mistake on command line 4.0 vs 4.1 but that shouldn’t influence this.)

marmarek · January 22, 2020, 3:59pm

Indeed 0 jobs were added.

Oops, that’s the issue, it should obviously be ARCH=x86_64.

Patrick · January 23, 2020, 12:50pm

That added the job.

openqa-client --host https://openqa.qubes-os.org isos post DISTRI=qubesos VERSION=4.0 FLAVOR=qubes-whonix ARCH=x86_64 BUILD=4.0-$(date +%Y%m%d%H) UPDATE=1 WHONIX_REPO=buster-developers

{
count => 3,
failed => [],
ids => [5640, 5641, 5642],
scheduled_product_id => 229,
}

Patrick · January 23, 2020, 1:58pm

As per OpenQA test result for build 2022042006-4.1 · Issue #2 · Whonix/updates-status · GitHub

WHONIX_REPO=buster-developers resulted in buster-buster-developers.

Try now:

openqa-client --host https://openqa.qubes-os.org isos post DISTRI=qubesos VERSION=4.0 FLAVOR=qubes-whonix ARCH=x86_64 BUILD=4.0-$(date +%Y%m%d%H) UPDATE=1 WHONIX_REPO=-developers

{
count => 3,
failed => [],
ids => [5643, 5644, 5645],
scheduled_product_id => 230,
}

marmarek · January 23, 2020, 2:04pm

Should be just WHONIX_REPO=developers (no dash)

Patrick · January 23, 2020, 2:42pm

openqa-client --host https://openqa.qubes-os.org isos post DISTRI=qubesos VERSION=4.0 FLAVOR=qubes-whonix ARCH=x86_64 BUILD=4.0-$(date +%Y%m%d%H) UPDATE=1 WHONIX_REPO=developers

Run just now.

marmarek · January 30, 2020, 10:16pm

My own recommended command is against this requirement… All 4.0 tests are now considered older than all 4.1, in practice meaning only 4.1 test runs are visible on the main page…
Can you change it in subsequent builds to put date first and -4.0 or -4.1 as suffix?

Patrick · January 31, 2020, 9:47am

Ok. Run just now.

openqa-client --host https://openqa.qubes-os.org isos post DISTRI=qubesos VERSION=4.0 FLAVOR=qubes-whonix ARCH=x86_64 BUILD=$(date +%Y%m%d%H)-4.0 UPDATE=1 WHONIX_REPO=developers