[HOME] [DOWNLOAD] [DOCS] [NEWS] [SUPPORT] [TIPS] [ISSUES] [DONATE]

Qubes-Whonix using openQA Instance by Qubes

#1

In Qubes we have openQA instance, which is running various tests. Basically there are two types of them:

  • external: run some script, check output and exit status - this is the majority of the tests, using python unittest framework (https://www.qubes-os.org/doc/automated-tests/)
  • native openQA: click somewhere, then compare (part of) screenshot - we have some tests for qubes manager and other GUI elements done this way; and also tests for installation itself

One of the tests run this way is whonixcheck in all Whonix-related VMs (sys-whonix, anon-whonix, whonix-* templates). Recent example: https://openqa.qubes-os.org/tests/5550

We run them in three modes:

  • new installation image - here only installation tests are run and only basic sanity checks on installed system
  • updates in current-testing repositories
  • proposed changes (open pull-requests)

Recently we also have reporting results back to github, like here: https://github.com/QubesOS/qubes-core-libvirt/pull/21#issuecomment-575955949 (pull request) or here: https://github.com/QubesOS/updates-status/issues/1589#issuecomment-575683166 (update).

What is missing and hard to automate is correlating individual test failures with a specific change (for example which package in current-testing caused problem). We don’t have resources to run test with each package separately and it would also be undesirable, as in the final system all those updates are together.

For Whonix templates, the second point includes also proposed-updates repository.
I can easily add another Whonix repository to the test system. And also some more commands to run and check results. But if some more elaborate tests would be desired, someone would need to write them. I can help with technicalities.

2 Likes
[looking for contributor] automated test suite
#2

Need to read this more carefully, but…

…immediately clear to me that I would like to see the buster-developers repository added there too. If resources allow also all the other repositories, buster-testers and buster (stable).

If I am somehow notified that a change in the developers repository breaks something - the challenge here is that there are many different version combinations (also TemplateBasedVM vs standalone vs DispVM etc.) - then of course I will fix that first. Upload a fixed package to the developers repository. And never migrate any broken package (combination) to any other repository.

1 Like
#3

Any preference? I already have a script commenting on github (existing issues/pull requests), but something else should be easy too (email to a mailing list? irc?)

#4

Added separate group on openqa.qubes-os.org for Qubes-Whonix tests. Example test run: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.1&build=4.1-2020011919&groupid=8
If you click on green dot on “system_tests_update” job, you’ll see:

  • Logs & Assets tab containing list of packages in each template
  • Settings tab with test settings, including WHONIX_REPO set here to developers

Failed “suspend” test is unrelated to Whonix (some issue in Xen).

I need from you:

  • info when should those tests be run (some schedule, like weekly? give you access to triggering them manually?)
  • how would you like be notified about results

Later we can expand the actual tests, but for now it at least verifies that packages successfully install and basic stuff still works (according to whonixcheck).

#5

Sounds great!

I need from you:

  • info when should those tests be run (some schedule, like weekly? give you access to triggering them manually?)

Ideally when new package tags are created but I guess that has to wait
until https://phabricator.whonix.org/T709 exists.

Weekly is good. Manual would be super useful too. If it’s either/or,
then I need manually more. If both are possible, I take that.

Any easy (little time effort on your side) way to trigger these from
command line (such as curl POST)? Then I could automate that with my
package build scripts.

  • how would you like be notified about results

Only one strong preference: I cannot follow (github watch) all of
https://github.com/QubesOS/updates-status/issues since it generates way
too many e-mails. Or is there a way to github e-mail subscribe to 1
specific tag only?

A public place preferred over private - to not add needlessly
centralization into me.

Some dedicated github repository? Great!

I could create a new dedicated mailing list too if that is required.
(Would not want to mix with whonix-devel.)

E-mail me privately: least preferred way since not public.

#6

Ok, I’ll setup both.

Yes, this is basically how it works. There is openqa-client tool for that (looks like it’s packaged in Fedora, but not in Debian), it pulls in ~250 perl packages…
Generally what is does it making a single POST request with fairly simple content, but authentication headers would need to be prepared manually (hmac of API path and timestamp using API secret). If using openqa-client directly would be an issue, I can probably write some ~10 lines of python doing the same.
Credentials request for openQA access sent in private email.

That can be the simplest option for now. Do you want to create one under Whonix org?

2 Likes
#7

I guess I could use qrexec service or fedora chroot for that.

Created https://github.com/Whonix/updates-status invited you with permission “maintain”, Invitation link: https://github.com/Whonix/updates-status/invitations

#8

openqa-client configuration is in ~/.config/openqa/client.conf (you need to create it):

[openqa.qubes-os.org]
key = APIKEY
secret = APISECRET

(replace with actual values you get from the web interface)

Then you can call:

openqa-client --host https://openqa.qubes-os.org isos post DISTRI=qubesos VERSION=4.1 FLAVOR=qubes-whonix ARCH=x64_64 BUILD=4.1-$(date +%Y%m%d%H) UPDATE=1 WHONIX_REPO=developers

Interesting variables:

  • BUILD - arbitrary build number, but needs to be unique and monotonic; it is listed on the main page, so should be meaningful. Doesn’t need to be fully numeric - for example some meaningful word as a suffix is fine
  • VERSION - qubes version to test on - currently supported 4.0 and 4.1
  • WHONIX_REPO - which repository to install updates from (appended to buster- or whatever base distro is there)

Others should stay the same

1 Like
#9

Where to find the API key?

What I can see in web interface:

Logged in as adrelanos

Operators Menu

Medium types Machines
Test suites
Job groups
Scheduled products
Assets
Workers
API help
Changelog
Logout

API help shows:

API help

For developers, devops, and similar folks to get routes is easy. Call script/openqa routes . Even with matching regexes (with -v option).

Hint: Calling any non-existant route from your browser will show an error page with all available routes as help. E.g. try this link.

#10

Try now, upgraded your user.

1 Like
#11 
openqa-client --host https://openqa.qubes-os.org isos post DISTRI=qubesos VERSION=4.0 FLAVOR=qubes-whonix ARCH=x64_64 BUILD=4.1-$(date +%Y%m%d%H) UPDATE=1 WHONIX_REPO=buster-developers

{ count => 0, failed => [], ids => [], scheduled_product_id => 228 }

Does not show up under https://openqa.qubes-os.org/tests running, scheduled or finished.

(Made mistake on command line 4.0 vs 4.1 but that shouldn’t influence this.)

#12

Indeed 0 jobs were added.

Oops, that’s the issue, it should obviously be ARCH=x86_64.

1 Like
#13

That added the job.

openqa-client --host https://openqa.qubes-os.org isos post DISTRI=qubesos VERSION=4.0 FLAVOR=qubes-whonix ARCH=x86_64 BUILD=4.0-$(date +%Y%m%d%H) UPDATE=1 WHONIX_REPO=buster-developers

{
count => 3,
failed => [],
ids => [5640, 5641, 5642],
scheduled_product_id => 229,
}

#14

As per https://github.com/Whonix/updates-status/issues/2

WHONIX_REPO=buster-developers resulted in buster-buster-developers.

Try now:

openqa-client --host https://openqa.qubes-os.org isos post DISTRI=qubesos VERSION=4.0 FLAVOR=qubes-whonix ARCH=x86_64 BUILD=4.0-$(date +%Y%m%d%H) UPDATE=1 WHONIX_REPO=-developers

{
count => 3,
failed => [],
ids => [5643, 5644, 5645],
scheduled_product_id => 230,
}

#15

Should be just WHONIX_REPO=developers (no dash)

1 Like
#16 
openqa-client --host https://openqa.qubes-os.org isos post DISTRI=qubesos VERSION=4.0 FLAVOR=qubes-whonix ARCH=x86_64 BUILD=4.0-$(date +%Y%m%d%H) UPDATE=1 WHONIX_REPO=developers

Run just now.

#17

My own recommended command is against this requirement… All 4.0 tests are now considered older than all 4.1, in practice meaning only 4.1 test runs are visible on the main page…
Can you change it in subsequent builds to put date first and -4.0 or -4.1 as suffix?

1 Like
#18

Ok. Run just now.

openqa-client --host https://openqa.qubes-os.org isos post DISTRI=qubesos VERSION=4.0 FLAVOR=qubes-whonix ARCH=x86_64 BUILD=$(date +%Y%m%d%H)-4.0 UPDATE=1 WHONIX_REPO=developers
[Imprint] [Privacy Policy] [Cookie Policy] [Terms of Use] [E-Sign Consent] [DMCA] [Investors] [Priority Support] [Professional Support]