In Qubes we have openQA instance, which is running various tests. Basically there are two types of them:
- external: run some script, check output and exit status - this is the majority of the tests, using python unittest framework (https://www.qubes-os.org/doc/automated-tests/)
- native openQA: click somewhere, then compare (part of) screenshot - we have some tests for qubes manager and other GUI elements done this way; and also tests for installation itself
One of the tests run this way is whonixcheck
in all Whonix-related VMs (sys-whonix, anon-whonix, whonix-* templates). Recent example: https://openqa.qubes-os.org/tests/5550
We run them in three modes:
- new installation image - here only installation tests are run and only basic sanity checks on installed system
- updates in current-testing repositories
- proposed changes (open pull-requests)
Recently we also have reporting results back to github, like here: https://github.com/QubesOS/qubes-core-libvirt/pull/21#issuecomment-575955949 (pull request) or here: https://github.com/QubesOS/updates-status/issues/1589#issuecomment-575683166 (update).
What is missing and hard to automate is correlating individual test failures with a specific change (for example which package in current-testing caused problem). We don’t have resources to run test with each package separately and it would also be undesirable, as in the final system all those updates are together.
For Whonix templates, the second point includes also proposed-updates repository.
I can easily add another Whonix repository to the test system. And also some more commands to run and check results. But if some more elaborate tests would be desired, someone would need to write them. I can help with technicalities.