automated test suite

Development
1

Not written in stone, means, we can discuss this further.

Tails has an automated test suite:
https://tails.boum.org/contribute/release_process/test/automated_tests/

Whonix needs an automated test suite as well.

  1. Check, if https://github.com/Whonix/Whonix/blob/master/CONTRIBUTING.md is acceptable.
  2. Create an automated test suite and/or adapt whatever is useful from the Tails’ automated test suite.
  3. Share it on git(hub).
  4. Optional: package as .deb.
1 Like
2

9 posts were split to a new topic: Qubes-Whonix using openQA Instance by Qubes

3

I am starting work on this

2 Likes
4

A few questions

  1. How would you like this repository set up? I have whonix forked, and could add automated testing in a seperate folder. I can create a public repository as well and am happy to give write access to @Patrick

  2. Do we want a jenkins server set up as a CI/CD pipe for source code PRs? Jenkins is a bit new to me but I am happy to take a stab at this.

  3. I would prefer to write ruby, and follow their model of

--Given /^I capture all network traffic$/ do
    @sniffer = Sniffer.new("sniffer", $vmnet)
    @sniffer.capture
    add_after_scenario_hook do
      @sniffer.stop
      @sniffer.clear
    end
  end

but without all the ivars

Once it is configured, I feel confident I can write effective tests productively. But getting it spun up may be a bit of a challenge

1 Like
5

This overlaps with this recent iniative:

Seems active. Last commit less than 24 hours ago.

Could you please join that forum thread and view that git repository for your oversight?

That is, because I am not the most senior python programmer and also this test suite seems uncharted territory (at least for me). Therefore it would be good if we can give good feedback to the students early so fundamental issues are noticed early.

2 Likes
6

I will join the thread and do my best. I was really hoping to do this in ruby, as I am very experienced with ruby (many thousands of hours). Ive only spent a few dozen hours or so with python. But the same principles should apply I think.

I will start the conversation with them.

1 Like
7

Automated Test Suite Updates:

Due to lack of communication from the college crew, I started throwing together a suite with @Patrick’s blessing.

I currently have CI setup and basic build checks passing.

Everyone out there on the world wide is welcome to follow along, contribute, critique, etc.

Next Steps:

  • get the VMs working properly (libvirt)
  • point and click functionality to perform true BDD (dogtail & sikuli)
  • create automated project board with test cases
  • take a stab at kernel testing while likely annoying patrick with lots of noob questions around the kernel (hooray for learning!)

Here’s the repo:

And the CI builds:
https://travis-ci.com/Mycobee/whonix_automated_test_suite/builds/

1 Like
8

Awesome!

Yes, we can have two automated test suites. Either fully independent or perhaps later integrated somehow (probably not important). The work on the other test suite shouldn’t block any progress here.

These questions would be ideally asked in this thread kernel recompilation for better hardening or if not suitable there in a separate thread that is because @madaidan, not me did most of that work.

1 Like
9

Project Updates:

Overview:

After discussion with @Patrick and @fahr3n, it seems as though we have a more clear idea of how to move forward and what we can do to actually make this happen. At a high level my primary objectives are:

  1. Successfully build Whonix VM with Vagrant in CI
  2. Configure CI to run @fahr3n’s feature tests
  3. Start writing Kernel Tests

Current Progress, Plans, and Challenges:

After successfully configuring vagrant and libvirt to work in my environment, I began researching heavily what it will take to run an automated testing suite with CI using a Vagrant VM. Essentially, I have having to pivot the CI tooling from TravisCI to Jenkins for this job.

Jenkins already has a plugin for Vagrant for this sort of thing, and it seems nearly impossible with TravisCI.

With that said, these are the action items required from me to move forward with objective #1.

  • Stand up local jenkins server (done)
  • Connect vagrant plugin with Jenkins to build VM in CI environment (in progress)
  • Successfully run automated tests in CI VM

After that is complete, I plan to start integrating test cases.

Feedback, comments, concerns, and questions are all welcome

1 Like
10

Any update?

11

The bringup/porting version of TailToaster to Whonix is available in below link …

  • bring-up is blocked in subcomponent (“remote-shell.rb”), responsible for remote command execution inside guest, fired from Host.
    For this Tails uses TCP socket as interface between Host & Guest

  • Under Whonix-Workstation context, I see only serial/console interface is available between Host & Guest. and virsh cli console command is working fine …

Whonix$ virsh -c qemu:///system console Whonix-Workstation
Connected to domain Whonix-Workstation
Escape character is ^]

user@host:~$ echo $(whoami)
user
user@host:~$ uname -a
Linux host 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2 (2020-04-29) x86_64 GNU/Linux
user@host:~$
  • But not sure howto do it via “ruby-libvirt” API

EfP95uI
EfP95uI1432×1051 172 KB

Any suggestions ?

2 Likes
12

Looks like an excellent initiative! Will look into it soon.

Where’s the source code for original TailsToaster? I would like to look for purpose of license compliance, comparison.

13

Thanks @Patrick

  • Source Code Link
1 Like
14

I recognize this is a huge repository and splitting out the toaster (test suite) part makes very much sense. Otherwise it’s hard to see through / understand / make changes indeed. However, for reasons of license compliance, authorship recognizance, and diff viewing… Could you please:

  • fork (gitlab clone) the repository as is ([1] doesn’t matter gitlab.com, github.com, that is easily changed later),
  • reupload that repository somewhere as is ([1]),
  • keep all git history,
  • the COPYING file must not be deleted,
  • go on a deletion spree (delete all files which are not going to be needed in the standalone WhonixToaster and git commit that mass deletion),
  • add changes on top which are required to make this Whonix specific (can be in a single commit)?

That would help to keep git history intact, authorship recognization intact, could answer the question a few years later “which commit was forked from” and “what are Whonix’s changes since”.

15

Thanks for working on this. There is a way to open a TCP tunnel between guest and host by adding this setting though I’ve never tried it:

1 Like
16

→

17

I propose we close this thread @Patrick to avoid any confusion, and see the thread you linked for progress updates.

1 Like