Hey all, we’re a team of CS students, considering Whonix for our senior capstone.
It’s required for us to reach out to the development team… and also I suppose I’m looking for maybe something a little more specific than the “Just get Started” directions.
We can help out on code or documentation. Experience in Python, C, Bash Scripting, and of course, competency in git.
Any recommendations for some of the easier (but possibly tedious) problems that need solving? We could help out on bug fixing, testing, or even adding a new feature. We’ve got a whole semester for this and our professor is expecting quite a bit of work.
Whonix consists of a lot components and endless possibilities of enhancing Whonix. I am sure we can find something.
One high priority item currently missing is an automated test suite. Not necessarily the same one that Tails is using. Just something that assists the development of Whonix through automated testing.
There are various Whonix build versions. For example some users started with
When the users upgrade, does some functionality break? When a release Whonix build was released, does something break?
For example recently to harden security we followed recommendation to enable hidepid. This resulted in pkexec based applications being broken such as for example synaptic, gparted, zulucrypt. This wasn’t noticed by any manual tester. And then also when that bug was finally reported, it wasn’t even easy to pinpoint the cause since so many things changed since then.
Therefore Whonix development is currently slowed down by time spent on manual testing as well as finding the causes for bugs.
An automated test suite would do things such as start Tor Browser, visit a clearnet website, visit an onion website; start XChat, connect to an IRC server; mount an encrypted USB disk with zulucrypt CLI, mount an encrypted USB disk with zulucrypt GUI and so forth.
I don’t know how difficult that would be. Didn’t look into it yet.
Bugs of that sort could be rather difficult to fix?
Maybe easier. A rewrite of sdwdate. The current code as historically grown isn’t the cleanest and there’s no unit testing at all. The code flow isn’t trivial to understand currently. Use of python classes might be overkill.
Since it’s already written it may not be too difficult to cleanly rewrite and create unit tests?
Unit testing sclockadj (part of sdwdate) might be a bit more tricky. Needs comparison of host clock with VM clock to check if VM clock was changed into the right direction as expected.
Tedious and simpler - also a lot to learn:
Creating a wiki page with Whonix source code documentation.
Similar to this but for other topics too.
Whonix source code consists of around 2800 source files.
(A lot are repetitive or not interesting files such as COPYRIGHT, changelog files etc.)
I’ve documented what Whonix does in part only in: Whonix Networking Implementation - Developer Documentation - Feedback Wanted!
Other non-networking related source files could use these source comments too:
@fahr3n Thanks for considering helping out Whonix!
Hope I’m not speaking out of place (because I’m not a core Whonix developer), but another unresolved bug which shouldn’t be too hard to figure out and might be fun to hunt down the thing that needs fixing, is this:
As already reported, I’ve tested several other Debian-based VMs (Mint, Ubuntu, Xubuntu, and most importantly, a vanilla Debian 10.2 VM) in the same VBox instance, and only the Whonix Gateway / Workstation VMs have this issue. Wouldn’t that mean it’s probably Whonix-specific?
As already reported, I’ve tested several other Debian-based VMs (Mint, Ubuntu, Xubuntu, and most importantly, a vanilla Debian 10.2 VM) in the same VBox instance, and only the Whonix Gateway / Workstation VMs have this issue. Wouldn’t that mean it’s probably Whonix-specific?
Whonix uses packages from Debian. 100% binary compatible. Any
hypothetical (or real) crash of any application from a Debian package
(such as xfce4-settings or something) can be reported to Debian. Ideally
things are reproduced outside of Whonix on plain Debian too but Debian
never said “Whonix bugs must not be reported here”. With Ubuntu vs
Debian it’s a different story. Ubuntu modifies Debian packages and
recompiles the whole repository.
For the sake of higher probabilities for bug fixes I would like to
suggest this: less pleading (such as about priorities and
responsibilities) and more contributions (such as contacting Debian
and/or VirtualBox).
Thanks for your help. I’ve looked into Tails implementation a great deal, because we definitely need to make sure this is the right difficulty.
We’ve talked it over and whatnot too, talked to my professor. In short, this seems feasible, very doable, for us to create something pretty similar to what Tails has going. The Tails system is really elegant it seems, with a focus on using other established programs, and their Automated Testing Suite linked those programs together in an intelligent fashion, really with some very thin code.
Cucumber is a really big deal in the Tails implementation, because it’s a very “non-technical” method for creating test cases using their formal language. It’s super cool because it prioritizes ease of use. That makes a lot of sense to us, because, whatever system we do design, it needs to be easy for other people to create test cases.
So, what my understanding is that we seem to be very capable of creating a Whonix variant that satisfies much of what the Tails system does, or maybe all of it? Or maybe better than what Tails does? See, so that’s where you come in. We’d like to work with you a bit to establish what exactly you want to build, and how we should package it. It’s important that it gets merged and actually used, after all.
Also, things like what types of software we choose are really important too. Packaging it as a .deb shouldn’t be a problem, but how would we go about getting it added? These are things our professor asked us to make sure. I’ve created a repository for our team, and posted our project proposal there, if you want to get a better detailed view as to what we have here. Happy to be working with you.
Doesn’t have to be all of it. From my perspective, everything that is better that what we have now (which is nothing) is better.
That would be nice but isn’t a requirement.
Great!
Right. Tails can be an inspiration but there isn’t much dependency on Tails. I mean, if any of Tails code can be re-used, the better. Re-using is better then re-invention. But if it seems impractical, if it seems making things more difficult, we need to re-use nothing from Tails.
There is also no requirement “must re-use from Tails”. Tails is an example. That’s about it. No hard dependency. The main goal is to invent something useful for Whonix (and perhaps useful for others too since Open Source). Tails is useful for inspiration on what kind of things are useful to test. I can certainly think off other tests which may be useful too. Such as as “more long term connectivity test”.
Makes packaging simpler. No more need to manually maintain ‘make install’ targets or distribution specific install files such as debian/pkg-name.install.
Files in etc/… in root source folder will be installed to /etc/…, files in usr/… will be installed to /usr/… and so forth. This should make renaming, moving files around, packaging, etc. very simple. Packaging of most packages can look very similar.
I think this could be separated into an “inside-vm” and “outside-vm” part.
The “inside-vm” part isn’t actually really necessarily “inside a VM”. It could be running on a Debian host, a Kicksecure host, a future Whonix-Host, inside a Whonix-Gateway VM or inside a Whonix-Workstation VM. This would be similar to whonixcheck.
I.e. “sudo apt install wats” could install a program which once started, tests various things. This is the most important part.
The “outside-VM” part could be separate. The task of this would be to automatically start a VM, runs commands/tests inside the VM, reboot the VM, shut down the VM. This is virtualizer specific. This is less important. If you could do that too, great, if not, also OK. Recently started a bit developing such a thing:
Could also be KVM (since there is Whonix KVM too) based. Ideally not hardcoded so any KVM function can later be re-implemented for VirtualBox too. (Keeping the “holes” open for that to happen aka “plugin”.)
There might be better, more clever ways to implement that which I am very much open for. This can maybe discussed later.
Let me know if you think such a separation between “test suite” and “run testsuite inside VM automation” makes sense. I don’t want to force any decisions which make this hard/impossible to implement. The implementation should be doable, fun and long term maintainable. That’s why this is a “little ego, high flexibility thing”.
The proposed software will be subsequently referred to as the WATS - Whonix Automated Testing Suite in this report.
This is OK however this might be actually more generic (Debian, Kicksecure, Whonix-Host, Whonix VMs). It doesn’t have to do all at once but perhaps to implement Debian support it would be as simple as deleting the Whonix specific tests. Or perhaps having Whonix specifc tests in a drop-in .d folder and/or having some command line switch or parameter to specify which tests should be run. Other distributions might appreciate to re-run the “basic tests” (if there will be such a thing).
License Proposal
I would like our code to be GPLv3 licensed. As opposed to say, MIT licensed, or Apache Licensed. I personally agree with the GNU foundation and support it’s mission, so their license is my preferred for that reason. Of course, we could discuss other licenses too.
Can be your Open Source license/copyright. This is very doable in this case. Whonix already includes code/licenses which are not under Whonix copyright / license. Nothing new. Examples:
Any of these licenses that you mentioned are OK.
(Others could be discussed.)
If GPL: GPLv3+ preferred over GPLv3 but just a wish, not a requirement.
My understanding is that since we are giving up all of our rights to our code to them, aka “it becomes their code” they could decide to license it under something else. I don’t think this could actually happen, or ever would happen, but it’s a thing.
You can keep the copyright too. All of this can be a written, signed agreement if you like. Copyright / licensing very most likely will find agreement.
(I doubt you are going to suggest proprietary-only licenses.)
However, we are not creating all the test cases. That is why software like Cucumber seems to be so important. If we create the WATS properly, others can contribute the Cucumber scripts which will perform the actual tests.
That sounds awesome and fits well into my “more generic” idea.
the WATS is a Continuous Integration Server?
Not necessarily. But possibly.
See my “test suite” and “run testsuite inside VM automation” idea above.
Are there any public server instances that Whonix could use?
If required I can also rent a server for that. Initially you might want to host it yourself and I and whoever else wants to reproduces according to your instructions the same server setup.
A “server” could also be a spare notebook of mine which I am using for manual testing now currently anyhow. Usually I have a number of VMs there. Whonix-Gateway-XFCE-15.0.0.6.6
Whonix-Workstation-XFCE-15.0.0.6.6
Whonix-Gateway-XFCE (a newer build)
Whonix-Workstation-XFCE (a newer build)
Whonix-Gateway-CLI (a newer build)
Whonix-Workstation-CLI (a newer build)
Kicksecure-XFCE (a newer build)
Kicksecure-CLI (a newer build)
And then I am wondering: does each of them boot, is each of the functional? Still functional after reboot? Can each of them be upgraded without issues from stable repository? Still after reboot? Ok, let’s reset that, upgrade from testers or developers repository? Does still work?
I.e. that’s two main tasks.
Testing a new build of Whonix if it is still fully functional.
Booting existing Whonix VMs, upgrade them and see if something breaks after upgrade.
It’s not so much “i do a commit”, “travis CI detects the commit”, “travis CI automatically builds, installs, tests the package”, “I get an e-mail if some commit broke something”.
Performance
I will need to contact Patrick on performance considerations, because this will limit our design. However, I suspect the performance won’t have be a major concern for User Interface simulation. BUT, I could be wrong.
Indeed performance is of almost no concern. This certainly should not be implemented in C or C++. Otherwise I am not fixated on which languages to use.
There could be a command line switch for normal running tests and short running tests + long running tests. (There are some strange bugs where connectivity for an existing connection breaks after 30 minutes or something but new connection are still possible.)
Patrick suggested we could take things from them. I don’t know if he means we can use their code but I suspect using any of their code is fair game.
Yes, if useful and doable.
For usability testing, we could pull a version of a Whonix package we know is broken (using past commits and the bug fix history), and write a test case that is designed to catch that bug.
That sounds good.
I see no reason to build any form of a GUI.
Indeed. A tool which developers and perhaps testers can run which outputs OK or which tests failed on the usual terminal output.
Let me know if there’s something that I forgot to address or that comes up.
Excellent, we seem to be on the same page for a lot of things.
You mention using “wats” more generically. Have you considered the implications of naming it something like - “whonix-ats”?
In my experience, when distros have tools that are useful, that they make available to other distros, it can drive more traffic to the distro. My immediate thought is with the arch wiki. I basically installed arch at one point just because of that wiki. Same goes for my knowledge of Redhat & Fedora. It was because I needed something (for my arch system) that they had built, which happened to be cross-compatible.
Just an idea, from my marketing knowledge. Brand-awareness right?
__
That sounds awesome and fits well into my “more generic” idea.
More generic I think is something that should result from good design. What we name it I think is separate from what it is capable of.
It would be super cool if we labeled our tests as to which components they work on.
“Debian Test - 012 - (human-readable-label) - Failed! - x xxxxx xxx xxxxxxxxxxx xxxx”
vs
“Whonix Test - 010 (human-readable-label) - Failed! - x xxxxx xxxx xxxxxxxxxx xxxxxxx xx”
We’d curve bug reports to the right people that way. I’m sure some tests wouldn’t quite be able to narrow it down, but it would be nice if we tested from the bottom up, I think.
That opens up the larger discussion as to how the tests should be ordered, and how the instructions for creating the tests should be.
I’m not so knowledgeable on that so I’m all ears for your advice.
Also, do you have any more specifics as to what this suite should/could be capable of? What kind of error information it should return?
–
As for the contract, I’ll make sure to confirm with my team. I don’t want to waste too much time on that, so I’ll get back to you.
Are there any public server instances that Whonix could use?
Personally, I have experience with AWS. It would be pretty neat if I could set up a docker container system, which just lets it all freely run through the cloud. Not sure if there are security concerns with that, I know people can get paranoid.
The key idea really, separating it from docker, would be that IF we went the CI server route, developing an image that could be hosted anywhere, easily, would be ideal.
I see the CI as more of “icing on the cake.” As in, it would come after, if time allows.
There could be a command line switch for normal running tests and short running tests + long running tests. (There are some strange bugs where connectivity for an existing connection breaks after 30 minutes or something but new connection are still possible.)
solid command line options will be desirable. That might be a point of continued communication.
Tails
Glad we’re on the same page.
Few Performance Constraints
Good.
Testing plan
Good.
No GUI
Good.
Inside vs Outside VM idea.
I’m familiar with whonixcheck. I wonder if the inside-vm and outside-vm could communicate?
For compatibility with virtualbox:
I believe libvirt offers an API that should significantly assist that process. Libvirt has some problems but it should be super useful.
I think one of my team members will be working with virtualbox. So, support will happen for it regardless.
Functionally speaking, I think we just need a priority for our tasks to get going. It sounds like the inside-vm would be the priority. Should CI support or outside-VM be priority next?
Once we get into writing up specific implementations I think we’ll have a better idea of what will cost what.
This test suite vs test-suite inside sounds like two different pieces of software.
One would interact heavily with libvirt, the other would interact with the GUI automator, like sikuli.
I wonder, could they communicate? I’m envisioning a server connection between host and guest, which allows commands to be sent from the outside-VM to the inside-VM software. This would allow more types of testing to be seamlessly utilized.
That software would be capable of performing the full installation process automatically, using the libvirt API. It could even test virtualbox + QEMU support. Interesting ideas.
Little ego, high flexibility
Absolutely, we’ll keep this agile.
Cool stuff, it seems like we’re getting somewhere.
No, I didn’t see it that way. Interesting! I always thought people are quick to dismiss. When they see “whonix-something” they immediately dismiss as “for Whonix - not for me (Debian)”. I will leave the naming to you. Rather minor point for me.
Yes.
Ordering of tests: perhaps quick and less likely to fail tests go first? Tests could be numbered.
somenumber_name-of-test
Numbers shouldn’t be direct decrescendos. I.e. not 1, 2, 3. Better 100_somehing. 200_something. If we later add something there will be enough “space” to insert numbers without having to rename existing files.
Both CLI and GUI tests. Did you find a list of tests that Tails is running?
Some ideas (higher → higher priority. Very unfixed priorities.):
Start Tor Browser, visit some websites, visit onion websites, close Tor Browser. Did it properly terminate with exit code 0 or some other? Did the process terminate or is it lingering?
change password
run upgrades
reboot
run whonixcheck --verbose --leak-tests --gui --cli, check if there is a whonixcheck CLI output, if there is a GUI window popup, if that is dismissable using OK or close button
other activities users are doing such as start dolphin, browse some files, open a file in a text editor, type something, save file.
The tests ideally would be aware in which boot mode (normal, recovery, future others) it would boot and which virtual terminal it would run the test in. Maybe that could be an abstraction.
Some tests are virtualizer specific. A lot tests will function in Qubes-Whonix hopefully (such as Tor Browser tests) but shared folder tests are different for VBox / KVM.
If it succeeded: maybe nothing (except when --verbose list all tests that succeeded).
Otherwise: if it failed: perhaps just write what failed, link to the test’s source file and perhaps (low priority) a screenshot (or video?)
Alright, it’s official. We’re clear to start working on the project. I’m not sure if you want to list our team (3 of us) as maintainers, but the odds of us not pushing this project to completion is quite slim. At this point we’d have to redo quite a bit of work if we wanted to do something else. So yeah, we’re moving.
Our current timeline is to push out a hello world super soon, which is where we will need to make sure that we have a way of storing our code that can eventually be merged in when completed. I’m assuming a github repo, correct? That seems to be what our professor desires. This is also our software setup as far as the different working pieces. We could touch base to just make sure it’s all packaged properly, and save us time later.
Then we’ll build a prototype, that contains basic functionality. That should be done by the end of the month, with time to spare for us to get back to you and ensure it’s what you’re looking for.
From there, our plan is to expand outwards, installing additional features (as far as command-line options, quality of life improvements, better support, etc), and eventually, hopefully, moving on to create the outside-vm module afterwards. It’d be nice to have the inside-vm component done by the end of the month in it’s entirety, and after that moving onwards to bigger and better features. One executable that allows the creation of multiple VM’s, and tests that run on those VM’s, would be wonderful. This would involve the outside-inside VM communication as previously discussed, probably over the virtual network using TCP/IP. This would come after the completion of the inside-vm component as a standalone.
From our design meeting: libvirt, and other features that start and stop VM’s would need to run on the host, so those features would actually be part of the outside-vm component, to be built later.
As for the inside-vm component, which is our first priority, software decisions currently are:
behave will be our cucumber-like component. It is python-based and semi-official. It will allow us to keep things in python, with a bash script to get it all going. We’ll eventually create some of your previously suggested test-cases when it comes time to create the prototype.
Dogtail is a possibility for our GUI tool, we’re unsure as to whether or not it’s sufficient, but it has python 3 support so. We’re looking into running that as our working GUI automation tool. Realistically, Dogtail seems to have slow development but it looks mature.
As for any legal stuff, just credit for our work is sufficient. We’ll probably link to the code repo from our github accounts. A resume addition, essentially. Do what you will with it, sell it, whatever.
We’ve also recognized that part of this project involves creating effective documentation as to how to create test-cases and make use of the tool. This should be something that anyone could read, and make sense of. This work is factored in to our plans. It should be easy for other contributors to add test-cases, and we see creating a proper set of test-cases as somewhat outside the scope of our project. We’re just looking to create the test-cases set you suggested, and generally ensure the software is working properly and has all required features. The suite of test cases, and the possible features for the test-case definitions will need to be expanded over time.
As long as it’s sufficient for us to use our own repo, to hopefully be merged when the work is completed, we’ll use the repo previously linked on this post - the one for the project proposal. Our documentation to accompany the software can go on the wiki for that repo.
As soon as there’s a hello world, any test of any slight usefulness, this can be merged.
(There are currently other components which aren’t ready for prime time due to missing integration, documentation, testing but already merged because that makes future work easier such as https://github.com/Whonix/hardened-kernel.)
Github is perfect.
(Other stuff could be considered if you see need.)
Yes. Sounds good.
(I wouldn’t even know any other place than the host to run the outside-vm component.)
Yes, sure. Whonix is an Open Source project with the goal of being an exemplary one and attribution is important. I certainly don’t want to be ever criticized for violating any customary attribution guidelines.
readme / wiki / etc. can list credits / authors
there can be be a credits / authors (or similar file name)
git history will include the name and e-mail address of the committer
on command line invocation the first (few) lines can show authors, year, copyright information (either following customary formats or own formats)
source files top comments design is up to you
anything else
Awesome!
(Btw as for selling, as far as I know, this is permitted by all FSF / OSI approved Free Software / Open Source licenses anyhow for anyone who received a copy as per https://www.gnu.org/philosophy/selling.en.html.)
Yes. For sure. That’s the usual git(hub) workflow anyhow.
After discussing with @Patrick, I would love to help you out with the project. I am a software engineer who works mainly with Rails during the day. I have extensive history in automated test suites, SDLC, agile workflows, CI/CD, etc.
I am happy to build a project board, PR templates, code reviews, etc. I think this would be a good thing for your learning as well, as it is very useful for working on professional teams.
But if you want to allow others to contribute you should probably include instructions for forking.
I’d argue that this information should be in the readme, not the wiki. Most developers (IME) want pretty, decipherable markdown in the app root of their project.
Hey @Mycobee, apologies for taking so long to get back to you! We’ve had a bit of chaos on our end but things are moving again now. I’ll have to check with my professor on you helping out, the only reason why I could see it being an issue is if he’s worried we would lean on your expertise a bit too much. It could possibly make it difficult for him to grade? I am unsure, but I’ll email him now.
About that documentation on the git repo - it’s just a rough placeholder I threw in. Our documentation guy is supposed to be hitting on that soon. We have a private repo we have to maintain for our college which has a more correct install. Thanks for the infographics though, definitely, I’m going to link them to our documentation guy.
@Patrick, to update you - we got a rough proof-of-concept going. It’s nothing special but it’s opening us up to the possibilities of what this can be. It seems to be satisfying all the needs you conveyed, and helped us understand the limitations of this software.
Honestly, I’m not sure how much updating you’d like from us, but - basic timeline stuff:
We have to have this done in about a month and a half (Poster of the completed project on April 14th). Also, it’s required for all of us to graduate, so, I think that’s why the professor is really specific about things.
As for the software:
Behave is super cool, fully in python - and so writing new tests will require nothing but some basic python code and the super easy Gherkin.
The vision, I’m seeing, is to make adding on to this as easy as possible. Realistically that means we need a sane architecture that is easy to use and expand upon, and robust documentation.
A good example of what I mean:
For Behave, feature files are the natural-language component, and all have to have their own corresponding step definition file ( in Python ).
However, feature files can use step definitions that were defined in other feature files, and no ambiguity is allowed.
So, I wrote a step that could be reused, and the step before it shows an example of how other developer’s feature files could interact with it.
DEBUG_MODE = True
TEST_TEXT = “Hello world!”
@given(‘we are testing nano and xterm’)
def step_impl(context):
# applications placed here can be increased, just don’t spell it wrong
context.testApplications = [“nano”,“xterm”]
@given(‘the application(s) to be tested are installed’)
def step_impl(context):
# execute dpkg on each application to check if it’s installed
for application in context.testApplications :
# execute, capture output and error
output = subprocess.check_output(["dpkg", "--status", application], encoding='ascii')
match=re.search(r'Status: install ok installed', output)
#no output found, throw an error
if match is None:
print(output)
raise RuntimeError('Success message not found in dpkg status output!')
else:
print("for program: %s dpkg found status %s" % (application, match.string)) if (DEBUG_MODE) else print("")
pass
So the idea is, as long as someone specifies which applications they are testing in their own feature file, ex:
@given('we are testing ‘an_application’)
def step_impl(context):
context.testApplications = [“an_application”]
They could write a Gherkin script with the lines
Given we are testing ‘an_application’
And the application(s) to be tested are installed
And they could reuse the step defined in a separate feature file, which does that testing.
So that’s opening up a larger discussion for us, and we’re studying other implementations. If you or @Mycobee (or whoever else) wants to weigh in on how this should be done, I’m sure it would be very helpful. But, we’re just a bunch of students so I don’t want to waste ya’lls time. Thanks for the help and sorry again for taking so long to get back. Apparently spam filters will eventually target Whonix.