Building Whonix in Qubes

Following are instructions to build whonix-gateway and whonix-workstation templates within a Qubes AppVM. Note that an install-able template will be available soon that you can install directly in dom0 using ‘[font=courier]qubes-dom0-update[/font]’ instead of needing to build from source.

I just re-ran the following steps and confirm they worked for me:

[ul][li]First I suggest you start completely over with a brand new AppVM. I named mine development-qubes. Base it on the ‘Fedora-20-x64’ template.[/li]
[li]Start AppVM from Qubes Manager[/li]
[li]Start a Dom0 Konsole session[/li]
[li]Increase disk size to 10GB per VM you going to build, so in Dom0 Konsole type ‘[font=courier]qvm-grow-private devleopment-qubes 25GB[/font]’[/li]
[li]Start a Terminal session for new AppVM (development-qubes) and enter the following:[/li][/ul]

git clone https://github.com/nrgaway/qubes-builder
cd qubes-builder/
git fetch
git checkout whonix
./README.whonix
  - Answer y <enter> to yum update question

Those steps will build both the gateway and workstation. You can edit ‘[font=courier]examples/whonix.conf[/font]’ (which will be linked as ‘[font=courier]builder.conf[/font]’ after running script) to change build options to build wheezy or jessie or add gnome to those as well (just un-comment out lines you want to build).

There should be no issues listed previously in this thread for any distro except Whonix can not yet update from template and you SHOULD run both the gateway and workstation as standalone until the update situation is resolved.

So, once the templates are built, the last line explains how to get them to Dom0. I actually save that line and created a script in Dom0 I use anytime I need to grab templates I built. So you can edit a file named ‘get-templates.sh’ on dom0 and include the following"

#!/bin/bash

qvm-run --pass-io development-qubes 'cat /home/user/qubes-builder/qubes-src/linux-template-builder/rpm/install-templates.sh' > install-templates.sh
chmod a+x install-templates.sh

Then make the file executable and run it to grab and install the templates on dom0

cd ~
mkdir bin
cd bin
vi get-templates.sh # Add text above
chattr a+x get-templates.sh
./get-templates.sh
cd /tmp
/home/user/bin/get-templates.sh # Grabs install script
./install-template.sh # Will download template rpm; remove old one (if installed); install new ones

In order to get the text I listed to dom0, here is a trick I figured out.

- Highlight text you want to copy.
- Right click with mouse and select 'copy'
- Press <SHIFT>+<CTRL>+<C>
- in dom0 Konsole type 'cat /run/qubes/qubes-clipboard.bin'
- the text you just copied will be displayed in dom0 terminal so now you can highlight it from dom0 terminal and paste it where ever

[HR]

Qubes manager options to select for gateway:

[ul][li]Name: whonix-gateway[/li]
[li]Template: whonix-gateway-experimental[/li]
[li]Type: Proxy VM[/li]
[li]NetVM: firewallvm[/li]
[li]Check Standalone[/li][/ul]

[ul][li]Name: whonix-workstation[/li]
[li]Template: whonix-workstation-experimental[/li]
[li]Type: AppVM[/li]
[li]NetVM: whonix-gateway[/li]
[li]Check Standalone[/li][/ul]

[HR]

Start the gateway from Qubes Manager. The setup screen should come up and ask you about repo and starting Tor. Note that the first two screens where it displays disclaimer that the buttons are not visible, so just press [font=courier][/font], then [font=courier][/font] again for second screen. If you happen to not press [font=courier][/font], or somehow focus gets messed up, VM will power off so you will need to try again.

When you are finished setup, if you get an error message from time proxy or that tor can not do a check since bootstrap not complete, try running ‘[font=courier]whonixcheck[/font]’ again, and it should succeed.

Do same for Workstation. Or you can use a regular AppVM and just select ‘[font=courier]whonix-gateway[/font]’ as its netvm.

Note the first run dialog says the password is ‘[font=courier]changeme[/font]’ when there is actually no password as per Qubes defaults.

[HR]

Some issues with Whonix applications:

[ul][li]Some don’t run maybe?[/li]
[li]Stuff like tor browser may need to be started from terminal at this point[/li]
[li]???[/li][/ul]

[HR]

So now you have your whonix-gateway and whonix-workstation installed remember it is experimental at this point.

[HR]

Test and keep a log of issues that need to be addressed and things that work compared to HVM version.

Do leak tests and report results. This is important.

Don’t use it for anything important yet until leak tests, etc are confirmed.

Please post any build issues or questions in this thread.

Using latest “whonix” git branch, all my previous mentioned build errors are now gone.

Next fatal build error I’m now encountering…

I’ve tried building ~20 times and every time I get this fatal error:

--> Downloading additional sources for vmm-xen... + make --quiet -C qubes-src/vmm-xen get-sources make[1]: *** [xen-4.1.6.1.tar.gz] Error 4 make: *** [get-sources] Error 1 [user@appvm qubes-builder]$

Difference experienced: Sometimes it’s “Error 4” or “Error 8” on that line…

make[1]: *** [xen-4.1.6.1.tar.gz] Error 4
make[1]: *** [xen-4.1.6.1.tar.gz] Error 8

[hr]

Speculative theory…

I’m building over Tor, so maybe that “xen-4.1.6.1.tar.gz” package is large and the script downloading it is not robust or patient enough for a Tor connection?

[FYI: Unrelated anecdotal: I was experiencing similar intermittent problems with packages not downloading over Tor upon first attempt with the Whonix --install-to-root, so Patrick built in an “auto retry” capability to help mitigate such fatal package download issues over Tor in Whonix 10 and beyond.]

I haven’t been able to look into it much yet, so that’s just my guess about what’s happening.

But 20 for 20 build attempts failed on this error for me so far.

Using:

  • Qubes R2
  • Tor Connection
  • Clean AppVM
  • Fully Updated fedora-20-x64
  • Latest “whonix” Git Branch

Build Steps:

git clone https://github.com/nrgaway/qubes-builder
cd qubes-builder/
git fetch
git checkout whonix
./README.whonix
  - Answer y <enter> to yum update question

[quote=“WhonixQubes, post:2, topic:647”]Speculative theory…

I’m building over Tor, so maybe that “xen-4.1.6.1.tar.gz” package is large and the script downloading it is not robust or patient enough for a Tor connection?[/quote]

I just briefly looked inside the code to find the source of the “xen-4.1.6.1.tar.gz” package.

In “/qubes-builder/qubes-src/vmm-xen/Makefile” it shows the source URL as being:

http://bits.xensource.com/oss-xen/release/4.1.6.1/xen-4.1.6.1.tar.gz

That package is 9.9 MB, so it is at least not very large.

But it could take up to a few minutes based on the Tor connection.

Just tested: 2m 37s download over Tor using wget in a same build AppVM as error received.

I see in “/qubes-builder/qubes-src/vmm-xen/xen.spec” that primary xen package looks like the first download from that domain, so I wonder if the other packages would successfully download from that website over Tor if the script were to get past the first failed package. Might try out-commenting the first one and see about the remaining ones next.

[quote=“WhonixQubes, post:2, topic:647”]Using latest “whonix” git branch, all my previous mentioned build errors are now gone.

Next fatal build error I’m now encountering…

I’ve tried building ~20 times and every time I get this fatal error:

Difference experienced: Sometimes it’s “Error 4” or “Error 8” on that line…

[hr]

Speculative theory…

I’m building over Tor, so maybe that “xen-4.1.6.1.tar.gz” package is large and the script downloading it is not robust or patient enough for a Tor connection?

[FYI: Unrelated anecdotal: I was experiencing similar intermittent problems with packages not downloading over Tor upon first attempt with the Whonix --install-to-root, so Patrick built in an “auto retry” capability to help mitigate such fatal package download issues over Tor in Whonix 10 and beyond.]

I haven’t been able to look into it much yet, so that’s just my guess about what’s happening.

But 20 for 20 build attempts failed on this error for me so far.

Using:

  • Qubes R2
  • Tor Connection
  • Clean AppVM
  • Fully Updated fedora-20-x64
  • Latest “whonix” Git Branch

Build Steps:

[code]
git clone GitHub - nrgaway/qubes-builder: Qubes Builder
cd qubes-builder/
git fetch
git checkout whonix
./README.whonix

  • Answer y to yum update question
    [/code][/quote]

Very strange.

Is your Tor netvm the Whonix gateway or the Qubes Tor gateway?

Is transparent TransPort working in the AppVM?

I will take a look later today building over Tor as I have not done so yet.

Different… Using Whonix-Gateway in a separate physically isolated box.

Yes, via physical isolation.

Ok, sounds good. Will be interested to see your results with this.

FYI: Error Observation…

Regarding the timeframe of behavior…

It usually takes about 15 to 20+ minutes between the beginning of this step:

--> Downloading additional sources for vmm-xen... + make --quiet -C qubes-src/vmm-xen get-sources

and the fatal error output:

make[1]: *** [xen-4.1.6.1.tar.gz] Error 4 make: *** [get-sources] Error 1

I finally had a bit of time to try building using the Qubes Whonix gateway.

I did come across the same issue as you but was able to solve it and build successfully.

# Assuming you are already in the qubes-builder directory:
cd qubes-src/vmm-xen
vi Makefile  # Or use any editor like nano

# search for wget, there is only one and change it from:
@wget -qN  $(URL) $(U...
# to
@wget $(URL) $(U...

# And save the file

So, really all I did was remove the -qN and it started to work. N turns on time-stamping. Really don’t know why it does not work with that option on, but it worked with it off.

The gateway seems to be working nicely. Only a few more issues to go then I will update it from 9.2. One weird issue is that some process keeps turning IP forwarding back on. I thought I had it sorted with Marek, but will have to dig a bit deeper.

[quote=“nrgaway, post:7, topic:647”]I did come across the same issue as you but was able to solve it and build successfully.

# Assuming you are already in the qubes-builder directory:
cd qubes-src/vmm-xen
vi Makefile  # Or use any editor like nano

# search for wget, there is only one and change it from:
@wget -qN  $(URL) $(U...
# to
@wget $(URL) $(U...

# And save the file

So, really all I did was remove the -qN and it started to work. N turns on time-stamping. Really don’t know why it does not work with that option on, but it worked with it off.[/quote]

Confirmed. Removing -qN gets past that “xen-4.1.6.1.tar.gz” error for me as well when building over Tor. Thanks!

Documentation of process:

After running “qubes-builder/README.whonix” the first time, it generates the “qubes-builder/qubes-src/vmm-xen/” directories and files, and throws the fatal “xen-4.1.6.1.tar.gz” package error:

--> Downloading additional sources for vmm-xen... + make --quiet -C qubes-src/vmm-xen get-sources make[1]: *** [xen-4.1.6.1.tar.gz] Error 4 make: *** [get-sources] Error 1 [user@appvm qubes-builder]$

After that, edit “qubes-builder/qubes-src/vmm-xen/Makefile” with nrgaway’s instructions:

Assuming you are already in the qubes-builder directory: cd qubes-src/vmm-xen vi Makefile # Or use any editor like nano

search for wget, there is only one and change it from:

@wget -qN $(URL) $(U…

to

@wget $(URL) $(U…

And save the file

Then run the build script again: “qubes-builder/README.whonix”

Another BUILDING OVER TOR consistent fatal package error I receive is with the “grub-0.97.tar.gz” package.

make[1]: *** [grub-0.97.tar.gz] Error 4 make: *** [get-sources] Error 1

The URL used for downloading this package is:

ftp://alpha.gnu.org/gnu/grub/grub-0.97.tar.gz

This ftp;// URL and Tor do not work well together and I never got it to download successfully, even after the “-qN” fix.

[Note: Other ftp:// package URLs do successfully work in this build script, so might just be an issue with this “alpha.gnu.org” domain/website.]

Thankfully this can easily be solved by changing the URL to use the http:// protocol instead.

Documentation of process:

After running “qubes-builder/README.whonix” the first time, it generates the “qubes-builder/qubes-src/vmm-xen/” directories and files, and throws the fatal “grub-0.97.tar.gz” package error:

make[1]: *** [grub-0.97.tar.gz] Error 4 make: *** [get-sources] Error 1

After that, edit “qubes-builder/qubes-src/vmm-xen/Makefile”:

Change the following line from:

GRUB_URL := ftp://alpha.gnu.org/gnu/grub/$(GRUB_FILE)

to:

GRUB_URL := http://alpha.gnu.org/gnu/grub/$(GRUB_FILE)

Save change.

Then run the build script again: “qubes-builder/README.whonix”

Note: this can and should be done at the same time as doing the “xen-4.1.6.1.tar.gz” package “-qN” fix, described here: Whonix Forum

Strange…

Still regarding BUILDING OVER TOR issues.

After addressing the “grub-0.97.tar.gz” package error with changing the ftp:// download link to http://…

The “grub-0.97.tar.gz” fatal error is still occurring, despite successful download over http with Tor.

However, another problem package that is not consistently downloading is “newlib-1.16.0.tar.gz”.

The download URL for this package is:

ftp://sources.redhat.com/pub/newlib/newlib-1.16.0.tar.gz

Unfortunately that server does not seem to accept alternative http:// URLs instead (Not Found: 404 Error).

Anyway, maybe this is a dependent or related package to grub?

Because upon “grub-0.97.tar.gz” downloading successfully, but “newlib-1.16.0.tar.gz” failing, the same fatal error as before is thrown:

make[1]: *** [grub-0.97.tar.gz] Error 4 make: *** [get-sources] Error 1

I’ve seen this error scenario happen 10+ times now, but I think, if I recall, I have seen the “newlib-1.16.0.tar.gz” package go through over Tor once or twice before.

Maybe just fickle Tor blocking policies on web servers?

Maybe another source domain URL has to be found for me to correct this error.

[quote=“WhonixQubes, post:9, topic:647”]Another BUILDING OVER TOR consistent fatal package error I receive is with the “grub-0.97.tar.gz” package.

to:

[quote]
GRUB_URL := http://alpha.gnu.org/gnu/grub/$(GRUB_FILE)
[/quote][/quote]

Another strange error. It did not happen for me but good to know you were able to work around it and you might as well just edit the file to make that change anyway while you are in it to remove the [font=courier]-qN.[/font]

Were you able to successfully build both package yet?

[HR]

Just so you know, the [font=courier]README.whonix[/font] is just a script that executes the [font=courier]Makefile[/font] process in specific order, so after running it for the first time and you reach the spot that gives your your first error, you can manually choose to only run specific [font=courier]Makefile[/font] functions. For example, once all the sources files are downloaded if you run the [font=courier]README.whonix[/font] again, it will attempt to download the sources again and merge them into existing sources (merged, not overwrite) but you can skip that step.

Following is a list of common [font=courier]Makefile[/font] commands:

To get the source files required to build templates. If sources already exist and new changes will be merged with existing sources. All source files (with exception to my personal branches which are not signed since they are temporary and will end up in Qubes repo) are verified automatically on download. If verification fails, the download process will exit with an error stating as such.

make get-sources

To build the Qubes modules that are requires to allow Whonix to operate as an AppVM. No harm in re-running this command either

make template-modules

To actually build the Whonix template (uses [font=courier]debootstrap[/font] to first download a base Debian system which is also verified, then load Whonix and finally the Qubes modules and then creates an RPM so you can install it via dom0). If you are interested all the template code to build Whonix is located in ‘[font=courier]~/qubes-builder/qubes-src/linux-template-builder/scripts_debian[/font]’

make template

Now, if you want to clean the sources, there are about 3 different commands you can you, each cleaning a bit more than the other

make clean
make mostlyclean
make clean-all

[font=courier]clean[/font] just removes complied source code. [font=courier]mostlyclean[/font] pretty much removes everything except the downloaded source files. [font=courier]clean-all[/font], removes everything including source.

[quote=“WhonixQubes, post:10, topic:647”]Strange…

Still regarding BUILDING OVER TOR issues.

After addressing the “grub-0.97.tar.gz” package error with changing the ftp:// download link to http://…

The “grub-0.97.tar.gz” fatal error is still occurring, despite successful download over http with Tor.

However, another problem package that is not consistently downloading is “newlib-1.16.0.tar.gz”.

The download URL for this package is:

ftp://sources.redhat.com/pub/newlib/newlib-1.16.0.tar.gz

Unfortunately that server does not seem to accept alternative http:// URLs instead (Not Found: 404 Error).

Anyway, maybe this is a dependent or related package to grub?

Because upon “grub-0.97.tar.gz” downloading successfully, but “newlib-1.16.0.tar.gz” failing, the same fatal error as before is thrown:

I’ve seen this error scenario happen 10+ times now, but I think, if I recall, I have seen the “newlib-1.16.0.tar.gz” package go through over Tor once or twice before.

Maybe just fickle Tor blocking policies on web servers?

Maybe another source domain URL has to be found for me to correct this error.[/quote]

I really don’t know why this is failing for you. As I had previously stated I was able to build over Tor just by removing the [font=courier]-qN[/font] from the wget.

Seems like your Whonix gateway is not configured to allow ftp connections?

Try this link: http://pkgs.fedoraproject.org/repo/pkgs/xen/newlib-1.16.0.tar.gz (replace in Makefile)

Not yet.

[hr]

[quote=“nrgaway, post:11, topic:647”]Just so you know, the [font=courier]README.whonix[/font] is just a script that executes the [font=courier]Makefile[/font] process in specific order, so after running it for the first time and you reach the spot that gives your your first error, you can manually choose to only run specific [font=courier]Makefile[/font] functions. For example, once all the sources files are downloaded if you run the [font=courier]README.whonix[/font] again, it will attempt to download the sources again and merge them into existing sources (merged, not overwrite) but you can skip that step.

Following is a list of common [font=courier]Makefile[/font] commands:[/quote]

I had assumed something like this was the case. Just hadn’t looked into the internals much yet. Thanks for the tips!

[quote=“nrgaway, post:12, topic:647”]I really don’t know why this is failing for you. As I had previously stated I was able to build over Tor just by removing the [font=courier]-qN[/font] from the wget.

Seems like your Whonix gateway is not configured to allow ftp connections?[/quote]

Nope. Other FTP links work fine for me through my Whonix-Gateway.

That link did not seem to point to a downloadable file. Rather a webpage.

Another alternative URL could be:

http://xenbits.xen.org/xen-extfiles/newlib-1.16.0.tar.gz

However, I’m wondering if alternative URLs might have problems with not providing expected signature files, as the original sources seem to.

Strange…

So I just checked 3 new build VMs that I launched a few hours ago.

I did not modify the “qubes-builder” repo files in these at all. No “-qN” fix. Nothing modified in Makefile, etc.

Still building over Tor. And all 3 of them are well past all of the previous errors we’ve discussed.

So, this time, while building over Tor, 3 VMs were able to do so without encountering those previous errors.

[hr]

Now 2 out of 3 of them have erred out based on other packages and downloads further on in the build process.

But the 3rd VM is still going strong retrieving a bunch of Whonix packages.

[hr]

These results tell me that it is the individual Tor circuits or exit nodes that we connect through that make or break our build process when building over Tor.

Since now, with a different random Tor circuit, none of the past errors occurred for me.

So the issues seem to be with the Tor circuits and/or exit nodes, and not the build script.

Although workarounds like “-qN” might help massage uncooperative Tor circuits/nodes.

[hr]

I have my fingers crossed on this remaining 3rd VM still running the build script.

@nrgaway

I’m wondering what your experience has been as far as the amount of time it takes for you to run the complete build script from start to finish.

[hr]

Me:

I have a generally fast i7 machine, but am building over Tor and using a HDD (non-SSD).

Seems to be taking 4+ hours. Haven’t finished yet.

[hr]

Wondering about how long it has taken you to do a complete build over both Tor and over Clearnet?

Also, are you using HDD or SSD?

Thanks!

[quote=“WhonixQubes, post:16, topic:647”]@nrgaway

I’m wondering what your experience has been as far as the amount of time it takes for you to run the complete build script from start to finish.

[hr]

Me:

I have a generally fast i7 machine, but am building over Tor and using a HDD (non-SSD).

Seems to be taking 4+ hours. Haven’t finished yet.

[hr]

Wondering about how long it has taken you to do a complete build over both Tor and over Clearnet?

Also, are you using HDD or SSD?

Thanks![/quote]

From a fresh clean AppVM takes aprox:
make get-sources – 5 minutes
make template-modules – ~20 minutes first run; 5 minutes once it has been run before
make template – 40 minutes per VM

So about an hour per VM. Usually I build about 6 of them before going to bed

I’m running on an i7 with SSD. I think Tor took about 10 minutes for sources; I only built once over Tor.

I did not track how much longer it took to install the debootstraped images in Tor.

Success! :smiley:

Today, I finally got a successful build and installation of the “whonix-gateway-experimental” and “whonix-workstation-experimental” VMs.

[hr]

I discovered that part of the problem I was having before was that my physically isolated Whonix-Gateway, which I was building over Tor with, was constantly maxing out its CPU at 100%. This was causing very slow Tor download rates and potentially even dropped packets.

So I started using Qubes TorVM to build over Tor.

Using TorVM I still encountered and bypassed the same previously documented issues with “qubes-builder/qubes-src/vmm-xen/Makefile”:

[hr]

The total build time for the default “README.whonix” build was about 11 hours, building over Tor with a HDD (non-SSD).

[hr]

One code bug I noticed inside: install-templates.sh

The VM name “development-qubes” is hardcoded into the script, where instead it should likely be ${name}.

qvm-run --pass-io development-qubes "cat ${path}/${file}" > ${file}

should likely be changed to:

qvm-run --pass-io ${name} "cat ${path}/${file}" > ${file}

[hr]

Quite excited to have these VMs up and running now! :smiley:

I will next proceed to further testing the Whonix-Gateway and Whonix-Workstation VMs.

[quote=“WhonixQubes, post:18, topic:647”]Success! :smiley:

Today, I finally got a successful build and installation of the “whonix-gateway-experimental” and “whonix-workstation-experimental” VMs.[/quote]

Yeah! Finally :slight_smile:

At least now you won’t be needing to re-download the complete sources again although I wonder which build you have.

My latest one has Whonix 9.4 but I don’t think its pushed to the whonix branch; its in wheezy. I think it will all be merged soon with the Qubes repos. Then maybe you can just download a rpm template from their community site.

Its been quiet around there so maybe Qubes R3 is days away?

One code bug I noticed inside: install-templates.sh

The VM name “development-qubes” is hardcoded into the script, where instead it should likely be ${name}.

qvm-run --pass-io development-qubes "cat ${path}/${file}" > ${file}

should likely be changed to:

qvm-run --pass-io ${name} "cat ${path}/${file}" > ${file}

Good catch. Fixed.

Quite excited to have these VMs up and running now! :D

I will next proceed to further testing the Whonix-Gateway and Whonix-Workstation VMs.

I have been running it for the last few days without any issues I could notice but I really don’t know what to use it for hehe. I just like to know its there if I need it.

Let me know how your tests work out. If you have the latest version both gateway and workstation can be setup as templates and update via tor (templates need to have netvm set to gateway). It won’t let you update over clearnet unless you enable networking I would think.

We need to focus on leak testing too.

[quote=“nrgaway, post:19, topic:647”]At least now you won’t be needing to re-download the complete sources again although I wonder which build you have.

My latest one has Whonix 9.4 but I don’t think its pushed to the whonix branch; its in wheezy.[/quote]

Yeah, I believe it is Whonix 9.2 that installed for me using the “whonix” branch in your git repo.

[hr]

Has this been talked about by the Qubes devs for your templates yet?

Do they plan to offer them in RPM community templates like this?

I remember a few months back Joanna saying a couple times on the Qubes mailing lists that she would like to at some point incorporate Whonix and TorVM into the official Qubes installer, pre-configured, with point and click ease.

[hr]

Could be. Looking forward to Qubes R3 soon. I wonder if R3 architecture changes will affect your current Whonix implementation or not.

[hr]

Wow. Amazing that you’ve developed and contributed this awesome new platform without a pressing personal need. Thank you! I personally am going to get a large amount of value from it, and know several others will to. Thank you! Thank you! Thank you! :smiley:

[hr]

Ok. Thanks.

[hr]

Yes. With leak testing, we will want to consider the state of leak testing results compared to updated source code or packaged versions.

For example, leak tests may check out right now, but new leaks could be opened through additional source code and/or Qubes dev packaging.

So just being conscious of versioning with leak testing and also considering workflow of when to do a full battery of leak tests. Whatever new major official or community releases are put out will likely warrant a new round of full leak testing.

[hr]

Also, some basic source code auditing should be done, for trust purposes, since new community code outside of Qubes and Whonix is being incorporated.