Anyone have any build tips to speed things up?

nrgaway · November 15, 2015, 7:42pm

Hi!

I am starting to work on implementing some encryption things into a branch based off of Whonix 8.6.6.0. My initial build takes a very long time (2+ hours).

This is my build script I use:

cd Whonix
sudo ~/Whonix/whonix_build \
  --build --tor-workstation \
  --64bit-linux \
  --terminal-only \
  --current-sources \
  --enable-whonix-apt-repository \
  --whonix-apt-repository-distribution stable \
  --vmram 128 \
  --vram 12 \
  --vmsize 20G \
  --qcow2

Also note that even though I select ’ --terminal-only’ I still do get a KDE login. I can live with that though.

So my questions are:

When I start adding / changing Whonix code, is there a way to build that can resuse everything that has already been downloaded, etc so I do not have to rebuild from scratch each time?
Do I need to remove the raw image files each time?
I saw a fast and fast2 option; should I use those?
Any other tips on rebuilding over and over again to keep my sanity

Thanks!

Patrick · November 15, 2015, 7:42pm

Whonix’s build script is step based to aid such endeavors as yours. The build steps in the build-steps.d folder can be run one by one.

So if you tell a me a little bit where and how you are attempting to sneak in your code, I can tell you a way on how to repeat only the necessary steps.

Using an apt cache is already automated and integrated into Whonix’s build script. When building for the second time, apt-get downloading packages from Debian should be faster. [You could override with your own http_proxy env var apt cache as well though.]

Usually no need to overwrite raw images, unless you’re not using whonix_build, but build-steps.d directory. When doing the latter you need to know a little bit about the steps.

I haven’t used the --fast in a while. They’re just for debugging and skipping specific steps. So it really depends on what you’re debugging if they’re useful. However they’re no general speed up that could be optionally used for anything.

I’ll made a note to check --terminal-only.

Patrick · November 15, 2015, 7:42pm

There was indeed a bug because packages and package names changed. Attempted to fix:

Not yet tested.

Patrick · November 15, 2015, 7:42pm

For the record…

[hr]

Ticket ‘Speedup Whonix 11 build time’ was implemented for Whonix 11. See:
https://phabricator.whonix.org/T295

[hr]

[quote=“Patrick, post:3, topic:458”][quote author=Patrick link=topic=480.msg3695#msg3695 date=1409462600]
I’ll made a note to check --terminal-only.
[/quote]
There was indeed a bug because packages and package names changed. Attempted to fix:

Not yet tested.[/quote]
There is some news on ‘–gui none’. See:
https://www.whonix.org/forum/index.php/topic,902.msg8123.html#msg8123

Bob · February 19, 2016, 3:11pm

Currently building the gateway requires 3 hours for me. I think most of the time is required for compilation (and installation) of the packages. There is a whonix_binary folder in the end with the final packages (I guess). Can I just copy this over to the folder of another build and the whonix build script will automatically detect that the final packages are already there, skip the compile step and just start the installation of the packages? Or do I need to run each of the whonix build steps one by one and stop e.g. after a certain step, make a backup and continue with the next step. I’ll also try out the --force-unsafe-io option but it think it only refers to the installation though compilation takes most of the time.

Patrick · February 21, 2016, 12:21am

No auto detection. If you have a rudimentary understanding, what each step is doing, then it becomes clear on when you can skip which steps. It depends on your final goal. If you just want to restart the build, I would rather go through a clean full build and wait. Not bother learning that. But if you want to contribute to Whonix, then it’s certainly worth to learn to speed up debugging.

https://www.whonix.org/wiki/Dev/Source_Code_Intro#Introduction

HulaHoop · March 13, 2016, 11:26pm

Most of the time is spent downloading the packages actually. Depending on whether its a slow or fast circuit and mirror load.

Since almost all packages are identical for GW and WS having apt-cacher-ng keep the packages downloaded once already for the second VM build can shorten build time a lot.

An example of debootstrap configured to use apt-cacher-ng:

sudo debootstrap --include=sudo,openssh-server,ntpdate,dosfstools,sysvinit,fbset,less,xserver-xorg-video-modesetting,task-xfce-desktop,hicolor-icon-theme,gnome-icon-theme,tango-icon-theme,i3-wm,i3status wheezy /mnt http://127.0.0.1:3142/ftp.ie.debian.org/debian/

Patrick · March 14, 2016, 5:05pm

Using an apt cache is possible and documented:

https://www.whonix.org/wiki/Dev/Build_Documentation/12_full#APT_Cache_.28Optional.29