ID: 577PHID: PHID-TASK-2d5wq5qut5vl3zuhdnroAuthor: HulaHoopStatus at Migration Time: resolvedPriority at Migration Time: Normal
I’ve looked at some ways to accelerate the build script to make building Whonix less painful. This is a very long-term item because breaking the script is serious problem and because other priorities but its definitely something worth doing at some point.
Low hanging fruit:
http://tinylittlelife.org/?p=283 (inspired by the post beneath it)John Tells All: Fast Parallel Downloading (for apt-get)
Taking this further the image generation step can be done in parallel too though time savings will be less compared to download time reduction.
Disqualified but worth noting:
I found the tool “apt-fast” which downloads one or two files quickly, by downloading with multiple streams per file. This is somewhat sketchy, as it requires installation of additional software, assumes the file gets spliced together correctly, and doesn’t gracefully handle network problems.
apt-fast - an apt wrapper that downloads package chunks from multiple mirrors in a “bittorrent-esque” way. If you are interested in it I could request the author gives it a license.
2016-12-13 10:44:56 UTC
avoid downloading the same things twice
Did you try using an apt-cache yet as per build documentation, chapter apt cache ?
It’s not needed to always run the full build script. If you have an overview on which build step is doing what, there is no need to always rerun the whole build script, no need to always rerun all build steps.
2016-12-14 09:47:58 UTC
Did you try using an apt-cache yet as per build documentation, chapter apt cache?
Hadn’t noticed before. Is this affecting the host in any way? Can I do this in the VM only?
sudo -E ./build-steps.d/1100_prepare-build-machine --install-to-root --tor-gateway
Can you please post an example that applies to both ws and gw?
Offtopic:
What does “–install-to-root” do?
2016-12-14 10:00:31 UTC
2016-12-14 17:23:00 UTC
–install-to-root is deprecated. That’s --target root for a few releases now. For physical isolation.
(So I don’t know where you got --install-to-root from. Either outdated build documentation or you got outdated sources.)
(For building KVM images, stay miles away from --target root.)
(Also --tor-gateway is outdated so I guess your sources are outdated.)
To get an overview of all switches, use the following and scroll up a bit.
sudo ./whonix_build --gnw -- --help
Or look here: https://github.com/Whonix/Whonix/blob/master/help-steps/parse-cmd#L106
An apt-cache can be used anywhere. When building on the host and also when building inside the VM.
Using an apt-cache affects the host or VM in as far as using an apt-cache usually does while Whonix is not involved. So in case of apt-cacher-ng the daemon will be running and cache files somewhere in the root filesystem (configurable but usually it works fine out of the box without issues).
sudo -E is useful to preserve environment variables. When you do
export http_proxy=…
sudo ./whonix_build
then whonix_build will not know about http_proxy because sudo clears the environment.
sudo http_proxy=... ./whonix_build should also work. Environment variables wrt Whonix build script work the same way as they do on any Linux (at very least on any Debian based operating system).
It should also alternatively also be possible to set the http_proxy variable through a build configuration snippet .
export http_proxy=http://192.168.0.1:3142 → http://10.152.152.11:3142
Probably neither. It’s 127.0.0.1 unless you want to run apt-cacher-ng on a different system/VM over (virtual) LAN.
Some examples:
sudo ./build-steps.d/1100_prepare-build-machine --build --flavor whonix-gateway --target qcow2
sudo ./build-steps.d/1150_export-libvirt-xml --build --flavor whonix-gateway --target qcow2
sudo ./build-steps.d/1700_install-packages --build --flavor whonix-workstation --target qcow2
2016-12-14 21:14:19 UTC
2016-12-15 00:22:12 UTC
Still acceleration required?
I am not sure it’s great to add new features to Whonix build script. (parallel stuff, speed up stuff [not creating Debian base image and Whonix packages twice]…)
Whonix build script is already a victim of TLDR. Making it more complex leads to even fewer people seeing through and/or wanting to work on it. It’s only used to build Non-Qubes-Whonix.
On the contary, I have been considering to throw out lots of non-essential Whonix build script features and/or to replace Whonix build script with some other VM image build system.
A list of features that Whonix build script currently features has been created just now. Grouped into essential, non-essential and undecided importance.
Whonix ™ Source Code Introduction
Feel free to start a discussion or creating an alternative list (changing group priorities).
Finding a VM build system that can do the minimum we decide essential features is a time consuming task to research, implement and test.
2016-12-16 03:17:29 UTC
Still acceleration required?
No. I consider this ticket complete. apt-cacher-ng is a pretty cool addon that satisfies this need without complicating the buildscript itself.
Whonix build script is already a victim of TLDR.
Don’t say that. It was just a matter of RTFM. Its extensive features are a great thing actually.
Making it more complex leads to even fewer people seeing through and/or wanting to work on it. It’s only used to build Non-Qubes-Whonix.
IMO its already feature complete
Finding a VM build system that can do the minimum we decide essential features is a time consuming task to research, implement and test.
True. Seems like waste of time to end up with something doing less than what we already have.