Terminology: building plain Debian using Whonix's build script --whodeb?

Patrick · October 16, 2014, 3:27am

The parameters…

--virtualbox
--qcow2
--install-to-root

…are called “build targets”. And mandatory. At least one has to be chosen. --virtualbox and --qcow2 can be combined.

Additionally I plan on introducing a mandatory “distribution” parameter.

--whonix
--whodeb

“–whonix” would be just what the build script builds as of Whonix 9. Usual Whonix build.

“–whodeb” would be a plain Debian system without Whonix’s Debian packages, without Whonix’s chroot scripts.

That would probably more generic and less confusing than a “–no-install-whonix-packages” parameter.

(“–whodeb” builds are needed for debugging purposes (ex: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=735932) and it’s low effort to add. Not planing to support this officially.)

Is “build targets” a good term for that use case?
Is “distribution” a good term for that use case?
Is “–whodeb” a good term for that use case? Other suggestions?

(Cannot use “–debian”. I’d like to use that term “whodeb” or else as part of the resulting file name. And one not being the Debian project must not redistribute images named “debian”. Debian trademark policy: Debian -- Debian Trademarks)

(Comparison: http://manpages.ubuntu.com/manpages/hardy/man1/ubuntu-vm-builder.1.html)

Patrick · October 16, 2014, 10:59pm

Since the builder has to choose a mandatory VM parameter such as

--whonix-gateway
--whonix-workstation
--whonix-custom-workstation

anyway, I guess a separate mandatory “distribution” parameter does not make sense.

What about an additional VM parameter

?

I am not sure “-plain-” implicates anything confusing.

JasonJAyalaP · October 17, 2014, 1:16am

Quick and dirty:

–debug-debian Builds without chroot scripts. Useful for reporting Debian bugs.

But this could be a good time to add a “debug” parameter.

–debug debian Builds without chroot scripts. Useful for reporting Debian bugs.

But I propose making this “flavor neutral”:

–debug no-whonix Builds without Whonix pakcages. Useful for reporting upstream bugs.

And in the future, you could add additional --debug options:

–debug blah Disables blah blah blah

A confounding issue is how to deal with Whonix flavors (when we get there).

My imaginary, ideal build parameters (though I’m probably missing something):

./whonix-build --vm gateway --flavor debian --target qcow2 --debug no-whonix

As you can see, I prefer “–parameter value” than --everything-is-a-parameter. But that’s a separate issue (though I am willing to design and or implement this CLI if you think it’s cleaner and more memorable).

Patrick · October 17, 2014, 2:08am

“flavor neutral” sounds good. You’ll never know if Whonix’s build script develops into a general tool to build VMs / os’s. I have a feeling, one day there might be options and maintainers for plain Debian, Fedora (+Whonix), (Hardened) Gentoo (+Whonix) and more. Either way it’s a good plan to leave that door open.

But to be really flavor neutral, “whonix” should be be an opt-in (–flavor whonix-…), not opt-out (–debug no-whonix). So what about this?

--flavor whonix-gateway
--flavor whonix-workstation
--flavor whonix-custom-workstation
--flavor whodeb

I totally agree with that and keep it in mind for further development.

The “-tg” is still a historic legacy from the early TorBOX build script.

As per a total redesign, it’s not a high priority for me, but feel free to go for designing / implementing it.

JasonJAyalaP · October 17, 2014, 7:08pm

Isn’t Debian/Fedora/Gentoo the “flavor” and workstation/gateway/custom the, uh, do we have a word for it? (I simply did --vm in my example but that’s not very clear).

In any case, I think you’re right: “no whonix” could be moved from debug to --vm (Maybe… “–whonix none”? That would be similar to what you’re doing now).

./build --flavor [Gentoo|Debian] --whonix [gateway|workstation|none]

Going with your current all-parameter-design, I suggest:
–whonix-gateway
–whonix-workstation
–whonix-none

Patrick · October 27, 2014, 4:52am

Yes, that’s it. I am looking for a word.

(I simply did --vm in my example but that's not very clear).

Yes, --vm is bad, because one could want to use --install-to-root.

I would like to think of the build script as for, well, hard to say. Not just a tool to build VMs, because it also supports --install-to-root. To think of it as a tool to build operating systems, physical or vm. For such a tool you wouldn’t have a --whonix parameter.

What about this?

(–suite is a term used by debootstrap and Ubuntu VM builder, and I think reusing them makes sense.)

Patrick · October 27, 2014, 7:01am

Ok, changed my mind. I want to fix this command line parameter mess relatively soon. The current state is confusing. Not to be advertised. And hinders wider adoption. Here comes a proposal.

Legend:

--old-param-notation
--another-old-param-notation
--> --new-parameter-proposal default-option|another-option|yet-another-option

Proposal:

--build
--clean
--> ???

new
--> --base-os debian|ubuntu|gentoo

--virtualbox
--qcow2
--install-to-root
--> --target virtualbox|qcow2|root

--whonix-gateway
--whonix-workstation
--whonix-custom-workstation
--"plain"
--> --extra whonix-gateway|whonix-workstation|whonix-custom-workstation|none

--all
--> ???

--internalrun
--> as is

--vmram 128
--vram 12
--vmsize 200G
--> as is

--terminal-only
--> --gui true|false

--no-default-applications
--> --default-applications true|false

--current-sources
--testing-frozen-sources
--> --sources-freshness frozen|current

--486-linux
--32bit-linux
--64bit-linux
--64bit-kfreebsd
--32bit-kfreebsd
--> --kernel-flavor linux|kfreebsd ????????????
--> --arch i386|amd64

--whonix-apt-repository-distribution wheezy
--whonix-apt-repository-distribution testers
--whonix-apt-repository-distribution developers
--> as is --whonix-apt-repository-distribution wheezy|testers|developers

--enable-whonix-apt-repository
--> ???

--minimal-report
-> minimal|full

--verifiable
--> true|false

--skip-sanity-tests
--> sanity-tests true|false

--help
--> as is

--no-validate-libvirt-xml
--> validate-libvirt-xml true|false

--fast1
--fast2
--> --fast 1|2

--file-system
--hostname
--os-password
--debopt

--auto-retry
--> auto-retry-max 0|1|2|...

--wait-auto-retry
--> auto-retry-wait 0|1|2|...

--dispatch-before-retry
--> auto-retry-dispatch-before script

--dispatch-after-retry
--> auto-retry-dispatch-after script

--ignore-uncommitted
--> --allow-uncommitted false|true

There are still some “???”. Please help fill them out. Please review. Looks good overall or like a new mess? I’d be also open to a separate proposal.

JasonJAyalaP · October 27, 2014, 3:21pm

--build
--clean
--> ???

These seem to be the “operating modes” (my term). Many CLIs use subcommands for
these situations: git commit git add. So it could be:

./whonix build --param v --param v
./whonix clean

But if that adds to much complexity to the CLI, you can keep them as
–build|–clean

--base-os debian|ubuntu|gentoo

Consider simply “–os”. The shorter the better (but I have no problem with
base-os)

./whonix build --os debian

--virtualbox
--qcow2
--install-to-root
--> --target virtualbox|qcow2|root

Agreed.

./whonix build --os debian --target virtualbox

--whonix-gateway
--whonix-workstation
--whonix-custom-workstation
--"plain"
--> --extra whonix-gateway|whonix-workstation|whonix-custom-workstation|none

How about “whonix” ?
–whonix gateway|workstation|custom|none

./whonix build --os debian --whonix gateway

--all
--> ???

I don’t like “all”. It sounds error prone. But you’ll have to answer: “Can the
user build more than one image at a time?”. For simplicity and sanity, I say no.
If you want to build multiple images in one command, use &&

--internalrun
--> as is

Hrm. Is this like “–dry-run” (simulate effects?)

--vmram 128
--vram 12
--vmsize 200G
--> as is

Agreed.

--terminal-only
--> --gui true|false

Agreed.

--no-default-applications
--> --default-applications true|false

–applications or --apps

--current-sources
--testing-frozen-sources
--> --sources-freshness frozen|current

–freshness

--486-linux
--32bit-linux
--64bit-linux
--64bit-kfreebsd
--32bit-kfreebsd
--> --kernel-flavor linux|kfreebsd ????????????
--> --arch i386|amd64

–kernel
Damn. The i and amd and x86_64 stuff always confuses me. i386 is 32-bit and
amb64 is 64-bit, right? Can we just say that? --arch (or --bit) 32|64. Hrmph.
But what about when we do ARM builds? --arch arm8|32|64 ?

--whonix-apt-repository-distribution wheezy
--whonix-apt-repository-distribution testers
--whonix-apt-repository-distribution developers
--> as is --whonix-apt-repository-distribution wheezy|testers|developers

–whonix-repo

--enable-whonix-apt-repository
--> ???

Merge with above:

–whonix-repo disable|wheezy|testers|developers

“off” or “disable”. You pick.

--minimal-report
-> minimal|full

–report minimal|full

--verifiable
--> true|false

--skip-sanity-tests
--> sanity-tests true|false

--help
--> as is

--no-validate-libvirt-xml
--> validate-libvirt-xml true|false

--fast1
--fast2
--> --fast 1|2

--file-system
--hostname
--os-password
--debopt

All agreed.

--auto-retry
--> auto-retry-max 0|1|2|...

--wait-auto-retry
--> auto-retry-wait 0|1|2|...

Or just --retry-wait and --retry-max

--dispatch-before-retry
--> auto-retry-dispatch-before script

--dispatch-after-retry
--> auto-retry-dispatch-after script

–retry-before ? Dunno.
–retry-after

--ignore-uncommitted
--> --allow-uncommitted false|true

Looks good.

Edit by Patrick:
Disabled smileys.

Patrick · October 27, 2014, 4:01pm

Consider simply "--os". The shorter the better (but I have no problem with base-os)

Ok.

How about "whonix" ? :) --whonix gateway|workstation|custom|none

Wrote this above, but eventually you missed it. "I would like to think of the build script as for, well, hard to say. Not just a tool to build VMs, because it also supports --install-to-root. To think of it as a tool to build operating systems, physical or vm. For such a tool you wouldn't have a --whonix parameter."

--all --> ???
I don’t like “all”. It sounds error prone. But you’ll have to answer: “Can the user build more than one image at a time?”. For simplicity and sanity, I say no. If you want to build multiple images in one command, use &&

Implemented this not long ago.

whonix_build <parameters for whonix_build> -- <parameters for help-steps/whonix_build_one>

Please see details usage here:
https://www.whonix.org/wiki/Dev/Build_Documentation/10_full#VM_Creation

Implementation on the source code level seems simple and sane to me. At doesn’t build at the same time, but in sequential order. And as soon one build breaks, it stops. Useful for test and redistributable builds. (Otherwise as a builder you end up with an extra script in your home folder that just does building multiple images one after another.) But if all cords break, this can be changed.

--internalrun --> as is
Hrm. Is this like “–dry-run” (simulate effects?)

No, more like “when recursively calling itself or steps that don’t care about gw|ws we can use --internalrun”.

--no-default-applications --> --default-applications true|false
–applications or --apps
--current-sources
--testing-frozen-sources
--> --sources-freshness frozen|current
–freshness

Ok.

Damn. The i and amd and x86_64 stuff always confuses me. i386 is 32-bit and amb64 is 64-bit, right?

Yes. But then there are also 486, 686, 686-pae and amd64 kernels.

Can we just say that? --arch (or --bit) 32|64. Hrmph. But what about when we do ARM builds? --arch arm8|32|64 ?

--arch is the term used by [grml-]debootstrap and others. Many possible values, those are listed here: https://www.debian.org/ports/

--whonix-apt-repository-distribution wheezy --whonix-apt-repository-distribution testers --whonix-apt-repository-distribution developers --> as is --whonix-apt-repository-distribution wheezy|testers|developers
–whonix-repo
--enable-whonix-apt-repository
--> ???
Merge with above:

–whonix-repo disable|wheezy|testers|developers

“off” or “disable”. You pick.

Agreed. (“off” is slightly better, because then it never gets enabled, therefore not really disabled.)

--auto-retry --> auto-retry-max 0|1|2|...
--wait-auto-retry
--> auto-retry-wait 0|1|2|...
Or just --retry-wait and --retry-max
--dispatch-before-retry
--> auto-retry-dispatch-before script

--dispatch-after-retry
--> auto-retry-dispatch-after script
–retry-before ? Dunno.
–retry-after

Agreed.

JasonJAyalaP · October 27, 2014, 9:26pm

Hmm. Apparently I keep saying --whonix and you say “but it’s more!” and then I forget and say “–whonix would work here”.

Well, what do you call this “layer” on top of the “base os”? spin; remix; flavor; layer…

Patrick · October 28, 2014, 1:24am

What about “blend”?

Comparison:
https://wiki.debian.org/DebianPureBlends
(Debian Pure Blends)

JasonJAyalaP · October 28, 2014, 3:03pm

According to their terminology, “flavor” is closer than “blend”.

I’m ok with flavor, but we might need a word that suggests much more than different defaults or KDE vs GNOME.

How about --system?

./build --os Gentoo --system hardened-experimental
./build --os Debian --system whonix-workstation

Patrick · October 28, 2014, 5:13pm

Good point. Do we want to replace

--gui true|false

with

-- gui kde|none|...

then?

Or perhaps we should have a --flavor that can be in theory used multiple times?

JasonJAyalaP · October 28, 2014, 8:17pm

They way you’re going, CLI switches won’t be enough to handle the complexity and length… We’ll need a makefile format. Whakefile And then parse, check for sanity, check for compatibility between the options.

Standard:
BASE=DEBIAN
SYSTEM=WHONIX-WORKSTATION
TARGET=VIRTUALBOX
GUI=KDE
VRAM=512

./whake common-builds.whake standard

Patrick · October 28, 2014, 8:45pm

All you can do using CLI, you can also do using (stackable) config files - and vice versa. What the CLI at the moment does is translating

--cli-switch param

to

some_variable="variable_content"

We kinda support something like something similra to Whakefiles already. See:
https://www.whonix.org/wiki/Dev/Build_Documentation/10_full#Introduction_.28Optional.29

buildconfig.d
../buildconfig.d
/etc/whonix_buildconfig.d

In recent versions the build script has been improved by me. I spared builders from needing to use those build config folders for “popular” switches, because those creating those build configs was apparently much more difficult for builders than just using cli. Those mechanism are still usable, though.

It would probably be trivial to add some --config /path/to/whonix_build.conf feature. I think I’ll add this anyhow. Personally I still prefer cli. But that doesn’t mean, we couldn’t easily support both use cases.

Patrick · October 31, 2014, 9:35am

This is done in:
10.0.0.0.9-developers-only

sudo ./whonix_build --help

## whonix_build main script
##
## Syntax:
## whonix_build <parameters for whonix_build> -- <parameters for help-steps/whonix_build_one>
##
## --all
## Builds Whonix-Gateway, Whonix-Workstation and Whonix-Custom-Workstation.
##
## --gnw
## Builds Whonix-Gateway and Whonix-Workstation.
##
## --flavor whonix-gateway|whonix-workstation|whonix-custom-workstation
## whonix-gateway: Builds Whonix-Gateway.
## whonix-workstation: Builds Whonix-Workstation.
## whonix-custom-workstation: Builds Whonix-Custom-Workstation.
##
## Run
## ./whonix_build --flavor whonix-gateway -- --help
## to find out whonix_build_one parameters.

sudo ./whonix_build --flavor whonix-gateway -- --help

## help-steps/whonix_build_one script
##
## Syntax:
## help-steps/whonix_build_one --flavor flavor --target target
##
## creates separate build folder
## /whonix_binary
##
## For verbose documentation on build configuration parameters please have a
## look at the full build documentation.
##
## FLAGS:
## --build
## Build Whonix.
##
## --clean
## Deletes build products (.raw, .ova and VirtualBox VM'\''s).
##
## --flavor whonix-gateway|whonix-workstation|whonix-custom-workstation|none
## whonix-gateway: Builds Whonix-Gateway.
## whonix-workstation: Builds Whonix-Workstation.
## whonix-custom-workstation: Builds Whonix-Custom-Workstation.
## none: not yet implemented
##
## --target virtualbox|qcow2|root
## virtualbox: Use this to build VirtualBox .ovas.
## qcow2: Use this to build qcow2 images.
## root: Use this for Whonix with physical or install to root installations.
##
## vmram, vram, vmsize examples.
## --vmram 128
## --vram 12
## --vmsize 200G
##
## --gui kde|none
## kde: Install KDE specific packages.
## none: Skip installation of gui specific packages.
##
## --apps true|false
## true: Install recommended packages.
## false: Install fewer packages. (Don'\''t complain about missing packages then.)
##
## --freshness frozen|current
## frozen: Use frozen sources.
## current: Use current sources.
##
## --whonix-repo off|wheezy|testers|developers
## Whonix APT Repository.
## off: Disabled by default.
## wheezy
## testers
## developers
##
## --report full|minimal
## full
## minimal: Skip most of the create-report build step. (Related to Verifiable Builds.)
##
## --verifiable true|false
## true
## false: Skip deletion of non-determinstic files in the cleanup chroot script.
##        (These files are later automatically re-created by First Run Initializer.)
##
## --sanity-tests true|false
## true
## false: Skip chroot script sanity tests.
##
## --retry-max 1|0|2|[...]
## attempts
##
## --retry-wait 5|1|2|[...]
## Seconds between attempts.
##
## --retry-before script
## --retry-after script
## Dispatch command or script.
##
## --allow-uncommitted false|true
## false: Break when uncommitted changes are found. Default.
## true: Do not use unless you know what you are doing.
##
## --arch
## Only of interest for VM builds. Not for installations to root.
## i386: default, best tested.
## amd64: less tested.
## kfreebsd-i386: entirely untested and most likely needs work.
## kfreebsd-amd64: entirely untested and most likely needs work.
##
## --kernel <Quoted space delimited list of packages.>
## Only of interest for VM builds. Not for installations to root.
## Defaults to linux-image-486 linux-image-686-pae.
## none: Do not install kernel packages.
## linux-image-486
## linux-image-686-pae
## linux-image-amd64
## Possibly others. (Unmaintained!)
##
## --headers <Quoted space delimited list of packages.>
## Only of interest for VM builds. Not for installations to root.
## Defaults to linux-headers-486 linux-headers-686-pae.
## none: Do not install kernel header packages.
## linux-headers-486
## linux-headers-686-pae
## linux-headers-amd64
## Possibly others. (Unmaintained!)
##
## --confdir /path/to/config/dir
## Will get sourced after
## $WHONIX_SOURCE_FOLDER/buildconfig.d /etc/whonix_buildconfig.d ../buildconfig.d
## and before --conffile.
##
## --conffile /path/to/config/file
## Will get sourced after all other files.

Textual strings are here:

Please review. I am sure --help sux.