Building images from source inside the VM Wiki page

Support
#1

Hot off the press.

https://www.whonix.org/wiki/Dev/BuildDocumentation_8_RC

Ideally this should lead to making building from source easier and to a more uniform, across-the-board debugging process. Not to mention that all our non-Debian stable friends can build now too!

All feedback and corrections are welcome.

#2

That’s what I forgot to answer in the other thread.

I am unhappy with the page name “8_RC”. Why keep RC in the name? The RC soon becomes the stable, and then? Name that page “https://www.whonix.org/wiki/Dev/BuildDocumentation_8”? That name is already taken.

I think the name of the page should reflect “Building images from source inside the VM Wiki page”.

As soon we found a new name, we can move the page content and redirect the old url to a new one (easy).

#3

Funny that you mentioned this. I was thinking just moments before that I chose the wrong name.

How about we name it “https://www.whonix.org/wiki/Dev/BuildDocumentation_8_VM”?

The RC becomes stable and all I got to do is to edit the RC references on the page to stable.

#4

What about “https://www.whonix.org/wiki/Dev/BuildDocumentation_8_inside_VM”?

Another idea. It could also replace this page:
https://www.whonix.org/wiki/Dev/BuildDocumentation_8_short

#5

I don’t know what the purpose of “8_short” really is. Removing the content and putting the VM content under is OK with me.

We can leave the 8RC for any further 8.x testing builds.

#6

It was an attempt to have as short and simple instructions as possible. Failed, though. Was probably a bad idea to put the build in VM edition there, since it’s not shortest possible.

I think the in_VM_8 makes most sense.

8.x will be _9.

#7

[quote=“adrelanos, post:6, topic:71”][quote author=Occq link=topic=52.msg428#msg428 date=1392585984]
I don’t know what the purpose of “8_short” really is.
[/quote]
It was an attempt to have as short and simple instructions as possible. Failed, though[/quote]

Leave the page. I’ll make short and simple instructions next week. Basically just commands with comments. Rehash of the VM page mostly, with no install section and VM refs removed.

Agreed. Transfer the page whenever.

8.x will be _9.
[/quote]

Good point.

#8

On the other hand, the future is in VM builds and more choices will confuse the user, not to mention template nightmare that we need to resolve sometime in the future.

Deleting _short and making a new in_VM_8 gets my vote.

#9

Ok, moved to “https://www.whonix.org/wiki/Dev/BuildDocumentation_8_in_VM”.

On the other hand, the future is in VM builds

For me, building inside a VM is just an addition to https://www.whonix.org/wiki/Dev/BuildDocumentation_8. It’s a personal choice whether you decide to build on a Debian stable host or inside a VM. For now, we don’t have to fix it any other way?

not to mention template nightmare that we need to resolve sometime in the future.
What needs to be solved there? The different git tags?
#10

Git tags, possibly signing images (different date for different ones).

I’ll rewrite the short documentation then.

#11

Done.

https://www.whonix.org/w/index.php?title=Dev/BuildDocumentation_8_short

#12

Perhaps we should label 8_short 8_in_VM alpha? And also note, that best way to get support is asking you in the forum?

I’ve got a mail by someone using 8_short using 7.7.8.9.

Strange locale warning:

+ true 'INFO: Installing whonix-gateway-packages-dependencies, because whonix_build_script_whonix_package does not include it... This may take a while...'
+ local apt_get_exit_code=0
+ chroot /home/user/whonix_binary/Whonix-Gateway_image apt-get -o Dir::Etc::sourcelist=/tmp/empty -o Dir::Etc::sourceparts=/var/lib/whonix/sources_temp_list.d -o Acquire::http::Timeout=180 -o Acquire::ftp::Timeout=180 -o Acquire::Retries=3 -o Acquire::Check-Valid-Until=false --yes --no-install-recommends install whonix-gateway-packages-dependencies
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  libevent-2.0-5 telnet tor torsocks ucspi-tcp
Suggested packages:
  mixmaster xul-ext-torbutton socat tor-arm polipo privoxy
Recommended packages:
  tor-geoipdb
The following NEW packages will be installed:
  libevent-2.0-5 telnet tor torsocks ucspi-tcp whonix-gateway-packages-dependencies
0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
Need to get 1570 kB/1576 kB of archives.
After this operation, 3833 kB of additional disk space will be used.
Get:1 http://snapshot.debian.org/archive/debian/20140208T161236Z/ stable/main libevent-2.0-5 i386 2.0.19-stable-3 [171 kB]
Get:2 http://snapshot.debian.org/archive/debian/20140208T161236Z/ stable/main tor i386 0.2.3.25-1 [1165 kB]
Get:3 http://snapshot.debian.org/archive/debian/20140208T161236Z/ stable/main torsocks i386 1.2-3 [77.1 kB]
Get:4 http://snapshot.debian.org/archive/debian/20140208T161236Z/ stable/main telnet i386 0.17-36 [65.8 kB]
Get:5 http://snapshot.debian.org/archive/debian/20140208T161236Z/ stable/main ucspi-tcp i386 1:0.88-3 [91.5 kB]
Fetched 1570 kB in 1s (892 kB/s)
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = "",
        LC_ALL = (unset),
        LANG = "en_GB.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory

Didn’t fail because of that. But failed later:

++ debsums --list-missing
+ debsums_output_1='perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = "",
        LC_ALL = (unset),
        LANG = "en_GB.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").'
+ '[' '!' 0 = 0 ']'
+ '[' 'perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = "",
        LC_ALL = (unset),
        LANG = "en_GB.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").' = '' ']'
+ '[' '' = true ']'
+ error 'ERROR: Packages (or debs) with missing MD5 sums file found!'
/var/lib/whonix/initial-chroot-scripts-pre.d/20_sanity_checks: line 76: error: command not found
++ error_handler
++ echo '
###########################################################
## chroot script: ERROR detected. Please report this bug! #
###########################################################
'

###########################################################
## chroot script: ERROR detected. Please report this bug! #
###########################################################

++ exit 1
run-parts: /var/lib/whonix/initial-chroot-scripts-pre.d/20_sanity_checks exited with return code 1

The “en_GB” is stalking me.

It happened in 1900_run-chroot-scripts-pre-d (20_sanity_checks ). That’s before 2000_install-files-packages, so the update-locale postinst.d script has not been run.

Any idea?

#13

Builder is using SolydK (http://solydxk.com/) a Debian based distro.

#14

I added an alpha warning and forum help referral.

All references and pictures referring to the selection of UK have been modified so that should take care of the problem.

I’m testing different locale settings and I’ll then try to build bare metal.

#15

Building bare-metal inside a VM
7.7.8.9
Debian 7.4

Settings:

Language - English
Location - India
Keymap - American English
No mirror selected

Log

 force_echo 'INFO /usr/share/whonix/postinst.d/70_locale: Changing your locale to en_US.UTF-8... Will only do this once.'
+ '[' '!' 1 = 1 ']'
+ '[' '!' 1 = 1 ']'
+ '[' '!' 1 = 1 ']'
+ force_echo 'INFO /usr/share/whonix/postinst.d/70_locale: Will run: update-locale LANG=en_US.UTF-8'
+ '[' '!' 1 = 1 ']'
+ '[' '!' 1 = 1 ']'
+ '[' '!' 1 = 1 ']'
+ update-locale LANG=en_US.UTF-8
*** update-locale: Error: invalid locale settings:  LANGUAGE="en_IN:en" LANG=en_US.UTF-8
++ error_handler
++ local exit_code=255
++ local 'last_err=update-locale LANG=en_US.UTF-8'
++ '[' '!' 1 = 1 ']'
++ '[' '' = '' ']'
++ output='## See above.'
++ '[' '!' 1 = 1 ']'
++ force_echo '
####################################################################
## e[31me[1mBEGIN ERROR in /usr/share/whonix/postinst.d/70_locale detected!e[0;10m
##
## e[4mERROR LOGe[0;10m:
## See above.
##
## e[4mBASH_COMMANDe[0;10m: update-locale LANG=en_US.UTF-8
## e[4mEXIT_CODEe[0;10m: 255
##
## e[31me[1mEND ERROR in /usr/share/whonix/postinst.d/70_locale detected!e[0;10m
## e[31me[1mPlease report this Whonix bug!e[0;10m
####################################################################
'
++ '[' '!' 1 = 1 ']'
++ '[' '!' 1 = 1 ']'
++ '[' '!' 1 = 1 ']'
++ exit 1
run-parts: /usr/share/whonix/postinst.d/70_locale exited with return code 1
++ error_handler
++ local exit_code=1
++ local 'last_err=run-parts --verbose --exit-on-error "/usr/share/whonix/postinst.d"'
++ '[' '!' 1 = 1 ']'
++ '[' '' = '' ']'
++ output='## See above.'
++ '[' '!' 1 = 1 ']'
++ force_echo '
####################################################################
## e[31me[1mBEGIN ERROR in /var/lib/dpkg/info/whonix-gateway-files.postinst detected!e[0;10m
##
## e[4mERROR LOGe[0;10m:
## See above.
##
## e[4mBASH_COMMANDe[0;10m: run-parts --verbose --exit-on-error "/usr/share/whonix/postinst.d"
## e[4mEXIT_CODEe[0;10m: 1
##
## e[31me[1mEND ERROR in /var/lib/dpkg/info/whonix-gateway-files.postinst detected!e[0;10m
## e[31me[1mPlease report this Whonix bug!e[0;10m
####################################################################
'
++ '[' '!' 1 = 1 ']'
++ '[' '!' 1 = 1 ']'
++ '[' '!' 1 = 1 ']'
++ exit 1
dpkg: error processing whonix-gateway-files (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 whonix-gateway-files
E: Sub-process /usr/bin/dpkg returned an error code (1)
++ error_handler_unchroot_unprevent_unmount
++ benchmark_time_end
+++ date +%s
++ benchmark_time_end=1392650575
++ benchmark_took_seconds=99
+++ convertsecs 99
+++ (( h=99/3600 ))
+++ (( m=(99%3600)/60 ))
+++ (( s=99%60 ))
+++ printf '%02d:%02d:%02d\n' 0 1 39
+++ true
++ benchmark_took_time=00:01:39
++ true
++ echo 'ERROR in ./build-steps.d/2000_install-files-packages. Aborting... (benchmark: 00:01:39)'
ERROR in ./build-steps.d/2000_install-files-packages. Aborting... (benchmark: 00:01:39)
cat /etc/default/locale
LANG="en_IN".
LANGUAGE="en_IN:en"
sudo update locale
[no output]
sudo which update-locale
/usr/sbin/update-locale
sudo update-locale LANG=en_US.UTF-8
***update-locale: Error: invalid locale settings: LANGUAG=E"en_IN:en" LANG=en_US.UTF-8.
sudo /usr/share/whonix/postinst.d/70_locale
BEGIN ERROR in /usr/share/whonix/postinst.d/70_locale detected.

ERROR LOG:
***update-locale: Error: invalid locale settings: LANGUAGE="en_IN:en" LANG=en_US.UTF-8.

BASH_COMMAND: update-locale LANG=en_US.UTF-8
EXIT_CODE: 255
END ERROR in /usr/share/whonix/postinst.d/70_locale detected.
sudo apt-get install $(grep -vE "^\s*#" grml_packages | tr "\n" " ")
...
BEGIN ERROR in /usr/share/whonix/postinst.d/70_locale detected.

ERROR LOG:
***update-locale: Error: invalid locale settings: LANGUAGE="en_IN:en" LANG=en_US.UTF-8.

BASH_COMMAND: update-locale LANG=en_US.UTF-8
EXIT_CODE: 255
END ERROR in /usr/share/whonix/postinst.d/70_locale detected.
...

Using anything else then LANG=en_US.UTF-8 breaks the script?

#16

Thank you for that analysis.

Using anything else then LANG=en_US.UTF-8 breaks the script?

Looks like.

It follows a sysadmin question. How to recover from…

sudo update-locale LANG=en_US.UTF-8 ***update-locale: Error: invalid locale settings: LANGUAG=E"en_IN:en" LANG=en_US.UTF-8.

Best would be, if we could run some commands, that enable us to run “sudo update-locale LANG=en_US.UTF-8”. As soon we found out, I’ll add it to the build script.

#17

I figured out. Has to do with /etc/locale.gen. Fix soon. Will be in Whonix 8.

#18

Hopefully fixed in:
https://github.com/Whonix/Whonix/commit/089b3d3c15301de9ab686a745d8dda8ade97838d

tag: nothing-to-see 7.7.9.0

I tested it as best I could from my running system. Still need to make real builds.

#19

Doesn’t work. Needs more work.

#20

New attempt.
https://github.com/Whonix/Whonix/commit/1317a8a6dca9de6214b4a38b76f6ed240ca96d74

tag: nothing-to-see 7.7.9.1

I kicked the locale thing out. See commit message. I am trying to build it.

And I couldn’t resist also merging the improved build script error handler.

For demonstration:

...
INFO: Setting... export UWT_DEV_PASSTHROUGH="1"
whonix_build_whonix_version_new: 7.7.8.8.23.g1317a8a6dca9de6214b4a38b76f6ed240ca96d74
whonix_build_new_changelog_version: 2:7.7.8.8-debpackage1
+ cd /home/user/whonix_dot/Whonix/help-steps
+ cd ..
...
+ xxxxxxxxxxxxx
./build-steps.d/1200_create-debian-packages: line 396: xxxxxxxxxxxxx: command not found
++ error_handler_general
++ error_handler_shared
++ last_exit_code=127
++ last_bash_command=xxxxxxxxxxxxx
++ '[' test -o xtrace = 0 ']'
++ set +x
ERROR in ./build-steps.d/1200_create-debian-packages detected!
Please have a look above "error_handler_general", note the command that failed, its output and last_exit_code.
- Please enter c and press enter to continue. (Recommended against!)
- Please press enter to cleanup and exit.

Will pause there and wait for input.

When pressing enter.

+++ caller
++ last_caller='91 pre'
++ last_script=./build-steps.d/1200_create-debian-packages
++ '[' test -o xtrace = 0 ']'
++ output_command=true
++ rm --force /etc/apt/sources.list.d/whonix_temp.list
++ rm --force /var/run/whonix/package_manager_lock
++ error_handler_exit
++ '[' false = true ']'
++ abort_or_continue=Aborted
++ true '
############################################################
ERROR in ./build-steps.d/1200_create-debian-packages detected! Aborted!
(benchmark: 00:00:00)
BASH_COMMAND: xxxxxxxxxxxxx
last_exit_code: 127
caller: 91 pre
ERROR in ./build-steps.d/1200_create-debian-packages! Aborted!
############################################################
'
++ '[' false = true ']'
++ exit 1

Or when pressing c and enter:

- Please press enter to cleanup and exit. c
+++ caller
++ last_caller='91 pre'
++ last_script=./build-steps.d/1200_create-debian-packages
++ '[' test -o xtrace = 0 ']'
++ output_command=true
++ rm --force /etc/apt/sources.list.d/whonix_temp.list
++ rm --force /var/run/whonix/package_manager_lock
++ error_handler_exit
++ '[' true = true ']'
++ abort_or_continue=Continue
++ true '
############################################################
ERROR in ./build-steps.d/1200_create-debian-packages detected! Continue!
(benchmark: 00:00:00)
BASH_COMMAND: xxxxxxxxxxxxx
last_exit_code: 127
caller: 91 pre
ERROR in ./build-steps.d/1200_create-debian-packages! Continue!
############################################################
'
++ '[' true = true ']'
++ '[' test -o xtrace = 0 ']'
++ set +x

You have chosen to ignore this error. Your build may be unstable!
This is recommended against unless you know what you are doing. Do not report
bugs, that are a result of this! Please press enter to continue.

Will pause there and wait for pressing enter. Then continue the build script.

This should help during development/testing. And allow advanced users to skip minor issues. And allow to note, what command failed with which exit code for those who haven’t a log enabled and can’t copy (due to cli-only).