hardened-malloc package build failed - Unpacking llvm-7-dev (1:7.0.1-8) ... - E: Sub-process /usr/bin/dpkg exited unexpectedly

Patrick · August 26, 2019, 12:28pm

Original bug report:

Continuing the discussion from Whonix Desktop Installer with Calamares - field report:

First, try to build package security-misc (or any other known good package for testing). That is a prerequisite exercise.

Then, move on to be building the failing hardened-malloc package the “easy” way.

See:

https://www.whonix.org/wiki/Dev/Build_Documentation/security-misc

The build process of building any of the packages by Whonix developers is always very similar. This very wiki page explain how to build package security-misc . By replacing the name of the package security-misc with the name of any other package you might be interested in building, the other package would be build.

Does that work for you?

If that works for you, let’s move on building a known good pacakge using cowbuilder.

build-steps.d/1100_prepare-build-machine sets up a chroot in /var/cache/pbuilder using cowbuilder.

Will follow up with instructions on how to build a package using cowbuilder.

onion_knight · August 26, 2019, 12:45pm

Ok, step by step.

First, building security-misc.
Fails at make deb-all-dep (" Build and Install the Package") stage:

make deb-all-dep
/usr/share/genmkfile/make-helper.bsh deb-all-dep
/usr/share/genmkfile/make-helper-one.bsh: INFO: deb-all-dep
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package apparmor-profile-anondist
make: *** [/usr/share/genmkfile/makefile-full:64: deb-all-dep] Error 100

Patrick · August 26, 2019, 1:54pm

That wasn’t the best example. It would require either

installing apparmor-profile-anondist from Whonix repository
or building that package from source code too https://github.com/Whonix/apparmor-profile-anondist

Try this package instead perhaps: https://github.com/Whonix/helper-scripts
because that does not have other Whonix packages it depends on which need to be (build/)installed first.

onion_knight · August 26, 2019, 2:36pm

OK, building helper-scripts worked without error.

How to build hardened-malloc? git clone command does not work

git clone https://github.com/Whonix/hardened-malloc.git
Cloning into 'hardened-malloc'...
Username for 'https://github.com':

Patrick · August 26, 2019, 4:41pm

Typo. Not -. It’s _.

git clone https://github.com/Whonix/hardened_malloc.git

onion_knight · August 27, 2019, 8:33am

Thanks!

make deb-all-dep
make: *** No rule to make target 'deb-all-dep'.  Stop.

Using

git describe
PQ3A.190705.001.2019.07.16.22

There is a Makefile inside the directory though.

Patrick · August 27, 2019, 8:41am

Wrong tag. I wonder why git master for you points at PQ3A.190705.001.2019.07.16.22

user@disp6713:~$ git clone https://github.com/Whonix/hardened_malloc.git
Cloning into 'hardened_malloc'...
remote: Enumerating objects: 20, done.
remote: Counting objects: 100% (20/20), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 1568 (delta 7), reused 12 (delta 6), pack-reused 1548
Receiving objects: 100% (1568/1568), 469.16 KiB | 514.00 KiB/s, done.
Resolving deltas: 100% (1000/1000), done.
user@disp6713:~$ cd hardened_malloc/
user@disp6713:~/hardened_malloc$ ls
Android.bp          h_malloc.c                  memory.h    random.h
calculate_waste.py  h_malloc.h                  mutex.h     README_generic.md
chacha.c            KERNEL_FEATURE_WISHLIST.md  new.cc      README.md
chacha.h            LICENSE                     pages.c     test
changelog.upstream  Makefile                    pages.h     third_party
CREDITS             make-helper-overrides.bsh   preload.sh  util.c
debian              memory.c                    random.c    util.h
user@disp6713:~/hardened_malloc$ git describe 
0.5-1

Symptom missing genmkfile (you don’t have this symptom):

user@disp6713:~/hardened_malloc$ make deb-all-dep
Makefile:147: /usr/share/genmkfile/makefile-full: No such file or directory
make: *** No rule to make target '/usr/share/genmkfile/makefile-full'.  Stop.

Did you clone from https://github.com/GrapheneOS/hardened_malloc or another place? That wouldn’t work - does not have Debian packaging.

In summary…

git clone https://github.com/Whonix/hardened_malloc.git

git tag -v 0.5-1

git checkout 0.5-1

onion_knight · August 27, 2019, 8:45am

building fails

+ sudo apt-get --yes --no-install-recommends install debhelper genmkfile clang llvm-7-dev clang-tidy '${shlibs:Depends}'
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package ${shlibs:Depends}
E: Regex compilation error - Invalid preceding regular expression
E: Couldn't find any package by regex '${shlibs'
+ apt_get_exit_code=100
+ true
+ exit 100
++ error_handler_pre
++ local exit_code=100
++ local 'last_err="$GENMKFILE_PATH/make-helper-one.bsh" "$@"'
++ '[' '!' 1 = 1 ']'
++ '[' '' = '' ']'
++ output='## See above.'
++ force_echo '
####################################################################
## BEGIN ERROR in /usr/share/genmkfile/make-helper.bsh detected!
##
## ERROR LOG:
## See above.
##
## BASH_COMMAND: "$GENMKFILE_PATH/make-helper-one.bsh" "$@"
## EXIT_CODE: 100
##
## END ERROR in /usr/share/genmkfile/make-helper.bsh detected!
## Please report this bug!
####################################################################
'
++ '[' '!' 1 = 1 ']'
++ '[' '!' 1 = 1 ']'
++ rm -f /tmp/tmp.GXYLT2yCAM
++ exit 1
make: *** [/usr/share/genmkfile/makefile-full:64: deb-all-dep] Error 1

seems ${shlibs:Depends} is not defined

Patrick · August 27, 2019, 8:57am

Just drop

${shlibs:Depends}

Patrick · August 27, 2019, 8:59am

Run

sudo apt-get --yes --no-install-recommends install debhelper genmkfile clang llvm-7-dev clang-tidy

As per:

Install build dependencies. It might be possible to let genmkfile do that.
make deb-all-dep
If that did not work, have a look in debian/control file and manually install all packages listed under Build-Depends and Depends .

Dependency installation isn’t a big deal. Isn’t done during actual package build.

make deb-all-dep is fixed in genmkfile git master. make deb-all-dep is just a convenience feature which unfortunately broke due to ${shlibs:Depends}.

onion_knight · August 27, 2019, 9:05am

Ok, removed ${shlibs:Depends} from debian/control

Build successful

What next

Patrick · August 27, 2019, 9:20am

This:

Patrick · August 27, 2019, 10:02am

After cowbuilder debugging we can try build known good packages using cowbuilder followed by building the failing package using cowbuilder.

Patrick · August 27, 2019, 11:49am

Also using cowbuilder for package builds can be disabled.

By setting make_use_cowbuilder=false build configuration.

sudo make_use_cowbuilder=false ./...

Add make_use_cowbuilder=false after sudo and before ./whonix_build.

Or by setting it through any other way as per build configuration.

Builds using cowbuilder are better:

makes sure no extraneous files are included in the build - since it builds them in chroot
makes sure that no dependencies are missing in debian/control
i.e. if package can be build in chroot, anyone should be able to build it in most cases - except a weird bug such as this one. This one is perhaps in top 3 most weird bugs ever.

It’s important (much better at least) that release builds are build that way but not so much for testing/debug builds.