cowbuilder debugging

Patrick · August 26, 2019, 1:58pm

(In response to Whonix Desktop Installer with Calamares - field report - #132 by onion_knight)

whonix_build_multiarch_package_item="amd64"
cow_folder="/var/cache/pbuilder/cow.cow_${whonix_build_multiarch_package_item}"
base_folder="/var/cache/pbuilder/base.cow_${whonix_build_multiarch_package_item}"
whonix_build_pbuilder_config_file="/home/user/whonix_binary/pbuilder.conf"
sudo cowbuilder \
   --execute "/bin/bash" \
   --basepath "$base_folder" \
   --buildplace "$cow_folder" \
   --configfile "$whonix_build_pbuilder_config_file"

Should show root@host:/# which is then a root shell inside the chroot.

Since not using

--save-after-login

changes will not be permanent.

Probably worth adding a script into Whonix help-steps folder to simplify invocation?

Then try for example:

sudo apt-get install clang-tidy

Or.

sudo apt-get install debhelper genmkfile clang llvm-7-dev clang-tidy

To leave the chroot, just type:

exit

onion_knight · August 27, 2019, 9:23am

E: cannot canonicalize filename /var/cache/pbuilder/base.cow_amd64, does not exist

Patrick · August 27, 2019, 9:30am

Not sure when _amd64 was introduced. But should already exist. Dropping _amd64 works?

Any folders in /var/cache/pbuilder/ exist?

ls -la /var/cache/pbuilder/

cowbuilder debugging presupposes that build-steps.d/1100_prepare-build-machine was already run and successful.

build-steps.d/1100_prepare-build-machine was recently split, though.

https://github.com/Whonix/Whonix/commit/8cd7bfa8606755539abe46180530fd32c0a0c7ae

In other words as per git tag 15.0.0.4.3-developers-only the first three build steps

build-steps.d/1100_sanity-tests
build-steps.d/1120_prepare-build-machine
build-steps.d/1130_cowbuilder-setup

should have already succeeded.

onion_knight · August 27, 2019, 9:50am

ls -la /var/cache/pbuilder/
total 32
drwxr-xr-x  8 root root 4096 Apr 27  2018 .
drwxr-xr-x 15 root root 4096 Apr 30 11:42 ..
drwxr-xr-x  2 root root 4096 Apr 23  2017 aptcache
drwxr-xr-x  2 root root 4096 Apr 23  2017 build
drwxr-xr-x  2 root root 4096 Apr 23  2017 ccache
drwxr-xr-x  2 root root 4096 Apr 23  2017 pbuildd
drwxr-xr-x  2 root root 4096 Apr 23  2017 pbuilder-mnt
drwxr-xr-x  2 root root 4096 Apr 23  2017 result

If I drop the first line (whonix_build_multiarch_package_item="amd64"), this error happens:
cannot canonicalize filename /var/cache/pbuilder/base.cow_, does not exist

Patrick · August 27, 2019, 9:52am

Try 15.0.0.4.5-developers-only

Change directory into Whonxi source folder first.

cd ~/Whonix

Then try.

sudo -E ./build-steps.d/1100_sanity-tests --build --redistribute --internalrun --target qcow2 --target iso

sudo -E ./build-steps.d/1120_prepare-build-machine --build --redistribute --internalrun --target qcow2 --target iso

sudo -E ./build-steps.d/1130_cowbuilder-setup --build --redistribute --internalrun --target qcow2 --target iso

Patrick · August 27, 2019, 9:56am

It would be without the underscore but never mind. Cowbuilder folder does not exist anyhow. So instructions from post #1 in this post won’t work.

Anyhow… Next step…

After following instruction in my above post folder /var/cache/pbuilder/base.cow_amd64 will exist.

Then instructions from original post will work.

I by mistake expected this test to be run in a VM that was previously used to try to build Whonix. Then cowbuilder chroot would already exist. Since it was probably done in a fresh VM (not bad) I missed to explain how to actually create the cowbuilder chroot.

onion_knight · August 27, 2019, 9:59am

You are right, this is done in a VM previously used to build Whonix.

I am trying now

Patrick:

Try 15.0.0.4.5-developers-only

sudo -E ./build-steps.d/1100_sanity-tests --build --redistribute --internalrun --target qcow2 --target iso

sudo -E ./build-steps.d/1120_prepare-build-machine --build --redistribute --internalrun --target qcow2 --target iso

sudo -E ./build-steps.d/1130_cowbuilder-setup --build --redistribute --internalrun --target qcow2 --target iso

onion_knight · August 27, 2019, 10:05am

The three scripts ran successfuly.

Patrick · August 27, 2019, 10:13am

Now try original post please

onion_knight · August 27, 2019, 11:03am

Works

I: Copying COW directory
I: forking: rm -rf /var/cache/pbuilder/cow.cow_amd64/cow.30970
I: forking: cp -al /var/cache/pbuilder/base.cow_amd64 /var/cache/pbuilder/cow.cow_amd64/cow.30970
I: removed stale ilistfile /var/cache/pbuilder/cow.cow_amd64/cow.30970/.ilist
I: Invoking pbuilder
I: forking: pbuilder execute --configfile /home/user/whonix_binary/pbuilder.conf --buildplace /var/cache/pbuilder/cow.cow_amd64/cow.30970 --mirror http://ftp.de.debian.org/debian/ --distribution sid --no-targz --internal-chrootexec 'chroot /var/cache/pbuilder/cow.cow_amd64/cow.30970 cow-shell' /bin/bash
W: /root/.pbuilderrc does not exist
I: Running in no-targz mode
I: copying local configuration
W: --override-config is not set; not updating apt.conf Read the manpage for details.
I: mounting /proc filesystem
I: mounting /sys filesystem
I: creating /{dev,run}/shm
I: mounting /dev/pts filesystem
I: redirecting /dev/ptmx to /dev/pts/ptmx
I: mounting /dev/pts/0 over /dev/console
I: policy-rc.d already exists
I: Obtaining the cached apt archive contents

Ends up with a root shell

Patrick · August 27, 2019, 11:03am

Did that also work?

onion_knight · August 27, 2019, 11:04am

Yes
sudo apt install clang-tidy
Reading package lists… Done
Building dependency tree
Reading state information… Done
The following additional packages will be installed:
clang-7 clang-tidy-7 clang-tools-7 lib32gcc1 lib32stdc++6 libc6-i386 libclang-common-7-dev libclang1-7 libedit2 libgc1c2 libllvm7 libobjc-8-dev libobjc4 libyaml-0-2 python-yaml
Suggested packages:
clang-7-doc
Recommended packages:
llvm-7-dev libomp-7-dev
The following NEW packages will be installed:
clang-7 clang-tidy clang-tidy-7 clang-tools-7 lib32gcc1 lib32stdc++6 libc6-i386 libclang-common-7-dev libclang1-7 libedit2 libgc1c2 libllvm7 libobjc-8-dev libobjc4 libyaml-0-2 python-yaml
0 upgraded, 16 newly installed, 0 to remove and 0 not upgraded.
Need to get 95.2 MB of archives.
After this operation, 430 MB of additional disk space will be used.
Do you want to continue? [Y/n]

Patrick · August 27, 2019, 11:04am

Installation finishes without error?

onion_knight · August 27, 2019, 11:05am

Yes

Patrick · August 27, 2019, 11:06am

Alright. Strange but… Let’s see.

cd into helper-scripts

cd helper-scripts

Folder ~/genmkfile-packages-result needs to exist.

mkdir ~/genmkfile-packages-result

Build using cowbuilder.

make_cowbuilder_dist_folder=~/genmkfile-packages-result make_use_cowbuilder=true make deb-pkg

Patrick · August 27, 2019, 11:07am

And if helper-scripts can be build successfully using cowbuilder, try building hardened malloc with it.

Patrick · August 27, 2019, 11:08am

Btw if you git cloned Whonix (to build Whonix images) then all sources of all packages are already on the disk in folder

~/Whonix/packages

onion_knight · August 27, 2019, 11:10am

OK this exited with 0 status make_cowbuilder_dist_folder=~/genmkfile-packages-result make_use_cowbuilder=true make deb-pkg

Everything seems OK, maybe I should try to build Whonix-Host 15.0.0.4.5?

Patrick · August 27, 2019, 11:10am

So both helper-scripts as well as hardened malloc could be build using cowbuilder successfully?

Patrick · August 27, 2019, 11:11am

Success also meaning… Packages are build and can be found in this folder…?

~/genmkfile-packages-result