Build failure of whonix-workstation Qubes template for R3.2 / add qubes-template-whonix to continuous integration service TravisCI

[phabricator-migrator]

Information

ID: 527
PHID: PHID-TASK-47vbywpp37o5fimoad2v
Author: marmarek
Status at Migration Time: resolved
Priority at Migration Time: Unbreak Now!

Description

Setting up whonix-workstation-packages-recommended (3:3.4-1) ...
dpkg: dependency problems prevent configuration of whonix-workstation-shared-packages-shared-meta:
 whonix-workstation-shared-packages-shared-meta depends on anon-workstation-packages-recommended; however:
  Package anon-workstation-packages-recommended is not configured yet.

dpkg: error processing package whonix-workstation-shared-packages-shared-meta (--configure):
 dependency problems - leaving unconfigured

(...)

Setting up qubes-whonix (1:5.5-1) ...
Adding 'diversion of /etc/qubes-rpc/qubes.SyncNtpClock to /etc/qubes-rpc/qubes.SyncNtpClock.anondist-orig by qubes-whonix'
Adding 'diversion of /etc/qubes-rpc/qubes.SetDateTime to /etc/qubes-rpc/qubes.SetDateTime.anondist-orig by qubes-whonix'
Adding 'diversion of /usr/lib/qubes/qubes-setup-dnat-to-ns to /usr/lib/qubes/qubes-setup-dnat-to-ns.anondist-orig by qubes-whonix'
Adding 'diversion of /usr/share/tinyproxy/default.html to /usr/share/tinyproxy/default.html.anondist-orig by qubes-whonix'
Adding 'diversion of /usr/share/qubes-updates-cache/errors/ERR_INVALID_URL to /usr/share/qubes-updates-cache/errors/ERR_INVALID_URL.anondist-orig by qubes-whonix'
Setting up qubes-whonix-shared-packages-recommended (1:5.5-1) ...
Setting up qubes-thunderbird (1.2.8-1+deb8u1) ...
Setting up qubes-gpg-split (2.0.23-1+deb8u1) ...
Setting up qubes-pdf-converter (2.0.5-1+deb8u1) ...
Setting up system-config-printer (1.4.6-1) ...
dpkg: dependency problems prevent configuration of qubes-whonix-workstation:
 qubes-whonix-workstation depends on whonix-workstation-shared-packages-shared-meta; however:
  Package whonix-workstation-shared-packages-shared-meta is not configured yet.

dpkg: error processing package qubes-whonix-workstation (--configure):
 dependency problems - leaving unconfigured

Full log: template-jessie+whonix-workstation+minimal+no-recommends.log · GitHub
(there is also gateway build log, but the build succeeded)

Comments

---

Patrick

2016-07-27 23:35:03 UTC

The culprit is probably this line:

qubes-template-whonix/whonix-gateway/04_install_qubes_post.sh at 4233ccab24e8f0ae4da22a66b20bb2789da1a886 · adrelanos/qubes-template-whonix · GitHub

I forget to set that to jessie since the builds created for Whonix 13 was blessed stable.

If you want just a debug build… If you can set this environment variable (not well tested)…

export whonix_repository_suite="jessie"

Or edit/commit to qubes-template-whonix/whonix-gateway/04_install_qubes_post.sh.

For a new testing or even stable release, I need to finish the updated qubes-whonix package that I am working on at the moment.

---

Patrick

2016-07-27 23:52:42 UTC

---

marmarek

2016-07-28 09:11:11 UTC

! In T527#9583, @Patrick wrote:
For a new testing or even stable release, I need to finish the updated qubes-whonix package that I am working on at the moment.

Ok, so I think for Qubes R3.2-rc2 I’ll use whonix templates from rc1. Is it ok?

---

Patrick

2016-07-28 14:49:49 UTC

Yes. (Whonixl legacy bind-directories should be disabled thanks to the
qubes-core-agent-linux upgrade and bind-dirs.sh should work fine.)

---

Patrick

2016-07-28 19:46:54 UTC

! In T527#9601, @marmarek wrote:

! In T527#9583, @Patrick wrote:
For a new testing or even stable release, I need to finish the updated qubes-whonix package that I am working on at the moment.

Ok, so I think for Qubes R3.2-rc2 I’ll use whonix templates from rc1. Is it ok?

! In T527#9602, @Patrick wrote:
Yes. (Whonixl legacy bind-directories should be disabled thanks to the
qubes-core-agent-linux upgrade and bind-dirs.sh should work fine.)

On a second thought that would not work. If you use templates from R3.3-rc1 those still and these get upgrades to newer qubes-core-agent, these would be missing the bind-directories legacy function that is not yet in Whonix stable repository. whonix-setup-wizard would rerun, lost Tor config, new Tor entry guards… Perhaps a compromise we can make.

What would work better would be a rebuild of Whonix 13 from Whonix stable repository with Qubes R3.2-rc2. These would get started with bind-dirs.sh from the first boot.

---

marmarek

2016-07-28 20:18:54 UTC

R3.2-rc2 is already released…
But - the whole thing applied only to workstation template - gateway build was ok and new one is included there - I’ve just checked and it has legacy function in /usr/lib/qubes-bind-dirs.d/41_qubes-whonix.conf.

BTW I think it worth adding template builds for automatic testing via travis-ci. It’s super easy - take a look at Debian one:
https://github.com/QubesOS/qubes-builder-debian/blob/master/.travis.yml

What repository would be good for it? qubes-template-whonix?

---

Patrick

2016-07-28 20:55:30 UTC

! In T527#9607, @marmarek wrote:
R3.2-rc2 is already released…
But - the whole thing applied only to workstation template - gateway build was ok and new one is included there - I’ve just checked and it has legacy function in /usr/lib/qubes-bind-dirs.d/41_qubes-whonix.conf.

I wonder where /usr/lib/qubes-bind-dirs.d/41_qubes-whonix.conf comes from. I haven’t added it to the stable repository.

BTW I think it worth adding template builds for automatic testing via travis-ci.

That would be awesome!

It’s super easy - take a look at Debian one:
https://github.com/QubesOS/qubes-builder-debian/blob/master/.travis.yml

What repository would be good for it? qubes-template-whonix?

Yes.

---

marmarek

2016-07-28 21:25:27 UTC

! In T527#9609, @Patrick wrote:

! In T527#9607, @marmarek wrote:
R3.2-rc2 is already released…
But - the whole thing applied only to workstation template - gateway build was ok and new one is included there - I’ve just checked and it has legacy function in /usr/lib/qubes-bind-dirs.d/41_qubes-whonix.conf.

I wonder where /usr/lib/qubes-bind-dirs.d/41_qubes-whonix.conf comes from. I haven’t added it to the stable repository.

Probably because of the line you’ve mentioned earlier the build used development repository…

---

Patrick

2016-07-28 22:37:23 UTC

Ah. Sure. However, an image build from the Whonix developers repository should not make it into any testing or even stable releases. That repo is very unstable where I do experiments that could break anything.

---

marmarek

2016-07-28 22:44:48 UTC

Hmm, this is all strange. As you can see in the build log, I’ve build from GitHub - QubesOS/qubes-template-whonix, master branch. And there I see [[ https://github.com/marmarek/qubes-template-whonix/blob/master/builder.conf#L14 | WHONIX_APT_REPOSITORY_OPTS ?= stable ]]
Is this setting ignored?

---

Patrick

2016-08-01 02:04:17 UTC

Added .travis.yml:

https://github.com/adrelanos/qubes-template-whonix/blob/master/.travis.yml

sudo ln -s sid /usr/share/debootstrap/scripts/stretch is failing. Log:

https://api.travis-ci.org/jobs/148761324/log.txt?deansi=true

Any idea? Is it even needed? I try without. (Without is also better for more consistent results.)

---

Patrick

2016-08-01 02:16:59 UTC

Package anon-workstation-packages-recommended is not configured yet.

This is likely just be a symptom. [I also would have wondered what could have broken dependencies.] The build breaks on purpose. “Failing closed.” The culprit is:

Failed to download: https://dist.torproject.org/torbrowser/5.5.5/sha256sums.txt.asc

I think we discussed this before, if so, you likely remember. Anyhow, I described that failing closed mechanism when TBB download fails during build here:
Tor Browser Essentials

qubes-template-whonix has been changed to whonix_repository_suite="jessie-proposed-updates" for now.

Even if I haven’t learned how to build Qubes R3.2 myself yet (Redirecting to Google Groups), chances are good, that you are already able to build Qubes R3.2 Whonix 13.

Whonix jessie-proposed-updates repository contains both a newer version of tb-updater (maintenance update with up to date version number) as well as an updated qubes-whonix package (fixes from T528).

---

Patrick

2016-08-01 02:31:12 UTC

Another Travis CI issue.

Makefile:45: *** Building template whonix-gateway not supported by any of configured plugins. Stop.

https://s3.amazonaws.com/archive.travis-ci.org/jobs/148762157/log.txt

(Using a minimal TravisCI config for now until these issues are sorted out until all variations get added.)

---

~3 years ago I had TravisCI builds somewhat working. --target root and package builds only. No vm image builds back then due to various TravisCI limitations. At one point I gave up on TravisCI, because there was too much effort and to keep it running and too much if Debian or Ubuntu code. Perhaps things have improved now. Hopefully! Generally, CI is very cool, but a Debian based CI platform is desirable.

---

marmarek

2016-08-01 09:16:13 UTC

! In T527#9673, @Patrick wrote:
Added .travis.yml:

https://github.com/adrelanos/qubes-template-whonix/blob/master/.travis.yml

sudo ln -s sid /usr/share/debootstrap/scripts/stretch is failing. Log:

https://api.travis-ci.org/jobs/148761324/log.txt?deansi=true

Any idea? Is it even needed? I try without. (Without is also better for more consistent results.)

It needs to be done after installing debootstrap.

Also, dist: jessie is not supported, only precise and trusty - and we need trusty to have less ancient tools (for example make is too old in precise).

Makefile:45: *** Building template whonix-gateway not supported by any of configured plugins. Stop.

You need to enable template-whonix builder plugin (in addition to builder-debian). Simply place BUILDER_PLUGINS="template-whonix builder-debian" in env.

---

Patrick

2016-08-01 13:55:04 UTC

Using that now.

Yet BUILDER_PLUGINS is missing it.

BUILDER_PLUGINS:

     builder-fedora, builder-debian, mgmt-salt,

Hence, build still failing.

Makefile:45: *** Building template whonix-gateway not supported by any of configured plugins. Stop.

log:
https://s3.amazonaws.com/archive.travis-ci.org/jobs/148863490/log.txt

---

marmarek

2016-08-01 14:02:56 UTC

! In T527#9675, @Patrick wrote:
Package anon-workstation-packages-recommended is not configured yet.

This is likely just be a symptom. [I also would have wondered what could have broken dependencies.] The build breaks on purpose. “Failing closed.” The culprit is:

Failed to download: https://dist.torproject.org/torbrowser/5.5.5/sha256sums.txt.asc

Ah, haven’t found this before. Anyway this was the fourth build in row which failed with the same message…

I think we discussed this before, if so, you likely remember. Anyhow, I described that failing closed mechanism when TBB download fails during build here:
Tor Browser Essentials

qubes-template-whonix has been changed to whonix_repository_suite="jessie-proposed-updates" for now.

Even if I haven’t learned how to build Qubes R3.2 myself yet (Redirecting to Google Groups), chances are good, that you are already able to build Qubes R3.2 Whonix 13.

Whonix jessie-proposed-updates repository contains both a newer version of tb-updater (maintenance update with up to date version number) as well as an updated qubes-whonix package (fixes from T528).

Ok, will try.

---

marmarek

2016-08-01 14:04:44 UTC

! In T527#9680, @Patrick wrote:
Using that now.

Yet BUILDER_PLUGINS is missing it.

BUILDER_PLUGINS:

     builder-fedora, builder-debian, mgmt-salt,

Hence, build still failing.

Makefile:45: *** Building template whonix-gateway not supported by any of configured plugins. Stop.

log:
https://s3.amazonaws.com/archive.travis-ci.org/jobs/148863490/log.txt

Try again (just retry build from travis UI)- I’ve fixed builder.conf to allow BUILDER_PLUGINS being overridden from env:
travis: switch back to official repo, allow override from .travis.yml · QubesOS/qubes-builder@f407369 · GitHub

---

Patrick

2016-08-01 14:55:12 UTC

Getting closer.

-> Updating sources for template-whonix...

--> Fetching from https://github.com/QubesOS/qubes-template-whonix.git master...

remote: Invalid username or password.

fatal: Authentication failed for 'https://github.com/QubesOS/qubes-template-whonix.git/'

make: *** [template-whonix.get-sources] Error 128

I could probably add a pre-script to add an override.conf that uses Whonix rather than QubesOS as github account name unless you have a better solution in mind.

---

Patrick

2016-08-01 15:32:07 UTC

I have the override file in place.

https://github.com/adrelanos/qubes-template-whonix/blob/master/.travis.yml

But it is ignored. Still failing for the same reason.

How do I make changes to override.conf take effect from command line? (usually I use ./setup, but that gui tool won’t work for TravisCI)

log:
https://api.travis-ci.org/jobs/148908578/log.txt?deansi=true

---

marmarek

2016-08-01 15:49:39 UTC

! In T527#9688, @Patrick wrote:
I have the override file in place.

https://github.com/adrelanos/qubes-template-whonix/blob/master/.travis.yml

But it is ignored. Still failing for the same reason.

How do I make changes to override.conf take effect from command line? (usually I use ./setup, but that gui tool won’t work for TravisCI)

override.conf is only a feature of ./setup - to apply modification some and and still be able to use setup UI. It isn’t a standard qubes-builder configuration.

But you can set of those settings using environment. Maybe even export xxx=yyy in before script will work (to avoid very long lines in env section).

---

Patrick

2016-08-01 16:53:12 UTC

That works. Sorting out gpg key import and verification was a bit difficult. And the before_script looks kinda complex now. Non-ideal, but it is building now. Let’s see.

---

Patrick

2016-08-01 17:10:51 UTC

The build succeeded, however there was an error at the end.

./create_template_list.sh: line 13: xenstore-read: command not found

log:
https://api.travis-ci.org/jobs/148930967/log.txt?deansi=true

---

marmarek

2016-08-01 17:50:20 UTC

! In T527#9691, @Patrick wrote:
The build succeeded, however there was an error at the end.

./create_template_list.sh: line 13: xenstore-read: command not found

Looks to be related to not running inside Qubes. Can be safely ignored.

---

Patrick

2016-08-01 19:10:45 UTC

Half of the builds succeeded. Both gateway and workstation builds succeeded. R3.1 and R3.2 builds succeeded.

Interestingly, only builds using USE_QUBES_REPO_TESTING=1 succeeded. These not using that failed.

Travis CI - Test and Deploy with Confidence

Is this something to worry about or something worth ignoring since fixed in testing?

---

marmarek

2016-08-01 19:39:24 UTC

For 3.1 - yes, this is expected, qubes-core-agent package currently is available only in testing repository (will be in stable in two days) - qvm-open-in-vm no longer opens URLs in target VM browser · Issue #2205 · QubesOS/qubes-issues · GitHub

For 3.2 - that’s strange:

Setting up brltty (5.2~20141018-5) ...
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
./functions.sh: line 68: 28742 Segmentation fault      (core dumped) /usr/sbin/chroot "${INSTALLDIR}" ${1+"$@"}

---

Patrick

2016-08-01 19:57:29 UTC

Where did that happen? I checked all successful logs and found that nowhere.

---

marmarek

2016-08-01 20:06:10 UTC

In the failed one (job #13.5)

---

Patrick

2016-08-02 12:25:33 UTC

Okay, that is very strange indeed. No idea. Perhaps a transient TravisCI issue?

Disabled the failing ones for now. Qubes testing R3.1 and R3.2 are building fine on a rebuild.

---

Patrick

2016-08-14 22:49:43 UTC

build Qubes-Whonix 13.0.0.1.0 templates for R3.2 final · Issue #2228 · QubesOS/qubes-issues · GitHub