I’ve updated all my Whonix Qubes from 15 to 16 except for 1. And I’d like to avoid rebuilding it from scratch.
I followed the instructions for upgrading, including all the sanity checks and was in the midst of “release-upgrade” when it crashed. This has happened 2 times now. But I have not been watching it when it actually happens. I just come back and my Terminal is gone. I can’t launch the Terminal or xterm. And qvm-ls reports the qube is still running.
If I shut it down it will not boot anymore. This is the second attempt on this Qube. I have not yet shut it down in case there is anything you can suggest I try first.
Hopefully I can help you track down the problem.
Thanks.
p.s. One other piece of info. I can still access the nginx server running on this qube via a browser but I get the default nginx message as if the website data is no longer there, which is strange.
You can append log output to a file log by appending the following command.
&> log
Also output before the crash might be useful. There’s a small chance due to user modifications (installed packages) that some Qubes specific package gets removed during release which is required.
didn’t get help in the qubes forum, so i set up an ssh connection and ran release-upgrade from there, here is the last output i got before it disconnected:
Preparing to unpack …/udisks2_2.9.2-2_amd64.deb …
Unpacking udisks2 (2.9.2-2) over (2.8.1-4) …
(Reading database … 92835 files and directories currently installed.)
Removing policykit-1-gnome (0.105-7) …
Removing policykit-1 (0.105-25) …
Removing libpam-systemd:amd64 (241-7~deb10u8) …
Removing libqt4-declarative:amd64 (4:4.8.7+dfsg-18+deb10u1) …
Removing libqt4-designer:amd64 (4:4.8.7+dfsg-18+deb10u1) …
Removing libqt4-help:amd64 (4:4.8.7+dfsg-18+deb10u1) …
Removing libqt4-xmlpatterns:amd64 (4:4.8.7+dfsg-18+deb10u1) …
Removing libqt4-network:amd64 (4:4.8.7+dfsg-18+deb10u1) …
Removing libqt4-scripttools:amd64 (4:4.8.7+dfsg-18+deb10u1) …
Removing libqt4-script:amd64 (4:4.8.7+dfsg-18+deb10u1) …
Removing libqt4-sql:amd64 (4:4.8.7+dfsg-18+deb10u1) …
Removing libqt4-svg:amd64 (4:4.8.7+dfsg-18+deb10u1) …
Removing libqt4-test:amd64 (4:4.8.7+dfsg-18+deb10u1) …
Removing libqtdbus4:amd64 (4:4.8.7+dfsg-18+deb10u1) …
Removing libqt4-xml:amd64 (4:4.8.7+dfsg-18+deb10u1) …
Removing libqtgui4:amd64 (4:4.8.7+dfsg-18+deb10u1) …
Removing qubes-whonix (1:16.5-1) …
Removing ‘diversion of /usr/share/qubes-updates-cache/errors/ERR_INVALID_URL to /usr/share/qubes-updates-cache/errors/ERR_INVALID_URL.anondist-orig by qubes-whonix’
Removing ‘diversion of /usr/share/tinyproxy/default.html to /usr/share/tinyproxy/default.html.anondist-orig by qubes-whonix’
Removing ‘diversion of /usr/lib/qubes/qubes-setup-dnat-to-ns to /usr/lib/qubes/qubes-setup-dnat-to-ns.anondist-orig by qubes-whonix’
Removing ‘diversion of /etc/qubes-rpc/qubes.SetDateTime to /etc/qubes-rpc/qubes.SetDateTime.anondist-orig by qubes-whonix’
Removing ‘diversion of /etc/qubes-rpc/qubes.SyncNtpClock to /etc/qubes-rpc/qubes.SyncNtpClock.anondist-orig by qubes-whonix’
Removing qubes-core-agent-networking (4.0.62-1+deb10u1) …
Removing qubes-vm-dependencies (4.0.11-1+deb10u1) …
Removing qubes-gui-agent (4.0.38-1+deb10u1) …
client_loop: send disconnect: Broken pipe
So far nothing wrong with that part of the log. The log is incomplete. It stops with:
Get:221 tor+https://deb.debian.org/
as well as it stops at a different point.
Most likely reason for incomplete log: Qubes global clipboard copy/paste has a maximum size. Unspecific to Whonix. → Information Booster might be Available!
The following packages will be REMOVED:
kicksecure-desktop-applications-xfce
qubes-core-agent qubes-core-agent-networking qubes-gui-agent
qubes-input-proxy-sender qubes-mgmt-salt-vm-connector qubes-usb-proxy
qubes-vm-dependencies qubes-whonix qubes-whonix-shared-packages-recommended
qubes-whonix-workstation salt-common salt-ssh systemcheck whonix-firewall
whonix-shared-packages-dependencies-cli
whonix-shared-packages-recommended-cli
whonix-workstation-shared-packages-shared-meta
The further debug, you’d need to either manually run the commands the build script would be running (look at /usr/sbin/release-upgrade) would be to edit the script and append qubes-whonix-workstation.
By removing the “–yes” you could abort before any above mentioned packages are installed. Removing the “–yes” everywhere in the script might be a good idea as well.
You could replace the experiment qubes-whonix-workstation by qubes-vm-dependencies.
The abstract task: you need to find out which package dependency of a custom installed package is created a conflict so APT decides to remove qubes-whonix-workstation / qubes-vm-dependencies instead of the custom installed package.
Not too easy to debug in Qubes. Time consuming. Since you cannot snapshot a VM and easily restore at least not as far as I know.
I copied /usr/sbin/release-upgrade and in the copy I deleted everything form the install line you suggest changing above down. Then I ran that script, doing everything before the problem occurs.
Then I tried installing each of those items manually one by one.
sudo apt-get --no-install-recommends libc6-dev
Says it will remove the qubes/whonix packages. So I say ‘n’ and don’t install. I go through and install the other 3 packages no problem.
I went back and tried installing libc6-def again. I get the same result, it wants to remove qubes-core-agent, etc. It was to upgrade 156, install 43 and remove 56 packages.
user@host:/usr/sbin$ sudo apt-get --no-install-recommends install libc6-dev qubes-whonix-workstation
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
qubes-whonix-workstation : Depends: qubes-whonix-shared-packages-recommended but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
Then I try installing libc6-def and qubes-whonix-workstation again. I get a similar message but the unmet dependency is now “qubes-mgmt-salt-vm-connector”
I’m going to stop here for now and see what you think of this. Should I keep installing the unmet dependencies?
Yes, please. You’ve got the correct mindset. Don’t do insecure stuff (installing unsigned packages, saying yes to strong warnings) (you didn’t do that) but do keep shuffling around with the packages. To fix or work around this, we need to figure out the the package conflict which APT is unable to resolve.
I installed qubes-mgmt-salt-vm-connector, tried installing libc6-dev and qubes-whonix-workstation. It wouldn’t let me because qubes-mgmt-salt-vm-connector is dependent on salt-ssh.
I tried
sudo apt install salt-ssh
And I got the same list as above I think. It wants to upgrade 158, install 46 and remove 53 packages including qubes-whonix, qubes-whonix-workstation, etc. So then I did:
Try to install qubes-vm-dependencies only. For now. Removal of qubes-whonix-workstation would be acceptable in the debugging process as long as no important Qubes packages such as qubes-gui-agent are removed. Would be interesting if qubes-vm-dependencies could be re-installed including all the packages which qubes-vm-dependencies Depends: on. See also:
apt-cache show qubes-vm-dependencies
Maybe some Qubes package recently changed dependencies and somehow became incompatible (packaging conflict) during the buster → bullseye upgrade? The same might be happening attempting to upgrade Qubes Debian buster → bullseye?
user@host:/usr/sbin$ sudo apt install qubes-vm-dependencies
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
gstreamer1.0-pulseaudio
Use 'sudo apt autoremove' to remove it.
Recommended packages:
qubes-vm-recommended
The following packages will be upgraded:
qubes-vm-dependencies
1 upgraded, 0 newly installed, 0 to remove and 1178 not upgraded.
Need to get 2,252 B of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 https://deb.qubes-os.org/r4.0/vm bullseye/main amd64 qubes-vm-dependencies amd64 4.0.11-1+deb11u1 [2,252 B]
Fetched 2,252 B in 7s (307 B/s)
(Reading database ... 92027 files and directories currently installed.)
Preparing to unpack .../qubes-vm-dependencies_4.0.11-1+deb11u1_amd64.deb ...
Unpacking qubes-vm-dependencies (4.0.11-1+deb11u1) over (4.0.11-1+deb10u1) ...
Setting up qubes-vm-dependencies (4.0.11-1+deb11u1) ...
Scanning processes...
Scanning candidates...
Scanning linux images...
No services need to be restarted.
No containers need to be restarted.
No user sessions are running outdated binaries.
Then as it recommended I tried:
user@host:/usr/sbin$ sudo apt install qubes-vm-recommended
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
cython3 : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
Recommends: python3-dev but it is not going to be installed
libsmbclient : Depends: samba-libs (= 2:4.13.5+dfsg-2) but it is not going to be installed
python3 : PreDepends: python3-minimal (= 3.7.3-1) but 3.9.2-3 is to be installed
Depends: python3.7 (>= 3.7.3-1~) but it is not going to be installed
Depends: libpython3-stdlib (= 3.7.3-1) but 3.9.2-3 is to be installed
python3-apt : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-cairo : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-cbor : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-cffi-backend : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-dbus : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-distutils : Depends: python3:any (>= 3.9.1-0~)
python3-dnf : Depends: python3-gpg but it is not going to be installed
Depends: python3-rpm (>= 4.14.0-1~) but it is not going to be installed
Recommends: deltarpm but it is not installable
python3-geoip : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-gi : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-gi-cairo : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-hawkey : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-kiwisolver : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-lib2to3 : Depends: python3:any (>= 3.9.1-0~)
python3-libapparmor : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-libcomps : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-libdnf : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-lz4 : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-markupsafe : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-matplotlib : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-msgpack : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-numpy : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-pil : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
Recommends: python3-olefile but it is not going to be installed
python3-psutil : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-pyqt5.sip : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-pyx : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-qubesdb : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-snappy : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-talloc : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-tk : Depends: python3 (>= 3.9.1-0~) but 3.7.3-1 is to be installed
python3-tornado : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-twisted-bin : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-ubjson : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-unbound : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-wsaccel : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-yaml : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-zmq : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
python3-zope.interface : Depends: python3 (>= 3.9~) but 3.7.3-1 is to be installed
qubes-vm-recommended : Depends: network-manager (>= 0.8.1-1) but it is not going to be installed
Depends: network-manager-gnome but it is not going to be installed
Depends: qubes-core-agent-network-manager but it is not going to be installed
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.
user@host:/usr/sbin$
release-upgrade command version 1.5 is available now. It’s provided by the whonix-legacy package which has been uploaded to all Whonix 15 / buster repositories. How to check release-upgrade command version version number is documented on the usual documentation wiki page for this release upgrade Release Upgrade Whonix 15 to Whonix 16 - Whonix.
In-place release upgrade should be fixed - but it assumes a non-broken APT state. So if you had created a VM snapshot (or clone or backup) before release upgrade attempt, you’re in luck now, revert to snapshot, perform a standard upgrade to get the updated/fixed release-upgrade and retry the release upgrade process.
Thanks Patrick, it seemed to have worked. I think I even caught it running the new check:
Conf qubes-vm-dependencies (4.0.12-1+deb11u1 Qubes Debian:deb.qubes-os.org [amd64])
+ true 'INFO: Checking if release upgrade is safe or would result in removal of the main meta package... This will take a moment...'
+ set +x
+ true 'INFO: Install release upgrade of dependency packages...'
+ apt-get-noninteractive --yes --no-install-recommends install sdwdate fasttrack-archive-keyring
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
fasttrack-archive-keyring
The following packages will be upgraded:
sdwdate
1 upgraded, 1 newly installed, 0 to remove and 1196 not upgraded.
Need to get 0 B/130 kB of archives.
After this operation, 6,144 B of additional disk space will be used.
Selecting previously unselected package fasttrack-archive-keyring.
(Reading database ... 91846 files and directories currently installed.)
Preparing to unpack .../fasttrack-archive-keyring_2020.12.19_all.deb ...
Unpacking fasttrack-archive-keyring (2020.12.19) ...
Preparing to unpack .../sdwdate_3%3a17.1-1_all.deb ...
Unpacking sdwdate (3:17.1-1) over (3:15.8-1) ...
It did end with errors, but seem manageable. exim4 packages have been a problem for me in in the past, usually fixed be reconfiguring with dpkg. I see someone else posted about this recently.
##########################################
## INFO: END : setup-wizard-dist postinst configure' '3:7.1-1
#################################################
'
+ exit 0
Setting up whonix-shared-default-applications-gui (3:22.0-1) ...
Setting up whonix-workstation-shared-packages-shared-meta (3:22.0-1) ...
Setting up qubes-whonix-workstation (3:22.0-1) ...
Errors were encountered while processing:
exim4-config
exim4-base
exim4-daemon-light
bsd-mailx
automysqlbackup
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)
+ true 'ERROR: An error was encountered during installation of release upgrade (1/2). See above.'
+ exit 1