Doing `dist-upgrade` just bricked my Whonix.

To upgrade in-place to the point release 15.0.0.8.9 I just did apt update and dist-upgrade, reboot, and it bricked my Whonix:

Kernel Panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

In my case I was even more severely locked out because I had set my grub timeout to 0, and pressing shift (etc) to access grub didn’t work.

I managed to attach my Workstation’s VDI as a second storage drive to another workstation VM, boot into the other VM, and after booting off that other VM’s grub it strangely loaded my broken workstation as the actual linux system. I could then easily change its particular grub timeout to 5 again instead of mounting its file system externally and doing a more advanced manual grub edit.

So with grub access restored, I booted back in using an older kernel version and did:

sudo dpkg --configure -a

which fixed it.

I can now boot into the Whonix with normal newest kernel 4.19.0-8, and the problem is resolved.


But if this will happen to others in the course of normal system upgrading, people not as advanced will suddenly lose their Whonix! That is terrible for Whonix remaining accessible to people who really need it.

As a possible clue in my case, I have had kernel updates fail at the end of updates for quite a while. (But if it’s happening in my Whonix, isn’t it happening in others?)

People shouldn’t be expected to backup their Whonix right before every dist-upgrade, just in case the upgrade bricks their Whonix. Unless dist-upgrade really is a major thing and people should only regularly perform upgrade without having unexpected concerns?

Either way, this was really bad, so here I am reporting it. What can be done about it?

Obviously.

Obviously.

Disagree. (Because the difference between dist-upgrade and upgrade is maybe different from what you might think.)

No reports of it happening by others.

Potential causes for bugs which only affect few users:

  • host hardware issues
  • host operating system bugs
  • virtualizer bugs

See also:

Theoretically: many things

Practically:

I’ve never had this happen to me when upgrading Whonix.

I’ve only had stuff like this happen when messing with apparmor-profile-everything or hardened-kernel which isn’t used by ordinary users yet.

Did you do a lot of configuration? Why exactly did your updates fail? Was there an error message?

We can’t really deduce much from “there was an error”.

1 Like

OK, if it’s quite possibly an uncommon incident, that makes me more relieved. Thank you for the information.

1 Like

In my in-place upgrade from 15.0.0.8.9 to 15.0.0.9.4, once again I had to do sudo dpkg --configure -a after sudo apt upgrade, in order to fix it. Any idea if it’s a common issue that’s commonly fixable? If I had not done that once again my Whonix would have been ‘bricked’ upon restart.

I do customize my Whonix quite a lot but it’s really not hardcore stuff that I change. If it comes up again next time I’ll try to get the terminal output before I apply the dpkg fix.

Not common. Might be fixable. Need to see it.

For sure needed.

1 Like

Yet again, this terrible bug has happened to me. I kept a backup of my broken VM before fixing it via the same exact method reported previously, and this time I’ll come back and share what the errors say so that hopefully it can be effectively troubleshooted.

Here is the info.

For several times when last performing sudo apt upgrade, my Whonix has finished the command with this error output:

Setting up initramfs-tools (0.133+deb10u1) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for initramfs-tools (0.133+deb10u1) ...
update-initramfs: Generating /boot/initrd.img-4.19.0-10-amd64
cryptsetup: WARNING: The initramfs image may not contain cryptsetup binaries 
    nor crypto modules. If that's on purpose, you may want to uninstall the 
    'cryptsetup-initramfs' package in order to disable the cryptsetup initramfs 
    integration and avoid this warning.
live-boot: core filesystems devices utils udev blockdev dnsE: /usr/share/initramfs-tools/hooks/live failed with return 1.
update-initramfs: failed for /boot/initrd.img-4.19.0-10-amd64 with 1.
dpkg: error processing package initramfs-tools (--configure):
 installed initramfs-tools package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 initramfs-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)

Because it returns to the prompt, it’s easy to not notice that system upgrading didn’t actually succeed.

If I then shut down the Workstation, it’s ‘bricked’, so to speak.

On next boot, it stops at an all black with single cursor screen. It quickly flashes a few kernel errors about unicode.pf2.

I do fairly normal things in my Whonix, just installing packages, not heavily modifying the system in a way that would reasonably ‘void warranty’ so to speak. I do have to apply my own fix to the GDebi policy bug, though.

What would be causing this?

1 Like

Now we have something which can be investigated.

If you see this message or a similar message about any other grub, initramfs or linux related package there is a probably that the system will not boot after reboot.

This needs to be debugged further. It can possibly be done both, before or after you have seen this error.

  1. Open /usr/share/initramfs-tools/hooks/live with root rights.

.

sudoedit /usr/share/initramfs-tools/hooks/live
  1. Enable xtrace / -x (for sh script debugging).

I.e. change the first line

#!/bin/sh

by appending -x. So it looks like this.

#!/bin/sh -x

(Btw this also works not only for #!/bin/sh shebang but also for #!/bin/bash shebang.)

  1. Save.

  2. Try to trigger this error again.

Command to dpkg finishing upgrades.

First one should work but if not try second one. Running both commands should be safe either way. Just in case run both commands.

sudo dpkg --configure -a

Or initramfs regeneration.

sudo update-initramfs -u
  1. Post the output of these commands here.

Thanks @Patrick Patrick, and sorry for the long delay.

I took the opportunity to upgrade my workstation again, and captured the xtrace output for those two commands as you’ve instructed. (Same errors occurred and same dpkg --configure solution needed as usual.)

Text outputs of sudo dpkg --configure -a and sudo update-initramfs -u are 1.5MB each. What’s the best way to share it with you? ZIP file? (Probably too long for Pastebin?) Hope we can solve this one. :slight_smile:

Output of sudo dpkg --configure -a and sudo update-initramfs -u are probably very, very similar besides minor differences.

I doubt all 1.5MB are needed. Any pastbin would be ok. Also any file hoster would be ok. Check this:
Kicksecure Forums Usage Instructions, Best Practices and FAQ

The broken script /usr/share/initramfs-tools/hooks/live is just 256 lines. It’s not that complex.

The mentioned error messages:

Interesting are just the last 1000 or 100 lines above these error messages. I.e. just the last 100 to 1000 lines of log would probably be sufficient. Which then could even be posted here in forums.