Some recent change breaks starting Whonix Workstation on Qubes - privleap suspected

marmarek · February 7, 2025, 6:34pm

Recent test run failed in several ways - most interactions with whonix-workstation-17 - based qubes fails.
Package diff compared to last good:

-whonix-workstation-17: ii  libqrexec-utils4                               4.3.2-1+deb12u1+devel1          amd64        Library of common functions of qrexec agent and daemon
+whonix-workstation-17: ii  libqrexec-utils4                               4.3.2-1+deb12u1+devel2          amd64        Library of common functions of qrexec agent and daemon
+whonix-workstation-17: ii  privleap                                       3:0.9-1                         all          Limited privilege escalation framework
-whonix-workstation-17: ii  python3-qrexec                                 4.3.2-1+deb12u1+devel1          amd64        Qrexec policy related python module
+whonix-workstation-17: ii  python3-qrexec                                 4.3.2-1+deb12u1+devel2          amd64        Qrexec policy related python module
-whonix-workstation-17: ii  qubes-core-qrexec                              4.3.2-1+deb12u1+devel1          amd64        Qubes qrexec agent
+whonix-workstation-17: ii  qubes-core-qrexec                              4.3.2-1+deb12u1+devel2          amd64        Qubes qrexec agent
-whonix-workstation-17: ii  tb-updater                                     3:36.0-1                        all          Tor Browser Downloader by Whonix developers
+whonix-workstation-17: ii  tb-updater                                     3:36.2-1

I noticed new privleap package.

I got some logs from there too:

[2025-02-06 19:51:45] [    5.323797] privleapd[972]: parse_config_files: CRITICAL: Failed to load config file '/etc/privleap/conf.d/tb-updater.conf'!
[2025-02-06 19:51:45] [    5.323918] privleapd[972]: Traceback (most recent call last):
[2025-02-06 19:51:45] [    5.323960] privleapd[972]:   File "/usr/lib/python3/dist-packages/privleap/privleapd.py", line 569, in parse_config_files
[2025-02-06 19:51:45] [    5.324004] privleapd[972]:     = pl.PrivleapCommon.parse_config_file(f.read())
[2025-02-06 19:51:45] [    5.324058] privleapd[972]:       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2025-02-06 19:51:45] [    5.324094] privleapd[972]:   File "/usr/lib/python3/dist-packages/privleap/privleap.py", line 948, in parse_config_file
[2025-02-06 19:51:45] [    5.324146] privleapd[972]:     raise ValueError(f"Unrecognized key '{config_key}' found")
[2025-02-06 19:51:45] [    5.324181] privleapd[972]: ValueError: Unrecognized key 'AuthorizedGroups' found
[2025-02-06 19:51:45] [    5.337165] systemd[1]: privleapd.service: Main process exited, code=exited, status=1/FAILURE
[2025-02-06 19:51:45] [    5.337279] systemd[1]: privleapd.service: Failed with result 'exit-code'.

and likely related several leapctl[1010]: ERROR: Could not connect to privleapd!

I’m not sure if directly related, but the actual problem is several things failing to start, most importantly any user session:

[2025-02-06 19:51:46] [e[0;1;31mFAILEDe[0m] Failed to start e[0;1;39muser@1000.…ee[0m - User Manager for UID 1000.

[2025-02-06 19:51:46] See 'systemctl status user@1000.service' for details.

[2025-02-06 19:51:46] [    5.536074] leapctl[1011]: ERROR: Could not connect to privleapd!
[2025-02-06 19:51:46] [    5.536235] systemd[1]: user@1000.service: Control process exited, code=exited, status=1/FAILURE
[2025-02-06 19:51:46] [    5.536306] systemd[1]: user@1000.service: Failed with result 'exit-code'.
[2025-02-06 19:51:46] [    5.536380] systemd[1]: Failed to start user@1000.service - User Manager for UID 1000.
[2025-02-06 19:51:46] [e[0;32m  OK  e[0m] Started e[0;1;39msession-c3.scopee[0m - Session c3 of User user.

[2025-02-06 19:51:46] [    5.544009] systemd[1]: Started session-c3.scope - Session c3 of User user.

And also Qubes GUI agent fails to start, likely for similar reason:

[2025-02-06 19:51:49] [    8.896939] qubes-gui-runuser[1000]: pam_unix(qubes-gui-agent:session): session closed for user user
[2025-02-06 19:51:49] [    8.897056] qubes-gui-runuser[1000]: pam_succeed_if(qubes-gui-agent:session): requirement "uid eq 0" not met by user "user"
[2025-02-06 19:51:49] [    8.945398] systemd[1]: qubes-gui-agent.service: Main process exited, code=exited, status=1/FAILURE
[2025-02-06 19:51:49] [    8.945496] qubes-gui[980]: Xorg exited in the meantime, aborting
[2025-02-06 19:51:49] [    8.945547] systemd[1]: qubes-gui-agent.service: Failed with result 'exit-code'.

marmarek · February 7, 2025, 6:37pm

I see some commits in privleap repo from few hours ago that looks to be related and hopefully fix stuff. I’ll re-run the test.

marmarek · February 7, 2025, 9:32pm

I still got:

+whonix-workstation-17: ii  privleap                                       3:0.9-1                         all          Limited privilege escalation framework

How can I get the fixed version?

arraybolt3 · February 7, 2025, 10:23pm

Follow-up on a conversation on Matrix, the fixed version is in the bookworm-developers repository still.

Root cause analysis that I posted on Matrix:

The reason it’s crashing is because I failed to realize there might be valid situations when there’s a user configured as permitted to run an action that doesn’t exist on the system.

So, in the name of not accepting invalid data, I made the server refuse to start if it hit a username that didn’t exist on the system during config file parsing.

That… turned out to be a bad idea, so now it just skips over those users, and also doesn’t lock up login if privleap fails to start.

Patrick · February 8, 2025, 11:42am

Fixed version has been uploaded to bookworm-developers just now.

marmarek · February 8, 2025, 11:53am

Did you mean bookworm-testers? The 0.9 version in bookworm-testers makes any template with that repo enabled quite broken…

Patrick · February 9, 2025, 11:16am

Was bookworm-developers only.

Now also in bookworm-testers.

Patrick · February 9, 2025, 9:03pm

A post was split to a new topic: The following packages have been kept back: privleap

marmarek · February 10, 2025, 1:24pm

systemcheck complains:

ERROR: You are unauthorized to run action 'tor-verify-config'.

I believe it’s related.
Full log:https://openqa.qubes-os.org/tests/128391/file/suspend-whonixcheck-sys-whonix.log

marmarek · February 13, 2025, 10:29pm

systemcheck still fails in sys-whonix (with testers repo) - log from today: https://openqa.qubes-os.org/tests/128689/file/suspend-whonixcheck-sys-whonix.log

I found some more info there (in addition to the above still present):

AVC apparmor="DENIED" operation="open" class="file" profile="/usr/bin/systemcheck" name="/run/privleapd/pid" comm="cat" requested_mask="r" denied_mask="r"

Feb 13 19:13:36 host sdwdate[1307]: 2025-02-13 19:13:36 - sdwdate - INFO - /usr/libexec/helper-scripts/onion-time-pre-script: WARNING: privleapd is not running. Cannot use privleap.

(but is it really not running? I don’t see privleapd startup errors anymore)

BTW, maybe privleap feature wants explicit check in systemcheck?

Patrick · February 14, 2025, 1:24am

The AppArmor issue is hopefully fixed.

Merged into testers repository just now.

systemcheck has been ported to privleap by @arraybolt3 already.

There is a separate test in systemcheck /usr/libexec/systemcheck/check_sudo.bsh which despite its non-ideal name (might update) now check privleap.

(It currently only tests privleap by I’ll modify it now to test both privleap and sudo. The latter only if applicable.)

marmarek · February 16, 2025, 11:02am

Yes, AppArmor issue seems to be fixed. But systemcheck still fails.
There is still:

ERROR: You are unauthorized to run action 'tor-verify-config'.

and I see also:

Feb 16 05:16:42 host PAM_tmpdir[1874]: /tmp/user/1000 owned by uid 0 instead of uid 1000. Failed to create safe $TMPDIR

with some more context:

[2025-02-16 00:16:42] [   29.463894] qubes-gui[1229]: Waiting on /var/run/xf86-qubes-socket socket...
[2025-02-16 00:16:42] [   29.558826] loginctl[1872]: Could not attach device: Failed to open device '/sys/devices/platform/pcspkr/input/*': No such device
[2025-02-16 00:16:42] [   29.654982] qrexec-agent[1864]: pam_unix(qrexec:session): session opened for user user(uid=1000) by (uid=0)
[2025-02-16 00:16:42] [   29.655091] qrexec-agent[1864]: pam_succeed_if(qrexec:session): requirement "uid eq 0" not met by user "user"
[2025-02-16 00:16:42] [   29.655250] qubes-gui-runuser[1865]: pam_unix(qubes-gui-agent:session): session opened for user user(uid=1000) by (uid=0)
[2025-02-16 00:16:42] [   29.655299] qubes-gui-runuser[1865]: pam_succeed_if(qubes-gui-agent:session): requirement "uid eq 0" not met by user "user"
[2025-02-16 00:16:42] [   29.658632] PAM_tmpdir[1874]: /tmp/user/1000 owned by uid 0 instead of uid 1000. Failed to create safe $TMPDIR
[2025-02-16 00:16:42] [   29.736356] systemd[1]: Created slice system-leapctl.slice - Slice /system/leapctl.
[2025-02-16 00:16:42] [   29.737651] systemd[1]: Created slice user-1000.slice - User Slice of UID 1000.
[2025-02-16 00:16:42] [   29.738825] systemd[1]: Starting leapctl@1000.service - leapctl - Enable access to privleap for each user...
[2025-02-16 00:16:42] [   29.751323] systemd[1]: Starting user-runtime-dir@1000.service - User Runtime Directory /run/user/1000...
[2025-02-16 00:16:42] [   29.786415] systemd-logind[1032]: New session c2 of user user.

Log from recent run: https://openqa.qubes-os.org/tests/128885/file/suspend-whonixcheck-sys-whonix.log

arraybolt3 · February 16, 2025, 7:11pm

This is probably the result of outdated packages in Kicksecure’s archive. I just did a test using locally built packages, using the tip of master from each of my forks of the following packages:

helper-scripts
security-misc
usability-misc
dist-base-files
privleap
setup-dist
setup-wizard-dist
systemcheck
sdwdate
sdwdate-gui

Each of these was built with genmkfile deb-pkg on a Kicksecure KVM virtual machine, then the built debs were copied into the whonix-workstation-17-exp and whonix-gateway-17-exp templates (which I had freshly cloned from whonix-workstation-17 and whonix-gateway-17). I installed all the debs onto both templates, then shut them down and restarted sys-whonix. I ensured both sys-whonix and anon-whonix were using the -exp templates, then ran systemcheck in both of them. In both cases, no errors were reported and systemcheck exited 0. I also tried running systemcheck in the -exp templates themselves for good measure, and it behaved properly there too.

arraybolt3 · February 16, 2025, 7:21pm

I can reproduce this issue with the packages in bookworm-testers. The issue is twofold there - there’s two tor-verify-config actions defined on accident (this is supposed to be an error, but earlier versions of privleap failed to detect it, the latest commits I pushed yesterday resolve this), and the one that systemcheck is mistakenly trying to use doesn’t permit user user to run it. I’ll see if this is a problem in Git and fix it if so.

I believe the reason using the tips of the master branch didn’t work to reproduce this is because I wasn’t building and installing anon-gw-anonymizer-config (the package with the conflicting privleap rule).

Patrick · February 21, 2025, 11:28am

Should be fixed.