kernel recompilation for better hardening

madaidan via Whonix Forum:

We should automate the versioning so we always get the latest LTS kernel and linux-hardened patch. New releases can sometimes come fast and go unnoticed so doing it manually is unreliable.

This can get the latest version as long as it starts with “4” which all LTS kernels will for a while:

curl https://github.com/anthraxx/linux-hardened/releases | grep "linux-hardened-4" | head -n1 | sed -e 's/.*linux-hardened-//g' | sed -e 's/\.a\.patch.*//g'

Better to not use any networking at all during build as per:

still contains various TODO.

1 Like

How else would we automate the versioning if not with networking?

It’s just a single curl command that downloads the html. It’s unlikely to fail and even if it did, we can add in a bunch of error checking.

It’s not like downloading the full kernel source.

1 Like

Merged. :slight_smile:

1 Like

I don’t think it’s advisable to automate fetching newest version. It creates many follow-up issues. Error checking is very doable but I don’t see any way for good error handling.

  • If it’s automated then developers cannot test the kernel before users. Any patch that results in a non-bootable kernel cannot be fixed before users using even the stable repository make their machines unbootable.
  • Either fail open and miss kernel upgrades or fail closed and break apt.
    • If it fails open then exit codes of apt-get dist-upgrade get unreliable. Exit success (exit code 0) wouldn’t guarantee that all upgrades were installed. Breaks automation of updates. Would require some status file indicating if upgrade was a success or failure when then automation scripts would check to check.
    • Failing closed, i.e. the compilation script exiting non-zero and thereby making APT exit non-zero would prevent the user from other package installation until that is fixed. Require to run sudo dpkg --configure -a. And if the download location is permanently down, things get more and more complicated. Failing closed, breaking APT is probably not an option?
  • Networking dependent: if networking is down, slow, etc. the update will fail.
    • (I plan to merge tb-starter, tb-updater, tb-default-browser and open-link-confirmation packages, add Tor Browser archive (and signature) to binaries-freedom package to make the only required networking by Whonix only APT and nothing else. I.e. once packages are fetched, there are no more external network connections required. This simplifies the build environment, tunneling all connections through Tor/onions during build and whatnot.)
  • gpg verification is a major hassle and security risk.
  • Upstreaming gets harder.
    • A package relying on resources unavailable from Debian main disqualifies for Debian main. (That is why torbrowser-launcher is in Debian contrib but not in Debian main.)
    • 2. The Debian Archive — Debian Policy Manual v4.6.2.0
    • Examples of packages which would be included in contrib are: free packages which require contrib , non-free packages or packages which are not in our archive at all for compilation or execution, and

  • Something special to cover for use case “download over onions only”.

What should be done (and easy to implement) is users overwriting variables, i.e. user ability to choose/hardcode/manually select any version numbers / urls as they desire so there is no hard dependency on a package upgrade before testing a newer kernel.

1 Like

The kernel and linux-hardened patches are already tested before being released. Using LTS makes it even more unlikely for there to be issues like that.

We can tell the users to fix their connection and update again if we can’t connect.

We can’t gpg verify the html anyway.

If we do upstream our config, it’d likely be using the linux-source package so we won’t have to worry about versioning.

This config won’t be upstreamed anytime soon either.

Can’t we just display the version in a text file on the Whonix onion service? e.g.

kver="$(curl http://dds6qkxpwdeubwucdiaord2xgbbeyds25rbsgr73tbfpqpt4a6vjwsyd.onion/kver.txt)"

A cron job could regularly update it.

Then an attacker can use a URL to a malicious kernel source or downgrade the kernel to a version with known vulnerabilities.

1 Like

To mitigate /proc/pid/sched spy on keystrokes - proof of concept spy-gksu we can probably unset CONFIG_SCHEDSTATS.

We can leave this to apparmor-profile-everything but then it’d just add compile time.

1 Like

I think apparmor-profile-everything is better suited. Trading off simplicity [1] against higher compile times.


[1] Avoiding complexity. apparmor-profile-everything seems simpler to understand, review, customize, test, etc. than any kernel compilation changes.

1 Like

LTS Linux kernel: good.

But what about linux-hardened? These patches target LTS Linux kernel. Good. But how stable are linux-hardened patches? Are these maintenance only for LTS kernels while newer security features / bigger changes happen for not-yet stable/LTS kernels only? In other words: does linux-hardened introduce new security features for LTS kernels?

These are exactly the thing which I am worried about here. A change by linux-hardened which makes a VM or machine unbootable due to the change.

Tested by whom and in which environments? Host only? Inside VMs? VirtualBox, KVM, Qubes? With hardened-kernel config? On Debian, which suite, buster? Inside Whonix? With apparmor-profile-everything? All of these components (virtualizer, apparmor, Debian, Whonix) add complexity and can result in non-bootable systems.

What I want to prevent here is breaking Whonix Stable Version User Experience.

I.e. a standard apt (kernel) upgrade resulting in breaking boot, connectivity, graphical desktop, apt-get or require manual fixes being applied.

I don’t think this is possible with APT.
APT can show any error messages, yes. But how can APT react? It can only either:

  • leave the system in a non-broken state (i.e. can re-run APT / upgrade / install any package), which is easily overlooked and cannot be easily automated (introducing a special case) or,
  • in a broken state (demanding the previous upgrade to finish, strange/hard to fix [users don’t know] error messages).

Once a package upgrade is done, there is no way to say “oh well, its actually not done, please upgrade again”.

I did not talk about gpg verification of the html with the version number yet. tb-updater has the same issue. Upstream won’t provide gpg signatures for RecommendedTBBVersions. Therefore there is an interactive Download Confirmation Notification. APT interactive is bad. To automate the build process and keep Qubes (specifically Disp)VMs updated these version numbers are hardcoded:

(With Tor Browser it is less severe since its internal updater is considered secure and reliable.)

With gpg verification is a major hassle and security risk. I was referring this this:

curl_opts="--tlsv1.2 --proto =https --location --remote-name-all --remote-header-name"

## TODO: do not use networking as per https://forums.whonix.org/t/kernel-recompilation-for-better-hardening/7598/214

curl $curl_opts "${kernel_source}" -o "${working_folder}/linux-${version}.tar.xz"

curl $curl_opts "${linux_hardened_patch}" -o "${working_folder}/linux-hardened-${version}.a.patch"

The System Security Level development goal for the Whonix software for source code and binary downloads by default is always software signatures verification security level.

If I can maintain an up to date version file on the server, why couldn’t I also maintain a deb package in Whonix APT repository? Then no networking outside of APT needs to be invented.

The problem with the tb-updater package currently is it mixes:

  • simple things such as hardcoded version numbers with
  • code changes that need the usual development process (repository: developers → testers → stable-proposed-updates → stable)

Therefore hardcoded version numbers cannot be easily updated in all repositories as soon as possible. A mistake to not repeat here.

There might also be the misconception about the security that can be expected from the whonix.org server. To clear that up, this chapter was expanded just now:
Placing Trust in Whonix

I should have said superroot.
This is always the case when there is write access. Either users are required to edit the script directly to use custom/testing versions which is harder and doesn’t stop an attacker or things are configurable through a configuration file or command line parameters. The latter is easier for users but doesn’t make an attack easier for an attacker. Since an attacker who has the skills for the attack, wouldn’t have an issue modifying the script directly either vs a config.

2 Likes

New security features aren’t actively being developed much anyway. I’d imagine they’d be ported to LTS kernels though.

The standard kernel has tons of testers due to being one of the most popular open source projects. It’d be tested in a whole load of environments.

The linux-hardened patch are likely tested by anthraxx, the maintainer. Those patches are also used in Arch Linux’s linux-hardened kernel and ClipOS.

With our config and apparmor-profile-everything though, only we will be testing it although the chance a newer kernel will result in these making the system unbootable is low.

Can’t we just do something like:

if ! curl ...; then
  echo "Couldn't connect. Fix your connection and retry."
  exit
fi

?

You’d still need to manually update the version numbers. Whereas with a cron job automatically updating the text file, you won’t need to do anything manually.

2 Likes

It’s possible but then APT would exit 0 and not indicate that the upgrade did not actually finish. (Except for the error message text.) Re-running apt-get dist-upgrade wouldn’t do anything either since the new package is already installed. From perspective of APT, there is nothing more to do. Users would have to read that error message during APT and then act accordingly by running commands manually to finish the upgrade.

Why would there be a new package for APT to upgrade to anyhow? Because a new package was uploaded to Whonix repository. But if a new package is uploaded, why doesn’t it already come with all that is needed, i.e. compilation script, kernel sources and linux-hardend patch?

Need to manually update version: yes
But how does a new package get uploaded to Whonix repository so it will result in a package upgrade? That’s a manual process too.
And if the kernel patch version number doesn’t get manually bumped in package source code, why would the package version number be bumped, the package re-build and uploaded to Whonix repository?

The question is how we inform the system of the user that a kernel upgrade is available? Usually the user notices when running APT upgrades since there is a new package. Ideally we can just use APT here too. Everything else expands into a lot complexity similar to inventing a new package manager. If that is required, better to look if any existing package manager would be suitable than inventing a new one. guix was discussed in past.

The idea is kinda “something should be updated faster than package upgrades are coming from Whonix repository”? Well, that would require a whole mechanism to check for updates, download updates, compile and install. Foreground or background…

For now I think it’s best to just add the kernel archive [1] and linux-hardened patch [1] to hardened-kernel package.

For those who want to use different / newer versions than whatever the hardened-kernel shipped, there can be superroot command line options, config files and/or replacing archive files on disk.


Automated download and gpg verification is an absolute mess:

  • endless data attacks (fill up user disk, crash system, could just be misconfigured web server)
  • rollback attacks (need to cache version numbers, remember, never downgrade)
  • infinite freeze attacks (always reporting outdated version to (targeted) users)
  • security risks of parsing strings
  • gpg signs and verifies files. Not file names. While a signature might be valid for the actual file, so would any signature be valid by the same white listed signing key owner that signed any other file)
  • key management (upstream keys expired, cycled, revoked)
  • signature freshness needs to be checked

Metadata of package managers needs to be signed, verified and fresh (valid-until). These are old, known attacks but not much inside security researchers consciousness. References:

It is all solvable in code but some thing which looks simple at first expands into a lot complexity.


[1] Plus gpg signatures. These would not be automatically checked by the compilation script [2] but are useful for auditors to manually check the hardened-kernel package source code. Signatures would be verified manually whenever developers add new archives to the package source code.
[2] At the time the compilation script which is installed and executed is trusted anyhow.

1 Like

https://github.com/clipos/bugs/issues/38

2 Likes

Good points. The problem now is we’d need to keep a very close eye on new releases to update the kernel quickly. There is already a new release from within 3 days of the last one https://github.com/anthraxx/linux-hardened/releases/download/4.19.94.a/linux-hardened-4.19.94.a.patch

Might be too fast to keep up.

1 Like

Are these fixing CVEs (similar to Debian kernel upgrades) (very time critical) or “only” more security hardening" (less time critical)?

2 Likes

The linux-hardened patches follow kernel releases. There was a 4.19.94 kernel release so linux-hardened also made a new release.

There’s been no new hardening since July https://github.com/anthraxx/linux-hardened/commits/4.19-lts so it is likely security fixes from upstream linux.

There’s been a 4.19.95 kernel release very recently so linux-hardened will likely make a new release very soon https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.19.95.tar.xz

2 Likes

https://github.com/Whonix/hardened-kernel/pull/22

https://github.com/Whonix/hardened-kernel/pull/23

1 Like

https://github.com/Whonix/hardened-kernel/pull/24

1 Like

Tested? I know from kernel module load hardening experiments that inability to auto load binfmt module broke XFCE startup.

Why needed if we already have that in security-misc package?

1 Like

I haven’t tested it yet but will later. I don’t see why XFCE would require custom binary formats though.

security-misc enables it after X starts which isn’t as good as having it enabled all the time.

https://github.com/Whonix/hardened-kernel/pull/25

https://github.com/Whonix/hardened-kernel/pull/26