kernel recompilation for better hardening

[Patrick]

@HulaHoop

I think the kernel config package should be replicated once for each different hypervisor to keep things clean and simple.

There’s no reason to have modules relevant to VBox and Xen enabled for KVM users and vice versa. If we’re about decreasing attack surface let’s go all the way.

Sounds good in theory but I vary the maintenance overhead.

@madaidan

But System wide MAC policy or verified boot would also prevent modprobe anyway?

The MAC policy can prevent module loading but currently it doesn’t.

Then let’s use MAC policy to prevent module loading so we don’t have to disable module support in kernel? Seems like a more popular, tested path rather than disabling module loading. Therefore should hopefully be more stable.

@madaidan

Run make menuconfig , load the .config file and save it. This is needed as my config doesn’t include a whole bunch of # CONFIG_X is not set lines for simplicity but those are needed for compiling.

Any way to make that non-interactive? Wouldn’t know what options to choose. Just nothing?, go with defaults? Non-interactive also needed for later automation.

[Patrick]

error: Cannot generate ORC metadata for CONFIG_UNWINDER_ORC=y, please install libelf-dev, libelf-devel or elfutils-libelf-devel

Installing libelf-dev helped. Edited your post. Compiling.

[Patrick]

Compilation with your config was successful but took ~ 1 hour in a Qubes VM. Did not test yet if the kernel boots. Due to long compile time, sounds more like a package to be added to Whonix repository?

[madaidan]

@HulaHoop

Module or built-in is besides the point. What I mean is: no vulnerable code is loaded by the GW and there is no way it can be so making a custom slimmed down kernel is not necessary to shrink attack surface.

I’m not sure what you mean. How would slimming the kernel config make the GW load vulnerable code?

@Patrick

Then let’s use MAC policy to prevent module loading so we don’t have to disable module support in kernel? Seems like a more popular, tested path rather than disabling module loading. Therefore should hopefully be more stable.

It would be a better approach although we’d need to create a systemd-modules-load profile so necessary modules can still be loaded at boot.

@Patrick

Any way to make that non-interactive? Wouldn’t know what options to choose. Just nothing?, go with defaults? Non-interactive also needed for later automation.

make menuconfig generates a new .config file with all the # X is not needed lines that we can use for the next time. I just use the config without all those lines for working on it.

You don’t need to chose any options. They’re all already chosen.

@Patrick

Compilation with your config was successful but took ~ 1 hour in a Qubes VM. Did not test yet if the kernel boots. Due to long compile time, sounds more like a package to be added to Whonix repository?

It took around an hour for me too. The config can still be slimmed down further and hopefully we can get it to compile faster.

[Patrick]

Quote BuildADebianKernelPackage - Debian Wiki

Using your current Debian kernel configuration as a starting point

Alternatively, you can use the configuration from a Debian-built kernel that you already have installed by copying the /boot/config-* file to .config and then running make oldconfig to only answer new questions.

When you’re ready… Could you please create a package hardened-vm-config (or so)… Then first add/commit the Debian default .config. Then another commit to make the changes? This is to illustrate in how far our config is different from Debian config. If that makes sense? Or compare with some other starting point?

[HulaHoop]

@madaidan

I’m not sure what you mean. How would slimming the kernel config make the GW load vulnerable code?

Let’s say an iptables module that is activated when some ruleset is triggered is not loaded by default and only happens when a some network pattern is detected. In a normal kernel it will be there, however with a slimmed down kernel where we haven’t tested every possible behavior/codepath it won’t and there is the hypothetical possibility of failing open.

Also we want to transition to nftables in the medium future and that requires a different set of modules.

@madaidan

It took around an hour for me too. The config can still be slimmed down further and hopefully we can get it to compile faster.

You want to make it use all available cores if not already.

[madaidan]

@HulaHoop

Let’s say an iptables module that is activated when some ruleset is triggered is not loaded by default and only happens when a some network pattern is detected. In a normal kernel it will be there, however with a slimmed down kernel where we haven’t tested every possible behavior/codepath it won’t and there is the hypothetical possibility of failing open.

Also we want to transition to nftables in the medium future and that requires a different set of modules.

I haven’t touched any netfilter modules except for disabling IPv6.

All iptables/nftables code will still be there.

[Patrick]

@HulaHoop

with a slimmed down kernel where we haven’t tested every possible behavior/codepath

That suggests better just harden workstation kernel and leave gateway kernel as per Debian default.

Hardened workstation kernel might become a default but I vary the gateway kernel unless we had a lot more sophisticate leak testing.

Otherwise some strange leak (related) might happen. We’re getting into mostly uncharted territory here. Not many (Debian) people using hardened kernel configurations. Otherwise if these things were well tested, it wouldn’t need to be invented here.

[madaidan]

I highly doubt a hardened kernel would cause any leaks. If anything, it would likely reduce the chances of them.

The config doesn’t touch any networking stuff except some unused protocols like DCCP which are disabled and IPv6.

[Patrick]

This is over my head. Input from any expert / actual kernel developer required.

[madaidan]

@Patrick

audio: workstation yes, gateway no.

We can have a separate config for the workstation and gateway with audio disabled in the gateway config and enabled in the workstation config. Other things can be disabled/enabled for different configs in the future if needed.

[madaidan]

Following hulahoop’s advice, I made the compilation use all cores assigned to the VM, by running make deb-pkg -j 4 which seemed to speed up the process.

For the package, we can use make deb-pkg -j $(nproc) so we use all of the user’s cores.

[madaidan]

@Patrick

When you’re ready… Could you please create a package hardened-vm-config (or so)… Then first add/commit the Debian default .config . Then another commit to make the changes? This is to illustrate in how far our config is different from Debian config. If that makes sense? Or compare with some other starting point?

https://github.com/madaidan/hardened-vm-kernel/commit/407dc6100bfc1708e89c4429daff59d35d0660ba

Debian packaging is beyond me.

[Patrick_mobile]

Thinking about packaging.
Build kernel and upload to Whonix repository?
No compilation on user’s system for all users, right?

Maybe have to fork this:

https://packages.debian.org/de/source/buster/linux-latest

Debian kernel team / linux-latest · GitLab

git clone. Checkout correct branch or tag. Fork new branch. Add your custom config. Commit. Build.

Also change kernel package name. Update debian/changelog with higher (debian packaging) version number.

But would prefer a more lightweight solution.

Also needs to be fully non interactive building. No graphical make menu config. But Debian must have sorted that already.

Also don’t use kernel signing / module signing during build? Another layer of complexity. Only the usual repository signing as for any package.

[madaidan]

@Patrick_mobile

Build kernel and upload to Whonix repository?
No compilation on user’s system for all users, right?

I would prefer to compile it on the user’s system for better security but currently the config isn’t slim enough to do that. I’ve heard of people reducing kernel compilation time to about 10 minutes which would be wonderful.

@Patrick_mobile

Maybe have to fork this:

Debian -- Informationen über Quellcode-Paket linux-latest in buster

No need. We can just create our own scripts that move a few files about, extracts the source, runs make deb-pkg and installs the packages. Much easier.

@Patrick_mobile

Also needs to be fully non interactive building. No graphical make menu config. But Debian must have sorted that already.

That was only needed to generate the full config out of the one I gave you (which removed unneeded lines for simplicity). https://github.com/madaidan/hardened-vm-kernel/blob/master/usr/share/hardened-vm-kernel/kernel-config is the full config and doesn’t require make menuconfig.

@Patrick_mobile

Also don’t use kernel signing / module signing during build? Another layer of complexity. Only the usual repository signing as for any package.

I didn’t think of that. We cannot store any private keys for signing on the user’s machine. Too risky.

If the kernel package is pre-compiled for users, the private key can be stored securely by whoever compiles it so users don’t need to worry about it too much.

Best option would be to get rid of the need for signing entirely which would require removing modules support.

[Patrick]

@madaidan

Best option would be to get rid of the need for signing entirely which would require removing modules support.

Signed kernel is also useful for verified boot (discussion), namely secure boot. But perhaps verified boot won’t rely on kernel signatures but rather checksums.

On the other hand, with apparmor-profile-everything manual modprobe by root user could be forbidden anyhow.

@madaidan

Best option would be to get rid of the need for signing entirely which would require removing modules support.

Which goes back to the discussion of disabling module load vs non-module load support.

@madaidan

Best option would be to get rid of the need for signing entirely which would require removing modules support.

At the moment not a blocker since:

• We’re not using signed kernel modules anyhow. Due to, see this very post: enforce kernel module software signature verification [module signing] / disallow kernel module loading by default - #23 by Patrick
• We’d have to integrate VirtualBox guest additions kernel module.

Both non-trivial. Not sure if realistic at all?

@madaidan

I would prefer to compile it on the user’s system …

Alright, let’s assume compile on the user’s system. I like that. However, I don’t have a solution yet.

I started working on a script. Initially just to ease my workflow but such things often become the basis for deployed scripts.

#!/bin/bash

## Copyright (C) 2012 - 2018 ENCRYPTED SUPPORT LP <adrelanos@riseup.net>
## See the file COPYING for copying conditions.

set -x

set -e

## TODO: This probably has to be converted to debian/control 'Build-Depends:'.
sudo apt-get --no-install-recommends --yes install linux-source build-essential libssl-dev libncurses-dev fakeroot libelf-dev

## TODO: Probably better using mktemp?
folder=~/kernel

mkdir -p "$folder"

pushd "$folder"

tar -xaf /usr/src/linux-source-4.19.tar.xz

popd

pushd "$folder"/linux-source-4.19/

xzcat /usr/src/linux-patch-4.19-rt.patch.xz | patch -p1

popd

## TODO: path to .config
cp ./.config "$folder"/linux-source-4.19/

#pushd "$folder"/linux-source-4.19/

## No more need....
#make menuconfig

#make deb-pkg

It would not be difficult for me to finish that script. Package it. Even run make deb-pkg the kernel during the postinst phase of installation script.

However, there is one blocker. The script would be run by apt/dpkg. The package creation would be happening during the installation of a pacakge while apt/dpkg is running. The result of that script would be Debian packages. The problem is, it is not possible to (sanely) possilble install a deb package while dpkg is already running. And it is already running.

A few years I invented GitHub - adrelanos/apt-during-apt: https://www.whonix.org/Impressum but it’s a giant hack. I don’t like it at all.

Somehow we’d need to install the packages after apt finished installing/upgrading hardened-vm-kernel.

How do we do that? Replace /usr/bin/apt and /usr/bin/apt-get with a wrapper? Not easy/clean since APT is already wrapped by uwt. And there are no stackable wrappers yet:
https://phabricator.whonix.org/T634

Utilize /usr/local/bin/? Ship /usr/local/bin/apt and /usr/local/bin/apt-get? Not clean. Also firejail is already using /usr/local/bin/ so that could set up for later conflicts.

Don’t wrap APT and tell users to use wapt /usr/bin/wapt (wrapped apt) instead? WAPT apt could be a wrapper to run APT normally but before and after APT any number of hooks could be run. One such hook could be “if hardened-vm-kernel created new packages, install these at the end”. Cleaner but also not great as we’d have to reach users to use WAPT rather than APT and constantly explain this.

@madaidan

That was only needed to generate the full config out of the one I gave you (which removed unneeded lines for simplicity). https://github.com/madaidan/hardened-vm-kernel/blob/master/usr/share/hardened-vm-kernel/kernel-config is the full config and doesn’t require make menuconfig .

Are you sure? make menuconfig creates other files too. To see for yourself: put the linux source folder under git version control before running make menuconfig. Delete .gitignore. Run make menuconfig. It creates other files inside the source folder too.

[Patrick]

git status
On branch master
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

  deleted:    include/config/tristate.conf

Untracked files:
  (use "git add <file>..." to include in what will be committed)

  scripts/kconfig/.conf.o.cmd
  scripts/kconfig/.mconf.o.cmd
  scripts/kconfig/conf.o
  scripts/kconfig/mconf.o

no changes added to commit (use "git add" and/or "git commit -a")

Last time it was more files.

Maybe not important. Maybe done automatically during make deb-pkg.

[Patrick]

Could you please check the diff between your original kernel config and the one based on the Debian kernel config?

[Patrick]

make menuconfig
  HOSTCC  scripts/basic/fixdep
  UPD     scripts/kconfig/.mconf-cfg
  HOSTCC  scripts/kconfig/mconf.o
  YACC    scripts/kconfig/zconf.tab.c
  LEX     scripts/kconfig/zconf.lex.c
  HOSTCC  scripts/kconfig/zconf.tab.o
  HOSTCC  scripts/kconfig/lxdialog/checklist.o
  HOSTCC  scripts/kconfig/lxdialog/inputbox.o
  HOSTCC  scripts/kconfig/lxdialog/menubox.o
  HOSTCC  scripts/kconfig/lxdialog/textbox.o
  HOSTCC  scripts/kconfig/lxdialog/util.o
  HOSTCC  scripts/kconfig/lxdialog/yesno.o
  HOSTLD  scripts/kconfig/mconf
scripts/kconfig/mconf  Kconfig


*** End of the configuration.
*** Execute 'make' to start the build or try 'make help'.

Any of these files required?