kernel recompilation for better hardening

I’m not sure about this one. Usually we should have as much things as unprivileged as possible but I doubt this would matter since were installing this kernel anyway. If an attacker compromises the kernel, it doesn’t matter if it was compiled as root once it’s running.

The only thing I can find on this is https://www.linuxquestions.org/questions/debian-26/compiling-the-vanilla-kernel-without-root-access-4175551137/ in which the general consensus seems to be to not do it and it’s apparently recommended against by kernel devs although there’s no link to them.

That’s not what they do. Only root can access /dev/mem regardless if these options are used. These options filters even root’s access to /dev/mem.

linux/lib/Kconfig.debug at master · torvalds/linux · GitHub

STRICT_DEVMEM:

If this option is disabled, you allow userspace (root) access to all of memory, including kernel and userspace memory. Accidental access to this is obviously disastrous, but specific access can be used by people debugging the kernel. Note that with PAT support enabled, even in this case there are restrictions on /dev/mem use due to the cache aliasing requirements.

If this option is switched on, and IO_STRICT_DEVMEM=n, the /dev/mem file only allows userspace access to PCI space and the BIOS code and data regions. This is sufficient for dosemu and X and all common users of /dev/mem.

IO_STRICT_DEVMEM:

If this option is disabled, you allow userspace (root) access to all io-memory regardless of whether a driver is actively using that range. Accidental access to this is obviously disastrous, but specific access can be used by people debugging kernel drivers.

If this option is switched on, the /dev/mem file only allows userspace access to idle io-memory ranges (see /proc/iomem) This may break traditional users of /dev/mem (dosemu, legacy X, etc…) if the driver using a given range cannot be disabled.

Default Debian has had this option enabled for years with no issues. I really don’t think we should be disabling important security features, especially when barely anything uses /dev/mem anymore.

2 Likes

madaidan via Whonix Forum:

Enable /dev/mem and disable panic_on_oops in debugging config by madaidan · Pull Request #30 · Kicksecure/hardened-kernel · GitHub

Merged.

2 Likes
2 Likes

You guys are on fire. Awesome stuff from ya @madaidan

1 Like

I’ve just tested a build with binfmt_misc disabled and it works fine.

Thanks!

1 Like

madaidan via Whonix Forum:

Disable bpf() syscall by madaidan · Pull Request #31 · Kicksecure/hardened-kernel · GitHub

Merged.

1 Like

What if we disable debugfs in the kernel but enable it in our new debugging kernel config?

Not every user of this config will be using apparmor-profile-everything so I think we should do as much reasonable kernel hardening as possible regardless of apparmor. Any lost debugging functionality can easily be re-enabled with the --debug flag if the user needs it.

1 Like
2 Likes

@Patrick Would it be possible to make the CI actually test booting the kernel via e.g. kexec?

kexec -l "/boot/vmlinuz-${version}" --append="root=$(grep -w '/' /proc/mounts | awk '{print $1}')" --initrd="/boot/initrd.img-${version}"
kexec -e

We can also test our custom boot parameters via the --append flag.

1 Like

I have a vision now how to implement this.

Installation through dpkg isn’t great anyhow? That doesn’t handle version upgrades. For example if someone installed a 5.x kernel from backports, this package would still force dpkg install a hardened 4.x kernel for no reason. By using APT, we’d never attempt a downgrade / install a lower versioned kernel.

dpkg also doesn’t well handle dependencies. If there ever was a conflicting dependency, dpkg would wreck APT (broken dependencies) which then requires manual commands to resolve. APT often warns against such situations, with the option to abort etc. Also dpkg does not properly call triggers such as DKMS?

How this could be implemented…

Terminology:

  • real APT: (non-Whonix, non-Kicksecure, plain Debian APT. Just the normal APT that any Debian user is using.)
  • wrapped APT (wapt): a wrapper that stacks various plugins such as rapt or hdapt
  • restricted APT (rapt): as per apparmor-profile-everything
  • hdapt: runs usual real apt dist-upgarde, apt update, apt-dist upgrade again

Most of these scripts can be implemented standalone without much thought about the other scripts.

When APT runs, some script could call the compilation script. And then create a local APT repository. That local APT repository would then be used with APT for proper processing of dependencies and triggers.

/etc/apt/sources.list.d/hardened-kernel-local.list:

deb [trusted=yes] file:/path/to/local/repository local main contrib non-free

similar to https://github.com/Whonix/Whonix/blob/master/help-steps/create-local-temp-apt-repo

User runs apt (or apt-get). What actually happens (without need to tell the user) (fully auditable in source code or by reading the files on the disk), that rapt (restricted APT) is executed - only if that is installed. It then passes on to hdapt (hardened kernel APT). Doing a usual real apt dist-upgrade. hdapt would then run “apt update” again to load the local APT repository which contains the hardened kernel and run real apt dist-upgrade again.

with apparmor-profile-everything:

wapt → rapt → hdapt → real apt update → real apt dist-upgrade → real apt update → real apt dist-upgrade

without apparmor-profile-everything:

wapt → hdapt → real apt update → real apt dist-upgrade → real apt update → real apt dist-upgrade


From user perspective:

  • user runs sudo apt update: just only that happens as per usual
  • user runs sudo apt dist-upgrade: usual sudo apt dist-upgrade happens + sudo apt update (fetching local APT repository with hardened kernel only) → another sudo apt --yes dist-upgrade is performed

(Fetching local (from hard drive) APT repository will take just 1 second or so.)

Same in other words:

  • user runs sudo apt update: just only that happens as per usual
  • user runs sudo apt dist-upgrade: usual sudo apt dist-upgrade happens → maybe hardened-kernel package is upgraded, compiles a new kernel, updates the local APT repository → + maybe sudo apt update (fetching local APT repository with hardened kernel only) → maybe another sudo apt --yes dist-upgrade is performed

maybe meaning: only if necessary (hardened-kernel package was upgraded).


Another layer of complexity that I haven’t mentioned yet: uwt stream isolation wrapper. Also something wrapped apt has to handle.


Just an idea yet. Dunno how it will look when implemented. Seems kinda complex. And messing that up would break APT. Users could still always run apt-get.anondist-orig (real apt).

And all this trouble for what? So users can keep typing what they’re accustomed to: apt or apt-get

Maybe rather than taking on that complexity with wrapped apt, we should rather have a one time popup teaching users to use hdapt from now?

hdapt and hdapt-get? Or only 1? I guess not much of extra work for both.


Maybe better to finish developing the concept of stackable wrappers first before implementing this?


What do we do with Whonix default redistributed (downloadable) builds in future? Pre-compile a hardened kernel and then re-compile it at first boot? Or just install Debian default kernel and compile a hardened kernel at first boot before going online for the first time?

An interesting option would also be to compile both, a non-debug and a debug kernel during the build process. Install both. Boot non-debug kernel by default. Debug kernel could be booted by choosing it in grub boot menu. Would waste some disk space and increase VM image size though.

madaidan via Whonix Forum:

What if we disable debugfs in the kernel but enable it in our new debugging kernel config?

Not every user of this config will be using apparmor-profile-everything so I think we should do as much reasonable kernel hardening as possible regardless of apparmor. Any lost debugging functionality can easily be re-enabled with the --debug flag if the user needs it.

Agreed. Debug kernel is required anyhow. And hardened-kernel might be sooner to go through completion, calling for testers, installed by default. Also more likely to be used by others than apparmor-profile-everything.

2 Likes

Instead of the complexity of all those wrappers, maybe we could just use a bash alias?

alias apt="hdapt"
alias apt-get="hdapt"

rapt runs apt which if hardened-kernel is being used would be an alias of hdapt. If not using hardened-kernel, it would be real apt.

We instruct users to use rapt when using apparmor-profile-everything and just apt when not.

Using ordinary apt can be done by running /usr/bin/apt.

1 Like

Since lockdown is not in our kernel version, I can create a patchset to implement some of its features in our kernel if you want.

I won’t implement the ones we’ve already mitigated with our config though as that’d be pointless so there won’t be too many changes.

1 Like

Did that.

https://github.com/madaidan/hardened-kernel/tree/lockdown/usr/share/hardened-kernel/patches/lockdown

The only one missing is the acpi_rsdp patch as that one is more complicated (it requires some reworking of the early boot code).

This isn’t for inclusion in linux-hardened as they’ll prefer the official lockdown LSM and these patches only complement our config (things such as LOCKDOWN_KPROBES would be needed for linux-hardened but not us).

1 Like

madaidan via Whonix Forum:

Disable notifier error injection by madaidan · Pull Request #32 · Kicksecure/hardened-kernel · GitHub

Enable sanity checks in virtual to page code by madaidan · Pull Request #33 · Kicksecure/hardened-kernel · GitHub

Merged.

1 Like

madaidan via Whonix Forum:

@Patrick Would it be possible to make the CI actually test booting the kernel via e.g. kexec?

I would love to have this.

Not only I would love to kexec boot the kernel but also run an automated test suite.

I don’t know if it is possible on travis CI. The current CI is rather
complex, hacky. Travis CI is based on Ubuntu. VM based. Dunno if kexec
would be supported there. The actual kernel build happens inside a
docker ( http://travis.debian.net/ ). Dunno if that docker could be
restarted with the newly built kernel either.

Also do we really have to invent this? Doesn’t kernel.org already have
automated testing / fuzzing on CI servers somewhere that we can re-use?

2 Likes

Absolutely.

Also I think this would make things easier to manage in case a user installs a regular unsecure kernel for hardware compatibility reasons?

Both to keep it in-line with user expectation and future extensibility. Think hapt-get purge. apt invokes a general man page, apt-get is one specific function of many.

Unnecessary. I’d consider it a working design document that describes what scope you implemented as you go along.

On my end I’m prepared to precompile binaries for distribution from our repos. I don’t know the feasibility of harnessing the reproducible build structure currently, but it should be a priority for user safety.

Debian has a dedicated debug kernel version so I think that’s the way to instead of either compromising th production version or making it undebuggable at all because of hardening.

2 Likes

We already have a separate kernel config for debugging.

hardened-kernel/usr/share/hardened-kernel/debugging-config at master · Kicksecure/hardened-kernel · GitHub

This is added onto the hardened config if the --debug flag is used during build.

Installing both by default would just be a usability feature.

Unrelated:

1 Like

I finished my lockdown patches.

The acpi_rdsp patch was actually not needed since it’s only an issue with CONFIG_KEXEC enabled which we disable.

It creates a CONFIG_SECURITY_LOCKDOWN option which when enabled locks down various ways userspace can escalate to kernel mode.

It cannot be disabled at runtime and the kernel_lockdown variable is marked read-only.

I’ve tested this and it works.

You can test that certain features are locked down. For example, you can use this to test if iopl() is locked down Debian paste error

user@host:~$ sudo ./iopl 
iopl: Operation not permitted
user@host:~$

We should now have prevented every way userspace can escalate to kernel mode, excluding vulnerabilities and modules (which we’ll eventually solve).

1 Like

madaidan via Whonix Forum:

Since lockdown is not in our kernel version, I can create a patchset to implement some of its features in our kernel if you want.

For quality control reasons and due to my very limited C programming
language skills (let alone kernel development) I need an independent
review of these patches. One option would be to get them merged in
another project with that capacity first for them to sign-off on so we
can go forward on this. Not sure if any is suited such as ClipOS or similar.

2 Likes