kernel recompilation for better hardening

Development
,
292

I have a vision now how to implement this.

Installation through dpkg isn’t great anyhow? That doesn’t handle version upgrades. For example if someone installed a 5.x kernel from backports, this package would still force dpkg install a hardened 4.x kernel for no reason. By using APT, we’d never attempt a downgrade / install a lower versioned kernel.

dpkg also doesn’t well handle dependencies. If there ever was a conflicting dependency, dpkg would wreck APT (broken dependencies) which then requires manual commands to resolve. APT often warns against such situations, with the option to abort etc. Also dpkg does not properly call triggers such as DKMS?

How this could be implemented…

Terminology:

  • real APT: (non-Whonix, non-Kicksecure, plain Debian APT. Just the normal APT that any Debian user is using.)
  • wrapped APT (wapt): a wrapper that stacks various plugins such as rapt or hdapt
  • restricted APT (rapt): as per apparmor-profile-everything
  • hdapt: runs usual real apt dist-upgarde, apt update, apt-dist upgrade again

Most of these scripts can be implemented standalone without much thought about the other scripts.

When APT runs, some script could call the compilation script. And then create a local APT repository. That local APT repository would then be used with APT for proper processing of dependencies and triggers.

/etc/apt/sources.list.d/hardened-kernel-local.list:

deb [trusted=yes] file:/path/to/local/repository local main contrib non-free

similar to https://github.com/Whonix/Whonix/blob/master/help-steps/create-local-temp-apt-repo

User runs apt (or apt-get). What actually happens (without need to tell the user) (fully auditable in source code or by reading the files on the disk), that rapt (restricted APT) is executed - only if that is installed. It then passes on to hdapt (hardened kernel APT). Doing a usual real apt dist-upgrade. hdapt would then run “apt update” again to load the local APT repository which contains the hardened kernel and run real apt dist-upgrade again.

with apparmor-profile-everything:

wapt → rapt → hdapt → real apt update → real apt dist-upgrade → real apt update → real apt dist-upgrade

without apparmor-profile-everything:

wapt → hdapt → real apt update → real apt dist-upgrade → real apt update → real apt dist-upgrade

From user perspective:

  • user runs sudo apt update: just only that happens as per usual
  • user runs sudo apt dist-upgrade: usual sudo apt dist-upgrade happens + sudo apt update (fetching local APT repository with hardened kernel only) → another sudo apt --yes dist-upgrade is performed

(Fetching local (from hard drive) APT repository will take just 1 second or so.)

Same in other words:

  • user runs sudo apt update: just only that happens as per usual
  • user runs sudo apt dist-upgrade: usual sudo apt dist-upgrade happens → maybe hardened-kernel package is upgraded, compiles a new kernel, updates the local APT repository → + maybe sudo apt update (fetching local APT repository with hardened kernel only) → maybe another sudo apt --yes dist-upgrade is performed

maybe meaning: only if necessary (hardened-kernel package was upgraded).

Another layer of complexity that I haven’t mentioned yet: uwt stream isolation wrapper. Also something wrapped apt has to handle.

Just an idea yet. Dunno how it will look when implemented. Seems kinda complex. And messing that up would break APT. Users could still always run apt-get.anondist-orig (real apt).

And all this trouble for what? So users can keep typing what they’re accustomed to: apt or apt-get

Maybe rather than taking on that complexity with wrapped apt, we should rather have a one time popup teaching users to use hdapt from now?

hdapt and hdapt-get? Or only 1? I guess not much of extra work for both.

Maybe better to finish developing the concept of stackable wrappers first before implementing this?

What do we do with Whonix default redistributed (downloadable) builds in future? Pre-compile a hardened kernel and then re-compile it at first boot? Or just install Debian default kernel and compile a hardened kernel at first boot before going online for the first time?

An interesting option would also be to compile both, a non-debug and a debug kernel during the build process. Install both. Boot non-debug kernel by default. Debug kernel could be booted by choosing it in grub boot menu. Would waste some disk space and increase VM image size though.

madaidan via Whonix Forum:

What if we disable debugfs in the kernel but enable it in our new debugging kernel config?

Not every user of this config will be using apparmor-profile-everything so I think we should do as much reasonable kernel hardening as possible regardless of apparmor. Any lost debugging functionality can easily be re-enabled with the --debug flag if the user needs it.

Agreed. Debug kernel is required anyhow. And hardened-kernel might be sooner to go through completion, calling for testers, installed by default. Also more likely to be used by others than apparmor-profile-everything.

2 Likes
293

Instead of the complexity of all those wrappers, maybe we could just use a bash alias?

alias apt="hdapt"
alias apt-get="hdapt"

rapt runs apt which if hardened-kernel is being used would be an alias of hdapt. If not using hardened-kernel, it would be real apt.

We instruct users to use rapt when using apparmor-profile-everything and just apt when not.

Using ordinary apt can be done by running /usr/bin/apt.

1 Like
294

Since lockdown is not in our kernel version, I can create a patchset to implement some of its features in our kernel if you want.

I won’t implement the ones we’ve already mitigated with our config though as that’d be pointless so there won’t be too many changes.

1 Like
295

Did that.

https://github.com/madaidan/hardened-kernel/tree/lockdown/usr/share/hardened-kernel/patches/lockdown

The only one missing is the acpi_rsdp patch as that one is more complicated (it requires some reworking of the early boot code).

This isn’t for inclusion in linux-hardened as they’ll prefer the official lockdown LSM and these patches only complement our config (things such as LOCKDOWN_KPROBES would be needed for linux-hardened but not us).

1 Like
296

madaidan via Whonix Forum:

Disable notifier error injection by madaidan · Pull Request #32 · Kicksecure/hardened-kernel · GitHub

Enable sanity checks in virtual to page code by madaidan · Pull Request #33 · Kicksecure/hardened-kernel · GitHub

Merged.

1 Like
297

madaidan via Whonix Forum:

@Patrick Would it be possible to make the CI actually test booting the kernel via e.g. kexec?

I would love to have this.

Not only I would love to kexec boot the kernel but also run an automated test suite.

I don’t know if it is possible on travis CI. The current CI is rather
complex, hacky. Travis CI is based on Ubuntu. VM based. Dunno if kexec
would be supported there. The actual kernel build happens inside a
docker ( http://travis.debian.net/ ). Dunno if that docker could be
restarted with the newly built kernel either.

Also do we really have to invent this? Doesn’t kernel.org already have
automated testing / fuzzing on CI servers somewhere that we can re-use?

2 Likes
298

Absolutely.

Also I think this would make things easier to manage in case a user installs a regular unsecure kernel for hardware compatibility reasons?

Both to keep it in-line with user expectation and future extensibility. Think hapt-get purge. apt invokes a general man page, apt-get is one specific function of many.

Unnecessary. I’d consider it a working design document that describes what scope you implemented as you go along.

On my end I’m prepared to precompile binaries for distribution from our repos. I don’t know the feasibility of harnessing the reproducible build structure currently, but it should be a priority for user safety.

Debian has a dedicated debug kernel version so I think that’s the way to instead of either compromising th production version or making it undebuggable at all because of hardening.

2 Likes
299

We already have a separate kernel config for debugging.

hardened-kernel/usr/share/hardened-kernel/debugging-config at master · Kicksecure/hardened-kernel · GitHub

This is added onto the hardened config if the --debug flag is used during build.

Installing both by default would just be a usability feature.

Unrelated:

1 Like
300

I finished my lockdown patches.

The acpi_rdsp patch was actually not needed since it’s only an issue with CONFIG_KEXEC enabled which we disable.

It creates a CONFIG_SECURITY_LOCKDOWN option which when enabled locks down various ways userspace can escalate to kernel mode.

It cannot be disabled at runtime and the kernel_lockdown variable is marked read-only.

I’ve tested this and it works.

You can test that certain features are locked down. For example, you can use this to test if iopl() is locked down Debian paste error

user@host:~$ sudo ./iopl 
iopl: Operation not permitted
user@host:~$

We should now have prevented every way userspace can escalate to kernel mode, excluding vulnerabilities and modules (which we’ll eventually solve).

1 Like
301

madaidan via Whonix Forum:

Since lockdown is not in our kernel version, I can create a patchset to implement some of its features in our kernel if you want.

For quality control reasons and due to my very limited C programming
language skills (let alone kernel development) I need an independent
review of these patches. One option would be to get them merged in
another project with that capacity first for them to sign-off on so we
can go forward on this. Not sure if any is suited such as ClipOS or similar.

2 Likes
302

Compare them to the upstream code.

0002-lockdown-efivar_ssdt_load.patch → LKML: Matthew Garrett: [PATCH V40 28/29] efi: Restrict efivar_ssdt_load when the kernel is locked down

0003-lockdown-pci-bar-access.patch → LKML: Matthew Garrett: [PATCH V40 11/29] PCI: Lock down BAR access when the kernel is locked down

0004-lockdown-perf.patch → LKML: Matthew Garrett: [PATCH V40 24/29] lockdown: Lock down perf when in confidentiality mode

0005-lockdown-tiocsserial.patch → LKML: Matthew Garrett: [PATCH V40 18/29] lockdown: Lock down TIOCSSERIAL

0006-lockdown-ioport.patch → LKML: Matthew Garrett: [PATCH V40 12/29] x86: Lock down IO port access when the kernel is locked down

0007-lockdown-pcmcia.patch → LKML: Matthew Garrett: [PATCH V40 17/29] lockdown: Prohibit PCMCIA CIS storage when the kernel is locked down

0008-lockdown-module-params.patch → LKML: Matthew Garrett: [PATCH V40 19/29] lockdown: Lock down module params that specify hardware parameters (eg. ioport)

They’re all very similar. They’re mostly just simple if statements and exits. You can see they do the same even with limited C skills.

The perf lockdown patch is slightly different as my patch locks down perf entirely instead of just PERF_SAMPLE_REGS_INTR as more bad things can be done with perf such as kernel address leaking with PERF_SAMPLE_IP Breaking KASLR with perf

I’m not aware of any other project these patches would make sense in due to the reasons above.

This isn’t for inclusion in linux-hardened as they’ll prefer the official lockdown LSM and these patches only complement our config (things such as LOCKDOWN_KPROBES would be needed for linux-hardened but not us).

ClipOS uses the official lockdown LSM so these patches don’t make sense for them.

303

Should we also disable coredumps and enable them in the debug kernel?

1 Like
304

madaidan via Whonix Forum:

Compare them to the upstream code.

I can’t even do that. For all of Whonix’s source code, for any reviewers
reporting any security issues or asking any reasonable questions, I can
can handle that. For kernel code, I could not even fix minor things.

2 Likes
305

For simplicity, I think Whonix redistributed (downloadable) versions
could come with Debian standard stable kernel by default. This would
also serve as “debug” kernel. I.e. those VMs that can’t boot with the
hardened kernel can at least boot with the Debian kernel. During the
first boot, the hardened kernel could be compiled and installed
automatically before any networking goes up and before the user can do
anything except cancel compilation of the kernel.

Not much of an enhancement to do same as above but ship with a hardened
kernel which layout is public knowledge (redistributed, downloadable,
public build) anyhow. Related to

madaidan via Whonix Forum:

Should we also disable coredumps and enable them in the debug kernel?

I guess yes. Debugging in hardened-kernel (non-debug) isn’t possible now
anymore?

2 Likes
306

Patrick via Whonix Forum:

[1] For simplicity, I think Whonix redistributed (downloadable) versions
could come with Debian standard stable kernel by default. This would
also serve as “debug” kernel. I.e. those VMs that can’t boot with the
hardened kernel can at least boot with the Debian kernel. During the
first boot, the hardened kernel could be compiled and installed
automatically before any networking goes up and before the user can do
anything except cancel compilation of the kernel.

[2] Not much of an enhancement to do same as above but ship with a hardened
kernel which layout is public knowledge (redistributed, downloadable,
public build) anyhow. Related to
Hide Kernel Symbols for Better Security vs Reproducible Builds

This also simplifies reproducible builds since it doesn’t introduce new
non-determinism. Our hardened kernel uses all available randomization
security options. Disabling these for redistributed (downloadable)
versions would add even more complexity. [1] is enough.

2 Likes
307

I could. Not much could go wrong in such little changes though.

Basic debugging is possible. Not really anything advanced though.

1 Like
308
1 Like
309
1 Like
310

madaidan via Whonix Forum:

Sign all kernel modules by madaidan · Pull Request #36 · Kicksecure/hardened-kernel · GitHub

Merged.

1 Like
311

All merged.

1 Like