kernel recompilation for better hardening

madaidan via Whonix Forum:

Fix debug build by madaidan · Pull Request #27 · Kicksecure/hardened-kernel · GitHub

Merged.

1 Like

madaidan via Whonix Forum:

Users can set kernel.panic_on_oops=0 if they need to disable it.

I see. That is good enough.

I was previously (probably falsely) assuming that this value would then
be hardcoded and users would need to recompile the kernel.

That isn’t really needed since we can just add those settings to security-misc but make them opt-in.

I.e. instruct users to copy from one place to another?

1 Like

No, this doesn’t get rid of the kernel.panic_on_oops sysctl.

No, as in telling them to do things like enable certain systemd services (e.g. hide-hardware-info) or uncomment certain lines (e.g. disabling SACK).

1 Like

Randomly found a message from Daniel Micay which talks about what kernel versions to use for security which is pretty relevant.

DanielMicay comments on GrapheneOS - an open source privacy and security oriented mobile OS with Android app compatibility

Relevant parts:

It’s important to follow along with the latest longterm / stable releases of the Linux kernel and the more recent branches are usually better choices because backporting is increasingly incomplete for the older branches. However, new kernel branches also have more attack surface and newer features / code that are less battle hardened. Of the supported upstream kernels (https://www.kernel.org/), I would go with 4.14 and soon 4.19 when it’s a little bit more mature. In terms of out-of-tree drivers, as long as they’re open source (they are), the only real sense it matters is acting as a blocker for migrating to newer LTS branches. Look at which branch grsecurity - Download uses for the production variant.

There are both security advantages (new security features, more bug fixes applied) and disadvantages (more attack surface, more complexity, more new bugs) to using a more recent kernel branch. I think the best choice for overall security is usually the most recent LTS branch, once it has been available for a bit, so the best choice would likely be the 4.19 LTS at the moment. There’s so much churn and so many regressions when following mainline even just in terms of what people run into during regular use and it certainly applies to the underlying security properties of the kernels.

It seems like we’re using the best kernel version (currently 4.19).

2 Likes

This is awesome! Thanks @madaidan! This will help to get more eyes on it and result in wider exposure / better testing.

2 Likes

I had archives in git repository and no download on user’s machine implemented but github crossed out my plans when trying to git push.

remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
remote: error: Trace: 980557540f5f13fe7ad779afff4213be
remote: error: See Managing large files - GitHub Docs for more information.
remote: error: File usr/src/hardened-kernel/files/linux-4.19.93.tar is 801.99 MB; this exceeds GitHub’s file size limit of 100.00 MB

Various options come to mind as a solution… No amazing option yet will keep looking…

1 Like

How did you come to that conclusion? I can’t find any documentation by hardened-linux.

2 Likes
git clone --depth=1 https://github.com/anthraxx/linux-hardened.git
cd linux-hardened
git fetch origin tag --depth=1 --no-tags 4.19.96.a
git verify-tag 4.19.96.a
git verify-commit 4.19.96.a^{commit}
git checkout 4.19.96.a
git branch 4.19.96.a

That should be as good as kernel archive from kernel.org + linux-hardened patch? Should b really the same?

Upon reflection… Maybe not fork linux-hardened. Just git clone and checking out the git tag which were are interested in. Then add linux-hardened as a git submodule to our hardened-kernel. That is because linux-hardened has .gitignore folder which contains the /debian folder and make deb-pkg creates the /debian folder. That conflicts with our hardened-kernel packaging. Rather than hack around that it seems better to not touch linux-hardened (except checkout proper version) and use a git submodule. Even though git submodules don’t have the best usability that would get the task done.

2 Likes

Upload the .tar.xz and instead of the .tar file.

madaidan via Whonix Forum:

Upload the .tar.xz and instead of the .tar file.

That might even work for now but it’s not a future proof solution. Linux
tarball seems to be growing. Breaking the 100 MB border soon.

97M linux-4.14.165.tar.xz
105M linux-5.4.12.tar.xz
167M linux-5.5-rc6.tar.gz

That would break with Linux 5.x series. And I’d rather not implement
something that will need changes soon.

kernel recompilation for better hardening - #216 by madaidan

I see. Well, arch linux might have different requirements.

1 Like
2 Likes
1 Like

Quote Linux Kernel Driver DataBase: CONFIG_RESET_ATTACK_MITIGATION: Reset memory attack mitigation

CONFIG_RESET_ATTACK_MITIGATION

Request that the firmware clear the contents of RAM after a reboot using the TCG Platform Reset Attack Mitigation specification. This protects against an attacker forcibly rebooting the system while it still contains secrets in RAM, booting another OS and extracting the secrets. This should only be enabled when userland is configured to clear the MemoryOverwriteRequest flag on clean shutdown after secrets have been evicted, since otherwise it will trigger even on clean reboots.

Specifically:

This should only be enabled when userland is configured to clear the MemoryOverwriteRequest flag on clean shutdown

Can you make head or tail of that?

since otherwise it will trigger even on clean reboots.

I would think this is a feature, not a bug.

1 Like

Both merged, thanks! :slight_smile:

Could you please re-enable this in the debug kernel config, i.e. no /dev/mem restriction in the debug kernel?

1 Like

Could you please also disable PANIC_ON_OOPS for debug kernel? And any other useful debug setting?

1 Like

Did you use that yet?

[+] Trying to detect architecture in “/usr/share/hardened-kernel/hardened-vm-kernel”…
[+] Detected architecture: X86_64
[+] Checking “/usr/share/hardened-kernel/hardened-vm-kernel” against hardening preferences…
option name | desired val | decision | reason | check result

CONFIG_BUG | y |defconfig | self_protection | OK
CONFIG_STRICT_KERNEL_RWX | y |defconfig | self_protection | OK
CONFIG_STACKPROTECTOR_STRONG | y |defconfig | self_protection | OK
CONFIG_SLUB_DEBUG | y |defconfig | self_protection | OK
CONFIG_STRICT_MODULE_RWX | y |defconfig | self_protection | OK
CONFIG_MICROCODE | y |defconfig | self_protection | OK
CONFIG_RETPOLINE | y |defconfig | self_protection | OK
CONFIG_X86_SMAP | y |defconfig | self_protection | OK
CONFIG_X86_UMIP | y |defconfig | self_protection | OK: CONFIG_X86_INTEL_UMIP “y”
CONFIG_IOMMU_SUPPORT | y |defconfig | self_protection | OK
CONFIG_SYN_COOKIES | y |defconfig | self_protection | OK
CONFIG_PAGE_TABLE_ISOLATION | y |defconfig | self_protection | OK
CONFIG_RANDOMIZE_MEMORY | y |defconfig | self_protection | OK
CONFIG_INTEL_IOMMU | y |defconfig | self_protection | OK
CONFIG_AMD_IOMMU | y |defconfig | self_protection | OK
CONFIG_VMAP_STACK | y |defconfig | self_protection | OK
CONFIG_RANDOMIZE_BASE | y |defconfig | self_protection | OK
CONFIG_THREAD_INFO_IN_TASK | y |defconfig | self_protection | OK
CONFIG_BUG_ON_DATA_CORRUPTION | y | kspp | self_protection | OK
CONFIG_DEBUG_WX | y | kspp | self_protection | OK
CONFIG_SCHED_STACK_END_CHECK | y | kspp | self_protection | OK
CONFIG_SLAB_FREELIST_HARDENED | y | kspp | self_protection | OK
CONFIG_SLAB_FREELIST_RANDOM | y | kspp | self_protection | OK
CONFIG_SHUFFLE_PAGE_ALLOCATOR | y | kspp | self_protection | FAIL: not found
CONFIG_FORTIFY_SOURCE | y | kspp | self_protection | OK
CONFIG_GCC_PLUGINS | y | kspp | self_protection | OK
CONFIG_GCC_PLUGIN_RANDSTRUCT | y | kspp | self_protection | OK
CONFIG_GCC_PLUGIN_LATENT_ENTROPY | y | kspp | self_protection | OK
CONFIG_DEBUG_LIST | y | kspp | self_protection | OK
CONFIG_DEBUG_SG | y | kspp | self_protection | OK
CONFIG_DEBUG_CREDENTIALS | y | kspp | self_protection | OK
CONFIG_DEBUG_NOTIFIERS | y | kspp | self_protection | OK
CONFIG_HARDENED_USERCOPY | y | kspp | self_protection | OK
CONFIG_HARDENED_USERCOPY_FALLBACK | is not set | kspp | self_protection | OK
CONFIG_MODULE_SIG | y | kspp | self_protection | OK
CONFIG_MODULE_SIG_ALL | y | kspp | self_protection | FAIL: “is not set”
CONFIG_MODULE_SIG_SHA512 | y | kspp | self_protection | FAIL: “is not set”
CONFIG_MODULE_SIG_FORCE | y | kspp | self_protection | FAIL: “is not set”
CONFIG_DEFAULT_MMAP_MIN_ADDR | 65536 | kspp | self_protection | OK
CONFIG_REFCOUNT_FULL | y | kspp | self_protection | OK
CONFIG_INIT_STACK_ALL | y | clipos | self_protection | OK: CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL “y”
CONFIG_INIT_ON_ALLOC_DEFAULT_ON | y | clipos | self_protection | FAIL: not found
CONFIG_INIT_ON_FREE_DEFAULT_ON | y | clipos | self_protection | FAIL: not found
CONFIG_SECURITY_DMESG_RESTRICT | y | clipos | self_protection | OK
CONFIG_DEBUG_VIRTUAL | y | clipos | self_protection | FAIL: “is not set”
CONFIG_STATIC_USERMODEHELPER | y | clipos | self_protection | FAIL: “is not set”
CONFIG_SLAB_MERGE_DEFAULT | is not set | clipos | self_protection | OK
CONFIG_GCC_PLUGIN_RANDSTRUCT_PERFORMANCE | is not set | clipos | self_protection | OK
CONFIG_GCC_PLUGIN_STACKLEAK | y | clipos | self_protection | FAIL: not found
CONFIG_STACKLEAK_METRICS | is not set | clipos | self_protection | FAIL: CONFIG_GCC_PLUGIN_STACKLEAK is needed
CONFIG_STACKLEAK_RUNTIME_DISABLE | is not set | clipos | self_protection | FAIL: CONFIG_GCC_PLUGIN_STACKLEAK is needed
CONFIG_RANDOM_TRUST_CPU | is not set | clipos | self_protection | OK
CONFIG_INTEL_IOMMU_SVM | y | clipos | self_protection | OK
CONFIG_INTEL_IOMMU_DEFAULT_ON | y | clipos | self_protection | OK
CONFIG_SLUB_DEBUG_ON | y | my | self_protection | FAIL: “is not set”
CONFIG_RESET_ATTACK_MITIGATION | y | my | self_protection | FAIL: not found
CONFIG_AMD_IOMMU_V2 | y | my | self_protection | OK
CONFIG_SECURITY | y |defconfig | security_policy | OK
CONFIG_SECURITY_WRITABLE_HOOKS | is not set |defconfig | security_policy | OK: not found
CONFIG_SECURITY_YAMA | y | kspp | security_policy | OK
CONFIG_SECURITY_LOADPIN | y | my | security_policy | FAIL: “is not set”
CONFIG_SECURITY_LOCKDOWN_LSM | y | my | security_policy | FAIL: not found
CONFIG_SECURITY_LOCKDOWN_LSM_EARLY | y | my | security_policy | FAIL: not found
CONFIG_LOCK_DOWN_KERNEL_FORCE_CONFIDENTIALITY| y | my | security_policy | FAIL: not found
CONFIG_SECURITY_SAFESETID | y | my | security_policy | FAIL: not found
CONFIG_SECCOMP | y |defconfig | cut_attack_surface | OK
CONFIG_SECCOMP_FILTER | y |defconfig | cut_attack_surface | OK
CONFIG_STRICT_DEVMEM | y |defconfig | cut_attack_surface | OK: CONFIG_DEVMEM “is not set”
CONFIG_MODULES | is not set | kspp | cut_attack_surface | FAIL: “y”
CONFIG_DEVMEM | is not set | kspp | cut_attack_surface | OK
CONFIG_IO_STRICT_DEVMEM | y | kspp | cut_attack_surface | OK: CONFIG_DEVMEM “is not set”
CONFIG_ACPI_CUSTOM_METHOD | is not set | kspp | cut_attack_surface | OK
CONFIG_COMPAT_BRK | is not set | kspp | cut_attack_surface | OK
CONFIG_DEVKMEM | is not set | kspp | cut_attack_surface | OK
CONFIG_COMPAT_VDSO | is not set | kspp | cut_attack_surface | OK: not found
CONFIG_BINFMT_MISC | is not set | kspp | cut_attack_surface | OK
CONFIG_INET_DIAG | is not set | kspp | cut_attack_surface | FAIL: “m”
CONFIG_KEXEC | is not set | kspp | cut_attack_surface | OK
CONFIG_PROC_KCORE | is not set | kspp | cut_attack_surface | OK
CONFIG_LEGACY_PTYS | is not set | kspp | cut_attack_surface | OK
CONFIG_HIBERNATION | is not set | kspp | cut_attack_surface | OK
CONFIG_LEGACY_VSYSCALL_NONE | y | kspp | cut_attack_surface | OK
CONFIG_IA32_EMULATION | is not set | kspp | cut_attack_surface | OK
CONFIG_X86_X32 | is not set | kspp | cut_attack_surface | OK
CONFIG_MODIFY_LDT_SYSCALL | is not set | kspp | cut_attack_surface | OK
CONFIG_X86_PTDUMP | is not set |grsecurity| cut_attack_surface | OK
CONFIG_ZSMALLOC_STAT | is not set |grsecurity| cut_attack_surface | OK
CONFIG_PAGE_OWNER | is not set |grsecurity| cut_attack_surface | OK
CONFIG_DEBUG_KMEMLEAK | is not set |grsecurity| cut_attack_surface | OK
CONFIG_BINFMT_AOUT | is not set |grsecurity| cut_attack_surface | OK: not found
CONFIG_KPROBES | is not set |grsecurity| cut_attack_surface | OK
CONFIG_UPROBES | is not set |grsecurity| cut_attack_surface | OK: not found
CONFIG_GENERIC_TRACER | is not set |grsecurity| cut_attack_surface | OK: not found
CONFIG_PROC_VMCORE | is not set |grsecurity| cut_attack_surface | OK: not found
CONFIG_PROC_PAGE_MONITOR | is not set |grsecurity| cut_attack_surface | OK
CONFIG_USELIB | is not set |grsecurity| cut_attack_surface | OK
CONFIG_CHECKPOINT_RESTORE | is not set |grsecurity| cut_attack_surface | OK
CONFIG_USERFAULTFD | is not set |grsecurity| cut_attack_surface | OK
CONFIG_HWPOISON_INJECT | is not set |grsecurity| cut_attack_surface | OK: not found
CONFIG_MEM_SOFT_DIRTY | is not set |grsecurity| cut_attack_surface | OK: not found
CONFIG_DEVPORT | is not set |grsecurity| cut_attack_surface | OK
CONFIG_DEBUG_FS | is not set |grsecurity| cut_attack_surface | FAIL: “y”
CONFIG_NOTIFIER_ERROR_INJECTION | is not set |grsecurity| cut_attack_surface | FAIL: “m”
CONFIG_ACPI_TABLE_UPGRADE | is not set | lockdown | cut_attack_surface | OK
CONFIG_ACPI_APEI_EINJ | is not set | lockdown | cut_attack_surface | OK
CONFIG_PROFILING | is not set | lockdown | cut_attack_surface | OK
CONFIG_BPF_SYSCALL | is not set | lockdown | cut_attack_surface | FAIL: “y”
CONFIG_MMIOTRACE_TEST | is not set | lockdown | cut_attack_surface | OK: not found
CONFIG_KSM | is not set | clipos | cut_attack_surface | OK
CONFIG_KALLSYMS | is not set | clipos | cut_attack_surface | FAIL: “y”
CONFIG_X86_VSYSCALL_EMULATION | is not set | clipos | cut_attack_surface | FAIL: “y”
CONFIG_MAGIC_SYSRQ | is not set | clipos | cut_attack_surface | FAIL: “y”
CONFIG_KEXEC_FILE | is not set | clipos | cut_attack_surface | OK
CONFIG_USER_NS | is not set | clipos | cut_attack_surface | FAIL: “y”
CONFIG_LDISC_AUTOLOAD | is not set | clipos | cut_attack_surface | FAIL: “y”
CONFIG_MMIOTRACE | is not set | my | cut_attack_surface | OK: not found
CONFIG_LIVEPATCH | is not set | my | cut_attack_surface | OK: not found
CONFIG_IP_DCCP | is not set | my | cut_attack_surface | OK
CONFIG_IP_SCTP | is not set | my | cut_attack_surface | OK
CONFIG_FTRACE | is not set | my | cut_attack_surface | OK
CONFIG_BPF_JIT | is not set | my | cut_attack_surface | FAIL: “y”
CONFIG_VIDEO_VIVID | is not set | my | cut_attack_surface | OK: not found
CONFIG_ARCH_MMAP_RND_BITS | 32 | clipos |userspace_hardening | OK

[+] config check is finished: ‘OK’ - 94 / ‘FAIL’ - 29

Looks quite good already! :slight_smile: I guess you worked your way through it 70-80% already. Please keep at your current style of learning about these options first and not just uncritically “copy/paste”.

2 Likes

madaidan via Whonix Forum:

There also seems to be an issue with the CI.

https://travis-ci.com/Whonix/hardened-kernel/builds/143787339

+sed -e s@^@  @g ..//*.changes

sed: can't read ..//*.changes: No such file or directory

The command "wget -O- https://raw.githubusercontent.com/adrelanos/travis.debian.net/gh-pages/script.sh | sh -" exited with 2.

finally managed to fix that

https://travis-ci.com/Whonix/hardened-kernel/builds/144913881

2 Likes

This feature is only meant to erase memory after an unclean shutdown. So on ordinary shutdowns, you are meant to disable memory clearing by clearing the MemoryOverwriteRequest flag.

I think it’s better to clear memory on ordinary shutdowns too at the expense of minor delays during shutdown. Disabling memory clearing on ordinary shutdowns allows an attacker to just shutdown the machine normally to perform the attack.

There’s more information in the actual spec https://www.trustedcomputinggroup.org/wp-content/uploads/Platform-Reset-Attack-Mitigation-Specification.pdf

We should still enable STRICT_DEVMEM and IO_STRICT_DEVMEM though.

Yes, many settings aren’t useful for us though.

CONFIG_SHUFFLE_PAGE_ALLOCATOR
CONFIG_INIT_ON_ALLOC_DEFAULT_ON
CONFIG_INIT_ON_FREE_DEFAULT_ON
CONFIG_GCC_PLUGIN_STACKLEAK
CONFIG_STACKLEAK_METRICS
CONFIG_STACKLEAK_RUNTIME_DISABLE
CONFIG_SECURITY_LOCKDOWN_LSM
CONFIG_SECURITY_LOCKDOWN_LSM_EARLY
CONFIG_LOCK_DOWN_KERNEL_FORCE_CONFIDENTIALITY
CONFIG_SECURITY_SAFESETID

These don’t exist in our kernel version.

CONFIG_PAGE_POISONING

We use linux-hardened’s CONFIG_PAGE_SANITIZE instead.

CONFIG_SLUB_DEBUG_ON

This has no advantage over the slub_debug boot parameter.

CONFIG_SECURITY_LOADPIN

We have no support for this yet.

CONFIG_DEBUG_FS

Mitigated by apparmor.

CONFIG_X86_VSYSCALL_EMULATION

Mitigated by CONFIG_LEGACY_VSYSCALL_NONE.

CONFIG_USER_NS

Mitigated by linux-hardened’s user namespace restrictions.

CONFIG_BPF_JIT

We harden it for now and haven’t decided to disable it or not.

CONFIG_MODULES
CONFIG_MODULE_SIG_*

We will fix this issue eventually. It mostly already is due to apparmor.

So, the only recommendations that might be useful are:

                 option name                 | desired val | decision |       reason       |   check result
=========================================================================================================================
CONFIG_DEBUG_VIRTUAL                         |      y      |  clipos  |  self_protection   |   FAIL: "is not set"
CONFIG_STATIC_USERMODEHELPER                 |      y      |  clipos  |  self_protection   |   FAIL: "is not set"
CONFIG_INET_DIAG                             | is not set  |   kspp   | cut_attack_surface |   FAIL: "m"
CONFIG_NOTIFIER_ERROR_INJECTION              | is not set  |grsecurity| cut_attack_surface |   FAIL: "m"
CONFIG_BPF_SYSCALL                           | is not set  | lockdown | cut_attack_surface |   FAIL: "y"
CONFIG_KALLSYMS                              | is not set  |  clipos  | cut_attack_surface |   FAIL: "y"
CONFIG_MAGIC_SYSRQ                           | is not set  |  clipos  | cut_attack_surface |   FAIL: "y"
CONFIG_LDISC_AUTOLOAD                        | is not set  |  clipos  | cut_attack_surface |   FAIL: "y"
CONFIG_BPF_JIT                               | is not set  |    my    | cut_attack_surface |   FAIL: "y"

I still need to research these though.

We’ve also talked about sysrq before.

2 Likes

Awesome!

CI got even better. Now using lintian and testing kernel package installation using dpkg.

https://travis-ci.com/Whonix/hardened-kernel/builds/144922548

Is it OK that we compile using the root user? I guess so? lintian complains about maintainer-address-is-root-use but this can just be ignored (I’ll add that). But on the actual user system… Well… Kernel sources need to be fully trusted. A specifically crafted kernel source file that would exploit gcc and therefore warrant compilation as non-root user seems is best considered out of threat model. Compilation as another user (which one? specific user only for that purpose) is possible but would make the compilation script more complex, error-prone, cost development time… Therefore ideally avoided if not needed.

Other doable, possible, future CI enhancements: I could probably automate compilation on Debian testing too. unstable/sid, other suites too but not sure that’s worthwhile.

Usually I would say debug should ease debugging maximally but I guess requiring (super)root to access /dev/mem isn’t a hindrance.

1 Like