kernel recompilation for better hardening

Patrick · January 15, 2020, 1:41pm

hardened-kernel 0.8-1 on Non-Qubes-Whonix Whonix-Gateway and Whonix-Workstation.

DKMS does not run during linux kernel package upgrade. Therefore after reboot LKRG and tirdad no longer functions.

Jan 15 13:36:06 host modprobe[213]: modprobe: ERROR: could not insert ‘p_lkrg’: Exec format error

Workaround until DKMS will be properly run after kernel package upgrade:

sudo apt install --reinstall lkrg-dkms tirdad-dkms

Tirdad is fixed after that procedure.

DKMS might not run since the package is installed directly by dpkg and not through APT?

Next issue after reboot:

sudo modprobe p_lkrg

modprobe: ERROR: could not insert ‘p_lkrg’: No buffer space available

madaidan · January 15, 2020, 8:21pm

Users can set kernel.panic_on_oops=0 if they need to disable it.

Not really.

That isn’t really needed since we can just add those settings to security-misc but make them opt-in.

madaidan · January 15, 2020, 8:35pm

Patrick · January 15, 2020, 9:01pm

madaidan via Whonix Forum:

https://github.com/Whonix/hardened-kernel/pull/27

Merged.

Patrick · January 15, 2020, 9:11pm

madaidan via Whonix Forum:

Patrick:

I see. Maybe better add this as a sysctl, in a package that gets only installed inside VMs? (then it would also be enabled in initramfs already) Reason is that this would be way more easy to disable / debug / port.

Users can set kernel.panic_on_oops=0 if they need to disable it.

I see. That is good enough.

I was previously (probably falsely) assuming that this value would then
be hardcoded and users would need to recompile the kernel.

Patrick:

Also we yet have to create security-paranoid(opt-in package for settings which are too experimental for security-misc default package?

That isn’t really needed since we can just add those settings to security-misc but make them opt-in.

I.e. instruct users to copy from one place to another?

madaidan · January 15, 2020, 9:20pm

No, this doesn’t get rid of the kernel.panic_on_oops sysctl.

No, as in telling them to do things like enable certain systemd services (e.g. hide-hardware-info) or uncomment certain lines (e.g. disabling SACK).

madaidan · January 15, 2020, 9:39pm

Randomly found a message from Daniel Micay which talks about what kernel versions to use for security which is pretty relevant.

https://old.reddit.com/r/linux/comments/bd2fm7/grapheneos_an_open_source_privacy_and_security/ekwtmxc/

Relevant parts:

It’s important to follow along with the latest longterm / stable releases of the Linux kernel and the more recent branches are usually better choices because backporting is increasingly incomplete for the older branches. However, new kernel branches also have more attack surface and newer features / code that are less battle hardened. Of the supported upstream kernels (https://www.kernel.org/), I would go with 4.14 and soon 4.19 when it’s a little bit more mature. In terms of out-of-tree drivers, as long as they’re open source (they are), the only real sense it matters is acting as a blocker for migrating to newer LTS branches. Look at which branch https://grsecurity.net/download.php uses for the production variant.

There are both security advantages (new security features, more bug fixes applied) and disadvantages (more attack surface, more complexity, more new bugs) to using a more recent kernel branch. I think the best choice for overall security is usually the most recent LTS branch, once it has been available for a bit, so the best choice would likely be the 4.19 LTS at the moment. There’s so much churn and so many regressions when following mainline even just in terms of what people run into during regular use and it certainly applies to the underlying security properties of the kernels.

It seems like we’re using the best kernel version (currently 4.19).

Patrick · January 16, 2020, 11:51am

This is awesome! Thanks @madaidan! This will help to get more eyes on it and result in wider exposure / better testing.

Patrick · January 16, 2020, 12:26pm

I had archives in git repository and no download on user’s machine implemented but github crossed out my plans when trying to git push.

remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
remote: error: Trace: 980557540f5f13fe7ad779afff4213be
remote: error: See http://git.io/iEPt8g for more information.
remote: error: File usr/src/hardened-kernel/files/linux-4.19.93.tar is 801.99 MB; this exceeds GitHub’s file size limit of 100.00 MB

Various options come to mind as a solution… No amazing option yet will keep looking…

Patrick · January 16, 2020, 1:51pm

git-annex (in Debian) can add files to git without really adding them. git push and git clone would work with any popular free git service such as github. However, those who clone the repository would not really have the required files. These would still have to be downloaded. Does not seems like the best solution for our use case.
github with git LFS. 1 GB storage (would be OK) with 1 GB traffic quota per month.
fork https://github.com/anthraxx/linux-hardened and git checkout 4.19.96.a. Then add our compilation script and packaging on top. When new version comes out, we just git rebase to new tag. We wouldn’t save any space in the repository but we would satisfy that particular github hardlimit.

How did you come to that conclusion? I can’t find any documentation by hardened-linux.

Patrick · January 16, 2020, 2:03pm

git clone --depth=1 https://github.com/anthraxx/linux-hardened.git
cd linux-hardened
git fetch origin tag --depth=1 --no-tags 4.19.96.a
git verify-tag 4.19.96.a
git verify-commit 4.19.96.a^{commit}
git checkout 4.19.96.a
git branch 4.19.96.a

That should be as good as kernel archive from kernel.org + linux-hardened patch? Should b really the same?

Upon reflection… Maybe not fork linux-hardened. Just git clone and checking out the git tag which were are interested in. Then add linux-hardened as a git submodule to our hardened-kernel. That is because linux-hardened has .gitignore folder which contains the /debian folder and make deb-pkg creates the /debian folder. That conflicts with our hardened-kernel packaging. Rather than hack around that it seems better to not touch linux-hardened (except checkout proper version) and use a git submodule. Even though git submodules don’t have the best usability that would get the task done.

madaidan · January 16, 2020, 4:35pm

Upload the .tar.xz and instead of the .tar file.

Patrick · January 16, 2020, 6:31pm

madaidan via Whonix Forum:

Patrick:

I had archives in git repository and no download on user’s machine implemented but github crossed out my plans when trying to git push.

Upload the .tar.xz and instead of the .tar file.

That might even work for now but it’s not a future proof solution. Linux
tarball seems to be growing. Breaking the 100 MB border soon.

97M linux-4.14.165.tar.xz
105M linux-5.4.12.tar.xz
167M linux-5.5-rc6.tar.gz

That would break with Linux 5.x series. And I’d rather not implement
something that will need changes soon.

Patrick:

How did you come to that conclusion? I can’t find any documentation by hardened-linux.

kernel recompilation for better hardening

I see. Well, arch linux might have different requirements.

madaidan · January 16, 2020, 10:40pm

madaidan · January 16, 2020, 10:52pm

Patrick · January 17, 2020, 7:28am

Quote https://cateee.net/lkddb/web-lkddb/RESET_ATTACK_MITIGATION.html

CONFIG_RESET_ATTACK_MITIGATION

Request that the firmware clear the contents of RAM after a reboot using the TCG Platform Reset Attack Mitigation specification. This protects against an attacker forcibly rebooting the system while it still contains secrets in RAM, booting another OS and extracting the secrets. This should only be enabled when userland is configured to clear the MemoryOverwriteRequest flag on clean shutdown after secrets have been evicted, since otherwise it will trigger even on clean reboots.

Specifically:

This should only be enabled when userland is configured to clear the MemoryOverwriteRequest flag on clean shutdown

Can you make head or tail of that?

since otherwise it will trigger even on clean reboots.

I would think this is a feature, not a bug.

Patrick · January 17, 2020, 7:30am

Both merged, thanks!

Could you please re-enable this in the debug kernel config, i.e. no /dev/mem restriction in the debug kernel?

Patrick · January 17, 2020, 9:53am

Could you please also disable PANIC_ON_OOPS for debug kernel? And any other useful debug setting?

Patrick · January 17, 2020, 10:07am

Did you use that yet?

[+] Trying to detect architecture in “/usr/share/hardened-kernel/hardened-vm-kernel”…
[+] Detected architecture: X86_64
[+] Checking “/usr/share/hardened-kernel/hardened-vm-kernel” against hardening preferences…
option name | desired val | decision | reason | check result

CONFIG_BUG CONFIG_STRICT_KERNEL_RWX CONFIG_STACKPROTECTOR_STRONG CONFIG_SLUB_DEBUG CONFIG_STRICT_MODULE_RWX CONFIG_MICROCODE CONFIG_RETPOLINE CONFIG_X86_SMAP CONFIG_X86_UMIP CONFIG_IOMMU_SUPPORT CONFIG_SYN_COOKIES CONFIG_PAGE_TABLE_ISOLATION CONFIG_RANDOMIZE_MEMORY CONFIG_INTEL_IOMMU CONFIG_AMD_IOMMU CONFIG_VMAP_STACK CONFIG_RANDOMIZE_BASE CONFIG_THREAD_INFO_IN_TASK CONFIG_BUG_ON_DATA_CORRUPTION CONFIG_DEBUG_WX CONFIG_SCHED_STACK_END_CHECK CONFIG_SLAB_FREELIST_HARDENED CONFIG_SLAB_FREELIST_RANDOM CONFIG_SHUFFLE_PAGE_ALLOCATOR CONFIG_FORTIFY_SOURCE CONFIG_GCC_PLUGINS CONFIG_GCC_PLUGIN_RANDSTRUCT CONFIG_GCC_PLUGIN_LATENT_ENTROPY CONFIG_DEBUG_LIST CONFIG_DEBUG_SG CONFIG_DEBUG_CREDENTIALS CONFIG_DEBUG_NOTIFIERS CONFIG_HARDENED_USERCOPY CONFIG_HARDENED_USERCOPY_FALLBACK CONFIG_MODULE_SIG CONFIG_MODULE_SIG_ALL CONFIG_MODULE_SIG_SHA512 CONFIG_MODULE_SIG_FORCE CONFIG_DEFAULT_MMAP_MIN_ADDR CONFIG_REFCOUNT_FULL CONFIG_INIT_STACK_ALL CONFIG_INIT_ON_ALLOC_DEFAULT_ON CONFIG_INIT_ON_FREE_DEFAULT_ON CONFIG_SECURITY_DMESG_RESTRICT CONFIG_DEBUG_VIRTUAL CONFIG_STATIC_USERMODEHELPER CONFIG_SLAB_MERGE_DEFAULT CONFIG_GCC_PLUGIN_RANDST CONFIG_GCC_PLUGIN_STACKLEAK CONFIG_STACKLEAK_METRICS CONFIG_STACKLEAK_RUNTIME_DISABLE CONFIG_RANDOM_TRUST_CPU CONFIG_INTEL_IOMMU_SVM CONFIG_INTEL_IOMMU_DEFAULT_ON CONFIG_SLUB_DEBUG_ON CONFIG_RESET_ATTACK_MITIGATION CONFIG_AMD_IOMMU_V2 CONFIG_SECURITY CONFIG_SECURITY_WRITABLE_HOOKS CONFIG_SECURITY_YAMA CONFIG_SECURITY_LOADPIN CONFIG_SECURITY_LOCKDOWN_LSM CONFIG_SECURITY_LOCKDOWN_LSM_EARLY CONFIG_LOCK_DOWN_KERNEL_ CONFIG_SECURITY_SAFESETID CONFIG_SECCOMP CONFIG_SECCOMP_FILTER CONFIG_STRICT_DEVMEM CONFIG_MODULES CONFIG_DEVMEM CONFIG_IO_STRICT_DEVMEM CONFIG_ACPI_CUSTOM_METHOD CONFIG_COMPAT_BRK CONFIG_DEVKMEM CONFIG_COMPAT_VDSO CONFIG_BINFMT_MISC CONFIG_INET_DIAG CONFIG_KEXEC CONFIG_PROC_KCORE CONFIG_LEGACY_PTYS CONFIG_HIBERNATION CONFIG_LEGACY_VSYSCALL_NONE CONFIG_IA32_EMULATION CONFIG_X86_X32 CONFIG_MODIFY_LDT_SYSCALL CONFIG_X86_PTDUMP CONFIG_ZSMALLOC_STAT CONFIG_PAGE_OWNER CONFIG_DEBUG_KMEMLEAK CONFIG_BINFMT_AOUT CONFIG_KPROBES CONFIG_UPROBES CONFIG_GENERIC_TRACER CONFIG_PROC_VMCORE CONFIG_PROC_PAGE_MONITOR CONFIG_USELIB CONFIG_CHECKPOINT_RESTORE CONFIG_USERFAULTFD CONFIG_HWPOISON_INJECT CONFIG_MEM_SOFT_DIRTY CONFIG_DEVPORT CONFIG_DEBUG_FS CONFIG_NOTIFIER_ERROR_INJECTION CONFIG_ACPI_TABLE_UPGRADE CONFIG_ACPI_APEI_EINJ CONFIG_PROFILING CONFIG_BPF_SYSCALL CONFIG_MMIOTRACE_TEST CONFIG_KSM CONFIG_KALLSYMS CONFIG_X86_VSYSCALL_EMULATION CONFIG_MAGIC_SYSRQ CONFIG_KEXEC_FILE CONFIG_USER_NS CONFIG_LDISC_AUTOLOAD CONFIG_MMIOTRACE CONFIG_LIVEPATCH CONFIG_IP_DCCP CONFIG_IP_SCTP CONFIG_FTRACE CONFIG_BPF_JIT CONFIG_VIDEO_VIVID CONFIG_ARCH_MMAP_RND_BITS | y |defconfig | self_protection | OK
| y |defconfig | self_protection | OK
| y |defconfig | self_protection | OK
| y |defconfig | self_protection | OK
| y |defconfig | self_protection | OK
| y |defconfig | self_protection | OK
| y |defconfig | self_protection | OK
| y |defconfig | self_protection | OK
| y |defconfig | self_protection | OK: CONFIG_X86_INTEL_UMIP “y”
| y |defconfig | self_protection | OK
| y |defconfig | self_protection | OK
| y |defconfig | self_protection | OK
| y |defconfig | self_protection | OK
| y |defconfig | self_protection | OK
| y |defconfig | self_protection | OK
| y |defconfig | self_protection | OK
| y |defconfig | self_protection | OK
| y |defconfig | self_protection | OK
| y | kspp | self_protection | OK
| y | kspp | self_protection | OK
| y | kspp | self_protection | OK
| y | kspp | self_protection | OK
| y | kspp | self_protection | OK
| y | kspp | self_protection | FAIL: not found
| y | kspp | self_protection | OK
| y | kspp | self_protection | OK
| y | kspp | self_protection | OK
| y | kspp | self_protection | OK
| y | kspp | self_protection | OK
| y | kspp | self_protection | OK
| y | kspp | self_protection | OK
| y | kspp | self_protection | OK
| y | kspp | self_protection | OK
| is not set | kspp | self_protection | OK
| y | kspp | self_protection | OK
| y | kspp | self_protection | FAIL: “is not set”
| y | kspp | self_protection | FAIL: “is not set”
| y | kspp | self_protection | FAIL: “is not set”
| 65536 | kspp | self_protection | OK
| y | kspp | self_protection | OK
| y | clipos | self_protection | OK: CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL “y”
| y | clipos | self_protection | FAIL: not found
| y | clipos | self_protection | FAIL: not found
| y | clipos | self_protection | OK
| y | clipos | self_protection | FAIL: “is not set”
| y | clipos | self_protection | FAIL: “is not set”
| is not set | clipos | self_protection | OK
RUCT_PERFORMANCE | is not set | clipos | self_protection | OK
| y | clipos | self_protection | FAIL: not found
| is not set | clipos | self_protection | FAIL: CONFIG_GCC_PLUGIN_STACKLEAK is needed
| is not set | clipos | self_protection | FAIL: CONFIG_GCC_PLUGIN_STACKLEAK is needed
| is not set | clipos | self_protection | OK
| y | clipos | self_protection | OK
| y | clipos | self_protection | OK
| y | my | self_protection | FAIL: “is not set”
| y | my | self_protection | FAIL: not found
| y | my | self_protection | OK
| y |defconfig | security_policy | OK
| is not set |defconfig | security_policy | OK: not found
| y | kspp | security_policy | OK
| y | my | security_policy | FAIL: “is not set”
| y | my | security_policy | FAIL: not found
| y | my | security_policy | FAIL: not found
FORCE_CONFIDENTIALITY| y | my | security_policy | FAIL: not found
| y | my | security_policy | FAIL: not found
| y |defconfig | cut_attack_surface | OK
| y |defconfig | cut_attack_surface | OK
| y |defconfig | cut_attack_surface | OK: CONFIG_DEVMEM “is not set”
| is not set | kspp | cut_attack_surface | FAIL: “y”
| is not set | kspp | cut_attack_surface | OK
| y | kspp | cut_attack_surface | OK: CONFIG_DEVMEM “is not set”
| is not set | kspp | cut_attack_surface | OK
| is not set | kspp | cut_attack_surface | OK
| is not set | kspp | cut_attack_surface | OK
| is not set | kspp | cut_attack_surface | OK: not found
| is not set | kspp | cut_attack_surface | OK
| is not set | kspp | cut_attack_surface | FAIL: “m”
| is not set | kspp | cut_attack_surface | OK
| is not set | kspp | cut_attack_surface | OK
| is not set | kspp | cut_attack_surface | OK
| is not set | kspp | cut_attack_surface | OK
| y | kspp | cut_attack_surface | OK
| is not set | kspp | cut_attack_surface | OK
| is not set | kspp | cut_attack_surface | OK
| is not set | kspp | cut_attack_surface | OK
| is not set |grsecurity| cut_attack_surface | OK
| is not set |grsecurity| cut_attack_surface | OK
| is not set |grsecurity| cut_attack_surface | OK
| is not set |grsecurity| cut_attack_surface | OK
| is not set |grsecurity| cut_attack_surface | OK: not found
| is not set |grsecurity| cut_attack_surface | OK
| is not set |grsecurity| cut_attack_surface | OK: not found
| is not set |grsecurity| cut_attack_surface | OK: not found
| is not set |grsecurity| cut_attack_surface | OK: not found
| is not set |grsecurity| cut_attack_surface | OK
| is not set |grsecurity| cut_attack_surface | OK
| is not set |grsecurity| cut_attack_surface | OK
| is not set |grsecurity| cut_attack_surface | OK
| is not set |grsecurity| cut_attack_surface | OK: not found
| is not set |grsecurity| cut_attack_surface | OK: not found
| is not set |grsecurity| cut_attack_surface | OK
| is not set |grsecurity| cut_attack_surface | FAIL: “y”
| is not set |grsecurity| cut_attack_surface | FAIL: “m”
| is not set | lockdown | cut_attack_surface | OK
| is not set | lockdown | cut_attack_surface | OK
| is not set | lockdown | cut_attack_surface | OK
| is not set | lockdown | cut_attack_surface | FAIL: “y”
| is not set | lockdown | cut_attack_surface | OK: not found
| is not set | clipos | cut_attack_surface | OK
| is not set | clipos | cut_attack_surface | FAIL: “y”
| is not set | clipos | cut_attack_surface | FAIL: “y”
| is not set | clipos | cut_attack_surface | FAIL: “y”
| is not set | clipos | cut_attack_surface | OK
| is not set | clipos | cut_attack_surface | FAIL: “y”
| is not set | clipos | cut_attack_surface | FAIL: “y”
| is not set | my | cut_attack_surface | OK: not found
| is not set | my | cut_attack_surface | OK: not found
| is not set | my | cut_attack_surface | OK
| is not set | my | cut_attack_surface | OK
| is not set | my | cut_attack_surface | OK
| is not set | my | cut_attack_surface | FAIL: “y”
| is not set | my | cut_attack_surface | OK: not found
| 32 | clipos |userspace_hardening | OK

[+] config check is finished: ‘OK’ - 94 / ‘FAIL’ - 29

Looks quite good already! I guess you worked your way through it 70-80% already. Please keep at your current style of learning about these options first and not just uncritically “copy/paste”.

Patrick · January 17, 2020, 11:33am

madaidan via Whonix Forum:

There also seems to be an issue with the CI.

https://travis-ci.com/Whonix/hardened-kernel/builds/143787339
+sed -e s@^@  @g ..//*.changes

sed: can't read ..//*.changes: No such file or directory

The command "wget -O- https://raw.githubusercontent.com/adrelanos/travis.debian.net/gh-pages/script.sh | sh -" exited with 2.

finally managed to fix that

https://travis-ci.com/Whonix/hardened-kernel/builds/144913881