Linux Kernel Runtime Guard (LKRG) - Linux Kernel Runtime Integrity Checking and Exploit Detection

VirtualBox host support is now on the horizon:
https://www.openwall.com/lists/lkrg-users/2020/01/24/1

1 Like

sysctl vs module parameters / Can sysctl be set before loading the module? | was: bug: LKRG kills VirtualBox host VMs

1 Like

UMH blocked when though lkrg.block_modules = 0

2 Likes

As for LKRG loading it as early as possible is non-trivial. Currently loading LKRG as early as possible would cause some issues. For example it would detect changes by tirdad as malicious and show a lot false positive messages. Asked upstream about it:

Compiling LKRG static into the Kernel / Loading LKRG kernel module as early as possible or after other modules?

1 Like

The wiki page got noticed by spender.

1 Like
2 Likes

Above fixed.


2 Likes

LKRG is compatible with Debian Linux buster-backports 5.4.8-1~bpo10+1 kernel

A Debian issue but unrelated to LKRG:
linux-image-amd64 vs linux-headers-amd64 Debian buster-backports version mismatch bpo.2 vs bpo.3

2 Likes

Related: T950: prevent kernel info leaks

1 Like

Rebased on upstream master branch.

This was addressed by upstream:
lkrg-users - Re: LKRG unfit for installation by default in Whonix / Kicksecure due to kernel boot console output - usability issue


A bug was privately reported and upstream is on it already:

https://bitbucket.org/Adam_pi3/lkrg-main/pull-requests/4

Added a workaround (useful anyhow) in meanwhile:

2 Likes

Quote lkrg-users - rootkit detection

Date: Sun, 14 Jun 2020 17:37:59 +0200
From: Solar Designer <solar@…nwall.com>
To: lkrg-users@…ts.openwall.com
Subject: rootkit detection

Hi,

Adam found this interesting Master’s Thesis of Juho Junnila, entitled
“Effectiveness of Linux Rootkit Detection Tools”:

http://jultika.oulu.fi/files/nbnfioulu-202004201485.pdf

It shows LKRG as the most effective kernel rootkit detector (of those
tested), as long as LKRG is loaded before the rootkit. It also shows
LKRG sometimes detect rootkits even when LKRG is loaded after the
rootkit (we never intended that to work, but it sometimes does anyway,
which is OK). Finally, it shows that LKRG never detects pure userspace
rootkits (as expected; it would probably be a bug if it did).

In particular, when LKRG is loaded before the rootkit, it detected 8 out
of 9 kernel rootkits tested: Diamorphine, Honey Pot Bears,
LilyOfTheValley, Nuk3 Gh0st, Puszek, Reptile, Rootfoo Linux Rootkit,
Sutekh. There were no false positives. No other tested rootkit
detector was anywhere close to LKRG’s effectiveness at detecting kernel
rootkits, with AIDE, OSSEC, and Rootkit Hunter detecting 2 out of 9
each, and Chkrootkit detecting none. These other detectors did,
however, detect some userspace rootkits, which LKRG doesn’t try to. The
combination of AIDE and LKRG is shown to be most effective, detecting 14
out of 15 rootkits total (both user and kernel space ones).

None of the tools detected Keysniffer, which the thesis describes as:

“Keysniffer is an LKM that can be classified as a kernel mode rootkit
due to its kernel hooking capabilities. It is a keylogger that records
keyboard events to debugfs, an in-memory file system used for Linux
kernel debugging. It does not feature any particular anti-detection
mechanisms except that it prevents accessing the log file by non-root
users and the module is named “kisni.ko” by default to avoid immediate
suspicion when listing kernel modules. (Jana, 2019.) While Keysniffer
may not meet the criteria of all rootkit definitions due to its lack of
specific hiding functionalities, it does meet the criteria by Harley and
Lee (2007) and is thus included in this study.”

While the above doesn’t say it, I think the non-detection of Keysniffer
by LKRG is due to it being too similar to non-malicious kernel modules.
The functionality described would conceivably be provided by a module
not intended for malicious use, and the module doesn’t need to mess with
the kernel’s integrity to provide this functionality. The non-detection
by other tools is probably because Keysniffer wasn’t popular enough, as
its signature-based detection is definitely practical.

Of course, none of these rootkits tried to bypass LKRG. If they wanted
to, they could e.g. simply “rmmod p_lkrg” before loading themselves into
the kernel. (Or the attacker could do that manually.) In LKRG
development, we’re currently more focused on exploits run before the
attacker got legitimate-looking privileged access to the system. We’re
not as focused on rootkits, which are loaded with legitimate-looking
privileged access.

That said, there are a few things we could do in the future to make it
more difficult and/or less reliable to bypass detection by LKRG-aware
rootkit loading. We could password-protect the module unloading and
sysctl interfaces of LKRG, and/or provide a way to disable those after
initial configuration. We could also provide setup instructions and/or
our own userspace components around LKRG where alerts would be sent to a
remote system, including alerts of LKRG being unloaded or reconfigured.

Alexander

2 Likes

A post was split to a new topic: rootkit detection system - AIDE

LKRG 0.8.1 is in all repositories since some time now.


https://www.openwall.com/presentations/OSTconf2020-LKRG-In-A-Nutshell/

When LKRG is installed on one kernel and then the user switches to another kernel, it breaks (e.g. installing it on hardened-kernel and then booting into vanilla):

lkrg-loader[249]: INFO: Running 'modprobe p_lkrg msr_validate=0 pcfi_validate=1 ' ...
lkrg-loader[249]: modprobe: FATAL: Module p_lkrg not found in directory /lib/modules/4.19.0-13-amd64
lkrg-loader[249]: ####################################################################
lkrg-loader[249]: ## BEGIN ERROR in /usr/lib/lkrg/lkrg-loader detected!
lkrg-loader[249]: ##
lkrg-loader[249]: ## ERROR LOG:
lkrg-loader[249]: ## See above.
lkrg-loader[249]: ##
lkrg-loader[249]: ## BASH_COMMAND: modprobe p_lkrg $lkrg_opts
lkrg-loader[249]: ## EXIT_CODE: 1
lkrg-loader[249]: ##
lkrg-loader[249]: ## END ERROR in /usr/lib/lkrg/lkrg-loader detected!
lkrg-loader[249]: ## Please report this bug!
lkrg-loader[249]: ####################################################################
1 Like

How did you install hardened-kernel? dpkg or APT? Could you share installation log please? I would like to see any messages regarding DKMS.

The LKRG Debian packaging supports DKMS. It’s already confirmed to work stable during Debian kernel upgrades. From LKRG / DKMS perspective, I wonder what is making the difference.

APT.

user@host:~$ sudo apt install lkrg
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  linux-headers-4.19.0-10-amd64 linux-headers-4.19.0-10-common linux-headers-4.19.0-11-amd64 linux-headers-4.19.0-11-common
  linux-image-4.19.0-10-amd64 linux-image-4.19.0-11-amd64
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  lkrg-dkms
The following NEW packages will be installed:
  lkrg lkrg-dkms
0 upgraded, 2 newly installed, 0 to remove and 1 not upgraded.
Need to get 97.0 kB of archives.
After this operation, 897 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 tor+https://deb.whonix.org buster-developers/main amd64 lkrg-dkms amd64 0:0.8.1.0-1 [89.9 kB]
Get:2 tor+https://deb.whonix.org buster-developers/main amd64 lkrg amd64 0:0.8.1.0-1 [7,064 B]
Fetched 97.0 kB in 3s (36.5 kB/s)
Selecting previously unselected package lkrg-dkms.
(Reading database ... 181991 files and directories currently installed.)
Preparing to unpack .../lkrg-dkms_0%3a0.8.1.0-1_amd64.deb ...
Unpacking lkrg-dkms (0.8.1.0-1) ...
Selecting previously unselected package lkrg.
Preparing to unpack .../lkrg_0%3a0.8.1.0-1_amd64.deb ...
Unpacking lkrg (0.8.1.0-1) ...
Setting up lkrg-dkms (0.8.1.0-1) ...
Loading new lkrg-0.8.1 DKMS files...
Building for 4.19.122
Building initial module for 4.19.122
Done.

p_lkrg.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/4.19.122/updates/dkms/

depmod...

DKMS: install completed.
Created symlink /etc/systemd/system/multi-user.target.wants/lkrg-dkms.service → /lib/systemd/system/lkrg-dkms.service.
Setting up lkrg (0.8.1.0-1) ...

The module is only present for a specific kernel. E.g. in my case it is installed in /lib/modules/4.19.122 but not anything else in /lib/modules. Further verified by:

user@host:~$ sudo modinfo p_lkrg
filename:       /lib/modules/4.19.122/updates/dkms/p_lkrg.ko
[...]

Whereas, on the other kernel:

user@host:~$ sudo modinfo lkrg
modinfo: ERROR: Module lkrg not found.

Thus, the modprobe command cannot find which module to load.

1 Like

Different command. p_lkrg. (I didn’t invent the p_. Upstream did.) But probably just a mistake. Not the reason.

So you had kernels 4.19.0 and 4.19.122 installed. The latter, the newer kernel version 4.19.122 was loaded. Kernel module was build for 4.19.122 only. Right?

I wonder how/if this can be fixed. See the DKMS file,debian/lkrg-dkms.dkms.

Then feel free to compare with this with other DKMS files.

Try:

sudo dkms status

Also interesting:

/var/lib/dpkg/info/lkrg-dkms.postinst

sudo sh -x /var/lib/dpkg/info/lkrg-dkms.postinst configure

Reveals what is required to debug further:

sudo sh -x /usr/lib/dkms/common.postinst lkrg 0.8.1 /usr/share/lkrg-dkms

The latter shows how list of kernels is generated. (Runs as root.)

KERNELS=$(ls /lib/modules/ 2>/dev/null || true)

Try this command.

sudo ls /lib/modules/

Perhaps apparmor-profile-everything or other issue which messes up setting of KERNELS? xtrace of /usr/lib/dkms/common.postinst might reveal that.

For me it builds modules for all installed kernel versions. I guess that is the default.

Dunno if DKMS in Debian only works for APT installed kernels but we’re already talking only about APT installed kernels here.

/etc/dkms/framework.conf

## Automatic installation and upgrade for all installed kernels (if set to a
## non-null value)
# autoinstall_all_kernels=""

Quote Bug #1844805 “virtualbox, virtualbox-dkms, virtualbox-qt fail du...” : Bugs : ubuntu-release-upgrader package : Ubuntu

I see there’s one caveat, the autoinstall_all_kernels variable in /etc/dkms/framework.conf can be set to build for all installed kernels, not just the current and newest. But the default is to build only for current and newest.

1 Like

They’re interchangeable and neither work on the second kernel while both work on the first.

Yes.

I compiled hardened-kernel manually and installed it via dpkg -i. I don’t think that would be classified as an “APT installed kernel”. hardened-kernel is the 4.19.122 one.

Setting

autoinstall_all_kernels="yes"

fixed it.

1 Like
1 Like

v0.9.0 was released.

https://www.openwall.com/lists/announce/2021/04/12/1

But not signed as previous releases. Perhaps because release this time was made by Solar Designer, not Adam.

Waiting for reply before pushing upgrade.