Detecting Loadable Kernel Modules
(LKM)
The purpose of this paper is cover LKM
basics, detecting "trojaned" LKM's and figuring out which LKM is
installed on your machine.
LKM
What is a LKM? Loadable Kernel Modules (LKM)
are files that contain dynamically loadable kernel components. LKM's are
normally used to load device drivers and other hardware drivers. LKM's can be
found on Linux, Solaris and BSD (Open, Free and Net). This paper will focus on
Linux.
Linux comes with various tools to assist
the system administrator in loading, listing and unloading of the kernel
modules. Before we cover the tools, lets look at some important files and
directories associated with Kernel Modules. The first directory we want to look
at is /lib/modules/"kernel_version". Lets look and see what files we
can find under /lib/modules."Kernel_version":
|
block build cdrom fs ipv4 misc modules.dep modules.isapnpmap modules.pcimap modules.usbmap net pcmcia scsi usb video |
Table
1. /lib/modules listing
Table 1 shows us both directories and files
related to LKM's. The only ones listed that are not directories are
modules.dep, modules.isapnpmap, modules.pcimap and mosules.usbmap. Those are
actual files that list modules within the system (combined). Lets take a quick
look at the net directory (This is
not a complete listing, do to space):
|
3c59x.o 3c90x.o 82596.o 8390.o ac3200.o acenic.o arlan-proc.o |
Table 2. net directory
There are
a couple of important bits of information we can gather from this table. The
first bit is, modules are listed as .o. Why? Because they are object files that
contain the actual module itself. The second bit is, the listing is not
complete. I did not have room to put all of the modules.
The second important directory/file we
want to look at it /etc/conf.modules. Conf.modules is the configuration file
that allows the system administrator to specify a variety of parameters that control
the loading of the modules. The conf.modules file typically looks like this:
|
alias parport_lowlevel parport_pc |
Table 3. conf.modules file
The
conf.modules file allows the administrator to assign alias to commonly used modules.
Note: This file is not required. A system administrator can create his/her own
configuration file. A system administrator can do that by using the modprobe –C
command.
Since I mentioned modprobe earlier, lets
briefly take a look at modprobe and the other tools available in Linux that
load and unload modules. The first tool is modprobe. Modprobe can load single
and multiple modules. It uses the modules.dep file to look up dependencies.
Depmod creates a “Makefile” like dependency file. This file is called
modules.dep. The last three commands I
will cover are lsmod, insmod and rmmod. lsmod provides the administrator a
listing of all modules currently loaded in the kernel. This list can also be
found at /proc/modules. This is helpful if you want to figure out what modules
are currently running. You will see a little later that lsmod is not always a
good tool to use for the detection of rootkit LKM’s. insmod loads modules(seems
simple). Some of the “trojaned” LKM code I have seen uses insmod -f to load the
“trojaned” module. rmmod is normally used to remove modules from the kernel.
Normally rmmod will NOT remove any “trojaned” LKM. We will discuss details of
that a little bit later. Hopefully, this section provided everybody the basics
of Loadable Kernel Modules. If this was not detailed enough see http://www.kernel.org/LDP/.
LKM
rootkits and kstat
Recently, there has been a lot of press
about adore. Well, the worm adore not the LKM adore. So I decided to look into
the LKM adore(everyone else has looked at the worm). BTW, if want to download
adore go to http://packetstorm.securify.com/filedesc/adore-0.34.html.
Adore is a Linux LKM rootkit. It is easy
to install and only requires a few minor adjustments when configuring. Adore
can be installed with a default configuration or the user can make changes to
some of the code. The readme file recommends that the user change the settings
for ELITE_CMD and for HIDDEN_PORT. When you run ./configure, adore asks you for
a password. This is for the backdoor port (hence, change the HIDDEN_PORT) If
you want more information on the install you will have to download it. I ran
the default installation, which consisted of running./configure and make. After
running make, you will see two files (ava, startadore). In order for adore too
execute you have to run startadore. Once you run startadore the user can then
run ava. Table 4 is the output from ./ava.
|
Usage: ./ava
{h,u,r,R,i,v,U} [file, PID or dummy (for U)] h hide file u unhide file r execute as root R remove PID forever U uninstall adore i make PID invisible v make PID visible |
Table 4. Ava output
Table
4 shows us the options that adore provides to us. I will not cover each switch
and what it does only because it does not help us detect this rootkit. Now that
we have covered the basics of adore (LKM) lets look at how we detect adore and
other rootkits.
Many rootkits hide processes, directories,
files and even connections. But many of them do so by modifying the source code
of binaries such as ps, df, netstat, top and lsof. There are a couple of ways
to detect these types of rootkits (i.e. t0rn):
1)
md5 checksums
2)
Compiling these binaries from a known good source (i.e. cd-rom, disk).
3)
Some rootkits have a default port that an administrator can focus in on.
4)
Running a program such as chkrootkit
Many
of the techniques used to detect rootkits like t0rn are not effective against
LKM rootkits (chkrootkit can detect some of them). Since LKM rootkits access
the kernel, they can hide processes, connections, directories and files without
modifying the binaries. MD5 checksums become useless because there are no files
being modified. This means the checksums will not change. Checking ports MIGHT
be an option BUT that only tells you that you have been “rooted”.
Well, how can I detect these monsters?
Easy. There is a program I highly recommend to everyone concerned with LKM
rootkits. This program is called KSTAT. You can find it at
http://s0ftpj.org/en/site.html. KSTAT
works by checking the memory (/dev/kmem) for information about the
host(including LKMs). Want to see what kstat looks like? Good. Here is the
output for kstat:
Usage: kstat [-i iff] [-P] [-p pid] [-M] [-m addr]
[-s]
-i iff may be
specified as 'all' or as name (e.g. eth0)
displays
info about the queried interface
-P displays
all processes
-p pid is
the process id of the queried task
-M displays
the kernel's LKMs' linked list
-m addr is
the hex address of the queried module
displays
info about the module to be found at addr
-s displays
info about the system calls' table
As you can see, kstat provides a person with many
options for detecting these rootkits. Lets go through some of the options and
from these options we will learn how to detect the following LKM rootkits:
1) knark
2) adore
3) rkit
BTW, if there’s any rootkits I have omitted please
let me know and I will update the paper. : ) Kstat –s seems to be the best way
to detect LKM rootkits. The other options are really great, but –s works all
the time. Here is an output from kstat –s:
|
SysCall Address sys_exit 0xc0117ce4 sys_fork 0xc0108ebc sys_read 0xc012604c sys_write 0xc0126110 sys_open 0xc0125c10 sys_close 0xc0125d60 sys_waitpid 0xc0117ff8 sys_creat 0xc0125ca4 sys_link 0xc012de60 sys_unlink 0xc012dc90 sys_execve 0xc0108f18 sys_chdir 0xc01254a0 sys_time 0xc01184b4 sys_mknod 0xc012d77c sys_chmod 0xc01256e4 |
Table 5. kstat –s
Table
5 is not a complete output from kstat –s, but it does give us an idea as too
what the output looks like. Remember –s provides us with a picture of the
sys_call_table. Keep this information fresh as we will revisit this again later.
Kstat –P is another switch that is very
effective. Kstat -P shows us all of the processes running at that time. This
includes the processes hidden by an LKM rootkit.
Table
6 is an example of kstat –P:
|
PID PPID
UID GID COMMAND 1 0 0 0 init 2 1 0 0 kflushd 3 1 0 0 kupdate 4 1 0 0 kpiod 5 1 0 0 kswapd 6 1 0 0 mdrecoveryd 241 1 1 0 portmap 256 1 0 0 lockd 257 256
0 0 rpciod 266 1 0 0 rpc.statd 280 1 0 0 apmd 331 1 0 0 syslogd |
Table
6. kstat –P
When
I first ran this program I wasn’t sure kstat could do what it said but…it
proved my doubts WRONG. Here is what I did to verify this switch. I used ava -i
241(portmap). I then ran kstat –P. As you can see from table 6, portmap(bold) shows up. I then ran
ps-ef(Table 7)and could not find it. Lsof also did not show it. Table 7 is the
output from ps –ef:
UID
PID PPID C STIME TTY TIME CMD
root
1 0 0 Mar30 ? 00:00:06
init [3]
root 2
1 0 Mar30 ? 00:00:00 [kflushd]
root
3 1 0 Mar30 ? 00:00:00
[kupdate]
root
4 1 0 Mar30 ? 00:00:00
[kpiod]
root
5 1 0 Mar30 ? 00:00:00
[kswapd]
root
6 1 0 Mar30 ? 00:00:00
[mdrecoveryd]
root
256 1 0 Mar30 ? 00:00:00 [lockd]
root
257 256 0 Mar30 ? 00:00:00 [rpciod]
root
266 1 0 Mar30 ? 00:00:00 rpc.statd
root
280 1 0 Mar30 ? 00:00:00 /usr/sbin/apmd -p 10 -w 5 -W -s
root
331 1 0 Mar30 ? 00:00:00 syslogd -m 0
root
340 1 0 Mar30 ? 00:00:00 klogd
nobody
354 1 0 Mar30 ? 00:00:00 identd -e -o
nobody
357 354 0 Mar30 ? 00:00:00 identd -e -o
nobody
359 357 0 Mar30 ? 00:00:00 identd -e -o
nobody
360 357 0 Mar30 ? 00:00:00 identd -e -o
nobody
361 357 0 Mar30 ? 00:00:00 identd -e -o
daemon
372 1 0 Mar30 ? 00:00:00 /usr/sbin/atd
root 386 1
0 Mar30 ? 00:00:00 crond
root
404 1 0 Mar30 ? 00:00:00 inetd
root
418 1 0 Mar30 ? 00:00:00 lpd
root
462 1 0 Mar30 ? 00:00:00 sendmail: accepting connections
root
477 1 0 Mar30 ? 00:00:00 gpm -t ps/2
root
491 1 0 Mar30 ? 00:00:01 httpd
xfs
531 1 0 Mar30 ? 00:00:00 xfs -droppriv -daemon -port -1
root
571 1 0 Mar30 tty1 00:00:00 login -- root
root
572 1 0 Mar30 tty2 00:00:00 /sbin/mingetty tty2
root
573 1 0 Mar30 tty3 00:00:00 /sbin/mingetty tty3
root
574 1 0 Mar30 tty4 00:00:00 /sbin/mingetty tty4
root
575 1 0 Mar30 tty5 00:00:00 /sbin/mingetty tty5
root 576
1 0 Mar30 tty6 00:00:00 /sbin/mingetty tty6
nobody
4290 491 0 Apr01 ? 00:00:00 httpd
nobody
4291 491 0 Apr01 ? 00:00:00 httpd
nobody
4292 491 0 Apr01 ? 00:00:00 httpd
nobody
4293 491 0 Apr01 ? 00:00:00 httpd
nobody
4294 491 0 Apr01 ? 00:00:00 httpd
nobody
4295 491 0 Apr01 ? 00:00:00 httpd
nobody
4298 491 0 Apr01 ? 00:00:00 httpd
nobody
4299 491 0 Apr01 ? 00:00:00 httpd
root
8073 571 0 Apr02 tty1 00:00:00 -bash
root
10659 8073 0 11:24 tty1 00:00:00 ps -ef
Table
7. ps –ef output
There
are two other switches we will go over, they are kstat –p and kstat –M. First,
lets look at kstat –p. kstat –p gives us more information about a process. In
order to run kstat –p you will need to provide a process id. For example: kstat
–p 241. This checks process 241 (portmap) and provide an output. Sometimes it
can provide us with more information about a LKM rootkit. TABLE 8 is the output
from kstat –p 241:
|
Name: portmap State: S (sleeping) Pid: 241 Ppid: 1 (init) Uid: 1 1 1 1 Gid: 0 0 0 0 Flags: PF_FORKNOEXEC PF_SUPERPRIV Crucial Capabilities Check Open Files 0 CHAR /dev/null 1 CHAR /dev/null 2 CHAR /dev/null 3 0.0.0.0:111 0.0.0.0:0 4 0.0.0.0:111 0.0.0.0:0 7 FIFO /// 8 FIFO /// 21 CHAR /dev/null |
Table
8. kstat –p output
Finally,
lets look at kstat –M. Under normal conditions an administrator can perform lsmod
or more /proc/modules and find out what modules he/she are running. When a
machine has been “rooted” you can’t trust lsmod for accurate information. Kstat
–M will catch many of the basic LKM rootkits that I have tested. Kstat –M list
all of the modules loaded. I have seen it where kstat –M did list anything for
knark, but…nothings perfect. The output from kstat –M is similar to lsmod.
So far we have
covered a lot of material about LKM’s, rootkits and kstat. Now we are going to
put all of that together and learn how to detect some of the LKM rootkits
available today.
The first LKM rootkit
we want to look at is knark. Knark is probably the best-known LKM rootkit and
one of the best written as well. Detecting it with kstat is fairly straight up
as well. Remember Table 5? Well in order to detect a knark LKM rootkit you will
need to run kstat –s. Once ran you need to look for the following:
|
sys_fork 0xc284652c WARNING! Should be at
0xc0108c88 sys_read 0xc2846868 WARNING! Should be at
0xc012699c sys_execve 0xc2846bb8 WARNING! Should be at
0xc0108ce4 sys_kill 0xc28465d4 WARNING! Should be at
0xc01106b4 sys_ioctl 0xc2846640 WARNING! Should be at
0xc012ff78 sys_settimeofday 0xc2846a8c
WARNING! Should be at 0xc0118364 sys_clone 0xc2846580 WARNING! Should be at
0xc0108ca4 |
Table 9. knark detection
Lets
take a closer look at this table and see what all of this mess means to us. First,
we see that there are seven (7) sys_call_table entries that have warnings. Lets
look at sys_settimeofday, here we see that currently it is making it’s home in
memory at 0xc2846a8c. kstat –s tells us that this is wrong and it should be at
0xc0118364. When knark is installed it
changes the sys_call_table and the memory locations where you can find:
sys_fork, sys_read, sys_execve, sys_kill, sys_ioctl, sys_settimeofday and
sys_clone. This is how you could detect knark. Knowing which sys_call_table
entries have been changed could help you identify the actual rootkit itself.
Knark changes seven(7) total. After running kstat –s, the next step would be to
run ps-ef along with kstat –P and compare the two. If the are any differences
in the two you can then take the correct action. Keep in mind that memory
locations can change from box to box. As I said earlier, detecting LKM
rootkits with kstat is quite simple.
Lets look at adore. Table 10 will show us
what adore changes:
|
sys_fork 0xc4051428 WARNING! Should be at
0xc0108c88 sys_write 0xc4051590 WARNING! Should be at
0xc01269b8 sys_close 0xc405163c WARNING! Should be at
0xc01264a4 sys_kill 0xc40514d0 WARNING! Should be at
0xc011060c sys_mkdir 0xc405172c WARNING! Should be at
0xc012e540 sys_clone 0xc405147c WARNING! Should be at
0xc0108ca4 sys_getdents 0xc40512a4 WARNING! Should be at
0xc013022c |
Table 10. Adore detection
Adore changes seven (7) entries as well (just like
knark). Knark and adore only change three (3) of the same sys_call_table
entries. They are sys_fork, sys_kill, sys_clone. This is a good thing because
it allows us the ability to detect each one separately. Again, once this has
been done, the administrator can then run kstat –P and ps-ef and compare the
two to figure out what processes are running and hiding.
The last rootkit we will look at is rkit. This
rootkit does not hide itself as well as the other two do. As a matter of fact it
only changes sys_setuid and can be found using kstat –M.
LKM rootkits can make a system administrator’s life a nightmare.
They are hard to detect, but using tools like kstat and understanding what the
rootkit changes can make our life easier. Since tools like kstat are available,
it would help systems administrators if they took a “picture” of the
sys_call_table after a fresh install and any upgrades. This will help in
identifying the good from the bad.
Although I have not had the time, one could write a shell | perl script
to automate this process and check weekly, daily, every 10 minutes or whatever.
http://s0ftpj.org/en/site.html
http://members.prestige.net/tmiller12/papers/KNARK.htm
LKM rootkits
http://packetstorm.securify.com/groups/thc/LKM_HACKING.html
Toby
Miller is a GIAC Certified Analyst and MCP. He is currently working towards his
CISSP and RHCE. Toby has contributed to 2 books, written papers for SANS,
Securityfocus, performs Risk Assessments and runs a lab for a living. For
entertainment Toby likes to analyze exploit signatures on his home network.
Toby can be reached at tmiller@va.prestige.net.