|
Analysis of a Compromised System
Introduction This report details the recent successful attack upon a Sun server, and the steps taken to investigate the damage and determine the path the attacker took to achive his goal. This machine was in a production environment, and the compromise went unnoticed for several weeks until the local administrators were alerted to it’s bad behaviour by administrators from another site. This machine came to our attention when we where contacted by the administrators of the site. They had previously been alerted to the fact that the machine was possibly compromised by the administrators of another site, and had pulled the machine from the network as a precaution. The data in this report has been sanitised of any information relating to the real installation. Initial Investigation We had as our only help the knowledge that the machine had indeed been compromised, the task being to find out how, not if. The first task was to comb through the log files looking for any activity or unusual gaps. An entry in the syslogd.7 file caught our attention: Sep 5 09:26:28 server-name sendmail[16060]: NOQUEUE: Null connection from bad-guy-1.organisation [172.16.0.1] Sep 5 20:16:02 server-name sendmail[18939]: UAA18939: from=root, size=283, class=0, pri=30283, nrcpts=1, msgid= <200009051216.UAA18939@server-name.good-guys.organisation>, relay=root@localhost Sep 5 20:16:15 server-name sendmail[18962]: UAA18939: to=bad-guy2@bad-guy2.organisation, ctladdr=root (0/1), delay=00:00:14, xdelay=00:00:12, mailer=esmtp, relay=mx1. bad-guy2.organisation. [172.16.10.10], stat=Sent (MAA27546 Message accepted for delivery) Sep 5 20:16:29 server-name sendmail[19096]: UAA19096: from=root, size=344, class=0, pri=30344, nrcpts=1, msgid= <200009051216.UAA19096@server-name.good-guys.organisation>, relay=root@localhost Sep 5 20:16:50 server-name sendmail[19098]: UAA19096: to=bad-guy3@bad-guy1.organisation, ctladdr=root (0/1), delay=00:00:35, xdelay=00:00:21, mailer=esmtp, relay=mail. bad-guy1.organisation. [172.16.0.2], stat=Sent (OAA16747 Message accepted for delivery)Our first noteworthy clue was that root on this machine should not have been sending mail to addresses outside of the organisation. We took note of the date for future reference. We ran standard Unix commands such as ps, netstat, arp and find etc, looking for anything out of the ordinary. We checked inetd.conf and noted that all of the default services where turned on, including chargen, echo, and time. Running find –perm – 4000 to search for files with the the setuid bit set we discovered that besides the files we expected to see, there was another interesting entry. /usr/lib/ldlibps.so -r-sr-xr-x 1 root root 26372 Sep 5 20:16 /usr/lib/ldlibps.soTaking note of the similarity between the date on this file and the date of the email posting, we decided this was worthy of more investigation. The strings command on this file produced what we took to be the contents of the ps command, which we checked against a known good machine for comparison. Execution of this file produced a process listing much as one would expect of the ps command. By using diff to compare the outputs of the two commands, we discovered more processes running than was first thought. root 190 1 0 Nov 10 ? 0:00 /usr/lib/lpsched root 254 1 0 Nov 10 ? 0:01 /usr/lib/lptd root 309 1 0 Nov 10 ? 0:00 /usr/lib/lpset -s -d 512 -i /dev/hme -o /dev/prom/sn.l root 252 1 0 Nov 10 ? 0:00 /usr/lib/lpset -s -d 512 -i /dev/hme -o /dev/prom/sn.l root 313 1 0 Nov 10 ? 0:02 /usr/lib/lptd root 311 1 0 Nov 10 ? 0:02 /usr/lib/lptdClearly the ps executable had been replaced, and the one that was run by default masked the extras the attacker had installed on the compromised system. Using the strings command on the ps executable produced the following output. /lib/ldlibps.so %s |grep -v %s|grep -v %s|grep -v %s|grep -v grep| grep -v ldlibps|grep -v -- -i sshd2 t0rnThis tells us the fake ps executable is using /lib/ldlibps.so to produce a process listing, and then the grep command to remove some of the output before it is displayed. This gave us our first modified file on the system and more items to investigate. Digging deeper We decided to investigate the /usr/lib/lpset process first, and the entry in the /dev/prom directory. The sn.1 file appeared to be the capture file of a network sniffer, which was identified as probably being a version of the popular Esniff sniffer. As can be seen by the switches after the executable in the process listing, it was configured to capture all traffic on the network interface and log it to the sn.l file. The log file is nearly 80mb in size, but appears to contain very little information of use to our attacker. We knew from the syslogd.7 log file that the mail had been sent on September the 5th, and decided to use the find command with the -mtime switch to see how many other files had been changed between the date of this mail and the current date. The experienced Unix administrators will see the flaw in using this approach, as the simple touch command can be used to alter the mtime value for a file and give it a time and date prior to the compromise, which would have neatly foiled our find command. However having noted that the attacker had failed to clean up the syslogd.7 file, and had not altered the mtime values on any of the files we had so far uncovered, this was thought to be a worthwhile exercise. We where in luck, and discovered from the output of the find command the following selection of files with date stamps around the time of the compromise: -r--rw---- 1 root daemon 134 Sep 5 09:26 /var/spool/calendar/callog.root.OTB -r--rw---- 1 root daemon 134 Sep 5 09:26 /var/spool/calendar/callog.root.CTN -r--rw---- 1 root daemon 134 Sep 5 09:26 /var/spool/calendar/callog.root.HRI -r--rw---- 1 root daemon 134 Sep 5 09:26 /var/spool/calendar/callog.root.QPP -r--rw---- 1 root daemon 3205 Sep 5 09:26 /var/spool/calendar/callog.root.KLJ -r--rw---- 1 root daemon 3205 Sep 5 09:26 /var/spool/calendar/callog.root.JNC -r--rw---- 1 root daemon 3205 Sep 5 09:27 /var/spool/calendar/callog.root.KHR -r--rw---- 1 root daemon 3205 Sep 5 09:27 /var/spool/calendar/callog.root.TNQ -r--rw---- 1 root daemon 3205 Sep 5 09:27 /var/spool/calendar/callog.root.JKD -r--rw---- 1 root daemon 3205 Sep 5 09:27 /var/spool/calendar/callog.root.ODN -r--rw---- 1 root daemon 3205 Sep 5 09:27 /var/spool/calendar/callog.root.CQS -r--rw---- 1 root daemon 3205 Sep 5 09:27 /var/spool/calendar/callog.root.APH -r--rw---- 1 root daemon 3205 Sep 5 09:27 /var/spool/calendar/callog.root.LJQ -r--rw---- 1 root daemon 3205 Sep 5 09:27 /var/spool/calendar/callog.root.DRQ -r--rw---- 1 root daemon 3205 Sep 5 09:27 /var/spool/calendar/callog.root.TRQ -r--rw---- 1 root daemon 3205 Sep 5 09:28 /var/spool/calendar/callog.root.HNMThese are the log files for the Calendar server process, which seemed excessively large given that this was not one of the functions of this machine. Listing the contents of one of the log files produced the following output. Version: 1 **** start of log on Tue Sep 5 09:26:41 2000 **** (access read "world" ) (access write ) (access delete ) (access exec ) (add "Thu Jan 1 08:00:00 1970" key: 1 what: "ªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªª ªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªª ªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªª ªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªªAnd lots more ‘a’ characters. Checking some of the many excellent archives of exploits on the internet gave us more positive evidence of the exploit, and an understanding of how the server was compromised. We concluded from the evidence and from our research that a buffer overflow of the Calendar Server daemon had occurred, probably leaving our attacker with a root shell. From here we can surmise he had sent email to a remote address with the password attached, which was then probably subjected to password cracking tools in an effort to uncover the root password. He had also downloaded other files to this now compromised server, and a note to let us know what he had done should we uncover his work. Hi Mr.System Admin, Your server was recently HACKED by X-ORG.. We patched the security hole used to hack the server, Please note, no data on your server was stolen or damaged, nor was this ever our intention. We simply installed some backdoors to permit us access to the server again. If you would like to contact us regarding any security issues or even simply for a chat, please email..XORG@mailroom.com X-Organisation. - EOF - X-ORG Members Internal release 4.0BETA-1Backdoors and other processes Checking our list of files with modified times around the time of the compromise we discovered that the /etc/inet/inetd.conf file had been altered. -r--r--r-- 1 root sys 4782 Sep 5 20:16 /etc/inet/inetd.confOur attacker had modified the inetd.conf file to prevent others from using the same means of access, but had left himself a backdoor. One of the backdoors is a modified sshd that we discovered when we listed the files that had a creation date around the time of our compromise, as shown below. -rwxr-xr-x 1 root root 264424 Sep 5 20:16 /usr/bin/sshd2 -rw------- 1 root root 525 Sep 5 20:16 /usr/bin/ssh_host_key -rw-r--r-- 1 root root 329 Sep 5 20:16 /usr/bin/ssh_host_key.pub -rw-r--r-- 1 root root 461 Sep 5 20:16 /usr/bin/sshd_configAfter reviewing the config file, we found that sshd was listening on port 13000, which we also noted by using netstat –a | grep LISTEN. Also of note is a modified binary that was installed as setuid on the system. It is assumed that it is another backdoor login containing root passwords. -r-sr-xr-x 1 root root 29512 Sep 5 20:16 /sbin/xloginWe discovered what appears to be an IRC bouncer running on this machine, which we believe to be called Psychoid Bouncer. -rwxr-xr-x 1 root root 191144 Sep 5 20:16 /var/lp/lpacct/lpacct -rw-r--r-- 1 root root 4 Sep 5 20:16 /var/lp/lpacct/psbnc.hosts -rw-r--r-- 1 root root 19 Sep 5 20:16 /var/lp/lpacct/psbnc.ini -rw------- 1 root root 6 Sep 5 20:16 /var/lp/lpacct/psbnc.pidTurning our attention to the other processes that appeared after running /usr/lib/ldlibps.so, and using the Unix truss command, we discovered that these processes sent data via UDP to an IP address registered to another organisation’s web server. They have been subsequently informed of the traffic and we will assist them with their investigations. After spending some time researching Internet documents on TFN, TFN2K and Stacheldraht, we came to the conclusion that the processes masquerading as lptd in the process listing where in fact the TFN daemon processes, and that this machine was acting as a TFN agent. Several other scripts were found on the system, presumably from a collection of common hacking tools or a rootkit. As well as the IRC bouncer mentioned above, these tools allowed the attacker to run a modified ident daemon, create DES passwords and included scripts to clean up the log files, which either had not worked or had not been used. We also discovered modifications to the startup scripts which automatically started the attackers installed scripts at boot time. Had the attacker been able to use these files to clean up the logs and modify the mtime values of the files he had installed, our investigation would have been much more difficult. Summary The attacker gained access to the machine via a vulnerable version of the Sun Calendar daemon. The calendar log files occurring on the 5th of September indicate that the calendar service was compromised with a buffer overflow. Having gained access to the system, the following changes were made. 1. Original copy of "ps" executable was moved to /usr/lib/ldlibps.so. A rootkit copy of ps was installed in /bin/ps that does not show all the running processes, thereby masking the extra processes that the attacker was running. 2. The attacker removed the vulnerable calendar service by removing the calendar server startup lines from inetd.conf. 3. The attacker installed administrative scripts in /dev/pts/01. These scripts performed the following functions. Clean log filesSome of the generated password hashes were found in /etc/ttyhash. 4. A modified SSH2 server was installed on the system, running on port 13000. It is assumed that this server would accept root logins without requiring the system root password. 5. A modified login binary was installed on the system as /sbin/xlogin. This binary was made setuid root, and it is assumed that it contains backdoor root passwords. 6. An ethernet sniffer was installed on the system as /usr/lib/lpset. This appears to be based on the esniff Unix sniffer. It was configured to start at boot time from /sbin/rc2, and logged captured traffic into /dev/prom/sn.l 7. A modified IRC server was installed on the system. This tool, identified as "psychoid bouncer" was presumably used to bounce attacks to other systems. Files relating to this were found in /var/lp/lpacct/* and /usr/lib/bnclp. 8. A modified ident server was installed on the system. This executable was placed on the machine in /usr/lib/lpsys, and started from the startup scripts. 9. A copy of the system netstat binary was made to the location /usr/lib/ldlibnet.so. The original netstat system binary was not replaced. It may have been the attacker's intention to replace the original system binary in the same way that ps was replaced. 10.A binary called "lptd" was installed on the system in /usr/lib/lptd and started at boot time. Truss output shows this binary sending data via UDP to an external IP address, then going into a wait state, awaiting response. 11.The startup files /sbin/rc2 and /sbin/rc3 were modified to start the installed trojan software. /sbin/rc2 was set to run the script /usr/lib/lpstart, which in turn started other trojan applications. Conclusion The machine was compromised through a well known vulnerability. The compromise itself did not require significant knowlege. The machine was not cleaned properly after compromise. The Calendar server logs showed that this service had been used in the compromise. There were some syslog entries from the compromise showing that sendmail had been used to send data out from the machine. File datestamps were not adjusted. The Unix touch binary allows attackers to alter the mtime value for a file. If this had been done, the intrusion would have been significantly more difficult to detect. Only one of the system binaries (ps) was rootkitted. While the lack of a reliable ps made detection difficult, a successful rootkit would have included netstat, find, ls and others. Strings information from the binaries lead us to believe that this machine was used as a TFN agent, and that the other scripts and binaries installed are standard utilities freely downloadable. There were many tags and identifying features embedded in the system binaries. This points to the conclusion that the attack came from low-grade 'script kiddie' hackers. Reference links Calendar Server vulnerabilities: http://packetstorm.securify.com/advisories/cert/CA-99-08-cmsd.txtEsniff: http://www.uha1.com/unix/unixsniff.htmlTFN: http://packetstorm.securify.com/distributed/tfn.analysis.txt http://staff.washington.edu/dittrich/misc/tfn.analysisTFN2K: http://packetstorm.securify.com/distributed/TFN2k_Analysis.htmStacheldraht: http://packetstorm.securify.com/distributed/stacheldraht.analysis http://www.sans.org/newlook/resources/IDFAQ/stacheldraht.htmSun Fingerprint Database: http://sunsolve4.sun.com/pub-cgi/show.pl?target=content/content7Man pages Online. http://www.freebsd.org/cgi/man.cgiUnix References: http://www.geek-girl.com/unix.html http://www.ussrback.com/files.html |
|||||||