INTRODUCTION | ||
---|---|---|
On Nov 25, 2000 two machines on our network were
compromised during one attack.
It appears the Suspect was searching specifically for Redhad 7 machines.
After a scan of our network to port 21, two machines were found to be running
Redhat 7.
This document presents the entire picture of the incident. Most of the information in this document refers to Victim#1, however Victim#2's compromise was identical (except where noted) The information is organized into three sections.
|
||
THE CAST | ||
The Bad
|
|
BadGuy IP#1 (205.177.91.2)
|
|
BadGuy IP #2 (210.94.114.48 ) | |
|
BadGuyIP #3 | |
|
BadGuyIP #4 | |
The Good
|
|
Victim#1 |
|
Victim#2 | |
THE LOGS | ||
SnortLogs
Snort was running inside the network on the backbone segment. The machines which were compromised were on 2 different segments of the network. The information analyzed by snort is therefore limited. This information offers detailed information on the network scans. TCPDump Logs
Firewall Logs
StartTime, EndTime, SrcIP, DstIP, SrcPort, DstPort, service, ConnectionDuration,PktsSent, PktsRcvd, FirewallActionHost (Victim) Logfiles Logfiles gathered from each compromised system provides information on the method of attack. Most importantly, they provide the information needed to understand the events which triggered various entries in the network logs. |
The Recon/Scan | |
OverviewObservations
|
|
Nov 5, 2000 - SNORT Portscan.logNov 5 13:14:07 205.177.91.2:1809 -> X.X.128.6:98 SYN **S*****Nov 5 13:14:07 205.177.91.2:1807 -> X.X.128.4:98 SYN **S***** Nov 5 13:14:07 205.177.91.2:1808 -> X.X.128.5:98 SYN **S***** Nov 5 13:14:08 205.177.91.2:1830 -> X.X.128.27:98 SYN **S***** Nov 5 13:14:08 205.177.91.2:1831 -> X.X.128.28:98 SYN **S***** Nov 5 13:14:08 205.177.91.2:1832 -> X.X.128.29:98 SYN **S***** Nov 5 13:14:08 205.177.91.2:1919 -> X.X.128.51:98 SYN **S***** Nov 5 13:14:10 205.177.91.2:2074 -> X.X.128.112:98 SYN **S***** Nov 5 13:14:10 205.177.91.2:2075 -> X.X.128.113:98 SYN **S***** Nov 5 13:14:10 205.177.91.2:2190 -> X.X.128.134:98 SYN **S***** Nov 5 13:14:10 205.177.91.2:2076 -> X.X.128.114:98 SYN **S***** Nov 5 13:14:12 205.177.91.2:1947 -> X.X.128.79:98 SYN **S***** Nov 5 13:14:12 205.177.91.2:2356 -> X.X.128.188:98 SYN **S***** |
|
Nov 25, 2000 - SNORT Portscan.LogNov 25 14:02:53 205.177.91.2:2654 -> X.X.128.4:23 SYN **S*****Nov 25 14:02:53 205.177.91.2:2656 -> X.X.128.5:23 SYN **S***** Nov 25 14:02:53 205.177.91.2:2664 -> X.X.128.6:23 SYN **S***** Nov 25 14:02:53 205.177.91.2:2701 -> X.X.128.27:23 SYN **S***** Nov 25 14:02:54 205.177.91.2:2762 -> X.X.128.60:23 SYN **S***** Nov 25 14:02:54 205.177.91.2:2794 -> X.X.128.70:23 SYN **S***** Nov 25 14:02:54 205.177.91.2:2807 -> X.X.128.81:23 SYN **S***** Nov 25 14:02:54 205.177.91.2:2813 -> X.X.128.79:23 SYN **S***** Nov 25 14:02:54 205.177.91.2:2808 -> X.X.128.80:23 SYN **S***** Nov 25 14:02:54 205.177.91.2:2817 -> X.X.128.76:23 SYN **S***** Nov 25 14:02:55 205.177.91.2:2945 -> X.X.128.131:23 SYN **S***** Nov 25 14:02:55 205.177.91.2:2949 -> X.X.128.132:23 SYN **S***** Nov 25 14:02:55 205.177.91.2:2951 -> X.X.128.134:23 SYN **S***** Nov 25 14:02:55 205.177.91.2:2952 -> X.X.128.133:23 SYN **S***** Nov 25 14:02:56 205.177.91.2:3052 -> X.X.128.188:23 SYN **S***** Nov 25 14:02:56 205.177.91.2:3104 -> X.X.128.210:23 SYN **S***** Nov 25 14:02:57 205.177.91.2:3125 -> X.X.128.219:23 SYN **S***** Nov 25 14:02:57 205.177.91.2:3119 -> X.X.128.225:23 SYN **S***** |
|
Nov 25 - SNORT ALERT LOG11/25-14:09:16.266354 0:0:EF:3:80:F0 -> 0:0:EF:3:4D:60 type:0x800 len:0x4A205.177.91.2:1173 -> X.X.210.38:23 TCP TTL:51 TOS:0x0 ID:36580 DF **S***** Seq: 0x1D02DE67 Ack: 0x0 Win: 0x7D78 TCP Options => MSS: 380 SackOK TS: 17364876 0 NOP WS: 0 =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 11/25-14:03:17.411780 0:0:EF:3:80:F0 -> 0:10:4B:2A:F1:14 type:0x800 len:0x4A 205.177.91.2:1548 -> X.X.133.7:23 TCP TTL:51 TOS:0x0 ID:59620 DF **S***** Seq: 0x654D3B7 Ack: 0x0 Win: 0x7D78 TCP Options => MSS: 380 SackOK TS: 17328985 0 NOP WS: 0 =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 11/25-14:03:17.423929 0:0:EF:3:80:F0 -> 0:50:DA:2C:12:9A type:0x800 len:0x4A 205.177.91.2:1562 -> X.X.133.15:23 TCP TTL:51 TOS:0x0 ID:59634 DF **S***** Seq: 0x62F807C Ack: 0x0 Win: 0x7D78 TCP Options => MSS: 380 SackOK TS: 17328986 0 NOP WS: 0 =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 11/25-14:03:18.374003 0:0:EF:3:80:F0 -> 0:1:2:29:8B:DD type:0x800 len:0x4A 205.177.91.2:1630 -> X.X.133.48:23 TCP TTL:51 TOS:0x0 ID:59748 DF **S***** Seq: 0x622515D Ack: 0x0 Win: 0x7D78 TCP Options => MSS: 380 SackOK TS: 17329085 0 NOP WS: 0 =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ 11/25-14:03:18.413957 0:0:EF:3:80:F0 -> 0:80:C8:1E:11:E1 type:0x800 len:0x4A 205.177.91.2:1640 -> X.X.133.52:23 TCP TTL:51 TOS:0x0 ID:59761 DF **S***** Seq: 0x684A0BC Ack: 0x0 Win: 0x7D78 TCP Options => MSS: 380 SackOK TS: 17329090 0 NOP WS: 0 |
|
Nov 25 - TCPDUMP14:02:53.601174 0:0:ef:3:80:f0 0:a0:36:0:85:cb 0800 74: 205.177.91.2.2654 > X.X.128.4.23: S 69699773:69699773(0) win 32120 <mss 380,sackOK,timestamp 17326609 0,nop,wscale 0> (DF)0x0000 4500 003c dd82 4000 3306 2a85 cdb1 5b02 E..<..@.3.*...[. 0x0010 5858 8004 0a5e 0017 0427 88bd 0000 0000 .....^...'...... 0x0020 a002 7d78 949f 0000 0204 017c 0402 080a ..}x.......|.... 0x0030 0108 6211 0000 0000 0103 0300 ..b......... 14:02:53.615573 0:0:ef:3:80:f0 0:a0:36:0:81:cb 0800 74: 205.177.91.2.2656 > X.X.128.5.23: S 58155330:58155330(0) win 32120 <mss 380,sackOK,timestamp 17326609 0,nop,wscale 0> (DF) 0x0000 4500 003c dd84 4000 3306 2a82 cdb1 5b02 E..<..@.3.*...[. 0x0010 5858 8005 0a60 0017 0377 6142 0000 0000 .....`...waB.... 0x0020 a002 7d78 bcc7 0000 0204 017c 0402 080a ..}x.......|.... 0x0030 0108 6211 0000 0000 0103 0300 ..b......... 14:02:53.635891 0:0:ef:3:80:f0 0:a0:36:0:84:32 0800 74: 205.177.91.2.2664 > X.X.128.6.23: S 63619730:63619730(0) win 32120 <mss 380,sackOK,timestamp 17326610 0,nop,wscale 0> (DF) 0x0000 4500 003c dd8c 4000 3306 2a79 cdb1 5b02 E..<..@.3.*y..[. 0x0010 5858 8006 0a68 0017 03ca c292 0000 0000 .....h.......... 0x0020 a002 7d78 5b1a 0000 0204 017c 0402 080a ..}x[......|.... 0x0030 0108 6212 0000 0000 0103 0300 ..b......... 14:02:53.750509 0:20:48:58:2:af 0:0:ef:3:80:f0 0800 62: X.X.128.23.23 > 205.177.91.2.2695: S 703593917:703593917(0) ack 55667634 win 8192 <mss 512,nop,wscale 0> 0x0000 4500 0030 a17c 0000 4006 9984 5858 8017 E..0.|..@....... 0x0010 cdb1 5b02 0017 0a87 29ef fdbd 0351 6bb2 ..[.....)....Qk. 0x0020 7012 2000 86ad 0000 0204 0200 0103 0300 p............... 14:02:53.865766 0:0:ef:3:80:f0 0:50:4:61:85:98 0800 74: 205.177.91.2.2701 > X.X.128.27.23: S 68115308:68115308(0) win 32120 <mss 380,sackOK,timestamp 17326627 0,nop,wscale 0> (DF) 0x0000 4500 003c ddb4 4000 3306 2a3c cdb1 5b02 E..<..@.3.*<..[. 0x0010 5858 801b 0a8d 0017 040f 5b6c 0000 0000 ..........[l.... 0x0020 a002 7d78 c1b0 0000 0204 017c 0402 080a ..}x.......|.... 0x0030 0108 6223 0000 0000 0103 0300 ..b#........ |
|
Nov 25 - FIREWALL LOGS(Format: StartTime, EndTime, SrcIP, DstIP, SrcPort, DstPort, service, ConnectionDuration in seconds,PktsSent, PktsRcvd, FirewallAction)14:01:37 14:02:54 205.177.91.2 X.X.128.10 2674 23 telnet 0
0 0 Deny
|
The Exploit | |
The SYN scan produced
2 vulnerable machines. Both were RedHat 7 Linux.
Victim #1 is X.X.8.60; Victim #2 is X.X.167.56 This log is from the firewall application. I have numbered each line of the logfile and reference these line numbers in the explaination that follows. 1. 14:04:08 14:37:57
205.177.91.2 X.X.159.254 4940 23 telnet 1949 624 0 Permit
Explaination of Firewall LogfileLine 1. The last scan entry from the firewall log indicates the connection closed at 14:37:57.Line 2. A telnet connection is made to Victim#1 from an BadGuyIP#2. Line 3,4. BadGuyIP#1 connects to LPR port (515) AND TCP 3879. Line 5-30. Followed by a series of connections alternating between port 515 and 3879. Line 40. We have root! Line 41. Notice the last connection to TCP515 is still active when a connection from Victim#1 port 1023 to BadGuyIP#3 port 514. Line 42-46. There are a few icmp attempts from BadGuyIP#2. Could he be testing to find out if our network allows incoming icmp? (We don't.) Line 47. BadGuy uses ICMP replies. (There are NO requests)
Line 49-51. A couple more outgoing ICMP connections. These become regular events from both Victim's over the next few days. Line 50, 52. But now there is something really odd. Two isolated incoming connection from BadGuy IP#2 to a different machine on our network (X.X.98.224). Line 53-58. Now we begin the attack on Victim#2. Observation: |
Victim#1 - LogFiles | |
/var/log/messages
1. Nov 25 14:39:09 Victim#1 SERVER[1616]: Dispatch_input: bad request line 'BBìóÿ ¿íóÿ¿îóÿ¿ïóÿ¿XXXXXXXXXXXXXXXXXX000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000048secursecurity000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000001074433944~P ~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P ~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P ~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P ~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P ~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P~P1Û1É1À°FÍ~@ ~Iå1Ò²f~IÐ1É~IËC~I]øC~I]ôK~IMü~MMôÍ~@1É~IEôCf~I]ìfÇEî^O'~IMð~MEì~IEøÆEü^P~IÐ~ MôÍ~@~IÐCCÍ~@~IÐCÍ~@~IÃ1ɲ?~IÐÍ~@~IÐAÍ~@ë^X^~Iu^H1À~HF^G~IE^L°^K ~Ió~MM^H~MU^LÍ~@èãÿÿÿ/bin/sh'<clipped> 2. Nov 25 14:42:55 Victim#1 SERVER[2393]: Dispatch_input: bad request line 'BBàóÿ ¿áóÿ¿âóÿ¿ãóÿ¿XXXXXXXXXXXXXXXXXX000000000000000000000000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000004800000001074433944security000000000000000000000000000000 0000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000001074401432^P ^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P ^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P ^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P ^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P ^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P^P1Û1É1À°FÍ^@ å1Ò²f Ð1É ËC ]øC ]ôK Mü^MMôÍ^@1É EôCf ]ìfÇEî^O' Mð^MEì EøÆEü^P Ð^MMôÍ^@ ÐCCÍ^@ ÐCÍ^@ Ã1ɲ? ÐÍ^@ ÐAÍ^@ë^X^ u^H1F^G E^L°^K ó^MM^H^MU^LÍ^@èãÿÿÿ/bin/sh'3. Nov 25 14:46:14 Victim#1 kernel: lpsched uses obsolete (PF_INET,SOCK_PACKET) 4. Nov 25 14:46:14 Victim#1 kernel: eth0: Setting promiscuous mode. 5. Nov 25 14:46:14 Victim#1 kernel: device eth0 entered promiscuous mode 6. Nov 25 14:50:00 Victim#1 CROND[2725]: (root) CMD ( /sbin/rmmod -as) 7. Nov 25 15:00:00 Victim#1 CROND[2737]: (root) CMD ( /sbin/rmmod -as) 8. Nov 25 15:01:00 Victim#1 CROND[2740]: (root) CMD (run-parts /etc/cron.hourly) 9. Nov 25 15:03:48 Victim#1 PAM_unix[2747]: authentication failure; (uid=0) -> root for system-auth service Line 1. This is the first entry of the buffer overflow attack. Even though I do not have the tcpdump of this attack, I can translate the characters to hex and confirm the exploit used was almost identical (if not the same) as the one posted on SANS GIAC Jan 22, 2001 by Chris Talianekte. The source code which I believe executed this attack includes identical hex code including the 0x0f27 which Chris suggests might be a clue to the odd alternating port connections during the buffer overflow. I am not a C programmer, however the source code does specifically refer to ports htons(515) and htons(3879). char shellcode[] = "\x31\xdb\x31\xc9\x31\xc0\xb0\x46\xcd\x80"
Again, the times recorded in this messages logfile were from Victim#1. The time difference in the various machines makes it difficult to synchronize the logfiles between machines. There were 682 entries between 14:39:09 and 14:42:55 in the /var/log/messages
file similar to this entry.
Line 3-5. This entry was made by the application lpsched .
|
|
/var/log/secure
1. Nov 25 13:54:04 Victim#1 xinetd[470]: START: telnet pid=1594
from=205.177.91.2
Line 1,2. The original scan connection which occured during the full network scan from BadGuyIP#1. Line 3. This is the telnet connection which immediately preceeded the attack. Observation:Line 4. This is interesting! We have an incoming telnet connection from Victim#2 TO Victim#1. This connection is also recorded on both victim machines in the BadGuy'ssniffer file on Victim#2. It is not recorded in Victim#1's sniffer file. |
|
Victim#2
- Beginning of captured sniffer file.
============================================ Victim#1- Begining of captured sniffer file ======================================= |
|
/var/log/maillog
Victim#1 Nov 25 14:46:15 victim1 sendmail[2717]: NOQUEUE: SYSERR(root): /etc/sendmail.cf:
line 108: readcf: map arith: class arith not available
Victim#2 Nov 25 14:53:03 victim2 sendmail[4611]: NOQUEUE: SYSERR(root): /etc/sendmail.cf:
line 108: readcf: map arith: class arith not available
|
|
The Exploitt | ||
I believe the attacker
exploited the vulnerability in LPRng, however I'm still unclear about what
part the connections to port 3879 had in the exploit. I've downloaded
several different exploit code for LPRng and have not been able to reprod
LPRng - The full code is available on BugTraq. /*
|
||
The Discovery/Compromise | ||
After compromise: [Snort Alert log]
[TCPdump of the above ICMP packets]
Monday, Nov 27, 2000 (9:30am) : I noticed a few scans in the weekend logfiles. The scans originated outside of our network. Scans are common in our environment (dot.edu). Normally I check a few other logs for signs of intrusion and if I find nothing, I simply make a few notes of the event. However my antena's were active and I had an odd feeling about these scans. With no evidence at hand, I decided to send an email message to several of our System Administrators and to the Managers of various System Administrators on our campus. In this message mentioned the scans and reminded them of the increased dangers during the Christmas season and encouraged them to take extra time to secure their servers. Wednesday, Nov 29, 2000 (5pm): An employee casually stops at my office door on his way out and asks if I know "any reason why 'ls' would stop working". Anyone reading this now will immediately recognize the clue. But
I've seen new unix administrators do some pretty weird things to their
machines. I kept the idea of a rootkit in the back of my mind while
I asked more questions.
Q: "What do you mean stopped working?"Now I've heard enough to make me curious. I ask him to show me what's happening. We walk to his office and he logs into his console with his nonpriviledged account. At the system prompt, he executes 'ls'. The system prompt returned, just as he had said. There was no listing. Me: "Do a 'pwd'"He is in his home directory Me: "Execute 'ls -al'"Again, the prompt returns promptly. He su's to root and excutes 'ls' again. This time there are files listed. Me: "Do a 'pwd'"He is in the same directory as before. Me: "Execute 'ls -al'"Afull listing returns. I verify the permissions are set correctly on . and .. Me: "Go back to the non priviledged account."He returns to his normal user login Me: "Type 'which ls'"/bin/ls returns. Me: "Type 'stat *'"We get the full status information on all the files in his directory. This means his account does not have a problem reading the directory information. Now he tries to 'ls' other directories, all result in the some return... nothing. About this time a Sr Unix Administrator walks by. I seize the opportunity and seek his insight. He followed the same path I had. Same questions, same commands. He also check the /dev/<filesystem> permissions and kmem permissions. These last items were not really expected to produce answers. We were now looking for correlation. The Sr Admin sat at the keyboard. The "new" Admin watched with bated breath, another unix/programmer had joined us. I was almost at peace with letting the Sr. Admin solve the mystery when I heard it. Five words spoken softlly, almost as if they were still in thought.. "PS does the same thing." There was a low hum as the gasps filled the air. I hear my name "Maarreeee". Then in almost a whisper the programmer brings me back to reality as he declared aloud what was now almost certain,"Rootkit!". The Sr. Admin and the programmer return to their offices. The new (now most surely VICTIM) admin gave me the root password to his machine, expressed regrets (he really did want to stick around for the show, but had other committments ), and went home. I logged into the console with the root account. Even though we have strong indication the system has been compromised, there has been no proof. It is getting late and I've been doing this too long to want to stay up all night working on this. My objective for the night is to find evidence the system has been compromised and remove any threat or danger the machine or our network may currently be in (in otherwords, verify the hack and quarantine the machine) At this time, I am still using commands which are on the suspect-machine. I know this is not the preferred method, but I wanted solid proof before I spent too much time working on this. My first thought is to capture network traffic going to or from the victim machine. I asked the Sr. Admin if he had any servers on the same segment of our
network as the Victim machine. He had serveral unix servers running
on that segment.
While he was installing tcpdump, I looked though the systems logfiles in /var/log. As I expected, the syslog.conf was running with 'default' settings. This means it will have very little logging. By viewing the /etc/syslog.conf file I can tell which files might possibly have logged interesting events. This information gets me started. Console#> cd /var/logThe command 'ls -alt' sorts the files by time modified with the most recently modified first. Console#> more /var/log/secureThis is where I noticed an unusual telnet connection from another machine on our campus. Still, this is not evidence of an attack or an intrusion, but I do make note of the date and time the connection occurred. Console#> more /var/log/messagesNothing unusual here. I notice the log was rotated on Nov 26 via cron. I remember the telnet connection from the/var/log/secure file which occured on Nov 25. I decide to check the previous message log file. Console#> more /var/log/messages.1BINGO! That didn't take long. After reading 1488 boring lines I found an entry where someone was attempting to exploit a buffer overflow. Now I have proof that someone TRIED to break into the machine. I still do not have proof they succeeded. I continued paging through the messages.1 file. There was line after line of the attempt. There were 811 lines in the logfile of the buffer overflow code. It seemed like forever before I reached the end of it. Finally, at the end of the buffer overflow entries was a log message indicating the ethernet interface was entering promisuous mode. I checked the ifconfig -a output and verified that promisuous mode was
currently enabled.
From another machine, I scanned Victim#1 with nmap.
(The 65517 ports scanned but not shown below are in state: closed)Port 15000 grabbed my attention. I telneted to the port (telnet victim1.edu 15000) and recieved a ssh banner. I suspect a backdoor sshd has been installed, however I cannot verify
this until I talk to the system owner (he's been known to do unusual things
while testing applications)
Now it is decision time. Do we unplug the machine from the network, or ride along for awhile. Normally I would have installed tcpdump on a neighboring machine to see what might be happening, but the Sr. Admin had not yet been able to install tcpdump. It was almost 7pm and my dinner was getting cold. I decided to unplug the machine from the network and go home. Before leaving the office, I blocked all access to and from the machine at the firewall. This was done to prevent the machine from being accidentally plugged back into the network. I spent the evening thinking about how to proceed the following day. One item which bothered me was the telnet connection to the victim machine from one of our machines. There was a possibility other machines were compromised also. DAY 2:
I made a list of the files I would need to review the damage on the machine (ls, find, ps, lsof, etc) and copied them from another linux server via floppy disk (remember, the machine is still unplugged from the network). I logged into the console and created a directory to copy my good files into. The login process took much longer than normal and triggered my suspicions. Console#> /gooddir/ls -al /sbin/login
Console#> /gooddir/find / -group lp -ls 47710210 0 dr-xr-xr-x 3 lp/lp
0 Dec 1 13:13 /proc/728
There are two ways to identify the rootkit files from the above output. First, they aare all dated Nov 25, 14:53 (Note: The listing used here was created for this report after the initial investigation was complete. The rootkit directory may have been modified during my investigation), which reflect Second, they are owner root, group lp. It is important to notice that there are also some legitimate files which are root/lp. These can be distinquished by their dates, however I did compare each file with an existing (uncompromised) Redhat 7 server. Using the above criteria, the list of files is narrowed down to the following: 202991 0 -rw------- 1 root/lp
0 Nov 25 14:53 /dev/dos
It's now pretty clear where the rootkit is installed; (/lib/security/.config).
At this point in the investigation, it is easy to lose focus. There is so much to look at! However, my priority is to determine the severity of this attack and if it has placed additional servers at risk. I know the ethernet card is in promiscuous mode (even though it is unplugged
from the wall).
./lpsched is runing.
I discover userid's and passwords in the output file. To determine if this data has been retrieved, I return to my firewall logs and start analyzing. I locate outgiong ICMP replies from victim #1 with no requests. I searched my tcpdump files and discovered I had captured the outgoing packets! I discovered the destination IP address was different from the attacking machine. I searched my logs for this new IP and located identical packets from Victim#2. There was no other traffic recorded for this IP (Badguy IP#4). I contacted the owner of this IP Address block and was informed that the subnet it was allocated to was a Dorm, however this particular address was not issued. I recieved a call from the owner of Victim#2. I explained the situation and asked him to search his machine for the rootkit directory (/lib/security/.config). THe directory was there. Victim #2 has now been confirmed. To create a complete picture I continued analyzing the logs.
I repeated the same procedure above but with Victim#2.
I returned to Victim#1 and archive the rootkit directory. I create a directory structure off of /goodir and mv trojaned files to the corresponding directory. I used mv to prevent the filestats from changing. Then I cp'd the /gooddir/<goodfilse> to the correct location. I began analyzing each file in the rootkit directory. I was fortunate to have the uncompromised Redhat box in the next office. This allowed me to quickly determine if the file was compromised. A few days into the analysis process it became clear the machine would
require a full reinstall of the operating system.
|
||
Rootkit Directory | ||
The rootkit is installed in a newly
created directory named /lib/security/.config .
drwx------ root/lp
0 2000-11-25 14:46:14 ./.config/
|
||
|