Analysis: AboveNet attacks

On April 25, '00, the web-hosting company AboveNet was taken down by a "hacker" who disabled the company's network equipment. AboveNet has avoided giving out details of this attack, citing an ongoing FBI investigation. I am a customer of AboveNet; my systems have recorded extensive information related to the attacks, and it is my profession to analyze such information. This document contains my analysis of the event.

Public information can be read in the following articles:

http://dailynews.yahoo.com/h/cn/20000426/tc/abovenet_victimized_by_attack_4.html
http://www.zdnet.com/zdnn/stories/news/0,4586,2555422,00.html
http://www.zdnet.com/zdnn/stories/news/0,4586,2556074,00.html
http://www.msnbc.com/news/399753.asp
http://www.computerworld.com/home/print.nsf/(frames)/000427D962?OpenDocument&~f

What the outage was
According to AboveNet, a "hacker" took down backbone switches in the main colos. A colo (collocation) is a building where customers put their webservers. It is a better solution than locating the servers in their own building because the colo company guarantees high-bandwidth and high-availability. These "switches" are high-end Cisco boxes handling gigabits/second of traffic. All the customers in a particular colo connect through the same "aggregation" switch before going out to the Internet.

One of the articles above writes:

Vixie said the exploit had nothing to do with a defect in the switch. He said the attacker exploited a flaw in the switch's configuration management process that the company has since changed.

This rules out the most obvious form of attack: exploiting unpatched vulnerabilities in vendor equipment. Such defects are regularly found in all networking products. Customers don't apply the fixes as they should, leaving the systems open for attack. There is a list of about 20 defects for the Cisco equipment listed at http://www.securityfocus.com/ (among a thousand more for other products, so you shouldn't interpret this as meaning Cisco is more vulnerable than anybody else). Several of these defects are "topical", so my normal inclination is to assume that the victim was late in applying the recommended fixes. However, AboveNet claims that this was not the case in the attack. They are instead saying that the intruder exploited a security hole left open by their own people.

Case Study: SNMP
The most obvious way that the intruder could have penetrated the system was through SNMP (Simple Network Management Protocol). AboveNet uses this protocol extensively in order to manage their equipment. However, SNMP is well known for its security deployment problems. Two common faults with deploying SNMP are to leave the default passwords of "public"/"private" enabled and accepting commands from the Internet (instead of just from local machines).

These problems are extremely common. Elite hackers have used this knowledge for years to play around with ISP equipment, such as the amusing feat of telling the ISP to hang up on a dial-up user. Unlike the February DDoS attacks, this type of SNMP hacking isn't widely done by script kiddies. This technique was recently re-publicized, which makes it a prime candidate. The main reason why this technique isn't used more often is the difficulty in mapping out the ISP looking for equipment to play with.

In the case of AboveNet, they actually tell everyone the IP addresses of their switches. They post to their website map the current status of all their equipment and Internet connections. They essentially publicize where to find the equipment and classify it in a well-known category of attacks that might bring it down.

Here is a possible scenario. Somebody read the recent postings about the SNMP vulnerability so it was fresh in their mind. While surfing AboveNet's site, they stumbled across the SNMP traffic graphs with the IP addresses labeled at the top. They opened their handy SNMP manager, typed in the IP addresses and "poof" found they could log in. They then walked through the configuration options until they found the item that would disable the machine.

There are also more subtle ways of hacking SNMP. I run a website hosted by AboveNet that runs an intrusion detection system (IDS). A month ago, the IDS triggered an alert from a network management system that tried to log into my machine using the password "AB-001". Even if AboveNet had correctly changed their password from "private" to "AB-001", they carelessly told me the new one.

According to AboveNet's comments, something similar may have happened with Telnet, HTTP, or T/FTP. In all likelihood, the technique used probably wasn't well known in the script kiddy world, but well known to anybody who maintains such equipment.

Broadcast Domains
Like most colos, AboveNet puts their customers on shared broadcast domains. While the switches isolate customers from seeing each other's primary traffic, it forces customers to see each other's secondary broadcast traffic. I host a site at AboveNet. I use the strong word "force" because not only does AboveNet force this traffic onto my machine, it appears that they are also billing me for it.

This is extraordinarily dangerous. If one customer gets hacked, the intruder can then use that machine as a base of operations to hack other machines. Network technologies are not protected against attacks across the local broadcast domain. It affords the hacker new ways of attacking machines that would not otherwise be possible remotely from the Internet.

By analyzing the traffic that AboveNet forces on my machine, I see the following interesting possible items:

• ARP poisoning and ARP spoofing can be used to tell the upstream router to redirect traffic. This means one customer can claim to be another customer's website, or even sit between the user and website in order to steal passwords and credit cards while relaying the traffic.
• ICMP redirects providing much the same access.
• Responding to the numerous DHCP and BOOTP requests in order to misconfigure other customer's devices. Of course, I'm assuming the DHCP/BOOTP broadcasts are actually from other customers and not from AboveNet (in which case, the situation would be even more disturbing).
• Connecting to the numerous WinNT servers who are broadcasting the availability of NetBIOS on private (192.168.x.x) addresses. Again, this is an attack that only works locally.
• A major website (as listed in http://www.mediametrix.com/Top500/Top500.html) uses WinNT clustering. I can bring down the cluster with my own broadcasts or possibly insert myself into the mix.
• Several nearby servers are announcing a likely vulnerability in their "Insight" manager program.
• One customer is apparently announcing that it is the "default router", which could confuse other people into relay their traffic through it.
• Another customer is trying to use SNAP/LLC encoding rather than standard DIX. I could probably exploit this and relay their traffic.

At minimum, I could bring down any of these machines by flooding them with extremely high rates of traffic (i.e. 148,800 SYNs/second) . The February attacks needed to be "distributed" because that was the only way to flood traffic at those high rates. A LDoS (Local DoS) is much easier than a DDoS.

Op-ed
I spend a lot of time in colos. I can tell you from first hand experience that colo network security is practically non-existent. I know of at least one major competitor to AboveNet who I can switch off with a touch of a button (not my own discovery, somebody else's). The situation in the industry is truly frightening. What happened to AboveNet can happen to any other colo at any time. My decision to host at AboveNet partially reflects my belief that more secure than others, but you should interpret this as meaning that I find them slight less lax/careless.

Here is a guide of some things that colos need to fix in their infrastructures:

• Turn off Internet access to their network equipment. Right now, I can ping the switch hosting my website from anywhere on the Internet. This is unacceptable; the equipment should only be reachable from the company's NOC (Network Operations Center).
• Restrict each customer to his/her own VLAN on their colo switches. This feature is built into their equipment already. This will isolate customers from each other so that they will have no more access to each other than anybody else from the Internet.
• Obtain regular audits from security companies. This has been standard practice in financial Accounting for eons, and should be used in security. Publish the name of the security auditors. If you get hacked, rake the auditor's name through the mud. Right now I'm finding gazillions of security holes just by analyzing the traffic AboveNet shoves down my throat; imagine what real auditors could find if they actively probed systems and reviewed designs.
• Install intrusion detection systems (IDS) monitoring network traffic. Of course, since I work for a company that builds what I believe to be the highest performing such system, I'm highly biased on this issue.

• Stop billing me for the broadcast traffic from other customers in the same building. This is carelessness bordering on the fraudulent. [NB: According to how I read the documents, AboveNet bills according to the greater of inbound vs. outbound traffic, and the inbound traffic is about 10% higher than our outbound, and consists mostly of local colo traffic. The machine in question receives millions of hits per month, so the amount of traffic we are talking about isn't small potatoes.]

Conclusion
AboveNet's SJ2 facility (where my equipment is located) is impressive. Their bandwidth is extremely high with redundant links. Their physical security is fairly tight with bank-like paranoia. They get their electricity from two different grids, then cleanse it with their own generators. In the off chance the Big One hits, they have commitments from diesel providers that will keep their facility up and running for months.

In contrast, their network security is nearly non-existent. True, it is slightly better than "industry norms", but industry norms suck right now. I expect that other colos will start experiencing the same problem as AboveNet until they start addressing the problems outlined above.

[April 29, '00]
...and another thing, the comparison with the February attacks is meaningless. The press loves the angle that mortals cannot defend themselves against hacker magic. The reality is that people don't pay much attention to security, and hackers find it laughably easy to penetrate the holes left open. Imagine burlgers laughing about how they rob houses by looking under the front mat for a key, and why don't these bozos get a clue and stop putting keys under mats? The DDoS attacks were an exception; generally attacks like AboveNet are something simple like leaving your door key under your mat. If you haven't, you ought to read my op-ed on the DDoS attacks.

[May 01, '00]
David Meissner points out that some ISPs allow customers to see each other's Apache access logs of all its other customers. This is pretty nasty, especially since you can often see authentication information within the logs.

[April 30, '00]
Ryan Russell has some good comments. First, he notes that as an amateur locksmith, the locked cages don't impress him (I would also point out that amateur gymnasists wouldn't find climbing over the cages difficult, and indeed I've seen their own NOC people climbing around like monkeys :-). I've seen no colo with better physical security, but they don't match the CIA, NSA, or a Swiss bank. Second, Ryan notes that there are ways around VLANs, many of which are listed on SecurityFocus.com. As I mention in my Sniffing FAQ, while Ethernet switches provide "soft" security, they cannot replace "hard" security. (i.e. your should enable VLANs, but don't assume that they can't be hacked).

[May 01, '00]
Lance Spitzner writes to tell me that the current rumor on NANOG is that the all the equipement shared the same password for Telnet access. The rumor is that this password was widely known, even by the sales people. The additional rumor is that the password was sniffed by MafiaBoy (the suspect in the DDoS attacks), but I seriously doubt it. Though sniffing such passwords is easy, the problem is interposing yourself at the right location. (Occam's Razor suggests that if even the sales guys know the answer, then it was unlikely to have been sniffed). The rumored password is "whY2Ghay/1Pee-Fr331y".

[May 03, '00]
Further rumors are that the intruder expoloited the bug described in Cisco bug ID CSCdr10025. As I've described above, I've never seen an intrusion technique that wasn't already published on lists like BUGTRAQ. One approach that a company can use to make their customers more comfortable is to post their fixes. If Cisco comes out with a patch, then the colo should patch their equipement and send a message to their customer list saying "We applied vendor patch X on our equipement on date X". Peer review is a powerful force to make things better and often overcomes the risks associated with exposing information.

[May 03, '00]
Cisco notes, as do I, that you need to turn off external access to your equipement. With Catalyst switches, use the command:

set ip permit <address> <mask> telnet
set ip permit enable