HIS.com Status: 1/24/2007 outage

[HIS System Status Updates]

As you undoubtedly know, we had an outage on Wednesday, 1/24, that  
completely or at least partially blocked access to mail.his.com and  
many of the web sites that we host.  Dialup access was also affected.

The cause of the problem was an attack on our central router at our
Kensington, MD facility using an exploit on Cisco IOS that Cisco
announced later that day (at 1 PM ...). The attacker was able to
download and run a script on one of our utility linux servers, using a
PHP script vulnerability.  The hack didn't involve a root exploit, but
unprivileged access was enough, and a high-bandwidth attack was executed
on an internal server port.

The attack caused the router to stop routing, then reboot, and because
we didn't know about the vulnerability we initially diagnosed the
problem as a hardware failure and replaced the router; this didn't help,
since the attack continued from its internal source.

We're extremely careful to stay current with Cisco IOS updates to
protect against this kind of problem, but this was a 'zero day' attack
that started about 4 hours before Cisco announced the vulnerability.

All of our routers are now up to date with IOS versions that are not
vulnerable to the exploits announced on Wednesday, and the utility
server has been taken offline until we can do a full analysis of its
security issues.

For the technically curious, there's more information on the Cisco  
vulnerability at these URLs:

http://blog.washingtonpost.com/securityfix/2007/01/time_to_reboot_the_internet_ag.html
http://www.internetnews.com/security/article.php/3656131
http://www.theregister.com/2007/01/25/cisco_ios_bug_fix/
http://isc.sans.org/diary.html?storyid=2097
http://www.us-cert.gov/current/current_activity.html#cisco0107

-- 
Paul Heller
ph at his.com