Thu, 18 Feb 2010
Tripping Myself Up
Caught with My Pants Down
At about 4:15 yesterday afternoon I received an unusal phone call. The guy at the other end didn’t identify himself at first. He just asked me if I was the site admin for eldoradotech.org, which I am. He then somewhat murkily explained that he’d received a phishing email from a third party. Further, the email linked to my web site. No big deal, except that my web site was in fact serving up a page that looked just like a JPMorgan Chase login page. Not good.
After determining that there was a problem, I immediately deleted the
unauthorized files from the server and then shut it down. Unfortunately, this
resulted in all of my web sites, email, and other services being unavailable,
which is a hassle for my millions thousands legions hundreds scores
dozens of two friends and fans. However, it was necessary because
whoever had put those files on my server the first time could always put them
back a second, and could further exploit not only that server, but the other
servers in my gleaming, high-tech ghetto basement datacenter
as well as the desktops and laptops around the house.
Fortunately, my workday was largely over; so I rushed out to my car at 5:05 to get home and check things out. All the way there I considered the many possible vectors an attacker might have used to break in to my server. Some of them are difficult and unlikely, while others would simply require access to my password. I generally keep close tabs on my password, but you can never rule out some kind of a slip-up. This is why changing your password regularly is a good idea, even if it is a pain in the ass.
During transit I also thought about the varying consequences of the attack: if it was just the one system, damage could be limited. However, if the attacker had been in the system a long time before using it for nefarious purposes, he or she might have logged other passwords, confidential business information, financial records, or other valuable data. This worried me.
When I arrived at home, I first turned off all the other computers in the house. That’s three other servers, three desktops, and two laptops, currently. This was to prevent the attacker from using one of them to re-infect the first system if he or she was already loose on my network.
After this first step, I booted up the infected server from a clean Knoppix CD image and analyzed the logs. It looks like two IPs were running a dictionary attack against a weakly-passworded mythtv user account:
58.177.188.213
172.173.83.246
Neither IP is responding to ping, now.
The attacker appears to have gained access to the brand-spanking-new mythtv account (no this server wasn’t being used for MythTV, but I keep accounts synchronized across my hosts to keep things simple) and then used a privilege escalation exploit to create a new user, ‘ftpd’. Then the attacker gave the new ftpd account a UID of ‘0’ (essentially, the same access level as root). From there, it was all down hill.
Logs don’t always tell the truth, because they can be edited, deleted, or corrupted. Having something to track back through was nice, but it’s not sufficient. Because for all I know the attacker was leaving a false trail, I elected to nuke the site from orbit. It’s the only way to be sure. So, for remediation, I wiped the system and reinstalled the OS and applications from known clean sources, removed the unauthorized ftpd account, changed passwords left-and-right, then restored user data from my latest backup.
I’m lucky that this was all it took. If my not-quite-anonymous caller hadn’t clued me in, it might’ve been several hours, or possibly several days, before I noticed a problem. And if the attacker had been more sophisticated about covering tracks, I might still not know what vector had been used to break in to my system. In other words, relatively little damage was done (at least to me; I can’t speak for people who may have been phished) and this was a relatively easy system to get back up and running. Now I just need to be more conscientious about my passwords.
posted at: 13:48 | permanent link to this entry
Marc Elliot Hall St. Peters, Missouri
Page created: 21 January 2002
Page modified: 31 December 2009
