It wasn’t cool, but thanks to Avi Alkalay who noticed it early and let me know, I caught it early enough.
The attackers probably used some kind of dictionary attack to guess the password to the FTP accounts of my website and another website virtually hosted on my server and opened FTP access to both accounts and uploaded files. To the other website they uploaded a PHP backdoor (that allows access to the local MySQL server and to the local file system with the permissions of the web server) and rootkit.
The rootkit was for an old Linux kernel, so it wouldn’t have worked, even if the attackers could get it to compile (which is impossible because unprivileged users have no access to compiler tools) and installed (which is impossible because the boot directory is unmounted). From the logs I can also see that they uploaded some binary files and deleted them later – probably because they didn’t work due to the
noexec setting on the virtual hosts file system.
The attackers also uploaded to both sites a PHP file which I assume is a mass mailer, but it was removed later and wasn’t there when I did my analysis this morning.
As I mentioned, the attack was performed over FTP. I think the passwords were brute forced, but my FTP server software – proftpd – does not log failed login attempts, and while my password used only lower case English letters and was somewhat based on a dictionary word (bad, I should have known better, and its better now), the other web site had a random combination of letters and numbers and brute forcing that would take some time I expect. After entry was achieved, the attackers uploaded the files to the web site, and accessed the PHP backdoor program. I’m not sure yet as to what they have done with it, but I have the logs and I’m examining them.
The attack was performed from a Moroccan ISP’s DSL connection, and they logged in at 15:00 Sunday (I’m not sure when the attack started, as I have no logs of failed attempts, as I explained), and I noticed it at 08:00 today. I started by removing the backdoor and defaced front page, then looked for the point of entry. I found it at about 9am and responded by blocking the entire DSL range of the attackers in the firewall. I then continued to remove all the uploaded files, changing passwords and disabling the login access to the other website.
I have all the modified files and logs backed up and I will examine them later. If anyone is interested in having a look, please contact me by e-mail (email@example.com).
The attackers did not gain access through brute forcing the FTP server as I thought. It was a much worse problem and it was entirely my fault.
The problem was that a recent update broke my weird login setup which uses custom PAM scripts to authenticate users using SHA1 passwords against a custom MySQL database. As pam_mysql is poorly maintained and was last updated over a year and a half, then as the operating system I use update their PAM setup with newer versions (currently 0.99.7.1) this brittle setup breaks in funny and interesting ways. The last change caused the proftpd pam.d configuration to give access no matter what password was used !! (and only that, for some reason – I just tested IMAP, POP and SSH and they work fine).
Luckily there aren’t a lot of FTP based attacks, this was probably just an opportunity probe that got really lucky. It was quite obvious after a short while that my initial estimate of brute forcing the password was wrong because no one attacked the SSH login, which would be a much more obvious target and if someone could find the password for an account, then the next obvious move would be to try SSH. The second website that was attacked, and where most of the action was, does not have SSH access enabled so no problem there, but I use mostly SSH on my account and it was quite obvious that the attackers didn’t have my password because they didn’t access my account through SSH.
Anyway – this is the end of another I’m stupid post, have fun y’all and watch those open ports 😉