My blog was defaced yesterday
It wasn’t cool, but thanks to Avi Alkalay who noticed it early and let me know, I caught it early enough.
The attackers probably used some kind of dictionary attack to guess the password to the FTP accounts of my website and another website virtually hosted on my server and opened FTP access to both accounts and uploaded files. To the other website they uploaded a PHP backdoor (that allows access to the local MySQL server and to the local file system with the permissions of the web server) and rootkit.
The rootkit was for an old Linux kernel, so it wouldn’t have worked, even if the attackers could get it to compile (which is impossible because unprivileged users have no access to compiler tools) and installed (which is impossible because the boot directory is unmounted). From the logs I can also see that they uploaded some binary files and deleted them later – probably because they didn’t work due to the noexec setting on the virtual hosts file system.
On my website the attackers put a new index file with the defacing message, which can be seen here:
And also a directory with a website in french, which I don’t understand its purpose – it can be seen here:
The attackers also uploaded to both sites a PHP file which I assume is a mass mailer, but it was removed later and wasn’t there when I did my analysis this morning.
As I mentioned, the attack was performed over FTP. I think the passwords were brute forced, but my FTP server software – proftpd – does not log failed login attempts, and while my password used only lower case English letters and was somewhat based on a dictionary word (bad, I should have known better, and its better now), the other web site had a random combination of letters and numbers and brute forcing that would take some time I expect. After entry was achieved, the attackers uploaded the files to the web site, and accessed the PHP backdoor program. I’m not sure yet as to what they have done with it, but I have the logs and I’m examining them.
The attack was performed from a Moroccan ISP’s DSL connection, and they logged in at 15:00 Sunday (I’m not sure when the attack started, as I have no logs of failed attempts, as I explained), and I noticed it at 08:00 today. I started by removing the backdoor and defaced front page, then looked for the point of entry. I found it at about 9am and responded by blocking the entire DSL range of the attackers in the firewall. I then continued to remove all the uploaded files, changing passwords and disabling the login access to the other website.
I have all the modified files and logs backed up and I will examine them later. If anyone is interested in having a look, please contact me by e-mail (oded@geek.co.il).
[Update]
The attackers did not gain access through brute forcing the FTP server as I thought. It was a much worse problem and it was entirely my fault.
The problem was that a recent update broke my weird login setup which uses custom PAM scripts to authenticate users using SHA1 passwords against a custom MySQL database. As pam_mysql is poorly maintained and was last updated over a year and a half, then as the operating system I use update their PAM setup with newer versions (currently 0.99.7.1) this brittle setup breaks in funny and interesting ways. The last change caused the proftpd pam.d configuration to give access no matter what password was used !! (and only that, for some reason – I just tested IMAP, POP and SSH and they work fine).
Luckily there aren’t a lot of FTP based attacks, this was probably just an opportunity probe that got really lucky. It was quite obvious after a short while that my initial estimate of brute forcing the password was wrong because no one attacked the SSH login, which would be a much more obvious target and if someone could find the password for an account, then the next obvious move would be to try SSH. The second website that was attacked, and where most of the action was, does not have SSH access enabled so no problem there, but I use mostly SSH on my account and it was quite obvious that the attackers didn’t have my password because they didn’t access my account through SSH.
Anyway – this is the end of another I’m stupid post, have fun y’all and watch those open ports 😉
Hi,
can you explain me who did you detected the modfiled files that uploaded? who did you detected the rootkit? do you have some kind of a checksum software?
I have a checksumming software – I use Mandriva’s msec, but it does not monitor the hosted websites, because files change there all the time (as people updated their websites), so I didn’t get a notification for that. My first clue was Avi’s email that he sent after checking my site and seeing that it was defaced. As the attackers didn’t do anything to my site other then put in an index.html, and as the bookmark I use to go to my site is directly to the blog, there would have been no other way for me to notice the problem.
After I identified the attack vector as the FTP access (I checked SSH and couldn’t find problems, and FTP was next), I looked in the logs for files that were recently uploaded to the compromised accounts, and just removed everything I found. I was lucky because I never use FTP to upload files to the site (only file transfers over SSH) and the second account is not very active – its a commercial website that doesn’t update often, so it was easy to identify malicious files. Once I got the files that were uploaded, I could analyze them – some were harmless HTML files, others were less harmless. The rootkit wasn’t successfully installed – I found the C file for the exploit that was uploaded by the attackers, but the attacker never managed to actually run the exploit.
Anyway – I was too early to rejoice. Apparently I didn’t block entirely enough of the attacker’s usable IP range, and worse – changing the password didn’t prevent them from logging in again, and so my website was targeted again – this time a phishing scam website was uploaded. I now blocked the attackers entire ISP and I need to figure out how they gain FTP access, ASAP.
I was once invited to analyze a Linux machine that was invaded. I ended up writing an article about it to the brazilian Linux Magazine.
The problem with the machine was a VERY weak root password. We could also find the tools they used to break that machine, cause they have installed it to attack other machines.
We could see a file containing about 18000 user+password combinations, a modified SSH client and a script that runs it all based on an IP range.
In our case, the attack was silent. They just wanted to use the machine to attack other machines. Pretty stupid.
It easy to learn about this attacks. Just connect to the Internet a machine with a plain Linux installation and ‘passw0rd’ as the root’s password, wait 1 or 2 weeks and your machine will be attacked. One way to verify the crackers are already in is to reinstall the netstat command (because they’ll modify your previous one) and see if there is some connection to IRC ports (around 6667).
If you investigate this IRC bot you’ll able to connect the IRC server, find the chat room, and actually talk to the cracker. I did this once and was not very funny.
Take care Oded !
[…] noticed Oded’s blog was attacked which make me remember some […]
To minimize risks of actually using FTP at all, you should configure/change to software that will allow you to lock IP / account after number of failed attempts. This will prevent brute forcing your server.
Avi: I actually had a server that was attacked and an IRC bot installed. It was fairly educational experience 😉
Leo: Thanks for the offer, I’ll try that. But the problem was not brute forcing the password as I initially thought: check the post update.