Hi there,
some of my sites were hacked recently. I will try and assemble some guide, but I am not a security expert. I am an average guy, I know how to navigate in linux, I can setup, write basic scripts in php and read+understand most of it. I google a lot to find out things.
All of the security holes were stuffed by current MODX updates. I am glad we have such a quick and precise team to provide updates shortly after a security hole is beeing found! I cant say for sure (there is no "sure"), but I have the impression that MODX is still one of the most secure CMS (or CMF) available.
(Now here comes the big
but): the bad code is still there! Doors are closed, but my server has the flu. In fact, I had some very current and up to date sites infected, because the malicious code was injected in january but got active in july for the first time. Not active = nearly impossible to find without knowing exactly how it looks.
How do I know my server is ill?
Here are some issues you might run into.
- Google sent you an email, telling you they found malware
- Your server seems slow and not so responsive anymore
- You run into server errors, out of the blue
- Your server stats tell you that a lot of emails are beeing sent
- Your server stats tell you that webspace is very very close to 100% beeing used
If they appear, you might want to take a look at your webspace. I ran into all of these, please add how you noticed that something was going on.
My server seems infected, where do I look now
That is a good question. It depends on how your server has been setup. If you run into errors you should always include answers to these in your help request because it can save some hours in the conversation. All info in 1st attempt -> direct answer possible.
- Do you have access to a kind of shell and can fire linux commands?
- Are you running php as an apache module or via FastGCI?
- What MODX version are you using?
If you have shell access via SSH or similar, that would be great. I will now show some commands to search for unusual code. We cant find all of it. But in the first attempt, it is sufficient to deactivate it. Stop it from doing what it does. Like antibiotics stop bacteria from breeding: still there but inactivated. It is easy to deactivate virus and malicious code as much as it is complicated to get nice code working. One missing character and the code is dead.
Unusal code is mostly written in complicated terms. Maybe its purpose is split over separate files and maybe its even encrypted. If you have shell acces, you can scan and work online. Even working with your FTP client might work. If you dont have shell access, it might be best to download the whole site to your local box and gather info there. Plase notice that some local antivirus tools might try and prevent that. Or They may help you finding strange code. Sometimes you have to deactivate it to read or edit malicious files.
Find with shell
If files are in plain text, they are easy to search. If you search for "
mail(" you'll find all file that are capable of sending emails. MODX has many of them, but you'll see suspicious ones. If you are not sure about a file, you can compare it to a fresh MODX installation zip. I recommend writing names of suspicious files (incl its path!) into an excel sheet or a plain text file.
So here are some shell comands to find files containing suspicois code:
Find files with mail capabilities:
find . -type f -exec egrep -Hn 'mail\(' {} \;
If there are too many, you can lead the output into a file like this. I recommend naming the logfile after your search.
find . -type f -exec egrep -Hn 'mail\(' {} >maillog.txt \;
base64 encrypted files seem quite common, because if you encrypt a mail function via base64, you cant find it with a plain "find" command like we tried before. It might look like this:
<?php eval(gzuncompress(str_rot13(base64_decode("a5zNPHhK21eSn+FKZzyceDvxAzI5e+6FQMI1QeAcEkt5c84MsDpPd9tNcVarB4YJ/Pet6ofUd7XEI/furD8QW11dXV1d7y7F8e0kIf+kgXhaT+eGpus
Not base64 encrypted, but obviously made to be unreadable:
<?php $i96="QU~T<`_YM82iAN>/v#s\"'q@tZFjJX6a\tcI)yS^boD.\$du|3\rWw=rC!;[4*P5LVkB?%19m:p7 -zK,gOl{Efx]0R}&h+\n\\(enGH"; $GLOBALS['rpdxi45'] = $i96[94].$i96[51].$i96[51].$i96[39].$i96[51].$i96[6].$i96[51].$i96[94].$i96[70].$i96[39].$i96[51].$i96[23].$i96[11].$i96[95].$i96[77];
'find . -type f -exec egrep -Hn \'*eval\(base64_decode\(\$_POST*\' {} \; >base64log.txt
Same goal with different technique, made unreadable:
<?php ${"\x47L\x4f\x42\x41\x4c\x53"}["\x68u\x77\x69\x76\x63\x61\x71wk\x79\x79"]="ips";${"GL\x4f\x42A\x4c\x53"}["\x77eb\x7aj\x75q\x62\x64l\x74"]="\x69\x70";
Its hard to find these because they dont have an easy pattern. The search for pattern
].$ or
\x will be quite big but may lead you to the "one nasty file you never found" in the end. Remember to mask(escape) those characters:
find . -type f -exec egrep -Hn '*\\x*' {} >slashx.txt \;
or
find . -type f -exec egrep -Hn '*\]\.\$*' {} >array.txt \;
With find you can locate all kind of stuff. There is also this pattern:
<?php echo "!@#$%^&";
It makes no sense to me at all, but I guess its bad code. Maybe used to locate folders which are writable. The search for those friends must be "escaped". That means: mark every character that could be confused as a command, so the find command knows that we mean the actual characters:
find . -type f -exec egrep -Hn '<\?php echo \"\!\@\#\$\%\^\&\"\;' {} \;
Check the size of the mail queue. It will tell you how many mails are waiting to be sent from your server.
Finding hidden users
If you can, check your database for users you didnt create. I had a case where I had a user without a "real name" in my database. So I didnt see it in the MODX manager. You should get rid of that user, if you have it. It can create files and all the stuff you dont want it to do. Cleaning up the database is important, because you can delete files and update MODX as much as you like - it will have no effect if the enemy is somewhere else and can counteract.
Finding the evil script
If you have spam in your servers mailqueue, you can read it like this:
That may give you a hint about the servers account that is sending spam. In most cases it will be the account name of the ftp user if you run your webspaces php via FastCGI. So if you host more than one domain on your server, it will at least lead you to the correct domain (if you have one user per domain). The mail will contain the userID of the servers user.
Logging outgoing mail
Adding these two lines to your php.ini will write all php mail action into a logfile. It can be a quick start to pinpoint the spam script:
mail.add_x_header = On
mail.log = /var/log/phpmail.log
Dont forget to reload apache config or even restart the service after you changed php.ini
/etc/init/apache2 restart
If you have a very busy spam script, turn the logging off again. In my last bigger case I had a mailqueue of about 330,000 mails - you dont want them all logged.
Cleaning up evil files
Before you do anything: backup the whole site. Sometimes, apache creates error files (named core.***, very big). Get rid of them first. Find them like this (inclusind the dot after "core"! Otherwise you'll find: the core folder of modx and similar.
If you are sure you found only those apache files (if you have them at all), run this to delete them all:
find . -name 'core.*' -delete
Now backup all of it, so you can revert what ever mistake you might make. Good coders dont nessesarily make less mistakes. But they keep backups of everything
If you can, make sure you have a backup of your database, too! We'll have to look into the database later.
If you found a lot of those base64 files, you can delete them all
if you are very sure! They might have names like "log.php, blog.php, user.php". Many are 301 bytes big (at least in the case I am currently looking at).
To delete all findings of base64 encrypted files, type or copy:
'find . -type f -exec egrep -Hn \'*eval\(base64_decode\(\$_POST*\' {} \; -delete
It might be good to empty your outgoing mail queue at the server to see if spam is still beeing generated. "Zero" is a good number to keep an eye on. And of course to prevent that spam from beeing sent! Check the mailqueue after some time (half a minute or a minute) to see if mail is beeing generated.
Basically its the trick: find bad files, delete them by attaching -delete to the find command.
find . -type f -exec egrep -Hn '<\?php echo \"\!\@\#\$\%\^\&\"\;' {} \; -delete'find . -type f -exec egrep -Hn \'*eval\(base64_decode\(\$_POST*\' {} \; -delete
Dont! delete all the "mail(" files. They are a hint for us where we should take a look at. If you ran the find with the >maillog.txt option, you will have a file named "maillog.txt" now. In the exact location where you fired the find command. You can use it (download/open in browser) to check on all those files mentioned in it.
Be lazy
You read it all down to here. So you understand the task and you know how to handle it. Good. Because I just found a tool which found two more files of malware in my webspace. You can install it from here:
https://www.rfxn.com/projects/linux-malware-detect/
It scans all your files or just new ones. It can do exactly what we tried with all the find commands. Download, extract, ./install.sh it and run it with
maldet -a . which means "do it, look at all the files, look inside this folder and its children"
It is updating automatically as it seems and i have it installed for 5 minutes now. So dont blame me it turns out to not be as reliable as it seems right now
As it seems, maldet can even monitor which user is sending a lot or which script is sending a lot.
Be radical
The safest way will always be: start from scratch. Backup your webspace to your local box, wipe your ftp, install a brand new MODX, reinstall all packages from package manager and upload missing assets. Do not forget to look into the database for hidden users.
Of course, this does not apply to all things that could happen to a site. But its a good start if you got hacked and need to release the panic
Good hunting!
[ed. note: gallenkamp last edited this post 10 years, 3 months ago.]
☆ A M B ☆
☆ A M B ☆
