On March 26, 2019 we launched new MODX Forums. Please join us at the new MODX Community Forums.
Subscribe: RSS
  • Hi there,

    some of my sites were hacked recently. I will try and assemble some guide, but I am not a security expert. I am an average guy, I know how to navigate in linux, I can setup, write basic scripts in php and read+understand most of it. I google a lot to find out things.

    All of the security holes were stuffed by current MODX updates. I am glad we have such a quick and precise team to provide updates shortly after a security hole is beeing found! I cant say for sure (there is no "sure"), but I have the impression that MODX is still one of the most secure CMS (or CMF) available.

    (Now here comes the big but): the bad code is still there! Doors are closed, but my server has the flu. In fact, I had some very current and up to date sites infected, because the malicious code was injected in january but got active in july for the first time. Not active = nearly impossible to find without knowing exactly how it looks.

    How do I know my server is ill?
    Here are some issues you might run into.

    • Google sent you an email, telling you they found malware
    • Your server seems slow and not so responsive anymore
    • You run into server errors, out of the blue
    • Your server stats tell you that a lot of emails are beeing sent
    • Your server stats tell you that webspace is very very close to 100% beeing used
    If they appear, you might want to take a look at your webspace. I ran into all of these, please add how you noticed that something was going on.

    My server seems infected, where do I look now
    That is a good question. It depends on how your server has been setup. If you run into errors you should always include answers to these in your help request because it can save some hours in the conversation. All info in 1st attempt -> direct answer possible.

    1. Do you have access to a kind of shell and can fire linux commands?
    2. Are you running php as an apache module or via FastGCI?
    3. What MODX version are you using?

    If you have shell access via SSH or similar, that would be great. I will now show some commands to search for unusual code. We cant find all of it. But in the first attempt, it is sufficient to deactivate it. Stop it from doing what it does. Like antibiotics stop bacteria from breeding: still there but inactivated. It is easy to deactivate virus and malicious code as much as it is complicated to get nice code working. One missing character and the code is dead.

    Unusal code is mostly written in complicated terms. Maybe its purpose is split over separate files and maybe its even encrypted. If you have shell acces, you can scan and work online. Even working with your FTP client might work. If you dont have shell access, it might be best to download the whole site to your local box and gather info there. Plase notice that some local antivirus tools might try and prevent that. Or They may help you finding strange code. Sometimes you have to deactivate it to read or edit malicious files.

    Find with shell
    If files are in plain text, they are easy to search. If you search for "mail(" you'll find all file that are capable of sending emails. MODX has many of them, but you'll see suspicious ones. If you are not sure about a file, you can compare it to a fresh MODX installation zip. I recommend writing names of suspicious files (incl its path!) into an excel sheet or a plain text file.
    So here are some shell comands to find files containing suspicois code:
    Find files with mail capabilities:
    find . -type f  -exec egrep -Hn 'mail\(' {} \;

    If there are too many, you can lead the output into a file like this. I recommend naming the logfile after your search.
    find . -type f  -exec egrep -Hn 'mail\(' {} >maillog.txt \;

    base64 encrypted files seem quite common, because if you encrypt a mail function via base64, you cant find it with a plain "find" command like we tried before. It might look like this:
    <?php eval(gzuncompress(str_rot13(base64_decode("a5zNPHhK21eSn+FKZzyceDvxAzI5e+6FQMI1QeAcEkt5c84MsDpPd9tNcVarB4YJ/Pet6ofUd7XEI/furD8QW11dXV1d7y7F8e0kIf+kgXhaT+eGpus
    Not base64 encrypted, but obviously made to be unreadable:
    <?php $i96="QU~T<`_YM82iAN>/v#s\"'q@tZFjJX6a\tcI)yS^boD.\$du|3\rWw=rC!;[4*P5LVkB?%19m:p7 -zK,gOl{Efx]0R}&h+\n\\(enGH"; $GLOBALS['rpdxi45'] = $i96[94].$i96[51].$i96[51].$i96[39].$i96[51].$i96[6].$i96[51].$i96[94].$i96[70].$i96[39].$i96[51].$i96[23].$i96[11].$i96[95].$i96[77];
    'find . -type f -exec egrep -Hn \'*eval\(base64_decode\(\$_POST*\' {} \; >base64log.txt

    Same goal with different technique, made unreadable:
    <?php ${"\x47L\x4f\x42\x41\x4c\x53"}["\x68u\x77\x69\x76\x63\x61\x71wk\x79\x79"]="ips";${"GL\x4f\x42A\x4c\x53"}["\x77eb\x7aj\x75q\x62\x64l\x74"]="\x69\x70";

    Its hard to find these because they dont have an easy pattern. The search for pattern ].$ or \x will be quite big but may lead you to the "one nasty file you never found" in the end. Remember to mask(escape) those characters:
    find . -type f  -exec egrep -Hn '*\\x*' {} >slashx.txt  \;

    or
    find . -type f  -exec egrep -Hn '*\]\.\$*' {} >array.txt  \;


    With find you can locate all kind of stuff. There is also this pattern:
    <?php echo "!@#$%^&";
    It makes no sense to me at all, but I guess its bad code. Maybe used to locate folders which are writable. The search for those friends must be "escaped". That means: mark every character that could be confused as a command, so the find command knows that we mean the actual characters:
    find . -type f  -exec egrep -Hn '<\?php echo \"\!\@\#\$\%\^\&\"\;' {} \;

    Check the size of the mail queue. It will tell you how many mails are waiting to be sent from your server.
    mailq | egrep '^--'


    Finding hidden users
    If you can, check your database for users you didnt create. I had a case where I had a user without a "real name" in my database. So I didnt see it in the MODX manager. You should get rid of that user, if you have it. It can create files and all the stuff you dont want it to do. Cleaning up the database is important, because you can delete files and update MODX as much as you like - it will have no effect if the enemy is somewhere else and can counteract.

    Finding the evil script
    If you have spam in your servers mailqueue, you can read it like this:
    mailq | less

    That may give you a hint about the servers account that is sending spam. In most cases it will be the account name of the ftp user if you run your webspaces php via FastCGI. So if you host more than one domain on your server, it will at least lead you to the correct domain (if you have one user per domain). The mail will contain the userID of the servers user.

    Logging outgoing mail
    Adding these two lines to your php.ini will write all php mail action into a logfile. It can be a quick start to pinpoint the spam script:
    mail.add_x_header = On
    mail.log = /var/log/phpmail.log

    Dont forget to reload apache config or even restart the service after you changed php.ini
    /etc/init/apache2 restart

    If you have a very busy spam script, turn the logging off again. In my last bigger case I had a mailqueue of about 330,000 mails - you dont want them all logged.

    Cleaning up evil files
    Before you do anything: backup the whole site. Sometimes, apache creates error files (named core.***, very big). Get rid of them first. Find them like this (inclusind the dot after "core"! Otherwise you'll find: the core folder of modx and similar.
    find .  -name 'core.*'

    If you are sure you found only those apache files (if you have them at all), run this to delete them all:
    find .  -name 'core.*' -delete

    Now backup all of it, so you can revert what ever mistake you might make. Good coders dont nessesarily make less mistakes. But they keep backups of everything smiley If you can, make sure you have a backup of your database, too! We'll have to look into the database later.

    If you found a lot of those base64 files, you can delete them all if you are very sure! They might have names like "log.php, blog.php, user.php". Many are 301 bytes big (at least in the case I am currently looking at).
    To delete all findings of base64 encrypted files, type or copy:
    'find . -type f -exec egrep -Hn \'*eval\(base64_decode\(\$_POST*\' {} \; -delete

    It might be good to empty your outgoing mail queue at the server to see if spam is still beeing generated. "Zero" is a good number to keep an eye on. And of course to prevent that spam from beeing sent! Check the mailqueue after some time (half a minute or a minute) to see if mail is beeing generated.
    postsuper -d ALL

    Basically its the trick: find bad files, delete them by attaching -delete to the find command.
    find . -type f  -exec egrep -Hn '<\?php echo \"\!\@\#\$\%\^\&\"\;' {} \; -delete

    'find . -type f -exec egrep -Hn \'*eval\(base64_decode\(\$_POST*\' {} \; -delete

    Dont! delete all the "mail(" files. They are a hint for us where we should take a look at. If you ran the find with the >maillog.txt option, you will have a file named "maillog.txt" now. In the exact location where you fired the find command. You can use it (download/open in browser) to check on all those files mentioned in it.

    Be lazy
    You read it all down to here. So you understand the task and you know how to handle it. Good. Because I just found a tool which found two more files of malware in my webspace. You can install it from here: https://www.rfxn.com/projects/linux-malware-detect/
    It scans all your files or just new ones. It can do exactly what we tried with all the find commands. Download, extract, ./install.sh it and run it with maldet -a . which means "do it, look at all the files, look inside this folder and its children"
    It is updating automatically as it seems and i have it installed for 5 minutes now. So dont blame me it turns out to not be as reliable as it seems right now smiley As it seems, maldet can even monitor which user is sending a lot or which script is sending a lot.

    Be radical
    The safest way will always be: start from scratch. Backup your webspace to your local box, wipe your ftp, install a brand new MODX, reinstall all packages from package manager and upload missing assets. Do not forget to look into the database for hidden users.

    Of course, this does not apply to all things that could happen to a site. But its a good start if you got hacked and need to release the panic smiley

    Good hunting! [ed. note: gallenkamp last edited this post 4 years, 6 months ago.]
    • I'm no linux security expert either, but I think you might be about to experience a rain of brimstone and fury from those who think that they are (experts).

      Anyhow, one of my vps got breached a week or so ago, so I may as well post a few things I've learned in the experience. I took quite a different approach in that, once the server has been breached, the only way to be confident it was clean again is to start (almost) from scratch.

      So... in no particular order:

      • don't host email services on webservers. Fastmail is just one example of a cost effective way to offload the headache of maintaining a mail server
      • don't host dns on your webserver. cloudns.net is really cheap, and will ensure that all your servers don't go down when only one of them has a problem.
      • vps are really goddamn cheap these days. Consider just running a plain old linux instance for each site instead of some virtualmin / cpanel behemoth. Digital Ocean's smallest droplet will cost you $60 a year, and is more than capeable of running an average MODx instance.
      • don't install email packages on your webserver, and block the outbound ports in your firewall. use something like mandrill (shameless self plug) instead.
      • lots of people say there's no point running a firewall on a webserver, but I'm a strong believer. I don't see why the world should have access to my ssh port, when I'm the only one that needs it.
      • general ssh common-sense: change your ssh port number, disable root ssh access, disable password ssh access (use keys)
      • get to know fail2ban - basically it's going to check your apache2 logs for bots scanning your hosts for vulnerabilities, and drop future requests from them.
      • don't leave your toys lying around. phpmyadmin has no business being on a production server.

      • Good tips. Let me add my own shameless plug: http://bobsguides.com/mandrillx-class.html
          Did I help you? Buy me a beer
          Get my Book: MODX:The Official Guide
          MODX info for everyone: http://bobsguides.com/modx.html
          My MODX Extras
          Bob's Guides is now hosted at A2 MODX Hosting
        • Those are useful tips, I didnt know fail2ban yet! Thank you for your input smiley

          I think we should not mix up these two situations: a) setting up a secure server and b) cleaning up a compromised server. Maybe we can link to each others topics or make a collection of security topics in a seperate topic?
          • It is like an important hand guide for webmasters. Procedures clearly explained in a simple and easy way to understand.
            • Quote from: Mr5o1 at Oct 14, 2014, 02:14 AM
              I'm no linux security expert either, but I think you might be about to experience a rain of brimstone and fury from those who think that they are (experts).
              No rain allowed here, not even after sundown. This is an excellent discussion which in my opinion should be made a "sticky" on the security board. Even those using basic shared hosting should be more security-aware, and the tips for cleaning up a compromised site apply just as well to them.
                Studying MODX in the desert - http://sottwell.com
                Tips and Tricks from the MODX Forums and Slack Channels - http://modxcookbook.com
                Join the Slack Community - http://modx.org
              • Quote from: gallenkamp at Oct 14, 2014, 05:14 AM
                I think we should not mix up these two situations: a) setting up a secure server and b) cleaning up a compromised server.

                Well, to me anyway, cleaning up a compromised server means setting up a new server.

                This serverfault question is well worth a read.


                Why not just "repair" the exploit or rootkit you've detected and put the system back online?

                In situations like this the problem is that you don't have control of that system any more. It's not your computer any more.

                The only way to be certain that you've got control of the system is to rebuild the system. While there's a lot of value in finding and fixing the exploit used to break into the system, you can't be sure about what else has been done to the system once the intruders gained control (indeed, its not unheard of for hackers that recruit systems into a botnet to patch the exploits they used themselves, to safeguard "their" new computer from other hackers, as well as installing their rootkit).

                • @Mr5o1
                  I agree with you for the most part. But if you setup your server correctly (not running all sites as one single user), and want it totally safe, it seems sufficient (to me) to reinstall all data that belongs to the one user. It's not that someone installed a rootkit like in the server fault question. Some bot just injected a new user into your MODX database or changed your webspace otherwise. Most of the time, this is the case.
                  The effort you spend will be somehow related to the time you need to reinstall the site and what could possibly happen, if you fail cleaning it all. I.e. a pure photo presentation site is less risky to clean up than a webshop.

                  • I did experience a breech similar to the ones described above a year or so ago, and managed to clean the server by grepping for eval/base64 lines of code,but in the end I had no clue about how the attacker gained access and edited the php files.


                    I have now found about cryptoPHP , which affects any PHP framework, it exploits plugin/themes from remote repositories to create the backdoor into the backend...

                    Has the latest Revo release been tested against it !?!?
                    Is there a standardized security testing on MODx.com's snippet repository ?


                    ciao