I struggled with the same issue - setting up a modx site on the same domain as the conventional site. I am using the following method now.
1) Install modx completely in the root of your domain - like any normal installation. If your old root start file is index.php, make sure to rename that before you start the installation, but a lot of times, this will be index.htm(l) and it can be left unchanged.
2) Make sure you don't use friendly URLS at this time. However, if you plan on using friendly URLS in the future, you can set it up now (including adjusting the .htaccess file), test it and have it ready for future use (I am assuming you are doing this sometime deep in the night - until step 4 is taken care of the old site will not work).
3) go into the manager and go to system settings. Set "Request Controller Filename" to anything-you-like.php, like secret-test.php and make sure that "Use Friendly URLs" is set to No.
5) rename index.php to anything-you-like.php, like secrest-test.php
6) rename, if necessary, the original index file from the old site to its old name (index.php).
At this point, your old site is working. The new site is accessed with
http://www.yoursite.com/secret-test.php. The wayfinder menu system works, you can access paged directly by adding "?id=1" (use any valid resource number) to the URL. When it is time to set the new modx site live, it's done in less than 30 seconds without any transfer of files or databases. And if something is not quite right, you can go back to the old site - also in a matter of seconds.
I see only ONE drawback: if someone knows or guesses the filename that starts the new website (secret-test.php), your new site is not so secret anymore. But with a cleverly chosen name that should not be the biggest issue for most of your clients.
Although I have tested the above, I just recently came up with this idea, so if you find any drawbacks, please let me know.
Also, perhaps you are already aware of this, if you use the MODX cloud, you will never have to deal with these issues anymore. It comes at a price, but I have tested it and it saves you from a lot of hassles and it includes extremely easy version updates of your installation.
[ed. note: joostmineur last edited this post 11 years, 3 months ago.]