I think it’s definitely something that can be done quite easily. Simply install modx in a subfolder of the Magento site and and have it’s 404 handler pass the request to the modx install.
Ryan Thrash, MODX Co-Founder
Follow me on Twitter at @rthrash or catch my occasional unofficial thoughts at thrash.me