If you look at both DBs (the old one and the new one) in PhpMyAdmin, you can see the character set of tables and fields on the Structure tab.
It's difficult to advise you without knowing the character for each DB and whether it's consistent within each DB.
utf8 is the recommended setting for MODX.
If the old DB is all utf8, the safest thing to do would be to delete the new DB and start over. When you create the new one, select utf8_general_ci or utf8_unicode_ci (whichever the old one is) for the character set. Then you can install MODX and copy the site_content table over with no problems.
If the old DB is all latin1 and the new one is utf8, it's more complicated. What I would do is back up the old DB, then use the script here:
http://bobsguides.com/convert-db-utf8.html to convert it to utf8. Make sure the old site looks OK before proceeding. Then drop the site_content table from the new DB and copy that table from the old DB.