Ok.. I have no experience from either of those, so I have to take your word for itbut from what I understood with a bit of googling is that MooTools is truly much better extensible than jQuery. jQuery’s biggest advantage though is that the syntax is very simple. You don’t have to write as complex code as you do with other libs. But the result from my googling session is that MooTools really looks the best fit for MODx. The basic idea in both MooTools and MODx is very similar.
EDIT: And I dream of the day that MODx has the same kind of download than MooTools has.. And what the little birds are saying, the wait might not be that long..

LOL, true!
Joomla has a much larger user community and prebuilt tools too.