There is no other configuration then the standard (with 0.9.6.3 final and the last TinyMCE version 3.2.1.1). The only one is setting entity encoding to raw because of using utf-8. All other things come directly from TinyMCE. No problems with unwanted br or paragraphs.
For editors it’s required to paste text (if not written directly in the RTE) as plain text without formatting, breaks and paragraphs. And of course NOT directly from an MS Office application. And after pasting or writing text THEN you should set paragraphs and formats.
It may be hard to understand that working with a CMS and a RTE should be learned like driving a car. You should have experience in traffic rules and in control a car. But operating MODx is a bit easier, like driving a dodgem car.
For my clients I suggest using Firefox as browser for editing content. Detecting empty (and unwanted) breaks and paragraphs I set up a RTE css file connected via the configuration. There are included some definitions for paragraphs, headings, lists and links to give the editior the "final" feeling of the text. And there are some additional definitions WORKING ONLY IN MODERN BROWSERS to detect paragraphs and headings:
p:after, h1:after, h2:after, h3:after, h4:after {
color:#990;
background-color:#ff9;
font-size:12px;
font-style:normal;
content:"\00B6\00A0";
}
Try this and you will see, and play around for your needs. Additionally you can see the effect from a live example (in German) with a customized RTE (deleted some buttons).