I’m also working with someone right now trying to figure out how to get this right... tried loads of options, and now suddenly it seems to work with everything I set..
Except static resources don’t seem to get the base dir added.. but that might not be related.
Currently its set to assets/content (the folder we want to limit it to) for *all* four fields: filemanager_path, filemanager_url, rb_base_dir and rb_base_url.
I also tried setting filemanager_path as an absolute url (and updating filemanager_path_relative to "no") which also seemed to work. When I tried to confirm my hypothesis that it needed the absolute url it also started working with all set to relative....
Updating TinyMCE to the latest version available (check with the Repo what the last version is!) also seem to solve some issues...