# The Friendly URLs part
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index.php?q=$1 [L,QSA,NC]
Just copy the .htaccess file from the MODx Manager subdirectory to your forum subdirectory to turn off mod_rewrite for URLs in there.
Options +FollowSymlinks
RewriteEngine On
RewriteBase /
# The Friendly URLs part
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index.php?q=$1 [L,QSA,NC]
RewriteRule ^(.*)\.png2$ $1off.png [nc]
# Rewrite www.domain.com -> domain.com
RewriteCond %{HTTP_HOST} .
RewriteCond %{HTTP_HOST} !^symphonicheaven\.com
RewriteRule (.*) http://symphonicheaven.com/$1 [R=301,L]
Options -MultiViews +FollowSymlinks -Indexes
#
# If mod_security is enabled, attempt to disable it.
# - Note, this will work on the majority of hosts but on
# MediaTemple, it is known to cause random Internal Server
# errors. For MediaTemple, please remove the block below
#
<IfModule mod_security.c>
# Turn off mod_security filtering.
# The below probably isn’t needed, but better safe than sorry.
</IfModule>
#
# MyBB "search engine friendly" URL rewrites
# - Note, for these to work with MyBB please make sure you have
# the setting enabled in the Admin CP and you have this file
# named .htaccess
#
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteRule ^forum-([0-9]+).html forumdisplay.php?fid=$1 [L,QSA]
RewriteRule ^forum-([0-9]+)-page-([0-9]+).html forumdisplay.php?fid=$1&page=$2 [L,QSA]
RewriteRule ^thread-([0-9]+).html showthread.php?tid=$1 [L,QSA]
RewriteRule ^thread-([0-9]+)-page-([0-9]+).html showthread.php?tid=$1&page=$2 [L,QSA]
RewriteRule ^thread-([0-9]+)-lastpost.html showthread.php?tid=$1&action=lastpost [L,QSA]
RewriteRule ^thread-([0-9]+)-nextnewest.html showthread.php?tid=$1&action=nextnewest [L,QSA]
RewriteRule ^thread-([0-9]+)-nextoldest.html showthread.php?tid=$1&action=nextoldest [L,QSA]
RewriteRule ^thread-([0-9]+)-newpost.html showthread.php?tid=$1&action=newpost [L,QSA]
RewriteRule ^thread-([0-9]+)-post-([0-9]+).html showthread.php?tid=$1&pid=$2 [L,QSA]
RewriteRule ^post-([0-9]+).html showthread.php?pid=$1 [L,QSA]
RewriteRule ^announcement-([0-9]+).html announcements.php?aid=$1 [L,QSA]
RewriteRule ^user-([0-9]+).html member.php?action=profile&uid=$1 [L,QSA]
RewriteRule ^calendar-([0-9]+).html calendar.php?calendar=$1 [L,QSA]
RewriteRule ^calendar-([0-9]+)-year-([0-9]+).html calendar.php?action=yearview&calendar=$1&year=$2 [L,QSA]
RewriteRule ^calendar-([0-9]+)-year-([0-9]+)-month-([0-9]+).html calendar.php?calendar=$1&year=$2&month=$3 [L,QSA]
RewriteRule ^calendar-([0-9]+)-year-([0-9]+)-month-([0-9]+)-day-([0-9]+).html calendar.php?action=dayview&calendar=$1&year=$2&month=$3&day=$4 [L,QSA]
RewriteRule ^calendar-([0-9]+)-week-([0-9]+).html calendar.php?action=weekview&calendar=$1&week=$2 [L,QSA]
RewriteRule ^event-([0-9]+).html calendar.php?action=event&eid=$1 [L,QSA]
<IfModule mod_env.c>
SetEnv SEO_SUPPORT 1
</IfModule>
</IfModule>
#
# If Apache is compiled with built in mod_deflade/GZIP support
# then GZIP Javascript, CSS, HTML and XML so they’re sent to
# the client faster.
#
<IfModule mod_deflate.c>
AddOutputFilterByType DEFLATE application/x-javascript text/css text/html text/xml
</IfModule>
What you probably need to do is to copy the .htaccess from your MODx Manager subdirectory (not the root one; the one that just turns off rewrites) to your MyBB admin subdirectory (so as to turn off rewriting in there also).