redirectToOnFailedAuth
Create a resource for unauthorized users. In the System Settings, set the unauthorized_page setting to the ID of your resource.
The (anonymous) group has to have load permission for your resources. If they can't load the protected resources they can't even see that they exist, thus getting the error_page.

Are you sure that your unauthorized page is published, and not protected?
Sorry, that's my full $.02 worth. Hopefully somebody else will have a better idea.
$registerId=`12'; /* ID of registration page */
$url = $modx->makeUrl($registerId, "", "", "full");
if (! $modx->user->isMember('someGroup')) {
$modx->sendRedirect($url);
}
return '';