Yes, that's definitely possible. I'm assuming that you're using Revolution.
There are two System Events: OnPageNotFound and OnPageUnauthorized. A plugin connected to either event should be able to check the resource group of the current resource and forward the user anywhere you want.
Making the unauthorized page work is tricky in Revo, because trying to access an unauthorized page actually triggers OnPageNotFound by default. There's a section here ("Unauthorized versus Error Page") on how to get around that:
http://bobsguides.com/revolution-permissions.html
The plugin code for resource group detection might look something like this:
<?php
$docId = $modx->resource->get('id');
$groupId = '12'; // ID of the resource group
$myNotFoundPage = '33'; // ID of specific "not found" page
$url = $modx->makeUrl($myNotFoundPage,'','','full');
if ($modx->getObject('modResourceGroupResource', array('document_group'=>$groupId, 'document'=>$docId)) ) {
/* resource is in group */
$modx->sendRedirect($url);
}