It might be possible to create a plugin connected to OnUserFormPrerender or OnUserFormRender that does it. You'd have to see if $modx->user is available at that point (and is the current user), and if the $user variable holds the user object for the user about to be edited. I'm not sure. If you get both of those, you can use
if ($user->isMember('somegroup')) {
}
to check memberships, and if they're not authorized, forward them somewhere with
$modx->sendRedirect($url)
The forward can be to a front-end page or some page in the Manager.
The user about to be edited should also be available as $_GET['id'], but you'd have to get the user object with this before calling isMember().
$u = $modx->getObject('modUser', $_GET['id']);
if ($u->isMember('somegroup')) {
}