Are your 2 examples there the same?
I always recommend avoiding output filters whenever there's another way to do this -- output filters can become tangled messes and are difficult to debug and impossible to cache, so they are often a bad tradeoff for appearances.
If I were you, I would write a custom snippet (which you'd be doing anyway if you were to write a custom output filter
http://rtfm.modx.com/display/revolution20/Input+and+Output+Filters+%28Output+Modifiers%29#InputandOutputFilters%28OutputModifiers%29-CreatingaCustomOutputModifier)
Just use runSnippet() to call one or the other variations of Wayfinder that you need (
http://rtfm.modx.com/display/revolution20/modX.runSnippet), e.g.
<?php
// UNTESTED
if ($modx->resource->get('isfolder')) {
$args = array();
// insert properties here
return $modx->runSnippet('Wayfinder', $args);
}
else {
$args = array();
// insert other properties here
return $modx->runSnippet('Wayfinder', $args);
}
Or, if you want to avoid the hassle of dealing with various behaviors, just peg your folders as folders and not as URLs -- you can do this by using the "getFirstChild" snippet, which redirects to the first child inside of a folder. I also saw a good trick the other day: create a webLink and set its target as the first child (for weblinks can also be treated as folders).
Hope that helps.