The parser sometimes has trouble with modifiers that have a 'then' clause with no 'else'. That could be the problem, or you could be right that it's the nesting. Output modifiers aren't really intended to solve problems that involve complex decision-making.
If I'm reading your code correctly and it worked as it should, the alias would be displayed twice if you're on the home page and it's published. Is that really what you want?
Since the Home page would always be published and its pagetitle would be Home (right?) the inner part seems unnecessary, but maybe I'm missing something.
Your code would probably work, and produce the same result if you did this:
[[*published:is=`1`:then=`
[[*alias]]']]
[[*pagetitle:is=`Home`:then=`[[*alias]]`]]
I was going to offer a simple snippet that would do the job much faster than nested conditional output modifiers, but I can't tell for sure what you're trying to accomplish.