This question has been answered by markh. See the first response.
You might be able of putting something together with a plugin (OnBeforeDocFormSave, check the number of children in the chosen parent, change it or return an error) but there's nothing for that out of the box.
If you post your final solution, this will be an excellent candidate for the Cookbook