hm, there must be some side effects in my view - when I only use a modal and an editor it works flawlessly
okay - I realized that I can properly set the height, but not under 100px for the editor, is there a way to make it even smaller?
add domElementAttachment="parent" to modal panel. It will prevent it from moving to the top of the DOM before showing. we not changed tinyMCE scripts in any way. And unfortunatelly tiny Editor initialized wrong if its container moved within the DOM tree.
Also after you defined this attribute you'll be able to use modal panel inside some parent forms. (liitation with its own form will not be actuall).