The option should read "-Xmx512m". not "?Xmx512m".
Oh, my Internet Browser changed "-Xmx512m" to "?Xmx512m".
I have set "-Xmx512m", but it doesn't work...
I suspect that it is more than the browser that is changing the "-" into a "?", hence your problem. (In other words, either Eclispe or the JBoss IDE is also converting "-" to "?".) Sounds like a keyboard mapping isse (I assume you are not using the US keyboard mapping). Unfortunately, I don't have any suggestions on what you could do to solve that issue, other than running JBoss from the command line.
There is no '?' in the Eclipse or JBossIDE ('?' was may be my typing fault too) - the error is:
Exception in thread "main"
I can start server manually (jboss-4.0.2\bin\run.bat), but I'd like to server navigator works properly...
You haven't mentioned which operating system you are using, or which JVM. If you have the Sun JVM, try setting the _JAVA_LAUNCHER_DEBUG environment variable to 1 and try starting JBoss again. You will have to set this globally before starting Eclispe because the server configuration dialog doesn't have any place for setting environment variables. This environment variable causes the Java launcher to print out detailed information about how it is interpreting the command line.
Once you have it set, start Eclipse, start Jboss via the Server Navigator. The Console panel will contain the Java launcher debug output, post that here and I will look at it.
Having worked extensively with the Java launcher, I know that it scans the command line options looking for the first one that doesn't start with a minus sign (-) and it uses that option as the class name. (Well, almost, if it sees the -classpath option, it knows the next option is the path list and that doesn't start with minus sign.)
The only possibilities I can think of for the "-Xmx512m" being taken as a class name is, first the option is really " -Xmx512m". The space before the minus sign will cause problems. The other possibility is that the minus sign is not really a minus sign (ASCII 0x2D), but some other character code that is displayed as a minus sign (hence my statement about the non-US keyboard layout).
I found it - you were right - it was some bad '-' sign, which were not ASCII 0x2D, I was copied it from file viewer which copied something strange to clipboard.
I didn't realized that if I see '-', it may be something else...
Thank you very much,