-
1. Re: JBoss 5.1 crash
wdfink Dec 18, 2010 12:11 PM (in response to sashka_nem)Could you describe a bit more detail?
- When the JBoss stop, during startup, under load or idle?
- How the CPU/process load at this time
- does debug/tract for log4j give any hint
- can JConsole view any information?
-
2. Re: JBoss 5.1 crash
peterj Dec 21, 2010 5:49 PM (in response to sashka_nem)Don't rely on Windows Service Manager giving you any meaningful information. Instead, look in the server.log file. Most likely something caused a shutdown command to be sent to the app server and it shut itself down.
Also, did you set the -Xrs JVM option? Setting this option will prevent unwanted "shut down" signals. See this the "Myterious Shutdowns" section on this wiki page: http://community.jboss.org/wiki/RunJBossAsAServiceOnWindows
-
3. Re: JBoss 5.1 crash
sashka_nem Dec 22, 2010 9:17 AM (in response to sashka_nem)Hello
Thanks for your responses, I have already set -Xrs property. I want to add that there is nothing in server.log (or any other) and there is no JVM crash error log.
here is description of my server:
JBoss 5.1JVM: Java HotSpot(TM) 64-Bit Server VM (1.6.0_17)
-Xms3000M
-Xmx3000M
-XX:PermSize=192M
-XX:MaxPermSize=192M
-Xrs
-XX:+HeapDumpOnOutOfMemoryError
-XX:+UseParallelOldGC
-XX:ParallelGCThreads=20
-Dsun.rmi.dgc.client.gcInterval=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000
-Dorg.jboss.resolver.warning=true
-Dfile.encoding=UTF-8
-Dcom.sun.management.jmxremote.port=3333
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Djava.endorsed.dirs=D:\UPS\JBoss\lib\endorsedThe problem that I see in JConsole:
Operating System: Windows 2003 5.2
Architecture: amd64
Number of processors: 4
Committed virtual memory: 5,335,164 kbytes
Total physical memory: 4,455,644 kbytes
Free physical memory: 159,752 kbytes
Total swap space: 10,373,196 kbytes
Free swap space: 4,720,480 kbytesCommitted virtual memory permanently grows and when no Free swap space is availible service is killed.
It looks like native memory leak. I don't use any native libraries by myself.
I have read about some issue with VFS leaking, beacause of Sun JDK bug.
But unfortunately there is no solution for this issue yet.
However I can't say exactly if this bug is growing my Committed virtual memory.
Are there any known issues? Or ways to profile native memory leaks on windows?
-
4. Re: JBoss 5.1 crash in the end (Native Memory Leak)
wdfink Dec 22, 2010 8:42 AM (in response to sashka_nem)Hi Alexander,
I'm a bit confused about 64-Bit Server VM (17.0-b16, mixed mode)
it looks like 1.7 JVM. If so could you retest with an actual 1.6 VM?
The heap usage looks not bad.
-
5. Re: JBoss 5.1 crash in the end (Native Memory Leak)
sashka_nem Dec 22, 2010 9:18 AM (in response to wdfink)VisualVM displays Version that way, 6u17 is the version (1.6.0_17)
I've update my previouse post, to make it clearer.
-
6. Re: JBoss 5.1 crash in the end (Native Memory Leak)
sashka_nem Dec 22, 2010 9:35 AM (in response to sashka_nem)For last 3.5 Hours
Committed virtual memory: 5,837,052 kbytes
memory has increased for 500mb, compared to what I wrote earlier.
-
7. Re: JBoss 5.1 crash in the end (Native Memory Leak)
mstruk Dec 22, 2010 12:42 PM (in response to sashka_nem)https://issues.jboss.org/browse/JBVFS-159 was reported as Sun JDK Linux version specific issue.
Try to use some memory profiling tool on your application to try to find out what consumes all this memory. Often you can dramatically reduce memory consumption by optimizing your classpath - standardize on specific library versions and move them from your.war/WEB-INF/lib to jboss_home/server/default/lib .
-
8. Re: JBoss 5.1 crash in the end (Native Memory Leak)
sashka_nem Dec 22, 2010 1:41 PM (in response to mstruk)Thank you Marko for your response.
I will take in mind your siggestion.
In fact I've profiled my application many times and and there are no memory leaks on application layer, as heap memory is normal, and there are no OOME in the application. I used Yourkit, visualVM, JConsole and all needed tools.
But is there any way to profile native memory consumed by whole JVM?
-
9. Re: JBoss 5.1 crash in the end (Native Memory Leak)
mstruk Dec 22, 2010 2:56 PM (in response to sashka_nem)I have no experience with tracking native memory leaks, but I'm pretty sure that kind of profiling is beyond the capabilities of Java profilers like those you mention: JConsole, VisualVM, YourKit ...
A quick web search, and I found this: http://www.oracle.com/technetwork/java/javase/memleaks-137499.html#gbyvk
Not for the faint hearted ...
You have to limit -Xmx to well below max physical memory of the machine. Once you run out of physical memory, and your java process still demands more heap (or extra heap memory for that matter) you're in deep trouble, as your java process will become tremendously slow, and your system will most likely become non-responsive.
-
10. Re: JBoss 5.1 crash in the end (Native Memory Leak)
sashka_nem Dec 22, 2010 4:27 PM (in response to sashka_nem)Still having the issue.
Can +UseParallelOldGC cousing the Virtual Memory growth?
Or 3GB Heap Size on 4GB total memory server?
Is the issue with VFS related only for Linux OS and not for Windows?
-
11. Re: JBoss 5.1 crash in the end (Native Memory Leak)
sashka_nem Dec 23, 2010 2:51 AM (in response to sashka_nem) -
12. Re: JBoss 5.1 crash in the end (Native Memory Leak)
wdfink Dec 23, 2010 3:09 AM (in response to sashka_nem)A hint for you, if jconsole is active it will consume (increasing) memory of the servers JVM!
-
13. Re: JBoss 5.1 crash in the end (Native Memory Leak)
sashka_nem Dec 23, 2010 3:11 AM (in response to wdfink)No it is defenitely not because of JConsole, or any other tool, they are not attached all the time
-
14. Re: JBoss 5.1 crash in the end (Native Memory Leak)
wdfink Dec 23, 2010 3:57 AM (in response to sashka_nem)Sorry, I meant this only at hint that jConsole might compound your problem, not that jConsole is the problem ;-)