-
15. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
apparaonali Dec 6, 2012 11:58 PM (in response to smarlow)I'm not sure how your EJB client code could cause this issue but you might check if your closing the InitialContext in a try/finally clause. I'm not saying to make that change now but it would be good to let us know if the context is getting closed.
From your other details, this issue does sound related to your EJB client code. Check to see if you forgot to close the InitialContext.
>>> I looked at the code and we are not closing the InitialContext. I added code to close the context after getting lookup and didn't observe any improvement.
-
16. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
smarlow Dec 7, 2012 10:01 AM (in response to apparaonali)Here are some resources that describe the process of finding memory leaks. This might be useful if you haven't read them yet (both from http://wiki.eclipse.org/index.php/MemoryAnalyzer#Finding_Memory_Leaks).
http://dev.eclipse.org/blogs/memoryanalyzer/2008/04/21/immortal-objects-or-how-to-find-memory-leaks/
https://www.sdn.sap.com/irj/sdn/weblogs?blog=/pub/wlg/6856
If the memory leak is caused by code in AS7, we should fix that. You can drill in deeper to find the root issue or create a standalone example that reproduces the leak. The standalone example could be attached to a bug report.
Scott
-
17. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
apparaonali Dec 10, 2012 11:56 AM (in response to smarlow)I wrote a sample application and I am not able to reproduce the issue using the sample application.
With sample application after 4 thousand bean method invocations there are only 2 org.xnio.nio.NioXnioWorker
However I am getting the issue in our application and our application ported from Jboss6 to Jboss7.
What could be the reason for accumating org.xnio.nio.NioXnioWorker objects in our application?
Do I need to debug the XNIO library to find reason for the objects accumation?
Any other suugestions?
-
18. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
smarlow Dec 10, 2012 3:03 PM (in response to apparaonali)By debugging, you could find out what is accumulating the org.xnio.nio.NioXnioWorker instances. Debugging with sources can be done with Eclipse or IntelliJ.
Debugging without sources can be done with JDB. You might also be able to get enough information from the eclipse memory analyzer tool. You could compare the two incoming references with the sample application, to the many incoming references to org.xnio.nio.NioXniowork in your ported app.
Also, http://www.redhat.com/support/ is another option (if you have a subscription for JBoss AS).
-
19. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
apparaonali Dec 10, 2012 4:06 PM (in response to smarlow)Hi Scott,
Thanks for your help.
Here are the references for the XNIO WorkerThread when I run the tests against our application.
When I an running the tests against the application I could see JMX related objects, however when I run some tests against sample application I couldn't find any references to JMX.
Here are refernces for XNIO WorkerThread when I run tets against sample application.
-
20. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
apparaonali Dec 10, 2012 6:27 PM (in response to apparaonali)Scott,
Can you suggest me which module should be debugged XNIO or Jboss Remoting?
-
21. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
smarlow Dec 12, 2012 9:07 AM (in response to apparaonali)Sounds likely to be storage associated with your remote EJB instances.
-
22. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
apparaonali Dec 12, 2012 2:31 PM (in response to smarlow)When I opened Jconsole I observed couple of things and here are my observations.
- If you look at below "jboss.remoting.handler" - has many obejcts of "management-client-#" and these objects are getting increased with remote method invocations.
- management-cleint=# object is having Xnio Worker Thread
- I commented all the code related to JMX in our applications code and still I am observing these objects.
- One more observation, I enabled trace in standalone.xml for XNIO and remoting packages, here is a difference. Please look at red and green colored words.
- Log statement where memory leak is there- Log collected during our application
- 13:22:27,562 TRACE [org.xnio.channels.framed] (Remoting "alh-anw7-lt:MANAGEMENT" read-1) Accepting java.nio.HeapByteBuffer[pos=0 lim=12 cap=8192] into java.nio.HeapByteBuffer[pos=0 lim=8196 cap=8196]
- 13:22:27,562 TRACE [org.xnio.channels.framed] (Remoting "alh-anw7-lt:MANAGEMENT" read-1) Accepting java.nio.HeapByteBuffer[pos=0 lim=12 cap=8192] into java.nio.HeapByteBuffer[pos=0 lim=8196 cap=8196]
- Log Statment where memory leak is not there - Log collected during test application
- 13:11:37,306 TRACE [org.xnio.channels.framed] (Remoting "alh-anw7-lt" read-1) Accepting java.nio.HeapByteBuffer[pos=0 lim=7 cap=8192] into java.nio.HeapByteBuffer[pos=0 lim=8196 cap=8196]
- 13:11:37,306 TRACE [org.xnio.channels.framed] (Remoting "alh-anw7-lt" read-1) Accepting java.nio.HeapByteBuffer[pos=0 lim=7 cap=8192] into java.nio.HeapByteBuffer[pos=0 lim=8196 cap=8196]
- Log statement where memory leak is there- Log collected during our application
-
23. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
smarlow Dec 12, 2012 3:52 PM (in response to apparaonali)Just divide and conquer. Your application leaks memory and your sample doesn't. Start removing things from your application until it doesn't leak anymore. Or add more of your application code to the sample. Keep adding until the leak appears.
-
24. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
apparaonali Dec 19, 2012 3:04 PM (in response to smarlow)My Environment was
JDK: 1.6 update 29, 32-bit
JBOSS7.1.2
PotsgreSql 9.1.1, 32-bit
Windows7 64-bit
When I used postgresql 64 bit then I am not observing the memory leak
After update here is my environment where I am not observing the memory leak.
JDK: 1.6 update 29, 32-bit
JBOSS7.1.2
PotsgreSql 9.1.1, 64-bit
Windows7 64-bit
Are there any issues in Jboss7 and postgresql 32-bit integration on 64-bit windows os?
-
25. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
apparaonali Dec 20, 2012 10:24 AM (in response to apparaonali)Finally found the reason for the memory leak.
It was not due to PostgreSQl and it was with 32 bit JDK.
When JDK is 32 bit, Jboss was string with option "-client".
When I added "-server" to JVM arguments in standalone.conf.bat then this memory leak issue is not there.
Thanks everyone for your support.
I would like to understand when to use "-client" and when to use "-server" option for jboss start.
Can any one provide some pointers?
-
26. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
smarlow Dec 20, 2012 10:41 AM (in response to apparaonali)Try checking the release notes for JDK 1.6 updates for memory leak fixes. Also try the latest JDK 1.6 update to see if that makes a difference when you run with the client jvm. I don't understand yet why your seeing this problem.
-
27. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
apparaonali Dec 20, 2012 5:08 PM (in response to smarlow)I downloaded latest jdk 1.16 update 38 and I obsrved the similar behavior as existing jdk 1.16 update 29.
I don't know whether should I start Jboss with "-server" option or not?
What are the issues in starting the jboss with "-server"?
What is recomended jvm options for Jboss start?
-
28. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
smarlow Dec 20, 2012 6:05 PM (in response to apparaonali)You can get details about using the -server option by searching for that. -client is typically used by client applications (hotspot will be less aggressive). -server is typically used by server applications (hotspot will be more aggressive about converting Java bytecode into executable machine code).
I don't understand what using -server versus -client has to do exactly with the OutOfMemory error that you are getting. Are you still getting the outofmemory error when you use the -client option? Or are you just seeing high memory usage with Eclise Memory Analyzer?
Scott
-
29. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
apparaonali Dec 20, 2012 9:07 PM (in response to smarlow)Thanks for your reply.
Here are comments
SCOTT>> I don't understand what using -server versus -client has to do exactly with the OutOfMemory error that you are getting. Are you still getting the outofmemory error when you use the -client option? Or are you just seeing high memory usage with Eclise Memory Analyzer?
apparaonali>> When I am starting jboss without setting "-server" option in standalone.conf.bat, then jboss jvm is taking around 350MB. After I run the
tests jboss jvm reaching to 1100MB. If I run the tests 4 times then it is throwing "OutOfMemoryError". I observed the jboss java process on task manager on windows and
the memory is getting increased continuously and it reaching to 1100MB before end of the test run.
When I am starting jboss after setting "-server" option in standaalone.conf.bat, then jboss jvm is taking aroud 550MB. After I run the tests
jboss jvm is around 550MB. I didn't run the tests 4 times. I will run and update. I thought as the jboss jvm memory is not getting increased/accumulated after the test run
hence I felt that there is no memory leak. I observed the jboss java process on task manager on windows and the memory is getting increasing and decrsing and at the end
of the test it coming back to around 550MB.
After this observation, I decided to start the jboss with "-server" option.
Before conclude on this, I would like to understand the following
What are the issues if I start the Jboss with "-server" option?
Why there was 200MB more during Jboss startup with "-server" option?