1 2 3 Previous Next 35 Replies Latest reply on Jan 11, 2013 2:49 PM by apparaonali Go to original post
      • 15. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
        apparaonali

        I'm not sure how your EJB client code could cause this issue but you might check if your closing the InitialContext in a try/finally clause.  I'm not saying to make that change now but it would be good to let us know if the context is getting closed.

         

              From your other details, this issue does sound related to your EJB client code.  Check to see if you forgot to close the InitialContext.

         

        >>> I looked at the code and we are not closing the InitialContext. I added code to close the context after getting lookup and didn't observe any improvement.

        • 16. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
          smarlow

          Here are some resources that describe the process of finding memory leaks.  This might be useful if you haven't read them yet (both from http://wiki.eclipse.org/index.php/MemoryAnalyzer#Finding_Memory_Leaks).

           

          http://dev.eclipse.org/blogs/memoryanalyzer/2008/04/21/immortal-objects-or-how-to-find-memory-leaks/

           

          https://www.sdn.sap.com/irj/sdn/weblogs?blog=/pub/wlg/6856

           

          If the memory leak is caused by code in AS7, we should fix that.  You can drill in deeper to find the root issue or create a standalone example that reproduces the leak.  The standalone example could be attached to a bug report.

           

          Scott

          • 17. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
            apparaonali

            I wrote a sample application and I am not able to reproduce the issue using the sample application.

            With sample application after 4 thousand bean method invocations there are only 2 org.xnio.nio.NioXnioWorker

             

            However I am getting the issue in our application and our application ported from Jboss6 to Jboss7.

             

            What could be the reason for accumating org.xnio.nio.NioXnioWorker objects in our application?

            Do I need to debug the XNIO library to find reason for the objects accumation?

            Any other suugestions?

            • 18. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
              smarlow

              By debugging, you could find out what is accumulating the org.xnio.nio.NioXnioWorker instances.  Debugging with sources can be done with Eclipse or IntelliJ. 

               

              Debugging without sources can be done with JDB.  You might also be able to get enough information from the eclipse memory analyzer tool.  You could compare the two incoming references with the sample application, to the many incoming references to org.xnio.nio.NioXniowork in your ported app.

               

              Also, http://www.redhat.com/support/ is another option (if you have a subscription for JBoss AS).

              • 19. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
                apparaonali

                Hi Scott,

                   Thanks for your help.

                Here are the references for the XNIO WorkerThread when I run the tests against our application.

                When I an running the tests against the application I could see JMX related objects, however when I run some tests against sample application I couldn't find any references to JMX.

                 

                XnioWorkerThreadReferencesAgainstApplication.png

                 

                Here are refernces for XNIO WorkerThread when I run tets against sample application.

                XnioWorkerThreadReferencesAgainstSampleApplication.png

                 

                • 20. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
                  apparaonali

                  Scott,

                    Can you suggest me which module should be debugged XNIO or Jboss Remoting?

                  • 21. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
                    smarlow

                    Sounds likely to be storage associated with your remote EJB instances.

                    • 22. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
                      apparaonali

                      When I opened Jconsole I observed couple of things and here are my observations.

                       

                      • If you look at below "jboss.remoting.handler" - has many obejcts of "management-client-#" and these objects are getting increased with remote method invocations.
                      • management-cleint=# object is having Xnio Worker Thread
                      • I commented all the code related to JMX in our applications code and still I am observing these objects.
                      • One more observation, I enabled trace in standalone.xml for XNIO  and remoting packages, here is a difference. Please look at red and green colored words.
                        • Log statement where memory leak is there- Log collected during our application
                          • 13:22:27,562 TRACE [org.xnio.channels.framed] (Remoting "alh-anw7-lt:MANAGEMENT" read-1) Accepting java.nio.HeapByteBuffer[pos=0 lim=12 cap=8192] into java.nio.HeapByteBuffer[pos=0 lim=8196 cap=8196]
                        • Log Statment where memory leak is not there - Log collected during test application
                          • 13:11:37,306 TRACE [org.xnio.channels.framed] (Remoting "alh-anw7-lt" read-1) Accepting java.nio.HeapByteBuffer[pos=0 lim=7 cap=8192] into java.nio.HeapByteBuffer[pos=0 lim=8196 cap=8196]

                      JconsoleAfterPerfTests.png

                      • 23. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
                        smarlow

                        Just divide and conquer.  Your application leaks memory and your sample doesn't.  Start removing things from your application until it doesn't leak anymore.  Or add more of your application code to the sample.  Keep adding until the leak appears.

                        • 24. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
                          apparaonali

                          My Environment was

                          JDK: 1.6 update 29, 32-bit

                          JBOSS7.1.2

                          PotsgreSql 9.1.1, 32-bit

                          Windows7 64-bit

                           

                          When I used postgresql 64 bit then I am not observing the memory leak

                          After update here is my environment where I am not observing the memory leak.

                          JDK: 1.6 update 29, 32-bit

                          JBOSS7.1.2

                          PotsgreSql 9.1.1, 64-bit

                          Windows7 64-bit

                           

                          Are there any issues in Jboss7 and postgresql 32-bit integration on 64-bit windows os?


                          • 25. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
                            apparaonali

                            Finally found the reason for the memory leak.

                             

                            It was not due to PostgreSQl and it was with 32 bit JDK.

                             

                            When JDK is 32 bit, Jboss was string with option "-client".

                            When I added "-server" to JVM arguments in standalone.conf.bat then this memory leak issue is not there.

                             

                            Thanks everyone for your support.

                             

                            I would like to understand when to use "-client" and when to use "-server" option for jboss start.

                            Can any one provide some pointers?

                            • 26. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
                              smarlow

                              Try checking the release notes for JDK 1.6 updates for memory leak fixes.  Also try the latest JDK 1.6 update to see if that makes a difference when you run with the client jvm.  I don't understand yet why your seeing this problem.

                              • 27. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
                                apparaonali

                                I downloaded latest jdk 1.16 update 38 and I obsrved the similar behavior as existing jdk 1.16 update 29.

                                 

                                I don't know whether should I start Jboss with "-server" option or not?

                                What are the issues in starting the jboss with "-server"?

                                What is recomended jvm options for Jboss start?


                                • 28. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
                                  smarlow

                                  You can get details about using the -server option by searching for that.  -client is typically used by client applications (hotspot will be less aggressive).  -server is typically used by server applications (hotspot will be more aggressive about converting Java bytecode into executable machine code).

                                   

                                  I don't understand what using -server versus -client has to do exactly with the OutOfMemory error that you are getting.  Are you still getting the outofmemory error when you use the -client option?  Or are you just seeing high memory usage with Eclise Memory Analyzer?

                                   

                                  Scott

                                  • 29. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
                                    apparaonali

                                    Thanks for your reply.

                                     

                                    Here are comments

                                     

                                    SCOTT>> I don't understand what using -server versus -client has to do exactly with the OutOfMemory error that you are getting.  Are you still getting the outofmemory error when you use the -client option?  Or are you just seeing high memory usage with Eclise Memory Analyzer?

                                     

                                    apparaonali>> When I am starting jboss without setting "-server" option in standalone.conf.bat, then jboss jvm is taking around 350MB. After I run the

                                                          tests jboss jvm reaching to 1100MB. If I run the tests 4 times then it is throwing "OutOfMemoryError". I observed the jboss java process on task manager on windows and

                                                          the memory is getting increased continuously and it reaching to 1100MB before end of the test run.

                                             

                                                         When I am starting jboss after setting "-server" option in standaalone.conf.bat, then jboss jvm is taking aroud 550MB. After I run the tests

                                                         jboss jvm is around 550MB. I didn't run the tests 4 times. I will run and update. I thought as the jboss jvm memory is not getting increased/accumulated after the test run

                                                         hence I felt that there is no memory leak. I observed the jboss java process on task manager on windows and the memory is getting increasing and decrsing and at the end

                                                         of the test it coming back to around 550MB.

                                     

                                    After this observation, I decided to start the jboss with "-server" option.

                                    Before conclude on this, I would like to understand the following

                                    What are the issues if I start the Jboss with "-server" option?

                                    Why there was 200MB more during Jboss startup with "-server" option?