5 Replies Latest reply on Mar 14, 2012 10:47 AM by peterj

    JBoss/JVM process RAM usage increasing steadily

    nambs

      On checking RAM usage for our JBoss/Java process using 'ps aux' command, RSS value= 12161968KB, and it is increasing steadily.

      We have set the Max heap size to 2GB while starting the process. We are checking heap size using 'jvisualvm' and it is below 2GB.

      (-Xss1m -XX:PermSize=128m -XX:MaxPermSize=256m -XX:NewRatio=2 -Xms2048m -Xmx2048m -XX:MaxPermSize=256m).

      Environment:

      OS: Red Hat Linux

      JVM: 1.6.0_30

      App Server: jboss-5.1.0.GA

       

      Need help on trouble shooting this issue.

        • 1. Re: JBoss/JVM process RAM usage increasing steadily
          peterj

          RAM for a Java app is consumed by more than just heap (RAM usage for the heap will never exceed 2048MB in your case).

           

          See this for a list of everything that can use up RAM:

          http://www.coderanch.com/t/111262/Linux-UNIX/read-memory-usage-process-running#560894

           

          How many threads are running? Have the number of threads increased? If so, that could account for the additonal RAM usage.

          • 2. Re: JBoss/JVM process RAM usage increasing steadily
            nambs

            We have tried to replicate this issue on Test bed with Heap size set to 1GB.

            On test bed as soon as JBoss is started we find that there are a lot of threads (JBoss System Thread(1) - around 300 threads) that are created.

            I have no idea what these threads are for, any information on this will be helpful.

            This is even before any request is made to the servers. But this is not observer on Production.

             

            Thread count is tabulated below. Though number of threads has decreased the RSS value is increasing.

             

            Date

            Time

            Total Thread

            Daemon Thread

            Non-Daemon Thread

            JBoss System Thread(1)

            http Thread

            VSZ

            RSS

            SZ

             

            2012-03-12

            17:55:30

            431

            403

            28

            300

            61

            2343840

            1028664

            585960

            mapped: 1835679K writeable/private: 1803527K shared: 0K

            2012-03-12

            18:22:42

            428

            399

            28

            300

            61

            2409304

            1500656

            602326

            mapped: 1835679K writeable/private: 1803527K shared: 0K

            2012-03-13

            09:52:52

            423

            395

            28

            300

            61

            2409304

            1501628

            602326

            mapped: 1835679K writeable/private: 1803527K shared: 0K

            • 3. Re: JBoss/JVM process RAM usage increasing steadily
              peterj

              You have the heap set to 2GB. The JVM will reserve that memory immediately, but will not start using it until it needs it. That would expalin the RSS increasing  - as the app runs and it uses the heap the JVM will start to make use of some of that memory that it has allocated and the OS will make it resident. Notice that after the app has been runnign for a while that the RSS rmains fairly constant.

               

              Note that the VSZ remain fairly constant. The increase after the initial size can be attributed to the perm gen increasing as classes are loaded.

               

              I don't see anything here that would cause me alarm.

               

              As far as what each of the threads are doing, you could take a JVM thread dump and see.

              • 4. Re: JBoss/JVM process RAM usage increasing steadily
                nambs

                Values mentioned in the above post were of server where Heap size was configured as 1GB.

                 

                 

                Below are values from the server where Heap is configured to 2GB but RSS=5.5GB and this keeps increasing till it uses up the entire RAM available.

                Also the thread count is almost constant for over 24hrs but RSS value is increasing.

                 

                Date

                Time

                Total

                threads

                Daemon

                threads

                Non daemon

                threads

                ‘JBoss

                System threads’

                HTTP

                threads

                VSZ

                RSS

                SZ

                mapped:

                writeable/private:

                shared:

                2012/3/13

                11:50:48

                277

                245

                32

                2

                208

                3644988

                3026036

                911247

                mapped: 3653180K

                writeable/private: 3403160K

                shared: 3272K

                2012/3/13

                14:32:03

                276

                244

                32

                2

                208

                3817224

                3186660

                954306

                mapped: 3825416K

                writeable/private: 3545084K

                shared: 3272K

                2012/3/13

                18:00:03

                276

                244

                32

                2

                208

                4175848

                3614728

                1043962

                mapped: 4184040K

                writeable/private: 3950492K

                shared: 3272K

                2012/3/14

                00:00:03

                276

                244

                32

                2

                208

                4696012

                4165484

                1174003

                mapped: 4704204K

                writeable/private: 4480168K

                shared: 3272K

                2012/3/14

                06:00:03

                276

                244

                32

                2

                208

                5168076

                4573556

                1292019

                mapped: 5176268K

                writeable/private: 4884224K

                shared: 3272K

                2012/3/14

                12:00:04

                276

                244

                32

                2

                208

                5953416

                5349432

                1488354

                mapped: 5961608K

                writeable/private: 5643848K

                shared: 3272K

                2012/3/14

                15:00:03

                276

                244

                32

                2

                208

                6349600

                5778032

                1587400

                mapped: 6357792K

                writeable/private: 6077580K

                shared: 3272K

                 

                • 5. Re: JBoss/JVM process RAM usage increasing steadily
                  peterj

                  Which JVM are you using? OpenJDK? Or the one from Oracle? If from Oracle, revert to version 1.6.0_26 and see if that helps (we have noticed various issues with versions more recent than that). If that doesn't help, you'll have to contact Oracle to see if there is a memory leak issue with the JVM itself.

                   

                  This almost sounds as if a system resource (such as a file handle or a socket connection) is not being released. But JBoss AS itself maintains pools of resources and reuses them, but who knows what the native JVM code is doing (only Oracle would know).