3 Replies Latest reply on May 30, 2008 5:09 PM by peterj

    HANamingService-OutOfMemory

    igain

      Hi All,

      We are using Jboss AS 4.2.0 GA version in clustered env. in production.
      There are 2 nodes in this cluster.

      I have started getting OutOfMemory exception for HANamingService. similar error i got for JbossMQ as well. i.e. OutOfMemory and unable to create new native threads


      Any help would be greatly appreciated.

      2008-05-30 11:21:36,335 ERROR [org.jboss.ha.jndi.HANamingService] Unexpected exception during accept
      org.jboss.util.threadpool.ThreadPoolFullException: java.lang.OutOfMemoryError: unable to create new native thread
       at org.jboss.util.threadpool.BasicThreadPool.execute(BasicThreadPool.java:417)
       at org.jboss.util.threadpool.BasicThreadPool.runTaskWrapper(BasicThreadPool.java:192)
       at org.jboss.util.threadpool.BasicThreadPool.run(BasicThreadPool.java:212)
       at org.jboss.util.threadpool.BasicThreadPool.run(BasicThreadPool.java:206)
       at org.jboss.ha.jndi.DetachedHANamingService$AcceptHandler.run(DetachedHANamingService.java:715)
       at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
      
      
       at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
       at java.lang.Thread.run(Thread.java:595)
      2008-05-30 11:21:39,482 INFO [STDOUT] jobList size --->0
      2008-05-30 11:21:42,957 INFO [STDOUT] jobList size --->0
      
      
      2008-05-30 11:49:22,173 ERROR [org.jgroups] uncaught exception in Thread[MERGE2.FindSubgroups thread (channel=ETProd-SFSBCache),5,JGroups threads] (thread group=org.jgroups.util.Util$1[name=JGroups threads,maxpri=10] )
      java.lang.OutOfMemoryError: unable to create new native thread
       at java.lang.Thread.start0(Native Method)
       at java.lang.Thread.start(Thread.java:574)
       at org.jgroups.protocols.PingWaiter.start(PingWaiter.java:51)
       at org.jgroups.protocols.Discovery.down(Discovery.java:301)
       at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:499)
       at org.jgroups.stack.Protocol.passDown(Protocol.java:533)
       at org.jgroups.protocols.MERGE2$FindSubgroups.findInitialMembers(MERGE2.java:333)
       at org.jgroups.protocols.MERGE2$FindSubgroups.run(MERGE2.java:291)
       at java.lang.Thread.run(Thread.java:595)
      2008-05-30 11:49:26,709 ERROR [org.jgroups] uncaught exception in Thread[PingWaiter,5,JGroups threads] (thread group=org.jgroups.util.Util$1[name=JGroups threads,maxpri=10] )
      java.lang.OutOfMemoryError: unable to create new native thread
       at java.lang.Thread.start0(Native Method)
       at java.lang.Thread.start(Thread.java:574)
       at org.jgroups.protocols.PingSender.start(PingSender.java:34)
       at org.jgroups.protocols.PingWaiter.findInitialMembers(PingWaiter.java:116)
       at org.jgroups.protocols.PingWaiter.run(PingWaiter.java:99)
       at java.lang.Thread.run(Thread.java:595)
      2008-05-30 11:49:27,655 ERROR [org.jgroups] uncaught exception in Thread[PingWaiter,5,JGroups threads] (thread group=org.jgroups.util.Util$1[name=JGroups threads,maxpri=10] )
      java.lang.OutOfMemoryError: unable to create new native thread
       at java.lang.Thread.start0(Native Method)
       at java.lang.Thread.start(Thread.java:574)
       at org.jgroups.protocols.PingSender.start(PingSender.java:34)
       at org.jgroups.protocols.PingWaiter.findInitialMembers(PingWaiter.java:116)
       at org.jgroups.protocols.PingWaiter.run(PingWaiter.java:99)
       at java.lang.Thread.run(Thread.java:595)
      2008-05-30 11:50:11,988 WARN [org.apache.tomcat.util.http.Parameters] Parameters: Invalid chunk ignored.
      2008-05-30 11:50:11,989 WARN [org.apache.tomcat.util.http.Parameters]
      


        • 1. Re: HANamingService-OutOfMemory
          peterj

          You did not say which operating system you are running. Is it Windows? Are you running with the /3GB boot option? If so, then the operating system does not have enough memory to allocate the data structures it needs to manage another thread. You will have to reboot without the /3GB option. If you are on Linux, it has a similar option, but I don't know what it is off-hand.

          • 2. Re: HANamingService-OutOfMemory
            igain

            Hi Peter,
            We are using Linux AS4 having 4GB of RAM.
            and using sun jdk 1.5.0_12.

            Heap size that we are using is 2GB.

            I did not get what you mean by " /3GB boot option"

            • 3. Re: HANamingService-OutOfMemory
              peterj

              When running a 32-bit OS, applications are limited to 4GB of addressable space. By default, the OS allocates 2GB of the address space to itself and 2GB for the application. (This is not a dividing up of the RAM, rather how the virtual address space for the application is divided up. Even if your have only 256MB of RAM, the app still has 4GB of addressable space with a 2GB/2GB split. The OS maps the virtual address space to both RAM and the swap space on disk.) The /3GB boot option in Windows causes it to change the address space allocation - the OS gets only 1GB and the app gets 3GB. Unfortunately, that leaves very little memory for the OS to do everything it needs to do, with typically a limit on the number of threads that can be allocated the limiting factor. And app servers love to create lots and lots of threads.

              If I recall correctly, the above applies to both Windows and Linux, except for the name of the option. And as I said, I don't recall the option in Linux - googling should reveal it.

              Note that this discussion has nothing to do with heap size. Native thread allocation does not use up heap space, so increasing that would not help. But based on the heap size you are using, I assume that your Linux box is running with the equivalent of the /3GB option set. Without that option you can usually get only a 1.5GB heap (give or take some megabytes).

              By the way, what is Linux AS4? What distro?