6 Replies Latest reply on Apr 9, 2004 2:58 AM by bergerarnaud

    UIL2 processes and SocketManager

    bergerarnaud

      Hi everyone !

      We are currently using JMS through UIL2.

      The max load we have is around 150-160 client applications, and the number of processes is around 600 at that time (in best case).

      The problem is that the number of processes keeps growing during the day and sticks to a high number at night (around 350 processes with no
      clients connected !).

      Within about two days, the amount of processes reaches 900 + and JBoss crashes with an "unable to create native Threads".

      The dev's instance doesn't display this problem (with the same simulated load, around 150 clients)

      The only thing where logs differ between the dev and the prod logs is that
      production's server.log displays the exact same stack every five minutes
      (even at night) :

      Example start of JBoss

      2004-04-07 09:12:50,744 INFO [org.jboss.system.server.Server] JBoss (MX MicroKernel) [3.2.3 (build: CVSTag=JBoss_3_2_3 date=200311301445)] Started in 31s:962ms
      2004-04-07 09:13:46,603 ERROR [org.jboss.mq.il.uil2.SocketManager] Failed to create ObjectInputStream
      java.io.EOFException
       at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2165)
       at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2634)
       at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:734)
       at java.io.ObjectInputStream.<init>(ObjectInputStream.java:253)
       at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:265)
       at java.lang.Thread.run(Thread.java:536)
      2004-04-07 09:13:49,147 DEBUG [org.apache.tomcat.util.threads.ThreadPool] Getting new thread data
      2004-04-07 09:13:55,411 DEBUG [org.apache.tomcat.util.threads.ThreadPool] Getting new thread data
      2004-04-07 09:18:46,429 ERROR [org.jboss.mq.il.uil2.SocketManager] Failed to create ObjectInputStream
      java.io.EOFException
       at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2165)
       at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2634)
       at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:734)
       at java.io.ObjectInputStream.<init>(ObjectInputStream.java:253)
       at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:265)
       at java.lang.Thread.run(Thread.java:536)
      2004-04-07 09:23:46,187 ERROR [org.jboss.mq.il.uil2.SocketManager] Failed to create ObjectInputStream
      java.io.EOFException
       at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2165)
       at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2634)
       at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:734)
       at java.io.ObjectInputStream.<init>(ObjectInputStream.java:253)
       at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:265)
       at java.lang.Thread.run(Thread.java:536)
      2004-04-07 09:28:46,688 ERROR [org.jboss.mq.il.uil2.SocketManager] Failed to create ObjectInputStream
      java.io.EOFException
       at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2165)
       at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2634)
       at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:734)
       at java.io.ObjectInputStream.<init>(ObjectInputStream.java:253)
       at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:265)
       at java.lang.Thread.run(Thread.java:536)
      2004-04-07 09:33:46,066 ERROR [org.jboss.mq.il.uil2.SocketManager] Failed to create ObjectInputStream
      java.io.EOFException
       at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2165)
       at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2634)
       at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:734)
       at java.io.ObjectInputStream.<init>(ObjectInputStream.java:253)
       at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:265)
       at java.lang.Thread.run(Thread.java:536)
      2004-04-07 09:38:46,494 ERROR [org.jboss.mq.il.uil2.SocketManager] Failed to create ObjectInputStream
      java.io.EOFException
       at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2165)
       at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2634)
       at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:734)
       at java.io.ObjectInputStream.<init>(ObjectInputStream.java:253)
       at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:265)
       at java.lang.Thread.run(Thread.java:536)


      So, does anybody know what could cause this stack, and if each of these errors may generate a superfluous process ?

      The second question being : is it normal that UIL2 shows far more processes than , say OIL for instance ( 2 or 3 times more ) ?

      Thanks in advance ,

      Best regards,


      Arnaud

        • 1. Re: UIL2 processes and SocketManager

          This is a bug which has already been fixed:
          http://cvs.sourceforge.net/viewcvs.py/jboss/jbossmq/src/main/org/jboss/mq/il/uil2/SocketManager.java?r1=1.1.4.8&r2=1.1.4.9

          UIL2 does use more threads than OIL making it more concurrent/scalable
          when there are multiple sessions on the same connection.
          It is on my list of enhancements to make use a shared thread pool for these threads.

          Regards,
          Adrian

          • 2. Re: UIL2 processes and SocketManager
            bergerarnaud


            Thanks a lot Adrian, I wasn't aware of it !

            So , should I install 3.2.4 RC1 or is it a way for me to upgrade 3.2.3 release with this modified class ?

            Best regards,

            Arnaud



            • 3. Re: UIL2 processes and SocketManager
              bergerarnaud

              Please forget last question , since we will wait for the 3.2.4 release.

              Having been through JBoss's Thread dump, it appears that this morning, before start of production, there is about 300 WriteTask threads left for only 5 or 6 ReadTask.

              Thread: UIL2.SocketManager.WriteTask#1314 : priority:5, demon:true
              Thread: UIL2.SocketManager.WriteTask#1316 : priority:5, demon:true
              Thread: UIL2.SocketManager.WriteTask#1318 : priority:5, demon:true
              Thread: UIL2.SocketManager.WriteTask#1320 : priority:5, demon:true
              Thread: UIL2.SocketManager.WriteTask#1322 : priority:5, demon:true
              Thread: UIL2.SocketManager.WriteTask#1324 : priority:5, demon:true
              Thread: UIL2.SocketManager.WriteTask#1326 : priority:5, demon:true
              Thread: UIL2.SocketManager.WriteTask#1328 : priority:5, demon:true
              Thread: UIL2.SocketManager.WriteTask#1330 : priority:5, demon:true
              Thread: UIL2.SocketManager.WriteTask#1332 : priority:5, demon:true
              Thread: UIL2.SocketManager.WriteTask#1334 : priority:5, demon:true
              Thread: UIL2.SocketManager.WriteTask#1336 : priority:5, demon:true
              Thread: UIL2.SocketManager.WriteTask#1338 : priority:5, demon:true
              Thread: UIL2.SocketManager.WriteTask#1340 : priority:5, demon:true
              Thread: UIL2.SocketManager.WriteTask#1342 : priority:5, demon:true
              Thread: UIL2.SocketManager.WriteTask#1344 : priority:5, demon:true
              



              Is this the direct consequence of the failed ReadTask(s) ?

              Also I've been wondering why the "ReadTask stacks" appear exactly each 5 minutes, and even at restart of JBoss ( with or without clients connected)


              Best regards,

              Arnaud


              • 4. Re: UIL2 processes and SocketManager

                No idea about the 5 minute thing, it sounds like you have some client
                connecting the UIL2 port, but it is not a messaging client.
                This is how we discovered this bug in the first place.

                Regards,
                Adrian

                • 5. Re: UIL2 processes and SocketManager
                  bergerarnaud


                  Thanks for the answers Adrian !

                  Regards,


                  Arnaud

                  • 6. Re: UIL2 processes and SocketManager
                    bergerarnaud


                    Just a note (it'll be the last) to tell you that some "nagiosed" task
                    did call the 8093 port of the server (which is in fact our UIL2's port).

                    That clearly explains the stacks.

                    Thanks again for the good pointer !

                    Regards,

                    Arnaud