7 Replies Latest reply on Sep 14, 2005 1:11 PM by clizarralde

    Infinite Thread generation in JBossMQ client?

    geir1

      It seems like the thread pool that the JBoss MQ is using is generating an infinite number of threads. We are seeing this on processes running over an extended time period with connects and disconnects. We are experiencing that a single process will acquire up to 500 threads in 20 hours, before our OS is stopping the process and the application fails with an OutOfMemoryError. The application has about 30 "working" threads.

      I have made some sample code to demonstrate the effect of reconnecting to a queue. This code never has more than 4 "working" threads.

      Is there a way of configuring a maximum size of the JbossMQ thread pool? Is there any obvious ways to stop the thread pool growing? Is it possible to turn off the Thread pooling? The optimizing effect of the pool is removed by the fact that the app now runs with too many threads.

      Geir H. Pettersen

      Some observations:
      1. The threadpool grows to about 34 if the sleep value is set to 2000; if it is set to 1000 it stabilizes on 65. If set lower it will grow more. My guess is that this differs between different systems.
      2. All the "idle" threads are doing the same thing:

      "Thread-5" daemon prio=5 tid=0x186bbf80 nid=0x8214 in Object.wait() [1936f000..1936fd88]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x1003df90> (a EDU.oswego.cs.dl.util.concurrent.LinkedNode)
       at EDU.oswego.cs.dl.util.concurrent.SynchronousChannel.poll(SynchronousChannel.java:353)
       - locked <0x1003df90> (a EDU.oswego.cs.dl.util.concurrent.LinkedNode)
       at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:707)
       at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:731)
       at java.lang.Thread.run(Thread.java:534)
      


      Example code:
      import javax.jms.JMSException;
      import javax.jms.Message;
      import javax.jms.MessageListener;
      import javax.jms.Queue;
      import javax.jms.QueueConnection;
      import javax.jms.QueueConnectionFactory;
      import javax.jms.QueueReceiver;
      import javax.jms.QueueSession;
      import javax.naming.InitialContext;
      import javax.naming.NamingException;
      
      /**
       * Class for testing the possible infinite Thread generation in
       * JBOSS-JMS client.
       *
       * @author Geir H. Pettersen
       * @version $Revision$
       */
      public class JMSClientThreads {
       private QueueConnectionFactory connectionFactory;
       private Queue queue;
       private int threadCount;
      //-----------------------------------------------------------------------------
       public JMSClientThreads() throws NamingException, JMSException {
       init();
       }
      //-----------------------------------------------------------------------------
       public void connect() throws JMSException {
       QueueConnection qConn;
       qConn = connectionFactory.createQueueConnection();
       QueueSession qSession = qConn.createQueueSession(false,QueueSession.CLIENT_ACKNOWLEDGE);
       QueueReceiver qReceiver = qSession.createReceiver(queue);
      
       qReceiver.setMessageListener(new MessageListener() {
       public void onMessage(Message message) {
       System.out.println("Recieved message");
       try {
       message.acknowledge();
       } catch (JMSException e) {
       System.out.println("Acknowledge failed");
       }
       }
       });
      
       qConn.start();
       try {
       Thread.sleep(100);
       } catch (InterruptedException e) {
       }
      
       qConn.close();
       qConn = null;
       }
      //-----------------------------------------------------------------------------
       public void init() throws NamingException, JMSException {
       System.setProperty("java.naming.factory.initial","org.jnp.interfaces.NamingContextFactory");
       System.setProperty("java.naming.provider.url","localhost:1199");
       System.setProperty("java.naming.factory.url.pkgs","org.jboss.naming:org.jnp.interfaces");
       InitialContext ic = new InitialContext();
       connectionFactory = (QueueConnectionFactory) ic.lookup("UILConnectionFactory");
       queue = (Queue) ic.lookup("queue/toExtQ");
       ic.close();
       }
      //-----------------------------------------------------------------------------
       private void debugThreads() {
       ThreadGroup threadGroup = Thread.currentThread().getThreadGroup();
       StringBuffer threadNames = new StringBuffer(128);
       int threadCount = threadGroup.activeCount();
       System.out.println("---");
       Thread[] threads = new Thread[threadCount];
       threadCount = threadGroup.enumerate(threads, true);
       System.out.println("Active threadcount: " + threadCount);
       for (int j = 0; j < threadCount; j++) {
       String threadName = threads[j].getName();
       threadNames.append(threadName);
       if (j+1 < threadCount) {
       threadNames.append(',');
       }
       }
       System.out.println("Active threads: " + threadNames);
       System.out.println("---");
       this.threadCount = threadCount;
       }
      //-----------------------------------------------------------------------------
       public void run() {
       while (true) {
       debugThreads();
       try {
       connect();
       } catch (JMSException e) {
       e.printStackTrace();
       }
       }
       }
      //-----------------------------------------------------------------------------
       public static void main(String[] args) throws NamingException, JMSException {
       JMSClientThreads t = new JMSClientThreads();
       t.run();
       }
      }
      
      



        • 1. Re: Infinite Thread generation in JBossMQ client?
          genman


          Looks like what your code does is connect, the disconnect. Is there some way you can avoid connecting and disconnecting so quickly? You can have one connection per application and create sessions per Thread even if multi-threaded.

          Use UIL2ConnectionFactory (see the FAQ) not UILConnectionFactory.

          You can also check out the source in CVS and modify settings yourself. Here are the settings I'm seeing:

          ./src/main/org/jboss/mq/il/uil2/SocketManager.java

          pool = new PooledExecutor(5);
          pool.setMinimumPoolSize(1);
          pool.setKeepAliveTime(1000 * 60);
          pool.runWhenBlocked();

          These values are hard coded, you may modify the keep alive time and build it yourself. Or, better, add a configuration via JMX to the connection factory and submit a patch.

          ./src/main/org/jboss/mq/il/uil2/UILServerILService.java

          • 2. Re: Infinite Thread generation in JBossMQ client?
            geir1

             

            Looks like what your code does is connect, the disconnect.

            Yep, This is true. This code only demonstrates a problem I am experiencing in a lager application.
            Is there some way you can avoid connecting and disconnecting so quickly?

            I have been trying to find a way... But I need each client to have their own session that can be started and stopped individually.

            Problem is that in our real application the connect/disconnect does not happen that fast. When the app starts about 10 connects will occur within notime. (After these connects the client has aquired over 50 threads!! ) After this a connect/disconnect occurs every few minutes. (threads increases steadily util app crashes)

            You can have one connection per application and create sessions per Thread even if multi-threaded.

            Hmm... How can I start and stop recieveing messages from a session if I want to use qReceiver.setMessageListener(...)? Now I am using start() and stop() on the queueConnetion.

            Will it work with qReceiver.close()? Probably.

            Ok, Ill try it out. Thanx.


            • 3. Re: Infinite Thread generation in JBossMQ client?
              genman


              If you want to use Connection.start and Connection.stop per client, then I see why you'd have a need for many of them.

              As I said, feel free to make changes to the JBoss source and send the patch my way. If it looks good, I'll make it available for 3.2.4.

              • 4. Re: Infinite Thread generation in JBossMQ client?
                geir1

                Hi,

                I have solved the problem by using a singelton wrapper for the Connection class. This will also enable all clients to be Exception listeners since the wrapper class distributes the onException() call.

                I am starting and stopping with, addMessageListener(...) and session.close().

                Seems to work nicely.

                Thanx,
                Geir.

                • 5. Re: Infinite Thread generation in JBossMQ client?
                  clizarralde

                  I am having the same problem as you.

                  I have 3 schedulers which are creating about 2 jms connections per second and close them 2 seconds later. The amount of threads (readtasks and writetasks) created by JBoss grows rapidly to 4 hundred when it stops growing. If we decrease the creation rate then the amount of threads decrease.

                  I have JBoss 4.0.1.sp1 and a I was wondering wich would be the best aproach:

                  a) Modify UIL2 SocketManager code.
                  b) Wrap the JMSConnection access with a singleton.

                  By the way, is it correct worrying about this issues? Shoudn't these be handled by JBoss? Are there any plans on changing this for future releases?

                  • 7. Re: Infinite Thread generation in JBossMQ client?
                    clizarralde

                     

                    "scott.stark@jboss.org" wrote:
                    http://jira.jboss.com/jira/browse/JBAS-1337
                    http://jira.jboss.com/jira/browse/JBAS-1338


                    Ok.

                    We are going to re-use the jms connections avoiding creating them.

                    Greetings