7 Replies Latest reply on Dec 1, 2004 8:04 PM by kalyan120

    ReadTask and Write Task Threads in UIL2

      Hi,

      We are running a test case where in the following happens:

      1. Client creates a temporary queue and sends the name to a stateless session bean
      2. The session bean does some work and puts back the result on the temporary queue.
      3. The client receives the message.

      When testing this behaviour with round about 100 clients (each in a separate jvm and each client sending a request to the server once in every 100 seconds, there by simulating 1 hit/sec on the server), the server becomes unusable. It doesn't die. It still processes the bean request, but was not able to send the message back to the client.

      We did a "kill -3" and figured out that there are many threads of ReadTask and WriteTask of uil2.SocketManager hanging. After some more testing we have noticed that, for every queue that's created, there are 3 ReadTask and 3 WriteTask threads being created. Once the client is killed, only 1 ReadTask and 1 WriteTask thread is being released. So, essentially, there are 4 threads (2 ReadTask and 2 WriteTask) threads hanging around even after the queue (and the client) are destroyed. At the time when the server reaches unusable state, there are around 1900 threads in the server. But there is still memory left and hence this is not a memory related issue. We are not sure as to what could be making the server unresponsive to JMS.

      Is it the number of clients that's causing the issue?
      Why are so many threads getting created for every connection (or queue), which are not released when the connection is closed (or killed)?

      Can someone throw some light on this?

      Thanks,
      Kalyan.

        • 1. non multicating(TCP), server dont see each other.

          hi all ,
          i'm running my JBOSS in cluster without multicasting that means i had configured my JBOSS in TCP ... and i'm running 2 JBOSS on different machine .... but the problem is that my JBOSS starts without any error oe exception but the problem is that they both didnt identify each other ... my cluster-service .xml file is :---


          <!-- UDP: if you have a multihomed machine,
          set the bind_addr attribute to the appropriate NIC IP address -->
          <!-- UDP: On Windows machines, because of the media sense feature
          being broken with multicast (even after disabling media sense)
          set the loopback attribute to true -->
          <TCP start_port="12800"/>
          <TCPPING initial_hosts="192.9.200.152[12800],192.9.200.150[12800]" port_range="5" timeout="3000" num_initial_members="3" up_thread="true" down_thread="true"/>
          <MERGE2 min_interval="5000" max_interval="10000"/>
          <FD shun="true" up_thread="true" down_thread="true" timeout="2500" max_tries="5"/>
          <VERIFY_SUSPECT timeout="3000" num_msgs="3" up_thread="true" down_thread="true"/>
          <pbcast.STABLE desired_avg_gossip="20000" up_thread="true" down_thread="true"/>
          <pbcast.NAKACK gc_lag="50" retransmit_timeout="300,600,1200,2400,4800" up_thread="true" down_thread="true"/>
          <UNICAST timeout="5000" window_size="100" min_threshold="10" down_thread="true"/>
          <FRAG frag_size="8192" down_thread="true" up_thread="true"/>
          <pbcast.GMS join_timeout="5000" join_retry_timeout="2000" shun="true" print_local_addr="true"/>
          <pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>



          according to documents in "initail hosts" i had given the IP of two machines and in machineA has also the same file and in machineB also have the same cluster-service.xml.

          pls tell me any solution ....

          bela...?

          Regards
          Raj........

          • 2. Re: ReadTask and Write Task Threads in UIL2
            starksm64

            A connection needs to be closed property to cleanup the ReadTask and WriteTask threads. There is only one ReadTask and WriteTask created per connection along with a thread pool for message delivery. When I run the temporary queue unit tests there are no ReadTask or WriteTask threads at the completion of the test. There are some msg delivery threads that will hang around for up to 60 seconds until the time to live expires.

            Create a bug report on the jbossmq project at the following url with your test code so we can see if there is a usage that is leaking threads.
            http://jira.jboss.com/jira/browse/JBMQ

            You can enable trace level logging on the org.jboss.mq category to see what is going on with the message delivery to the clients.

            • 3. Re: ReadTask and Write Task Threads in UIL2

              Thanks for the reply Scott

              Yes, we are aware that a connection has to be closed once it's usage is over. In fact, there are no threads hanging in the server if the clients are all killed. That part of the functionality is working fine. The problem is with the number of threads alive when the clients are still connected to the JBoss MQ server. Though, I'm repeating the steps again, they are more granular, so, here's what we do:

              1. Client creates a temporary queue and creates a receiver for this queue
              2. It caches the temorary queue in a hashtable (useful as it receives messages from server on the same queue often)
              2. Client binds this queue object to the jndi with the queue name
              3. Client invokes a statless session bean with the queue name
              4. Session Bean creates a string of 100KB (typically)
              5. It looks up the queue object from the jndi, for the given queue name.
              6. It creates a session and there by a sender for this queue.
              7. It sends the message
              8. The registered MessageListener on the client gets notified.

              Client sends requests to the server for every 100 secs (typically). The next time it tries to send the request, it doesn't create another temp queue. It reuses the same queue that's created earlier. Similarly, the server uses the same sender info that it has created earlier.

              Following are the code snippets for your interest:

              Client code:

              QueueConnectionFactory qconFactory =
               (QueueConnectionFactory) context.lookup(qConnFactory);
              
               QueueConnection connection = qconFactory.createQueueConnection();
               connection.setExceptionListener(
               new JMSConnectionExceptionListener(serverUrl));
               QueueSession session = connection.createQueueSession(false,
               Session.AUTO_ACKNOWLEDGE);
               Queue queue;
               // creating temporary queue
               queue = (Queue) session.createTemporaryQueue();
               queueName = queue.getQueueName();
               connection.start();
               QueueReceiver receiver = session.createReceiver(queue);
               receiver.setMessageListener(this);
               context.rebind(queue.getQueueName(), queue);


              Server code:

              holder = new QueueConnectionHolder();
               myQueueTable.put(queueName, holder);
              
               Queue queue = (Queue) myContext.lookup(queueName);
              
               QueueConnectionFactory factory =
               (QueueConnectionFactory) myContext.lookup(qConnFactory);
              
               QueueConnection connection = factory.createQueueConnection();
               holder.setConnection(connection); // remember queue connection
               QueueSession session =
               connection.createQueueSession(false, sessionMode);
               holder.setSession(session); // remember the session
              
               QueueSender sender = session.createSender(queue);
               holder.setSender(sender); // remember sender
              
               connection.setExceptionListener(this);
               connection.start();


              And finally sends the message this way:

              QueueConnectionHolder holder
               = (QueueConnectionHolder) myQueueTable.get(queueName);
               holder.getSender().send(message);


              We are not sure what could be causing the problem in this code. When I monitored the log on the server today, I still noticed that there are 3 "Begin WriteTask" and 3 "Being ReadTask" statements in the log. When I kill the client, I see all correspondily 3 "End WriteTask" and 3 "End ReadTask" log statements in the log. But if I don't kill the client, these threads are around in the server.

              Please advise on this. Meanwhile, I shall also report the test case in the bug report.

              Thanks for your time.

              Regards,
              Kalyan.

              • 4. Re: ReadTask and Write Task Threads in UIL2
                starksm64

                Every connection has a read/write task thread so to have 3 you have 3 connections. I only see two so its not clear where the third is coming from. On the server side, you can eliminate on UIL2 connection in favor of a INJVM connection if the session bean and JMS server are collocated. This is accessed using the java:/ConnectionFactory jndi name rather than ConnectionFactory.

                In the bug report indicated the os and jvm as the behavior and scalability with the number of threads will depend on this combo.

                • 5. Re: ReadTask and Write Task Threads in UIL2

                  Thanks for your reply Scott.

                  We have figured out that the deadlock is because of the method validateSoftReferenceDepth method in MessageCache. We have commented out the calls to that method and the server works fine. We have run 225 clients , each hitting the server every 100 secs for 2.5 days and still the performance is good. So, looks like that solved the problem for us.

                  I had opened a bug at : http://jira.jboss.com/jira/browse/JBMQ-4

                  We are using NullPersistenceManager as suggested in one of the wiki pages. Even then there is this business of Soft Referencing happening in the MessageCache. Is this required when there is no requirement for storing the messages?

                  I shall conduct a similar test with JVMIL and will let you know the results.

                  Thanks & Regards,
                  Kalyan.

                  • 6. Re: ReadTask and Write Task Threads in UIL2
                    genman


                    In 3.2.6 you can turn off soft referencing using the JMX attribute "MakeSoftReferences" on the message cache.

                    • 7. Re: ReadTask and Write Task Threads in UIL2
                      genman


                      By the way, the message cache is used to page messages out of memory, regardless if you want to persist them or not. This is to avoid out of memory conditions.

                      From looking at the stack trace in the bug, it seems to want to page out a message that hasn't been added to the server yet, which is odd.