0 Replies Latest reply on Oct 9, 2008 7:41 AM by Ivan Sevastianov

    Blocked reception / dispatch

    Ivan Sevastianov Newbie

      Blocked reception / dispatch

      Hi gurus!

      We are using JBoss in big distributed system.

      Versions:
      OSs RHEL 4.1 (central server), Solaris 9 (regions)
      jboss 4.2.1 GA
      jboss messaging 1.4.0 SP3 (Oracle10G persistence)
      jboss remoting 2.2.2 SP7

      Many servers (clients) are connected to a central server (central server) on which the jms provider.
      Library versions (messaging and remoting) are identical on both the client and server.

      Every client sends and receives messages synchronously by
      timeout in EJB Timers using transacted connection factory.

      Connection factory and queue searched using jms resource adapter and External JNDI

      Every client caches two QueueConnection (one to send and one to receive)

      First, we had about 10 simultaneous clients, and everything worked fine.

      Then the number of clients has increased dramatically to about 100
      and on the most of clients timers is blocked. :-(

      I tried to increase the value JBM_clientMaxPoolSize to 2000 but this did not help.

      Perhaps exhausted threadpool or connection pool on the central server?

      Here are the client logs (logger level for messaging and remoting is set to TRACE):


      2008-10-09 14:32:26,136 INFO [ru.tower.ejb.jms.timeinfo.SenderInfo] Time is out .... sending
      2008-10-09 14:32:26,306 INFO [ru.tower.ejb.jms.cons.RemoteMsgConsumer] Preparing consumer ... Queue name: queue/Region38
      2008-10-09 14:32:26,306 DEBUG [ru.tower.ejb.jms.cons.RemoteMsgConsumer] About to look up the connection factory
      2008-10-09 14:32:26,781 DEBUG [ru.tower.ejb.jms.cons.RemoteMsgConsumer] About to create connection
      2008-10-09 14:32:26,810 DEBUG [ru.tower.ejb.jms.cons.RemoteMsgConsumer] About to look up the queue queue/Region38
      2008-10-09 14:32:27,588 DEBUG [ru.tower.ejb.jms.cons.RemoteMsgConsumer] About to create queue session
      2008-10-09 14:32:33,823 DEBUG [org.jboss.jms.ConnectionFactoryHelper] using connection factory: JBossConnectionFactory->ConnectionFactoryDelegate[jboss.messaging.connectionfactory:service=SlowConnectionFactory, SID=0]
      2008-10-09 14:32:33,823 DEBUG [org.jboss.jms.ConnectionFactoryHelper] using username/password: system/-- not shown --
      2008-10-09 14:32:34,324 DEBUG [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[902a8b, bisocket://1.99.14.56:4457] setting client socket wrapper class name to org.jboss.jms.client.remoting.ClientSocketWrapper
      2008-10-09 14:32:34,325 DEBUG [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[902a8b, bisocket://1.99.14.56:4457] setting shouldCheckConnection to false
      2008-10-09 14:32:34,325 DEBUG [org.jboss.remoting.transport.socket.SocketClientInvoker] SocketClientInvoker[902a8b, bisocket://1.99.14.56:4457] setting timeout to 0
      2008-10-09 14:32:34,326 DEBUG [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[902a8b, bisocket://1.99.14.56:4457] constructed
      2008-10-09 14:32:34,326 DEBUG [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[902a8b, bisocket://1.99.14.56:4457] setting client socket wrapper class name to org.jboss.jms.client.remoting.ClientSocketWrapper
      2008-10-09 14:32:34,326 DEBUG [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[902a8b, bisocket://1.99.14.56:4457] setting shouldCheckConnection to false
      2008-10-09 14:32:34,326 DEBUG [org.jboss.remoting.transport.socket.SocketClientInvoker] SocketClientInvoker[902a8b, bisocket://1.99.14.56:4457] setting timeout to 0
      2008-10-09 14:32:34,328 DEBUG [org.jboss.remoting.transport.bisocket.BisocketClientInvoker] Setting ping frequency to: 214748364
      2008-10-09 14:32:34,329 DEBUG [org.jboss.remoting.MicroRemoteClientInvoker] SocketClientInvoker[902a8b, bisocket://1.99.14.56:4457] connecting
      2008-10-09 14:32:34,329 DEBUG [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] Creating semaphore with size 50
      2008-10-09 14:32:34,330 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[902a8b, bisocket://1.99.14.56:4457] added new pool ([]) as ServerAddress[1.99.14.56:4457, NO enableTcpNoDelay timeout 0 ms, maxPoolSize=50]
      2008-10-09 14:32:34,330 DEBUG [org.jboss.remoting.MicroRemoteClientInvoker] SocketClientInvoker[902a8b, bisocket://1.99.14.56:4457] connected
      2008-10-09 14:32:34,540 TRACE [org.jboss.jms.client.delegate.DelegateSupport] ConnectionFactoryDelegate[jboss.messaging.connectionfactory:service=SlowConnectionFactory, SID=0] invoking ConnectionFactoryGetClientAOPStackRequest[eab576] synchronously on server using Client[19896082]
      2008-10-09 14:32:34,541 TRACE [org.jboss.remoting.MicroRemoteClientInvoker] SocketClientInvoker[902a8b, bisocket://1.99.14.56:4457](1) invoking InvocationRequest[167519d, JMS, ConnectionFactoryGetClientAOPStackRequest[eab576]]
      2008-10-09 14:32:34,541 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[902a8b, bisocket://1.99.14.56:4457] obtained semaphore: 49
      2008-10-09 14:32:34,541 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[902a8b, bisocket://1.99.14.56:4457] creating socket
      2008-10-09 14:32:34,582 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[902a8b, bisocket://1.99.14.56:4457] created socket: Socket[addr=/1.99.14.56,port=4457,localport=38699]
      2008-10-09 14:32:34,587 TRACE [org.jboss.remoting.transport.socket.SocketWrapper] constructing org.jboss.jms.client.remoting.ClientSocketWrapper instance for Socket[addr=/1.99.14.56,port=4457,localport=38699], using timeout 0
      2008-10-09 14:32:34,588 TRACE [org.jboss.remoting.transport.socket.SocketWrapper] ClientSocketWrapper[Socket[addr=/1.99.14.56,port=4457,localport=38699].1f6fa3f] setting timeout to 0
      2008-10-09 14:32:34,588 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[902a8b, bisocket://1.99.14.56:4457] writing version 22 on output stream
      2008-10-09 14:32:34,588 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[902a8b, bisocket://1.99.14.56:4457] writing invocation to marshaller
      2008-10-09 14:32:34,588 TRACE [org.jboss.jms.wireformat.JMSWireFormat] Writing InvocationRequest[167519d, JMS, ConnectionFactoryGetClientAOPStackRequest[eab576]]
      2008-10-09 14:32:34,588 TRACE [org.jboss.jms.wireformat.JMSWireFormat] Stream is a DataOutputStream
      2008-10-09 14:32:34,588 TRACE [org.jboss.jms.wireformat.JMSWireFormat] JBM Request
      2008-10-09 14:32:34,589 TRACE [org.jboss.jms.wireformat.JMSWireFormat] Writing packet: ConnectionFactoryGetClientAOPStackRequest[eab576]
      2008-10-09 14:32:34,589 TRACE [org.jboss.jms.wireformat.JMSWireFormat] Wrote packet
      2008-10-09 14:32:34,589 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[902a8b, bisocket://1.99.14.56:4457] done writing invocation to marshaller
      2008-10-09 14:32:34,589 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[902a8b, bisocket://1.99.14.56:4457] reading version from input stream
      2008-10-09 14:32:51,696 TRACE [org.jboss.jms.server.recovery.MessagingXAResourceRecovery] org.jboss.jms.server.recovery.MessagingXAResourceRecovery@c1a0eb hasMoreResources
      2008-10-09 14:32:51,723 TRACE [org.jboss.jms.server.recovery.MessagingXAResourceRecovery] org.jboss.jms.server.recovery.MessagingXAResourceRecovery@c1a0eb getXAResource
      2008-10-09 14:32:51,723 DEBUG [org.jboss.jms.server.recovery.MessagingXAResourceWrapper] Recover java:/RemoteJMSProvider
      2008-10-09 14:33:45,968 INFO [ru.tower.ejb.jms.timeinfo.SenderInfo] 23 message(s) for sending
      2008-10-09 14:33:45,970 INFO [ru.tower.ejb.jms.Sender] Initializing sender
      2008-10-09 14:33:45,974 DEBUG [ru.tower.ejb.jms.Sender] About to look up the connection factory
      2008-10-09 14:33:45,974 DEBUG [ru.tower.ejb.jms.Sender] About to create queue connection
      2008-10-09 14:33:45,974 DEBUG [ru.tower.ejb.jms.Sender] About to create queue session
      2008-10-09 14:33:46,032 DEBUG [org.jboss.jms.ConnectionFactoryHelper] using connection factory: JBossConnectionFactory->ConnectionFactoryDelegate[jboss.messaging.connectionfactory:service=SlowConnectionFactory, SID=0]
      2008-10-09 14:33:46,033 DEBUG [org.jboss.jms.ConnectionFactoryHelper] using username/password: system/-- not shown --
      
      
      



      any ideas?