7 Replies Latest reply on Feb 12, 2007 11:55 AM by clebert.suconic

    JBoss Messaging thread BLOCKED at method org.jboss.jms.clien

    rcjboss

      Hi,

      Environment: JBoss App server 4.0.5GA(build: CVSTag=Branch_4_0 date=200610162340) running the JMS remote client and connecting to another server:
      JBoss Messaging server 1.0.1.GA on another JBoss 4.0.5GA.
      OS: SuSE Linux 9.x
      Messaging DB - Oracle 10g.

      Multiple threads have been blocked, as the following thread dump lists at the bottom.
      Looking at the code for "org.jboss.jms.client.JBossConnectionFactory.ensureAOPConfigLoaded" there is a Synchronized block of code in the class where everyone is stuck.
      In my application newer requests that put a message on jboss messaging get stuck and timeout. As they are not able to get a connection from the "queueConnectionFactory".

      The implementation I have is that I get JndiContext and QueueConnectionFactory in a static block and use the same thruout the life of application.

      Thread: http-0.0.0.0-8080-4 : priority:5, demon:true, threadId:352, threadState:BLOCKED, threadLockName:java.lang.Class@11ee18eorg.jboss.jms.client.JBossConnectionFactory.ensureAOPConfigLoaded(JBossConnectionFactory.java:227)org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:196)org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:108)org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:103)com.wizcom.mcdb.channel.LDB2CDBRecordSender.createConnection(Unknown Source)com.wizcom.mcdb.channel.LDB2CDBRecordSender.getQueueSession(Unknown Source)com.wizcom.mcdb.channel.LDB2CDBRecordSender.sendRecord(Unknown Source)com.wizcom.mcdb.business.bo.AbstractBaseBo.sendToCdb(Unknown Source)com.wizcom.mcdb.business.bo.AddressBO.saveAddress(Unknown Source)com.wizcom.mcdb.business.bo.AddressBO.handle(Unknown Source)com.wizcom.mcdb.business.bo.AbstractBaseBo.process(Unknown Source)com.wizcom.mcdb.business.delegate.Delegator.process(Unknown Source)com.wizcom.mcdb.presentation.action.AbstractBaseAction.execute(Unknown Source)org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:419)org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:224)org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)javax.servlet.http.HttpServlet.service(HttpServlet.java:717)javax.servlet.http.HttpServlet.service(HttpServlet.java:810)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)com.wizcom.mcdb.presentation.filter.CharsetFilter.doFilter(Unknown Source)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)org.apache.catalina.valves.FastCommonAccessLogValve.invoke(FastCommonAccessLogValve.java:495)org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)java.lang.Thread.run(Thread.java:595)
      
      


        • 1. Re: JBoss Messaging thread BLOCKED at method org.jboss.jms.c
          clebert.suconic

          Can you post your thread dump again... with a better format now?


          i couldn't understand your post before... and something even broke the HTML on this page (I had to edit and use code quotes).

          You should use the code tag in between [] to your thread dump. On this forum you can click the Code button and you will have the proper tag open and close syntxes.

          • 2. Re: JBoss Messaging thread BLOCKED at method org.jboss.jms.c
            rcjboss

            I get number of threads blocked on the call mentioned in the thread dump. After that my application does not respond.
            Sorry, here is the thread dump

            Thread: http-0.0.0.0-8080-1 : priority:5, demon:true, threadId:111, threadState:BLOCKED, threadLockName:java.lang.Class@11ee18e
            org.jboss.jms.client.JBossConnectionFactory.ensureAOPConfigLoaded(JBossConnectionFactory.java:227)
            org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:196)
            org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:108)
            org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:103)
            com.wizcom.framework.audit.AuditRecordSender.createConnection(Unknown Source)
            com.wizcom.framework.audit.AuditRecordSender.getQueueSession(Unknown Source)
            com.wizcom.framework.audit.AuditRecordSender.audit(Unknown Source)
            com.wizcom.mcdb.business.bo.AbstractBaseBo.audit(Unknown Source)
            com.wizcom.mcdb.business.bo.AddressBO.saveAddress(Unknown Source)
            com.wizcom.mcdb.business.bo.AddressBO.handle(Unknown Source)
            com.wizcom.mcdb.business.bo.AbstractBaseBo.process(Unknown Source)
            com.wizcom.mcdb.business.delegate.Delegator.process(Unknown Source)
            com.wizcom.mcdb.presentation.action.AbstractBaseAction.execute(Unknown Source)
            org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:419)
            org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:224)
            org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)
            org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)
            javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
            javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
            org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
            org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
            com.wizcom.mcdb.presentation.filter.CharsetFilter.doFilter(Unknown Source)
            org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
            org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
            org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
            org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
            org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
            org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
            org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
            org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
            org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
            org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
            org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
            org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
            org.apache.catalina.valves.FastCommonAccessLogValve.invoke(FastCommonAccessLogValve.java:495)
            org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
            org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
            org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
            org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
            org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
            org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
            java.lang.Thread.run(Thread.java:595)
            
            Thread: TP-Processor1 : priority:5, demon:true, threadId:112, threadState:BLOCKED, threadLockName:java.lang.Class@11ee18e
            org.jboss.jms.client.JBossConnectionFactory.ensureAOPConfigLoaded(JBossConnectionFactory.java:227)
            org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:196)
            org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:108)
            org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:103)
            com.wizcom.mcdb.channel.LDB2CDBRecordSender.createConnection(Unknown Source)
            com.wizcom.mcdb.channel.LDB2CDBRecordSender.getQueueSession(Unknown Source)
            com.wizcom.mcdb.channel.LDB2CDBRecordSender.sendRecord(Unknown Source)
            com.wizcom.mcdb.business.bo.AbstractBaseBo.sendToCdb(Unknown Source)
            com.wizcom.mcdb.business.bo.AddressBO.saveAddress(Unknown Source)
            com.wizcom.mcdb.business.bo.AddressBO.handle(Unknown Source)
            com.wizcom.mcdb.business.bo.AbstractBaseBo.process(Unknown Source)
            com.wizcom.mcdb.business.delegate.Delegator.process(Unknown Source)
            com.wizcom.mcdb.presentation.action.AbstractBaseAction.execute(Unknown Source)
            org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:419)
            org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:224)
            org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)
            org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)
            javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
            javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
            org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
            org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
            com.wizcom.mcdb.presentation.filter.CharsetFilter.doFilter(Unknown Source)
            org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
            org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
            org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
            org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
            org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
            org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
            org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
            org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
            org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
            org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
            org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
            org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
            org.apache.catalina.valves.FastCommonAccessLogValve.invoke(FastCommonAccessLogValve.java:495)
            org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
            org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
            org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
            org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
            org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
            org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
            org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
            org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
            java.lang.Thread.run(Thread.java:595)
            
            



            • 3. Re: JBoss Messaging thread BLOCKED at method org.jboss.jms.c
              rcjboss

              On running further tests here is a thread that probably causes all the others to get blocked.

              Thread: TP-Processor7 : priority:5, demon:true, threadId:120, threadState:RUNNABLE, threadLockName:null
              java.net.SocketInputStream.socketRead0(Native Method)
              java.net.SocketInputStream.read(SocketInputStream.java:129)
              java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
              java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
              java.io.BufferedInputStream.read(BufferedInputStream.java:313)
              java.io.FilterInputStream.read(FilterInputStream.java:90)
              org.jboss.serial.io.JBossObjectInputStream.checkSignature(JBossObjectInputStream.java:110)
              org.jboss.serial.io.JBossObjectInputStream.(JBossObjectInputStream.java:94)
              org.jboss.remoting.serialization.impl.jboss.JBossSerializationManager.createInput(JBossSerializationManager.java:59)
              org.jboss.remoting.transport.socket.ClientSocketWrapper.createInputStream(ClientSocketWrapper.java:83)
              org.jboss.remoting.transport.socket.ClientSocketWrapper.createStreams(ClientSocketWrapper.java:76)
              org.jboss.remoting.transport.socket.ClientSocketWrapper.(ClientSocketWrapper.java:54)
              sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
              sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
              sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
              java.lang.reflect.Constructor.newInstance(Constructor.java:494)
              org.jboss.remoting.transport.socket.SocketClientInvoker.createClientSocket(SocketClientInvoker.java:162)
              org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:699)
              org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:319)
              org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:125)
              org.jboss.remoting.Client.invoke(Client.java:589)
              org.jboss.remoting.Client.invoke(Client.java:581)
              org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.invoke(ClientConnectionFactoryDelegate.java:199)
              org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$getClientAOPConfig_8697532701842707646.invokeNext(ClientConnectionFactoryDelegate$getClientAOPConfig_8697532701842707646.java)
              org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.getClientAOPConfig(ClientConnectionFactoryDelegate.java)
              org.jboss.jms.client.JBossConnectionFactory.ensureAOPConfigLoaded(JBossConnectionFactory.java:233)
              org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:196)
              org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:108)
              org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:103)
              com.wizcom.framework.audit.AuditRecordSender.createConnection(Unknown Source)
              com.wizcom.framework.audit.AuditRecordSender.getQueueSession(Unknown Source)
              com.wizcom.framework.audit.AuditRecordSender.audit(Unknown Source)
              com.wizcom.mcdb.business.bo.AbstractBaseBo.audit(Unknown Source)
              com.wizcom.mcdb.business.bo.AddressBO.saveAddress(Unknown Source)
              com.wizcom.mcdb.business.bo.AddressBO.handle(Unknown Source)
              com.wizcom.mcdb.business.bo.AbstractBaseBo.process(Unknown Source)
              com.wizcom.mcdb.business.delegate.Delegator.process(Unknown Source)
              com.wizcom.mcdb.presentation.action.AbstractBaseAction.execute(Unknown Source)
              org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:419)
              org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:224)
              org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)
              org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)
              javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
              javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
              org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
              org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
              com.wizcom.mcdb.presentation.filter.CharsetFilter.doFilter(Unknown Source)
              org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
              org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
              org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
              org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
              org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
              org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
              org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
              org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:175)
              org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:74)
              org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
              org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
              org.jboss.web.tomcat.tc5.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:156)
              org.apache.catalina.valves.FastCommonAccessLogValve.invoke(FastCommonAccessLogValve.java:495)
              org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
              org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
              org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
              org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
              org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
              org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
              org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
              org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
              java.lang.Thread.run(Thread.java:595)
              


              • 4. Re: JBoss Messaging thread BLOCKED at method org.jboss.jms.c
                timfox
                • 5. Re: JBoss Messaging thread BLOCKED at method org.jboss.jms.c
                  sathish_06

                  Hi,
                  Does Jboss Connection factory (org.jboss.jms.client.JBossConnectionFactory class) provides any kind of "pooling" mechanism, which pools TopicConnection/TopicSession objects ?

                  The reason for asking this is, in our Producer piece of code, we do create multiple TopicConnection objects and re-use them whenever required to publish messages, instead of destroying & create new objects each time.

                  But, its not good approach in load balancing environment(with more than one JBoss Messaging service running)

                  Question is:-
                  Do we need to manage JMS Connection pooling, or does existing Jboss Connection factory implementation provides ?

                  Please suggest.

                  PS: Our Env details: JBoss Messaging 1.0.1 SP2 runing in jboss-4.0.4.GA AS. OS: Linux.

                  Thx, Sathish




                  • 6. Re: JBoss Messaging thread BLOCKED at method org.jboss.jms.c
                    timfox

                     

                    "sathish_06" wrote:
                    Hi,
                    Does Jboss Connection factory (org.jboss.jms.client.JBossConnectionFactory class) provides any kind of "pooling" mechanism, which pools TopicConnection/TopicSession objects ?


                    If you use the JMS JCA resource adaptor, it will provide pooling:

                    http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossJMSRA

                    You can only use the resource adaptor from within the app server, i.e. from inside an EJB, servlet or mbean service.

                    • 7. Re: JBoss Messaging thread BLOCKED at method org.jboss.jms.c
                      clebert.suconic

                      I have talked to Kabir and opened http://jira.jboss.org/jira/browse/JBAOP-360