10 Replies Latest reply: Jan 9, 2012 4:47 AM by Hushen Savani RSS

    javax.jms.JMSException: Timed out waiting for response when sending packet 30

    Hushen Savani Novice

      Hi,

         

           I have my application running on JBossAS-5.1.0 with Hornetq-2.2.5. My application is running on Solaris SPARC Machine. The application works fine with lower number of messages. i.e. 50. But when it increases to 400+, following exception is coming:

       

       

      2011-12-29 10:32:35,154 ERROR [STDERR] (Thread-15 (group:HornetQ-client-global-threads-1834164732)) javax.jms.JMSException: Timed out waiting for response when sending packet 30
      2011-12-29 10:32:35,156 ERROR [STDERR] (Thread-15 (group:HornetQ-client-global-threads-1834164732))     at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:276)
      2011-12-29 10:32:35,156 ERROR [STDERR] (Thread-15 (group:HornetQ-client-global-threads-1834164732))     at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:695)
      2011-12-29 10:32:35,156 ERROR [STDERR] (Thread-15 (group:HornetQ-client-global-threads-1834164732))     at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:264)
      2011-12-29 10:32:35,156 ERROR [STDERR] (Thread-15 (group:HornetQ-client-global-threads-1834164732))     at org.hornetq.jms.client.HornetQConnection.authorize(HornetQConnection.java:589)
      2011-12-29 10:32:35,156 ERROR [STDERR] (Thread-15 (group:HornetQ-client-global-threads-1834164732))     at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:694)
      2011-12-29 10:32:35,157 ERROR [STDERR] (Thread-15 (group:HornetQ-client-global-threads-1834164732))     at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:121)
      2011-12-29 10:32:35,157 ERROR [STDERR] (Thread-15 (group:HornetQ-client-global-threads-1834164732))     at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:116)
      2011-12-29 10:32:35,157 ERROR [STDERR] (Thread-15 (group:HornetQ-client-global-threads-1834164732))     at com.elitecore.messageservice.ejb.base.utils.hornetq.HornetQ.startConnection(HornetQ.java:84)
      

       

      Following are my observations:

       

      1) There is no network latency as consumer and producer are on the same machine.

      2) This works fine on Linux machine for very higher number of messages. i.e. 250000+ Messages.

      3) All the HornetQ Configurations are same on Solaris & Linux Machine.

       

      Following are my configurations:

       

      hornetq-configuration.xml:

       

      <!--
        ~ Copyright 2009 Red Hat, Inc.
        ~  Red Hat licenses this file to you under the Apache License, version
        ~  2.0 (the "License"); you may not use this file except in compliance
        ~  with the License.  You may obtain a copy of the License at
        ~     http://www.apache.org/licenses/LICENSE-2.0
        ~  Unless required by applicable law or agreed to in writing, software
        ~  distributed under the License is distributed on an "AS IS" BASIS,
        ~  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
        ~  implied.  See the License for the specific language governing
        ~  permissions and limitations under the License.
        -->
      
      
      <configuration xmlns="urn:hornetq"
                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                     xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">
      
      
         <!--  Don't change this name.
               This is used by the dependency framework on the deployers,
               to make sure this deployment is done before any other deployment -->
         <name>HornetQ.main.config</name>
      
      
         <log-delegate-factory-class-name>org.hornetq.integration.logging.Log4jLogDelegateFactory</log-delegate-factory-class-name>
         
         <bindings-directory>${jboss.server.data.dir}/hornetq/bindings</bindings-directory>
      
      
         <journal-directory>${jboss.server.data.dir}/hornetq/journal</journal-directory>
         
         <journal-min-files>10</journal-min-files>
      
      
         <journal-type>ASYNCIO</journal-type>
      
      
         <large-messages-directory>${jboss.server.data.dir}/hornetq/largemessages</large-messages-directory>
      
      
         <paging-directory>${jboss.server.data.dir}/hornetq/paging</paging-directory>
         
         <message-counter-enabled>true</message-counter-enabled>
         
         <message-counter-max-day-history>7</message-counter-max-day-history> 
         
         <!-- sample the queues every minute (1ms) -->
                <message-counter-sample-period>29000</message-counter-sample-period>
      
      
         <connectors>
            <connector name="netty">
               <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
      <!--         <param key="host"  value="${jboss.bind.address:localhost}"/> -->
                 <param key="host"  value="192.168.1.76"/> 
                 <param key="port"  value="${hornetq.remoting.netty.port:5445}"/> 
            </connector>
            
            <connector name="netty-throughput">
               <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
          <!--     <param key="host"  value="${jboss.bind.address:localhost}"/> -->
                <param key="host"  value="192.168.1.76"/> 
               <param key="port"  value="${hornetq.remoting.netty.batch.port:5455}"/>
               <param key="batch-delay" value="50"/>
            </connector>
      
      
            <connector name="in-vm">
               <factory-class>org.hornetq.core.remoting.impl.invm.InVMConnectorFactory</factory-class>
               <param key="server-id" value="${hornetq.server-id:0}"/> 
            </connector>
      
      
         </connectors>
      
      
         <acceptors>   
            <acceptor name="netty">
               <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
              <!-- <param key="host"  value="${jboss.bind.address:localhost}"/> -->
                <param key="host"  value="192.168.1.76"/> 
               <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
            </acceptor>
            
            <acceptor name="netty-throughput">
               <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
              <!--  <param key="host"  value="${jboss.bind.address:localhost}"/> -->
                <param key="host"  value="192.168.1.76"/> 
               <param key="port"  value="${hornetq.remoting.netty.batch.port:5455}"/>
               <param key="batch-delay" value="50"/>
               <param key="direct-deliver" value="false"/>
            </acceptor>
      
      
            <acceptor name="in-vm">
              <factory-class>org.hornetq.core.remoting.impl.invm.InVMAcceptorFactory</factory-class>
              <param key="server-id" value="0"/>
            </acceptor>
      
      
         </acceptors>
      
      
         <security-settings>
            <security-setting match="#">
               <permission type="createNonDurableQueue" roles="guest"/>
               <permission type="deleteNonDurableQueue" roles="guest"/>
               <permission type="consume" roles="guest"/>
               <permission type="send" roles="guest"/>
            </security-setting>
         </security-settings>
      
      
         <address-settings>
            <!--default for catch all-->
            <address-setting match="#">
               <dead-letter-address>jms.queue.DLQ</dead-letter-address>
               <expiry-address>jms.queue.ExpiryQueue</expiry-address>
               <redelivery-delay>0</redelivery-delay>
               <max-size-bytes>3221225472</max-size-bytes>       
               <message-counter-history-day-limit>10</message-counter-history-day-limit>
                <page-size-bytes>104857600</page-size-bytes>
               <address-full-policy>PAGE</address-full-policy>
      
      
            </address-setting>
         </address-settings>
      
      
      </configuration>
      

       

       

      hornetq-jms.xml:

       

      <configuration xmlns="urn:hornetq"
                  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                  xsi:schemaLocation="urn:hornetq /schema/hornetq-jms.xsd">
      
      
         <connection-factory name="NettyConnectionFactory">
            <xa>true</xa>
                          <client-failure-check-period>-1</client-failure-check-period>
            <connectors>
               <connector-ref connector-name="netty"/>
            </connectors>
            <entries>
               <entry name="/XAConnectionFactory"/>
            </entries>
         </connection-factory>
         
         <connection-factory name="NettyConnectionFactory">
            <xa>false</xa>
                          <client-failure-check-period>-1</client-failure-check-period>
            <connectors>
               <connector-ref connector-name="netty"/>
            </connectors>
            <entries>
               <entry name="/ConnectionFactory"/>
            </entries>
      
      
         </connection-factory>
         
         <connection-factory name="NettyThroughputConnectionFactory">
            <xa>true</xa>
                          <client-failure-check-period>-1</client-failure-check-period>
            <connectors>
               <connector-ref connector-name="netty-throughput"/>
            </connectors>
            <entries>
               <entry name="/XAThroughputConnectionFactory"/>
            </entries>
         </connection-factory>
         
         <connection-factory name="NettyThroughputConnectionFactory">
            <xa>false</xa>
                <client-failure-check-period>-1</client-failure-check-period>
            <connectors>
               <connector-ref connector-name="netty-throughput"/>
            </connectors>
            <entries>
               <entry name="/ThroughputConnectionFactory"/>
            </entries>
      
      
         </connection-factory>
         
         <connection-factory name="InVMConnectionFactory">
            <xa>true</xa>
                          <client-failure-check-period>-1</client-failure-check-period>
            <connectors>
               <connector-ref connector-name="in-vm"/>
            </connectors>
            <entries>
               <entry name="java:/XAConnectionFactory"/>
            </entries>
         </connection-factory>
      
      
         <connection-factory name="InVMConnectionFactory">
            <xa>false</xa>
                          <client-failure-check-period>-1</client-failure-check-period>
            <connectors>
               <connector-ref connector-name="in-vm"/>
            </connectors>
            <entries>
               <entry name="java:/ConnectionFactory"/>
            </entries>
         </connection-factory>
      
      
         <queue name="DLQ">
            <entry name="/queue/DLQ"/>
         </queue>
         
         <queue name="ExpiryQueue">
            <entry name="/queue/ExpiryQueue"/>
         </queue>
      
      
                <queue name="testQueue">
            <entry name="/queue/testQueue"/>
                </queue>
        
      </configuration>
      

       

      Pl. have some pointers on the same. Thanks.

        • 1. Re: javax.jms.JMSException: Timed out waiting for response when sending packet 30
          Yong Hao Gao Master

          Any error message at the server side?

           

          Also are you creating connections/sessions each time you send a message?

           

          Howard

          • 2. Re: javax.jms.JMSException: Timed out waiting for response when sending packet 30
            Hushen Savani Novice

            Hi Howard,

             

            Any error message at the server side?

             

            Following log messages are coming on server side:

             

            011-12-29 18:29:26,814 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from /127.0.0.1:45505. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]

            2011-12-29 18:29:26,814 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Client connection failed, clearing up resources for session a82f4a9c-321c-11e1-80a5-b1c6e8464921

            2011-12-29 18:29:26,815 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Cleared up resources for session a82f4a9c-321c-11e1-80a5-b1c6e8464921

            2011-12-29 18:29:26,815 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Client connection failed, clearing up resources for session a82f98bd-321c-11e1-80a5-b1c6e8464921

            2011-12-29 18:29:26,815 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Cleared up resources for session a82f98bd-321c-11e1-80a5-b1c6e8464921

            2011-12-29 18:29:26,822 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from /127.0.0.1:45506. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]

            2011-12-29 18:29:26,826 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Client connection failed, clearing up resources for session a830aa2f-321c-11e1-80a5-b1c6e8464921

            2011-12-29 18:29:26,826 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Cleared up resources for session a830aa2f-321c-11e1-80a5-b1c6e8464921

            2011-12-29 18:29:26,826 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Client connection failed, clearing up resources for session a830f850-321c-11e1-80a5-b1c6e8464921

            2011-12-29 18:29:26,826 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Cleared up resources for session a830f850-321c-11e1-80a5-b1c6e8464921

             

            However, with this warn messages, it was working fine on Linux Machine.

             

            Also are you creating connections/sessions each time you send a message?

             

            Yes, Connections/Sessions are created each time.

            • 3. Re: javax.jms.JMSException: Timed out waiting for response when sending packet 30
              Yong Hao Gao Master

              Re: However, with this warn messages, it was working fine on Linux Machine.

               

              This error messages suggests that your client didn't close the connections properly. They should be closed when is not used.

               

              Re: Yes, Connections/Sessions are created each time.

               

              This is very inefficient. You should cache connections/sessions/consumers/producers whenever possible. Because creating those objects are expensive calls.

              • 4. Re: javax.jms.JMSException: Timed out waiting for response when sending packet 30
                Hushen Savani Novice

                Hi Howard,

                This is very inefficient. You should cache connections/sessions/consumers/producers whenever possible. Because creating those objects are expensive calls.

                 

                I am recreating sessions and connection each time as my application is running on JEE Application Server. And according to following wiki article, it is not an anti-pattern:

                 

                http://community.jboss.org/wiki/ShouldICacheJMSConnectionsAndJMSSessions

                 

                The mystry is that the same configuration and the same code is working very fine on Linux machine. No such "timed out" exception appeared even once. But on Solaris, it appears.

                 

                Can there be any possible solution for this exception?

                • 5. Re: javax.jms.JMSException: Timed out waiting for response when sending packet 30
                  Yong Hao Gao Master

                  com.elitecore.messageservice.ejb.base.utils.hornetq.HornetQ.startConnection(HornetQ.java:84)

                   

                  I ask the question because I see this line in the error message. If your application is running inside JEE app server, you should use JCA api to get JMS resources. However this line seems to me you directly create the connections, which are not managed by JCA. Is that true?

                   

                  Re: Can there be any possible solution for this exception?

                   

                  We need to find out what's the cause first and then come up with a solution. So far I'm not clear what's happened.

                  • 6. Re: javax.jms.JMSException: Timed out waiting for response when sending packet 30
                    Hushen Savani Novice

                    Hi Howard,

                     

                    Following is the code snippet where I start the connection:

                     

                    @Resource(mappedName = "java:JmsXA")

                              private ConnectionFactory _cfConnFactory = null;

                     

                     

                     

                              public void startConnection(Context initialContecxt, String hornetqQueue) throws BusinessTaskException {//NamingException, JMSException {

                     

                                        String strMethodName ="startConnection():: ";

                     

                                        System.out.println(strLogPreffix + strMethodName + "HornetQ Queue Name:: " + hornetqQueue);

                     

                                        try {

                                                  this._ctx = initialContecxt;

                     

                                                  String queue = "";

                                                  if(hornetqQueue!=null) {

                                                            //Set HornetQ Queue

                                                            queue = "/queue/" + hornetqQueue;

                                                            this._queue = (Queue)_ctx.lookup(queue);

                     

                     

                                                            //Set Connection

                                                            _cfConnFactory = (ConnectionFactory) _ctx.lookup("/ThroughputConnectionFactory");

                                                            this._conn = _cfConnFactory.createConnection();

                     

                                                            //Set session

                                                            this._session = _conn.createSession(false, Session.CLIENT_ACKNOWLEDGE);

                                                            _producer = _session.createProducer(_queue);

                                                _producer.setDisableMessageID(true);

                                                _producer.setDisableMessageTimestamp(true);

                                               

                                                //Start Service

                                                if(_conn!=null)

                                                                      _conn.start();

                                                            else

                                                                      System.err.println("[ HornetQ:: startConnection() ]Error in Connection!");                           

                                                    

                                                  }

                                                  else {

                                                            System.err.println("[HornetQ :: startConnection() Queue Name is not provided.]");

                                                  }

                                        }

                                        catch (Exception e) {

                                                  e.printStackTrace();

                                        }

                              }

                     

                    com.elitecore.messageservice.ejb.base.utils.hornetq.HornetQ.java is a POJO which is being called from an MDB on its onMessage().

                     

                    If you see problem in this code, pl. provide the reference for the correction.

                     

                    Thanks.

                    • 7. Re: javax.jms.JMSException: Timed out waiting for response when sending packet 30
                      Yong Hao Gao Master

                      //Set Connection

                      _cfConnFactory = (ConnectionFactory) _ctx.lookup("/ThroughputConnectionFactory");

                      this._conn = _cfConnFactory.createConnection();

                       

                      This doesn't sound right to me. You should look up something like "JmsXA". like

                       

                      _cfConnFactory = (ConnectionFactory) _ctx.lookup("/JmsXA");

                      this._conn = _cfConnFactory.createConnection();

                       

                      This way you will get a managed connection. And then each time after the message is sent you should close it. like:

                       

                      this._conn.close();

                       

                      This doesn't necessarily close the jms connection underneath but it allows the connection resource to be reused.

                       

                      Howard

                      • 8. Re: javax.jms.JMSException: Timed out waiting for response when sending packet 30
                        Andy Taylor Master

                        once youve mapped the factory like

                         

                        @Resource(mappedName = "java:JmsXA")

                                  private ConnectionFactory _cfConnFactory = null;

                         

                        you dont have to look it up, thi is actually doingg that for you.

                         

                        In your code you are using a different (non JCA) factory so a new connection is being created for every message processed and then not closed. These are onl;y closed via GC so depending on the VM its liikely you will get memory problems causing timeouts etc

                        • 9. Re: javax.jms.JMSException: Timed out waiting for response when sending packet 30
                          Hushen Savani Novice

                          Thanks a lot Howard and Andy. Issue resolved. I have shifted above code in a Session Bean instead of a POJO and taken care of not looking up the connection factory on each call.

                           

                          Happy New Year Guys! :-)

                          • 10. Re: javax.jms.JMSException: Timed out waiting for response when sending packet 30
                            Hushen Savani Novice

                            Hi Guys,

                             

                                 With reference to the above problem, as Howard and Andy suggested, I have moved message producing logic to a Session Bean from the POJO, so that I will not have to look up the connection factory on each message, and also, I get rid of connection/session/producer switching anti-pattern if I use a session bean. Hence, moved to session bean, all went well. But, when a great load is generated on application, it shows the following exception:

                             

                            2012-01-08 21:38:16,726 ERROR [com.elitecore.messageservice.ejb.base.utils.logger.EJBLogger] (Thread-610 (group:HornetQ-client-global-
                            threads-1848021541)) [ PROCESSING_ENTITY_LISTNER_MDB ] enqueueProcessingEntity():: javax.naming.NameNotFoundExceptionjavax.naming.Name
                            NotFoundException: HornetQSessionBean not bound
                            

                             

                            How come this session bean automatically gets unbound if I produce so many messages using this session bean?

                             

                            I doubt that session bean pools are exhausted if I look up this session bean for producing messages heavily.

                             

                            Please suggest your pointers on the same... Thanks!