1 2 Previous Next 16 Replies Latest reply on Dec 10, 2009 10:31 PM by sv_srinivaas

    Hornetq ejb3 mdb reconnect issue

    sv_srinivaas

      Hi, I am facing the exact same issue that is reported in HORNETQ-2. I tried the jca-config example that comes with hornetq. Everything worked fine as long as the network was connected and when I disconnected the server from the network and reconnected it, I got the below error in my server within 3 minutes and after that the mdb stopped consuming messages

      [exec] WARNING: Connection failure has been detected: Did not receive ping from /172.18.xxx.yyy:2100. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. The connection will now be closed. [code=3]

      Pls let me know how to fix this issue. Thanks!!!

      Environment Details: Windows Xp (Service Pack2)
      version: Hornetq2.0.0 BETA5

        • 1. Re: Hornetq ejb3 mdb reconnect issue
          timfox

          Please provide full details of how to replicate this issue and someone will take a look.

          Or just wait a short time until the next release is out since it's probably been fixed weeks ago anyway on trunk.

          • 2. Re: Hornetq ejb3 mdb reconnect issue
            sv_srinivaas

            HI i tried this time with the latest version Hornetq 2.0.0.CR1 and still having issues with mdb not reconnecting in case of a network outage.

            Environment details:
            JBoss 5.0.0 GA
            Hornetq 2.0.0.CR1
            OS: Windows Xp (service pack2)

            Installed Jboss 5.0.0. GA and Hornetq. Executed jca-config example that comes with HornetQ after making the below changes to run the example using the ip address instead of localhost (that comes by default).

            Steps to replicate the issue:

            1. Modified jca-config\build.xml to run the jboss server using -b option to specify the ip address as shown below



            2. Modified jca-config\config\jndi.properties to use the ip address instead of loaclhost as below

            java.naming.provider.url=jnp://xxx.yy.zzz.aaa:1199

            3. Modified jca-config\server2\hornetq-configuration.xml to use the ip address for both the acceptor and connector (as i'm running the server and mdb client in the same machine)





            4. Modified jca-config\server2\jms-remote-ds.xml to use the ip address for connection parameters

            <config-property name="ConnectionParameters" type="java.lang.String">host=xxx.yy.zzz.aaa;port=5445</config-property>

            5. Modified jca-config\server2\ra.xml to use the ip address for acceptor and connector
            <config-property>
            <config-property-name>ConnectionParameters</config-property-name>
            <config-property-type>java.lang.String</config-property-type>
            <config-property-value> host=xxx.yy.zzz.aaa;port=5445</config-property-value>
            </config-property>


            6. Then using the \jca-config\build.xml, started both the servers 1 and 2 using the commands "build.bat deploy" and then "build.bat deploy2". The consumer count also increased from 0 to 15 for queue B in server2.

            7. Sent a message to queue B using "build.bat run".

            8. Message was sent successfully and in the server 2 console (as I’m interested only in remote adapter test case) I was able to see the info that " (MDBQueueB.java) Message received using the remote adapter. Message = "Hello there MDB!" that confirmed that the MDB picked the message from remote queue B

            9. Now I disconnected the network cable few seconds and I could the see the below exception in server2 console

            [exec] 12:14:44,958 ERROR [STDERR] Dec 7, 2009 12:14:44 PM org.hornetq.core
            .logging.impl.JULLogDelegate warn
            [exec] WARNING: Handling JMS exception failure: org.hornetq.ra.HornetQRAMan
            agedConnection@488831
            [exec] javax.jms.JMSException: HornetQException[errorCode=0 message=Netty e
            xception]
            [exec] at org.hornetq.jms.client.HornetQConnection$JMSFailureListener.c
            onnectionFailed(HornetQConnection.java:582)
            [exec] at org.hornetq.core.client.impl.FailoverManagerImpl.callFailureL
            isteners(FailoverManagerImpl.java:741)
            [exec] at org.hornetq.core.client.impl.FailoverManagerImpl.failoverOrRe
            connect(FailoverManagerImpl.java:703)
            [exec] at org.hornetq.core.client.impl.FailoverManagerImpl.handleConnec
            tionFailure(FailoverManagerImpl.java:554)
            [exec] at org.hornetq.core.client.impl.FailoverManagerImpl.connectionEx
            ception(FailoverManagerImpl.java:296)
            [exec] at org.hornetq.integration.transports.netty.NettyConnector$Liste
            ner$2.run(NettyConnector.java:632)
            [exec] at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.ru
            n(OrderedExecutorFactory.java:96)
            [exec] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Thread
            PoolExecutor.java:885)
            [exec] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPool
            Executor.java:907)
            [exec] at java.lang.Thread.run(Thread.java:619)
            [exec] Caused by: HornetQException[errorCode=0 message=Netty exception]
            [exec] at org.hornetq.integration.transports.netty.HornetQChannelHandle
            r.exceptionCaught(HornetQChannelHandler.java:105)
            [exec] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(S
            impleChannelHandler.java:140)
            [exec] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(D
            efaultChannelPipeline.java:560)
            [exec] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannel
            HandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
            [exec] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptio
            nCaught(SimpleChannelUpstreamHandler.java:148)
            [exec] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUp
            stream(SimpleChannelUpstreamHandler.java:122)
            [exec] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(D
            efaultChannelPipeline.java:560)
            [exec] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(D
            efaultChannelPipeline.java:555)
            [exec] at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels
            .java:619)
            [exec] at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.ja
            va:86)
            [exec] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamin
            gRunnable.java:108)
            [exec] at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRu
            nnable.java:46)
            [exec] at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunn
            able.run(VirtualExecutorService.java:179)
            [exec] ... 3 more
            [exec] Caused by: java.net.SocketException: Connection reset
            [exec] at java.net.SocketInputStream.read(SocketInputStream.java:168)
            [exec] at java.net.SocketInputStream.read(SocketInputStream.java:182)
            [exec] at java.io.FilterInputStream.read(FilterInputStream.java:66)
            [exec] at java.io.PushbackInputStream.read(PushbackInputStream.java:122
            )
            [exec] at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.ja
            va:77)
            [exec] ... 5 more
            [exec] 12:14:44,958 WARN [TxConnectionManager] Connection error occured: o
            rg.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListene
            r@11768ab[state=NORMAL mc=org.hornetq.ra.HornetQRAManagedConnection@488831 handl
            es=0 lastUse=1260168260708 permit=false trackByTx=false mcp=org.jboss.resource.c
            onnectionmanager.JBossManagedConnectionPool$OnePool@17f75f1 context=org.jboss.re
            source.connectionmanager.InternalManagedConnectionPool@1182c3d xaResource=org.ho
            rnetq.ra.HornetQRAXAResource@1d8c699 txSync=null]
            [exec] javax.jms.JMSException: HornetQException[errorCode=0 message=Netty e
            xception]
            [exec] at org.hornetq.jms.client.HornetQConnection$JMSFailureListener.c
            onnectionFailed(HornetQConnection.java:582)
            [exec] at org.hornetq.core.client.impl.FailoverManagerImpl.callFailureL
            isteners(FailoverManagerImpl.java:741)
            [exec] at org.hornetq.core.client.impl.FailoverManagerImpl.failoverOrRe
            connect(FailoverManagerImpl.java:703)
            [exec] at org.hornetq.core.client.impl.FailoverManagerImpl.handleConnec
            tionFailure(FailoverManagerImpl.java:554)
            [exec] at org.hornetq.core.client.impl.FailoverManagerImpl.connectionEx
            ception(FailoverManagerImpl.java:296)
            [exec] at org.hornetq.integration.transports.netty.NettyConnector$Liste
            ner$2.run(NettyConnector.java:632)
            [exec] at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.ru
            n(OrderedExecutorFactory.java:96)
            [exec] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Thread
            PoolExecutor.java:885)
            [exec] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPool
            Executor.java:907)
            [exec] at java.lang.Thread.run(Thread.java:619)
            [exec] Caused by: HornetQException[errorCode=0 message=Netty exception]
            [exec] at org.hornetq.integration.transports.netty.HornetQChannelHandle
            r.exceptionCaught(HornetQChannelHandler.java:105)
            [exec] at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(S
            impleChannelHandler.java:140)
            [exec] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(D
            efaultChannelPipeline.java:560)
            [exec] at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannel
            HandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
            [exec] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptio
            nCaught(SimpleChannelUpstreamHandler.java:148)
            [exec] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUp
            stream(SimpleChannelUpstreamHandler.java:122)
            [exec] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(D
            efaultChannelPipeline.java:560)
            [exec] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(D
            efaultChannelPipeline.java:555)
            [exec] at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels
            .java:619)
            [exec] at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.ja
            va:86)
            [exec] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamin
            gRunnable.java:108)
            [exec] at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRu
            nnable.java:46)
            [exec] at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunn
            able.run(VirtualExecutorService.java:179)
            [exec] ... 3 more
            [exec] Caused by: java.net.SocketException: Connection reset
            [exec] at java.net.SocketInputStream.read(SocketInputStream.java:168)
            [exec] at java.net.SocketInputStream.read(SocketInputStream.java:182)
            [exec] at java.io.FilterInputStream.read(FilterInputStream.java:66)
            [exec] at java.io.PushbackInputStream.read(PushbackInputStream.java:122
            )
            [exec] at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.ja
            va:77)
            [exec] ... 5 more


            10. Reconnected the network cable and sent a new message using the "build.bat run" command and the jmx console confirmed that the message was sent to queue B in server2 but the MDB consumer count was showing as 0 and the message was NOT picked by the MDB.

            11. Also the server 2 console displayed the below error message

            [exec] 13:08:28,812 ERROR [STDERR] Dec 7, 2009 1:08:28 PM org.hornetq.core.
            logging.impl.JULLogDelegate warn
            [exec] WARNING: Connection failure has been detected: Did not receive ping
            from /172.18.164.138:1895. It is likely the client has exited or crashed without
            closing its connection, or the network between the server and client has failed
            . The connection will now be closed. [code=3]
            [exec] 13:08:28,812 ERROR [STDERR] Dec 7, 2009 1:08:28 PM org.hornetq.core.
            logging.impl.JULLogDelegate warn
            [exec] WARNING: Client connection failed, clearing up resources for session
            3aeb9c8f-e303-11de-a4e1-002170a2b761

            12. When I restarted server2 MDB started working and picked the message.

            • 3. Re: Hornetq ejb3 mdb reconnect issue
              ataylor

              i will take a look

              • 4. Re: Hornetq ejb3 mdb reconnect issue
                timfox

                What did you set, reconnectAttempts to in the jca config? Default is zero

                • 5. Re: Hornetq ejb3 mdb reconnect issue
                  sv_srinivaas

                  reconnectAttempts is set to -1 in hornetq-jms.xml as below. I dont see much difference with or without this setting.

                  <connection-factory name="ConnectionFactory">

                  <connector-ref connector-name="netty"/>






                  <retry-interval>1000</retry-interval>
                  <retry-interval-multiplier>1.0</retry-interval-multiplier>

                  <!-- Try reconnecting an unlimited number of times (-1 means "unlimited") -->
                  <reconnect-attempts>-1</reconnect-attempts>
                  </connection-factory>

                  • 6. Re: Hornetq ejb3 mdb reconnect issue
                    timfox

                    That's not the JCA config.

                    It's the JCA adaptor making a connection, so you need to set reconnect attempts in there.

                    • 7. Re: Hornetq ejb3 mdb reconnect issue
                      sv_srinivaas

                      Tim, can you pls let me know where exactly should I make this change? Is it in the ra.xml under hornetq-ra.rar\META-INF and if so, is it in the inbound-resourceadapter? Thanks!

                      • 8. Re: Hornetq ejb3 mdb reconnect issue
                        ataylor

                        you need to configure server2/ra.xml and server2/jms-ds.xml. Server2 ids the remote erver that sends and consumes from server1

                        • 9. Re: Hornetq ejb3 mdb reconnect issue
                          ataylor

                          fyi i have changed the default configuration to always try reconnection, see http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4269537#4269537

                          • 10. Re: Hornetq ejb3 mdb reconnect issue
                            sv_srinivaas

                            Hi, clicking on the above url refreshes the same page. Pls help.

                            • 11. Re: Hornetq ejb3 mdb reconnect issue
                              timfox

                               

                              "sv_srinivaas" wrote:
                              Hi, clicking on the above url refreshes the same page. Pls help.


                              Andy was going to add something to the user manual regarding this, so it should help you out.

                              • 12. Re: Hornetq ejb3 mdb reconnect issue
                                ataylor

                                sorry i posted the wrong link, it was meant to be for the Jira which is now fixed in trunk. https://jira.jboss.org/jira/browse/HORNETQ-240.

                                If you follow my config that i provided earlier this should work fine. I have tested this myself.

                                • 13. Re: Hornetq ejb3 mdb reconnect issue
                                  sv_srinivaas

                                  Andy, I have not worked in jca before, can you pls paste the code for ra.xml and jms-ds xml for this fix? I tried based on your comment but i dont see any difference in the output and i'm sure i'm missing something.

                                  • 14. Re: Hornetq ejb3 mdb reconnect issue
                                    timfox

                                    Probably a good place to start is to read the chapter on JCA in the user manual.

                                    That explains how to configure the resource adaptor.

                                    1 2 Previous Next