1 2 Previous Next 16 Replies Latest reply on Jul 19, 2014 9:31 PM by jbertram

    JBoss MDB not receiving messages from HornetQ

    zcabir

      We are seeing a weird issue in the JBoss that intermittently MDB's are not receiving messages from HornetQ.   This happens when we are run JBoss in a KVM (16 GB RAM, 2 vCpu) and the problem goes away when we restart the jboss.  In bare metal, this issue is not seen.  Below is our environment details

       

      - JBoss EAP 6.0.0.GA (AS 7.1.2.Final-redhat-1)

      - HornetQ Server version 2.2.16.Final

      -  JDK 1.7.0_25 (Oracle)

      - Cent OS: Linux xsd-mvdclnx57 2.6.32-358.18.1.el6.x86_64 #1 SMP Wed Aug 28 17:19:38 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

       

      We have two MDB's with queue as destination (jms/queue/Q1 and jms/queue/Q2) and one MDB with topic  as destination (jms/topic/T1).  None of them receives messages in this state.

       

      Noticed that MDB's and hornetQ started properly.

       

      14-Oct-2013 21:33:04,053 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-4) HornetQ Server version 2.2.16.Final (HQ_2_2_16_FINAL, 122) [349a958f-3518-11e3-891d-8bed1a580245]) started

      :

      14-Oct-2013 21:33:19,966 INFO  [org.jboss.as.ejb3] (MSC service thread 1-2) JBAS014142: Started message driven bean 'Q1MDB' with 'hornetq-ra' resource adapter

      14-Oct-2013 21:33:19,966 INFO  [org.jboss.as.ejb3] (MSC service thread 1-4) JBAS014142: Started message driven bean 'Q2MDB' with 'hornetq-ra' resource adapter

       

      When the messages got published, we do not see any error messages in the server.log, but there are warning messages like below.

       

      14-Oct-2013 21:39:57,673 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]

      14-Oct-2013 21:39:57,675 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Client connection failed, clearing up resources for session 45b0aac0-3518-11e3-891d-8bed1a580245

      14-Oct-2013 21:39:57,679 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Cleared up resources for session 45b0aac0-3518-11e3-891d-8bed1a580245

      14-Oct-2013 21:39:57,709 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]

      14-Oct-2013 21:39:57,709 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Client connection failed, clearing up resources for session 456a2c91-3518-11e3-891d-8bed1a580245

      14-Oct-2013 21:39:57,710 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Cleared up resources for session 456a2c91-3518-11e3-891d-8bed1a580245

      14-Oct-2013 21:39:57,710 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]

       

      If anyone has seen this issue, please share your thoughts.

        • 1. Re: JBoss MDB not receiving messages from HornetQ
          ataylor

          I can tell you what is happening but not why,  the client pings the server every n seconds, if the server doen receive the ping in a certain time then it closes the connection, in this instance it is an invm connection from the MDB. The usual causes for this are the server or client becoming unresponsive because of overload, memory issues etc, maybe your KVM isnt configured correctly

          1 of 1 people found this helpful
          • 2. Re: JBoss MDB not receiving messages from HornetQ
            zcabir

            Hi Andy,

            Thanks very much for your response.  If KVM configuration is a problem, can you please recommend any particular starting point to diagnose this issue?  Our automated deployment  does the following:

            1.  Installs qcow2 file to create the VM and powers it on.

            2.  Installs a set of rpm's (Including JBOSS rpm)

            3.  Starts each of the components in a specific order.

            4.  Run tests (The tests are failing because of this MDB/HornetQ issue).

             

            If we restart jboss, everything works fine.

             

            Initially, i thought memory could be an issue and tried in a 16GB KVM (when it  failed in 8GB KVM).  Let me check this problem happens in 32/64GB KVM.

             

            Thanks,

            Cabir

            • 3. Re: JBoss MDB not receiving messages from HornetQ
              zcabir

              Setting clientFailureCheckPeriod and connectionTTL seems to fix the issue.

              • 4. Re: JBoss MDB not receiving messages from HornetQ
                cpuffalt

                Andy,

                 

                We just had this happen to us in production as well.  is there some reason why HornetQ doesn't recover from this situation?

                 

                Here's what we see in the logs:

                 

                00:50:28,915 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]

                00:50:29,208 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Client connection failed, clearing up resources for session e33d5fe6-caf3-11e2-bc0f-3d92824f2653

                00:50:29,235 WARN  [org.hornetq.core.server.impl.ServerSessionImpl] (hornetq-failure-check-thread) Cleared up resources for session e33d5fe6-caf3-11e2-bc0f-3d92824f2653

                [the above two messages are repeated a number of times in rapid succession...and then finally we get...]

                00:50:30,187 WARN  [org.hornetq.jms.server.recovery.RecoveryDiscovery] (Thread-18647 (HornetQ-client-global-threads-2106576636)) Notified of connection failure in xa discovery, we will retry on the next recovery: HornetQException[errorCode=2 message=Channel disconnected]

                        at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:381) [hornetq-core-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]

                        at org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:203) [hornetq-core-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]

                        at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [hornetq-core-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]

                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_07]

                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_07]

                        at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_07]

                00:50:29,240 WARN  [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Thread-18651 (HornetQ-client-global-threads-2106576636)) Notified of connection failure in xa recovery connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0], discoveryGroupConfiguration=null], connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0, backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=2 message=Channel disconnected]

                        at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:381) [hornetq-core-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]

                        at org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:203) [hornetq-core-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]

                        at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [hornetq-core-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]

                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_07]

                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_07]

                        at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_07]

                00:50:38,633 INFO  [org.hornetq.jms.server.recovery.RecoveryDiscovery] (HornetQ Recovery Discovery Reinitialization) Starting RecoveryDiscovery on XARecoveryConfig [transportConfiguration = [org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0], discoveryConfiguration = null, username=null, password=null]

                00:50:38,636 INFO  [org.hornetq.jms.server.recovery.RecoveryDiscovery] (HornetQ Recovery Discovery Reinitialization) RecoveryDiscovery started fine on XARecoveryConfig [transportConfiguration = [org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0], discoveryConfiguration = null, username=null, password=null]

                 

                From this point onward, we still get messages queued but nothing gets delivered to our MDB.  Is there some way to recover from this automatically?  Also, if this is a complete failure scenario how come none of the log messages are logged at either ERROR or FATAL levels?

                 

                Thanks,

                Corey

                • 5. Re: JBoss MDB not receiving messages from HornetQ
                  ataylor

                  its not a complete failure, its the transaction manager checking for XA transactions in ddoubt and not being able to connect to HornetQ, why i dont know, maybe too much load or network issues. Like it says it will retry on the next pass

                  • 6. Re: JBoss MDB not receiving messages from HornetQ
                    cpuffalt

                    Unfortunately it didn't retry though.  It was several days later when we were investigating and the JMS queue was full of undelivered messages.  We had to restart JBoss completely to get things to recover.

                    • 7. Re: Re: JBoss MDB not receiving messages from HornetQ
                      cpuffalt

                      I've created WFLY-2608 to track this issue.

                      • 8. Re: Re: JBoss MDB not receiving messages from HornetQ
                        jbertram

                        From what I can tell from the log snippets (and based on others who have seen this problem), the MDB's sessions are being cleaned up by HornetQ because they are idle for too long.  I'm not sure why the MDB's sessions wouldn't be responding to pings, but you should be able to work around this problem.

                         

                        Try these settings on the MDB to disable connection checking, add the following annotations:

                         

                          @ActivationConfigProperty(propertyName = "clientFailureCheckPeriod", propertyValue = "600000")

                          @ActivationConfigProperty(propertyName = "connectionTTL", propertyValue = "-1")

                         

                        To be clear, the "clientFailureCheckPeriod" can be any number > 0. The value "600000" equates to 10 minutes which seems reasonable (the default is 30 seconds).

                         

                        In any event, I don't believe this is a Wildfly problem so you should close WFLY-2608.

                        • 9. Re: JBoss MDB not receiving messages from HornetQ
                          cpuffalt

                          From a user standpoint it seems to me the most reasonable expectation is for the container to automatically recreate the MDB sessions which is why I opened the ticket.  Having to add extra configuration parameters to get a reliable config for what's a simple in-container queue and MDB isn't very reasonable.

                          • 10. Re: JBoss MDB not receiving messages from HornetQ
                            jbertram

                            I agree with you.  I didn't mean to imply that the problem wouldn't (or shouldn't) be fixed.  I'm just trying to help you work around the problem in the mean-time.  I also wanted to tell you that the JIRA was filed in the wrong project since this is not a Wildfly issue but rather a HornetQ one (at least as far as I can see).

                            • 11. Re: JBoss MDB not receiving messages from HornetQ
                              cpuffalt

                              Thanks for the clarification.  I did indeed misinterpret your earlier response.  I've left the ticket open against Wildfly for now.  I noticed on another  similar issue the decision was made to clone the ticket into the HornetQ JIRA presumably so the issue could still be tracked against Wildfly and would then show up in release notes when fixed.

                              • 12. Re: JBoss MDB not receiving messages from HornetQ
                                roko98

                                Maybe this will help somebody.  I don't know if its an error on hornet or not, but I think the same thing happened to me on EC2.  I keep getting periodical hornetq-failure-check-thread and hornet doesn't recover from that.  In my case was a clock sync issue.  For some reason, my EC2 instance loose clock sync very quickly, so in a matter of hours got 20-30 seconds behind.  When the NTP sync kicks in, the machine clock get a new value in the "future" and I think the TTL algorithm complains.

                                 

                                I haven't try myself, but I think you can reproduce this issue very easily playing with your machine time.

                                • 13. Re: JBoss MDB not receiving messages from HornetQ
                                  cpuffalt

                                  Justin,

                                   

                                  We just had another occurrence of this issue even with the suggested configuration properties as a workaround.  Can you provide some suggestions in terms of where to look in the code to resolve this permanently?  It doesn't look like the reported issue has gotten any traction...

                                   

                                  Thanks,

                                  Corey

                                  • 14. Re: JBoss MDB not receiving messages from HornetQ
                                    jbertram

                                    We just had another occurrence of this issue even with the suggested configuration properties as a workaround.

                                    As far as I can tell from the code it shouldn't be possible for an MDB to have its session(s) timed out by the broker using the activation configuration properties detailed above.  I've worked with numerous users who've run into this particular issue and as far as I know they've all been remedied using the previous work-around or by upgrading to a later version where connection validation is disabled for in-vm connections (fixed via HORNETQ-1314).  Out of curiosity can you post the configuration for the MDB that's experiencing this problem?

                                     

                                    Can you provide some suggestions in terms of where to look in the code to resolve this permanently?

                                    As I indicated previously, this issue should be remedied either by using the aforementioned activation configuration properties or by upgrading HornetQ.  Until I understand your specific use-case better I can't make any additional recommendations.

                                     

                                    It doesn't look like the reported issue has gotten any traction...

                                    Are you talking about WFLY-2608?  If so, it's not surprising that it hasn't gotten any traction since, as I indicated before, it's not a WildFly problem (it's a HornetQ one).  If you're not talking about WFLY-2608 please clarify which issue you're referencing.

                                    1 2 Previous Next