1 2 3 4 Previous Next 59 Replies Latest reply on Jul 18, 2009 7:44 AM by nbhatia

    "Connection failure has been detected" message

    nbhatia

      I am having trouble receiving messages reliably using JBM 2.0.0.BETA3 deployed in JBoss AS 5. Every once in a while I am getting a "Connection failure has been detected" message and then I start loosing messages. Can someone help me identify the issue?

      I have simplified my web application down to the bare bones. At this point I can send 1000 messages as fast as I can and then see if I receive all of them. Few times I have gotten all 1000, but several times I get much less, say about 600. I get the following warnings in between:

      22:58:26.968 WARNING [Thread-44 (group:JBM-client-global-threads-27401341)] [org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has been detected Did not receive ping from server:3
      ...
      23:02:56.984 WARNING [Thread-3 (group:JBM-scheduled-threads-10153586)] [org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl] Connection failure has been detected Did not receive ping from client. It is likely a client has exited or crashed without closing its connection, or the network between the server and client has failed. The connection will now be closed.:3
      23:02:56.984 WARNING [Thread-3 (group:JBM-scheduled-threads-10153586)] [org.jboss.messaging.core.server.impl.ServerSessionImpl] Client connection failed, clearing up resources for session ceb2cc8d-69d8-11de-9025-0016d433fcb5
      23:02:57.000 WARNING [Thread-3 (group:JBM-scheduled-threads-10153586)] [org.jboss.messaging.core.server.impl.ServerSessionImpl] Cleared up resources for session ceb2cc8d-69d8-11de-9025-0016d433fcb5
      


      The message listener code is as simple as this:

      @MessageDriven(activationConfig={
       @ActivationConfigProperty(propertyName="destinationType", propertyValue="javax.jms.Queue"),
       @ActivationConfigProperty(propertyName="destination", propertyValue="queue/jmstest")
       })
      public class TestMessageListener implements MessageListener {
      
       private static final Logger logger =
       LoggerFactory.getLogger(TestMessageListener.class);
      
       @Override
       public void onMessage(Message message) {
      
       if (message instanceof TextMessage) {
       String messageText = null;
       try {
       messageText = ((TextMessage)message).getText();
       logger.debug("Received message: {}", messageText);
       }
       catch (JMSException e) {
       throw new RuntimeException(e);
       }
       }
       }
      }
      


      I can send my small Maven project if someone could try it out.

      Thanks.
      Naresh

        • 1. Re:
          clebert.suconic

          Can you open a JIRA and attach the test please?



          https://jira.jboss.org/jira/browse/JBMESSAGING

          • 2. Re:
            nbhatia
            • 3. Re:
              clebert.suconic

              I couldn't replicate your issue.

              However I had to add the acceptors and connectors at the config.


              I have used this configuration here instead (which just have the netty and in-vm acceptors and connectors:


              http://anonsvn.jboss.org/repos/messaging/trunk/examples/javaee/mdb/server/jbm-configuration.xml


              Also.. you don't need to narrow anything on the lookup. And since you are on the application server, you don't need also to use properties. You could do just:

              jndiContext = new InitialContext();
              
               logger.debug("Looking up ConnectionFactory...");
               connectionFactory = (ConnectionFactory)
               jndiContext.lookup(CONNECTION_FACTORY);
              
               logger.debug("Looking up test queue...");
               testQueue = (Queue)
               jndiContext.lookup(TEST_QUEUE);
              



              (But I had tested without these changes also, and I couldn't replicate).



              Also.. you should probably use the JCA connection factory inside the application server. you are currently using /ConnectionFactory.



              And all this ran really fast.


              I will close as can't replicate for now.. if you provide more data on replicating we can reopen it.

              • 4. Re:
                nbhatia

                Clebert, thanks for testing my app. I will try your suggestions and let you know if it makes any difference. Quick followup question - how do I use the JCA connection factory? Is there one existing in the default configuration and what is its JNDI name?

                Thanks.
                Naresh

                • 5. Re:
                  clebert.suconic

                  Take a look at jms-ds.xml


                  The JNDI is java:/XAConnectionFactory.

                  (You need to use the local JNDI for that... just do new InitialContext(). As you may known java:/ means local VM).


                  you can just do:


                  InitialContext ctx = new IntialContext();
                  ConnectionFactory cf = (ConnectionFactory)ctx.lookup("java:/XAConnectionFactory");
                  




                  • 6. Re:
                    nbhatia

                    Clebert, I have now tried the same code (with the modifications you suggested) on 3 machines - Windows XP, Windows Vista and Windows NT. The only machine without the connection failures is Windows NT. XP and Vista seem to start giving issues after receiving 500 messages. What OS did you run your test on?

                    - Any other next steps you suggest?
                    - Can I monitor the connections somehow?
                    - What does the error message mean ("Connection failure has been detected Did not receive ping from server:3")?

                    Thanks.
                    Naresh

                    • 7. "Connection failure has been detected" message
                      timfox

                      Pinging changed between beta1/2 and beta3.

                      Are you sure you're not using a jar from beta1/beta2 with different jars from beta3?

                      E.g. you have jars from an earlier release on the client side and different jars on the server side. Or maybe you are using the rar from a different release.

                      All jars used *everywhere* must be from the same release.

                      • 8. Re:
                        timfox

                         


                        However I had to add the acceptors and connectors at the config.


                        The fact that Clebert had to add these acceptors and connectors in order to get your example to work and you didn't, tells me you can't be using beta3.

                        Otherwise you would have had to do the same too to get the example to work.

                        • 9. Re:
                          timfox

                           

                          "clebert.suconic@jboss.com" wrote:
                          Take a look at jms-ds.xml


                          The JNDI is java:/XAConnectionFactory.

                          (You need to use the local JNDI for that... just do new InitialContext(). As you may known java:/ means local VM).


                          you can just do:


                          InitialContext ctx = new IntialContext();
                          ConnectionFactory cf = (ConnectionFactory)ctx.lookup("java:/XAConnectionFactory");
                          




                          That is NOT correct.

                          If you lookup java:/XAConnectionFactory you will NOT be using JCA.

                          There is a section on using JCA in the docs:

                          http://labs.jboss.com/file-access/default/members/jbossmessaging/freezone/docs/usermanual-2.0.0.beta3/html_single/index.html#d0e5931



                          • 10. Re:
                            nbhatia

                            Hi Tim,

                            To answer your questions:

                            - I am not mixing jars from beta1/2 with beta 3. I have completely separate JBoss installations for JBoss 5.1.0 + JBM 2.0.0.BETA1 and JBoss 5.1.0 + JBM 2.0.0.BETA3. The JBoss installations were created by running ant in JBM's config\jboss-as directory with appropriate versions installed. Also since my app runs on JBoss, my ear does not contain any JBM jars.

                            - I also have acceptors and connectors on my beta 3 configurations just like clebert. We had talked about that issue on a separate thread, so I am aware of it. Specifically, I have copied jboss-messaging-2.0.0.BETA3\examples\javaee\mdb\server\jbm-configuration.xml to JBOSS_HOME\server\default-with-jbm2\deploy\messaging.sar\jbm-configuration.xml.

                            - I tried doing a JCA configuration as you suggested, but that crashed and burned! Specifically, the ant task that generates default-with-jbm2 creates a ra.xml that is fairly close to the example in the documentation. The default jms-ds.xml also creates a tx-connection-factory with jndi-name = JmsXA. So I simply changed my jndi lookup as follows:

                             jndiContext = new InitialContext();
                             connectionFactory = (ConnectionFactory)
                             jndiContext.lookup("/JmsXA");
                            


                            I am not sure how to configure the listener part. Following the documentation example I changed my message listener as follows:

                            @MessageDriven(name = "TestMessageListener", activationConfig = {
                             @ActivationConfigProperty(propertyName="destinationType", propertyValue="javax.jms.Queue"),
                             @ActivationConfigProperty(propertyName="destination", propertyValue="queue/jmstest"),
                             @ActivationConfigProperty(propertyName="acknowledgeMode", propertyValue="Auto-acknowledge"),
                             @ActivationConfigProperty(propertyName="ConnectorClassName", propertyValue="org.jboss.messaging.integration.transports.netty.NettyConnectorFactory"),
                             @ActivationConfigProperty(propertyName="ConnectionParameters", propertyValue="jbm.remoting.netty.port=5545")
                             })
                            public class TestMessageListener implements MessageListener {
                             ...
                            }
                            


                            Of course, this doesn't work! I get the following exception:

                            Caused by: MessagingException[errorCode=2 message=Unable to connect to server using configuration org-jboss-messaging-integration-transports-netty-NettyConnectorFactory?jbm-remoting-netty-port=5545]
                             at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.createSession(ConnectionManagerImpl.java:294)
                             at org.jboss.messaging.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:962)
                             at org.jboss.messaging.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:706)
                             at org.jboss.messaging.ra.JBMResourceAdapter.createSession(JBMResourceAdapter.java:1261)
                             at org.jboss.messaging.ra.inflow.JBMActivation.setupSession(JBMActivation.java:359)
                             at org.jboss.messaging.ra.inflow.JBMActivation.setup(JBMActivation.java:306)
                             at org.jboss.messaging.ra.inflow.JBMActivation.start(JBMActivation.java:269)
                            


                            What next?

                            BTW, couple of questions:

                            1) I see that we are creating a tx-connection-factory in jms-ds.xml. Where do we create an "rx-connection-factory"? Do these have to be seperate?

                            2) Why do we have low-level details like jbm.remoting.netty.port=5545 in annotations? Is there any way to avoid it. My earlier configuration simply had this:

                            @MessageDriven(activationConfig={
                             @ActivationConfigProperty(propertyName="destinationType", propertyValue="javax.jms.Queue"),
                             @ActivationConfigProperty(propertyName="destination", propertyValue="queue/jmstest")
                             })
                            


                            which seems very logical, but I don't know how it figured out which connection factory to use. I would appreciate some clarification on this.

                            Thanks.
                            Naresh

                            • 11. Re:
                              nbhatia

                              Hi Tim,

                              To answer your questions:

                              - I am not mixing jars from beta1/2 with beta3. I have completely separate JBoss installations for JBoss+JBM.BETA1 and JBoss+JBM.BETA3. The JBoss installations were created by running ant in JBM's config\jboss-as directory with appropriate versions installed. Also since my app runs on JBoss, my ear does not contain any JBM jars.

                              - I also have acceptors and connectors on my beta 3 configuration just like clebert. We had talked about that issue on a separate thread, so I am aware of it. Specifically, I have copied jboss-messaging-2.0.0.BETA3\examples\javaee\mdb\server\jbm-configuration.xml to JBOSS_HOME\server\default-with-jbm2\deploy\messaging.sar\jbm-configuration.xml.

                              - I tried doing a JCA configuration as you suggested, but that crashed and burned! Specifically, the ant task that generates default-with-jbm2 creates a ra.xml that is fairly close to the example in the documentation. The default jms-ds.xml also creates a tx-connection-factory with jndi-name = JmsXA. So I simply changed my jndi lookup as follows, hope that's correct:

                              jndiContext = new InitialContext();
                              connectionFactory = (ConnectionFactory)
                               jndiContext.lookup("/JmsXA");
                              


                              I am not sure how to configure the listener part. Following the documentation example I changed my message listener as follows:

                              @MessageDriven(name = "TestMessageListener", activationConfig = {
                               @ActivationConfigProperty(propertyName="destinationType", propertyValue="javax.jms.Queue"),
                               @ActivationConfigProperty(propertyName="destination", propertyValue="queue/jmstest"),
                               @ActivationConfigProperty(propertyName="acknowledgeMode", propertyValue="Auto-acknowledge"),
                               @ActivationConfigProperty(propertyName="ConnectorClassName", propertyValue="org.jboss.messaging.integration.transports.netty.NettyConnectorFactory"),
                               @ActivationConfigProperty(propertyName="ConnectionParameters", propertyValue="jbm.remoting.netty.port=5545")
                               })
                              public class TestMessageListener implements MessageListener {
                               ...
                              }
                              


                              Of course, this doesn't work! I get the following exception:

                              Caused by: MessagingException[errorCode=2 message=Unable to connect to server using configuration org-jboss-messaging-integration-transports-netty-NettyConnectorFactory?jbm-remoting-netty-port=5545]
                               at org.jboss.messaging.core.client.impl.ConnectionManagerImpl.createSession(ConnectionManagerImpl.java:294)
                               at org.jboss.messaging.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:962)
                               at org.jboss.messaging.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:706)
                               at org.jboss.messaging.ra.JBMResourceAdapter.createSession(JBMResourceAdapter.java:1261)
                               at org.jboss.messaging.ra.inflow.JBMActivation.setupSession(JBMActivation.java:359)
                               at org.jboss.messaging.ra.inflow.JBMActivation.setup(JBMActivation.java:306)
                               at org.jboss.messaging.ra.inflow.JBMActivation.start(JBMActivation.java:269)
                              


                              What next?

                              BTW, couple of questions:

                              1) I see that we are creating a tx-connection-factory in jms-ds.xml. Where do we create an "rx-connection-factory"? Do these have to be seperate and different?

                              2) Why do we have low-level details like "jbm.remoting.netty.port=5545" in annotations? Is there any way to avoid them. My earlier configuration simply had this:

                              @MessageDriven(activationConfig={
                               @ActivationConfigProperty(propertyName="destinationType", propertyValue="javax.jms.Queue"),
                               @ActivationConfigProperty(propertyName="destination", propertyValue="queue/jmstest")
                               })
                              


                              which seems quite logical, but I don't know how it figured out which connection factory to use. I would appreciate some clarification on this.

                              Thanks.
                              Naresh

                              • 12. Re:
                                nbhatia

                                Sorry about the double post. I did not realize that the first one had gone to page 2.

                                • 13. Re:
                                  timfox

                                  Hi Naresh-

                                  I don't have a Windows machine at hand (as you can imagine most of us don't use Windows) to try and replicate your issue, but Clebert and Andy have - perhaps they can try and replicate it.

                                  Regarding the JCA config - there's a chapter in the user manual on how to configure this, also there should be a fully working example in the distro that demonstrates an MDB in action.

                                  Andy is currently rewriting the JCA chapter (since it's a bit terse) so should be able to help you out with your config.

                                  • 14. Re:
                                    ataylor

                                    Hi Naresh,

                                    heres a quick overview of the JCA configuration.

                                    The JCA configuration is found in ra.xml under the jms-ra.rar. This configures the incoming and outgoing connection factories.

                                    The incoming is what is used by your MDB's. All these properties can be over ridden via the activation properies on your MDB. you have done this with the ConnectorClassName parameters to use a netty connector. By default the the ra.xml is configured to use th eINVM connectors since the MDB's are in the same VM. You only need to over ride them like you have if you want your MDB's to consume from a remote server.


                                    The outgoing is what is used to look up connection factories within servlets etc and are configured via the jms-ds.xml file. again all the properties can be over ridden in here. The jndi-name is the name you should look up this connection factory by.

                                    Can you review what you have done and if you are still getting errors provide me with a test MDB/configuration and i'll run it on my windows box.

                                    Andy

                                    1 2 3 4 Previous Next