6 Replies Latest reply on Mar 2, 2012 9:29 AM by ataylor

    Connectivity issues

    vchmakov

      Hi,

       

      We are trying to integrate different parts of our system written in different languages using a messaging backbone. We have chosen HornetQ as a message broker and running it under the JBoss AS 7.0.2.

      On a very high level the system looks like as follows:

      1. Server1 (Delphi) is listening on Queue1 and sends to Queue2 via STOMP
      2. Server2 (Java) is listening on Queue2 and sends to Queue1 via native HornetQ protocol using core API (not JMS)

      The problem is that we are experiencing weird disconnects and suspect the STOMP side. The weirdness is in the fact that we do not see any error messages in logs, but HornetQ is creating something which looks very much like a second buffer/consumer on the Queue1 which results in Server1 receiving every second message which obviously is breaking the application level protocol. Please forgive me if I’m not explaining myself clearly. Has anyone been experiencing something similar? Are there any suggestions as to how we could diagnose/troubleshoot the problem to better understand of what is going on?

       

      JBoss xml for HornetQ looks like this:

       

              <subsystem xmlns="urn:jboss:domain:messaging:1.0">

                  <connection-ttl-override>

                      86400000

                  </connection-ttl-override>

                  <journal-type>

                      NIO

                  </journal-type>

                  <journal-file-size>

                      102400

                  </journal-file-size>

                  <acceptors>

                      <netty-acceptor name="netty" socket-binding="messaging"/>

                      <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">

                          <param key="batch-delay" value="50"/>

                          <param key="direct-deliver" value="false"/>

                      </netty-acceptor>

                      <in-vm-acceptor name="in-vm" server-id="0"/>

                      <netty-acceptor name="stomp-acceptor" socket-binding="stomp">

                          <param key="protocol" value="stomp"/>

                      </netty-acceptor>

                  </acceptors>

       

      Sometimes when we restart jBoss we see messages like this:

       

      07:50:55,657 WARN  [org.jboss.messaging] (MSC service thread 1-7) failed to destroy jms queue: RuleEngineOutQueue: HornetQException[errorCode=104 message=Cannot delete queue jms.queue.RuleEngineOutQueue on binding jms.queue.RuleEngineOutQueue - it has consumers = org.hornetq.core.postoffice.impl.LocalQueueBinding]

                      at org.hornetq.core.server.impl.HornetQServerImpl.destroyQueue(HornetQServerImpl.java:1083) [hornetq-core-2.2.7.Final.jar:]

                      at org.hornetq.jms.server.impl.JMSServerManagerImpl.destroyQueue(JMSServerManagerImpl.java:710) [hornetq-jms-2.2.7.Final.jar:]

                      at org.jboss.as.messaging.jms.JMSQueueService.stop(JMSQueueService.java:68) [jboss-as-messaging-7.0.2.Final.jar:7.0.2.Final]

                      at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1909) [jboss-msc-1.0.1.GA.jar:1.0.1.GA]

                      at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1872) [jboss-msc-1.0.1.GA.jar:1.0.1.GA]

                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_26]

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_26]

                      at java.lang.Thread.run(Thread.java:662) [:1.6.0_26]

       

      I can understand the meaning of this, but can this add to our troubles?

       

      I’d highly appreciate ANY idea/insight on this. I’m at lost, but the issue is very time-critical.

       

      Thanks,

       

      Vlad

        • 1. Re: Connectivity issues
          clebert.suconic

          I'm not sure how's the integration on AS7, but I believe it will call destroy once the queue is stopped. Destroy doesn't mean delete the queue messages, but it means undeploy...etc.

           

           

          It seems that there's a dependency issue here, since Stomp should be stopped before these queues, since Stomp could be holding consumers, causing the issue you are seeing.

           

           

          I guess you should open a JIRA on AS7 project.

           

           

          Workaround would be to kill the server when using stomp (if AS7)

          • 2. Re: Connectivity issues
            vchmakov

            Hi Clebert,

             

            AS7 errors is not that much of concern to me. My real problem is a STOMP connection  stability and HornetQ creating a secondary consumer as described above. Is there any insight with that resect?

             

            Thanks,

             

            Vlad

             

            • 3. Re: Connectivity issues
              vchmakov

              Hi,

               

              Just a heads up for the community.

              Our STOMP connectivity issues have gone away 3 weeks ago since we switched off to ActiveMQ. Also ActiveMQ has a very simple yet powerful administration console. I cannot imagine MQ-related development without having such a tool. Just a glimpse on the screen gives you a clear picture as to what is going on in your system in communication area. And yes, I know that there is an extensive management API in HornetQ - we are very close to creating our own monitoring/testing tool.

               

              To HornetQ team: our team is ready to help with all means in helping to resolve the STOMP connectivity issue as we DO want to use JBoss AS in our implementations including integration of the heterogeneous systems. For non-Java application the STOMP protocol perhaps the only option here.

               

              Thanks,

               

              Vladimir

               

              • 4. Re: Connectivity issues
                ataylor

                could you provide something simple we can run to recreate this issue?

                 

                also, does this also happen in AS 7.1?

                • 5. Re: Connectivity issues
                  vchmakov

                  Hi Andy,

                   

                  We are preparing a simple application for you to investigate. What is the best way to send it to you?

                   

                  Thanks,

                   

                  Vladimir

                  • 6. Re: Connectivity issues
                    ataylor

                    just attach it to this forum post