2 Replies Latest reply on May 2, 2012 8:20 AM by Andrea Ippolito

    Distributed queues - what am I doing wrong?

    Andrea Ippolito Newbie

      I'm using JBoss 5.0.1 EAP, Java 6 x64 and win7 x64.


      Here is what I want to do:

      a 3-node cluster with two of them having a distributed queue (node1 and node1b); the third node (node2) acts as a producer for the queue, and the queues should be load balanced. A second producer node (node2b) will be added later.

      The "funny" thing is, I also got it working, but now somehow it does no more.

      I'm posting my configuration so please tell me if there something wrong with it.


      I'm working on a single machine, so I had to start the 3 instances with default, 01 and 02 port sets. Binding is to "localhost" for all of them; config is "all".

      The queue is deployed only on node1 and node1b, with the Clustered attribute set to true on both. I'm not looking for a singleton queue, so I understood this is the right way to do it.

      On node2 (producer) I'm using this code to perform cluster-wide JNDI lookup, because otherwise the producer will be looking for a local ConnectionFactory to connect to a remote queue, which I've learned won't work.


      private static Context getContext() throws NamingException {
              Properties p = new Properties();
              p.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory");
              p.put(Context.URL_PKG_PREFIXES, "jboss.naming:org.jnp.interfaces");
              // HA-JNDI is registered under the partition name passed to JBoss via -g  
              String partitionName = System.getProperty("jboss.partition.name", "DefaultPartition");
              p.put("jnp.partitionName", partitionName);
              return new InitialContext(p);


      (I haven't used the -g switch on any instance, so they're all in the same partition). I read in clustering guide that this is the best way to perform remote lookups since you don't have to specify the remote hosts IP address(es).

      This is the code I use to perform the actual lookup (I know it's not very efficient):


                  ctx = getContext();
                  System.out.println("ctx: " + ctx);
                  Queue queue = (Queue) ctx.lookup(fileProperties.getString("QUEUE"));
                  System.out.println("queue: " + queue);
                  QueueConnectionFactory factory = (QueueConnectionFactory) ctx.lookup("RemoteConnectionFactory");
                  System.out.println("factory: " + factory);
                  cnn = factory.createQueueConnection();


      "RemoteConnectionFactory" is a copy-and-paste of ClusteredConnectionFactory with renamed "name" attribute and jndi-bindings, and it is defined on node1 and node1b only, so the producer will always get a CF on a node which also has the distributed queue deployed.


        <mbean code="org.jboss.jms.server.connectionfactory.ConnectionFactory"
            <depends optional-attribute-name="ServerPeer">jboss.messaging:service=ServerPeer</depends>
            <depends optional-attribute-name="Connector">jboss.messaging:service=Connector,transport=bisocket</depends>
            <attribute name="JNDIBindings">
            <attribute name="SupportsFailover">true</attribute>
            <attribute name="SupportsLoadBalancing">true</attribute>      



      Next step: configuring the PostOffice to be clustered, otherwise failover and loadbalancing won't work, say the WARN messages upon startup.


      Here is the last part of the PostOffice config in messaging/db2-persistence-service.xml. Also this configuration only affects node1 and node1b


      <attribute name="Clustered">true</attribute>
      <!-- All the remaining properties only have to be specified if the post office is clustered.
      You can safely comment them out if your post office is non clustered -->
      <!-- The JGroups group name that the post office will use -->
      <attribute name="GroupName">${jboss.messaging.groupname:MessagingPostOffice}</attribute>
      <!-- Max time to wait for state to arrive when the post office joins the cluster -->
      <attribute name="StateTimeout">5000</attribute>
      <!-- Max time to wait for a synchronous call to node members using the MessageDispatcher -->
      <attribute name="CastTimeout">50000</attribute>
      <!-- Set this to true if you want failover of connections to occur when a node is shut down -->
      <attribute name="FailoverOnNodeLeave">false</attribute>
      <depends optional-attribute-name="ChannelFactoryName">jboss.jgroups:service=ChannelFactory</depends>
      <attribute name="ControlChannelName">jbm-control</attribute>
      <attribute name="DataChannelName">jbm-data</attribute>
      <attribute name="ChannelPartitionName">${jboss.partition.name:DefaultPartition}-JMS</attribute>



      However with this configuration I get an error on this instruction:


      cnn = factory.createQueueConnection();

      (Utility.java:97, see stacktrace)


      java.lang.IllegalStateException: This invocation should not be handled here!
          at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientClusteredConnectionFactoryDelegate$createConnectionDelegate$aop(ClientClusteredConnectionFactoryDelegate.java:260)
          at org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
          at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
          at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101)
          at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95)
          at it.poste.oracolo.service.common.Utility.accodamento(Utility.java:97)
          at it.poste.oracolo.service.servlet.OracoloWSEXecute.service(OracoloWSEXecute.java:56)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
          at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
          at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
          at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
          at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)
          at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
          at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
          at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
          at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
          at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
          at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
          at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
          at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
          at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)
          at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:601)
          at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
          at java.lang.Thread.run(Unknown Source)


      I'm sure that I got it working, because after producing 50 msgs, I could see 20 of them on the queue on node1 and the others on node1b.

      Now the only way I can write to the queue without seeing exceptions is setting the PostOffice to be non clustered on both node1 and node1b, but this prevents the load balancing from working (in fact this way I can see all msgs being delivered to only one node).

      I really don't know what I could have changed to break it.


      Any help is greatly appreciated, thanks

        • 1. Re: Distributed queues - what am I doing wrong?
          Andrea Ippolito Newbie

          Nevermind, I started all over again and now it works.

          Maybe I messed up something, but I'll never know...

          • 2. Re: Distributed queues - what am I doing wrong?
            Andrea Ippolito Newbie

            I think I understood the problem.

            When it didn't work, the method I was invoking to send JMS messages (second code block in my first post) was in the Utility class and was directly invoked from the doPost method of a Servlet.

            Now that it works, instead, I created a Stateless Session Bean, whose implementation is the same as the previous Utility class, and which is accessed from the doPost method of the Servlet via its local interface.


            So apparently you can't just do whatever you want from within a Servlet. Maybe the deployment assembly plays a role too: previously I had everything within the webapp's war, now I have a webapp.war + the ejb .jar both within a same .ear.

            Hope this will help someone...