JMS-based client-side load balancing: doc unclear?| JBoss.org Content Archive (Read Only)

15. Re: JMS-based client-side load balancing: doc unclear?

kennardconsulting Jun 15, 2011 7:06 PM (in response to ataylor)

Andy,

In response to your suggestion, I tried following the instructions in "32.5. Configuring the JBoss Application Server to connect to Remote HornetQ Server". Here is my ra.xml:

   <resourceadapter>
      <resourceadapter-class>org.hornetq.ra.HornetQResourceAdapter</resourceadapter-class>
      <config-property>
         <description>The transport type</description>
         <config-property-name>ConnectorClassName</config-property-name>
         <config-property-type>java.lang.String</config-property-type>
         <config-property-value>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</config-property-value>
      </config-property>
      <config-property>
         <description>The transport configuration. These values must be in the form of key=val;key=val;</description>
         <config-property-name>ConnectionParameters</config-property-name>
         <config-property-type>java.lang.String</config-property-type>
         <config-property-value>host=${avant.hornetq.host:localhost};port=${hornetq.remoting.netty.port:5445}</config-property-value>
      </config-property>

And here is my jms-ds.xml:

<tx-connection-factory>
      <jndi-name>JmsXA</jndi-name>
      <xa-transaction/>
      <rar-name>jms-ra.rar</rar-name>
      <connection-definition>org.hornetq.ra.HornetQRAConnectionFactory</connection-definition>
      <config-property name="SessionDefaultType" type="java.lang.String">javax.jms.Topic</config-property>
      <config-property name="ConnectorClassName" type="java.lang.String">org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</config-property>
      <config-property name="ConnectionParameters" type="java.lang.String">host=${avant.hornetq.host:localhost};port=${hornetq.remoting.netty.port:5445}</config-property>
      <max-pool-size>20</max-pool-size>
      <security-domain-and-application>JmsXARealm</security-domain-and-application>
   </tx-connection-factory>
</connection-factories>

Note I modified the existing 'JmsXA' rather than create a new 'RemoteJmsXA' as suggested in 32.1 - is that a problem?

Anyway remember I have 3 nodes. The doc isn't explicit on what to do if you have more than 1 node connected to the same remote queue. I assume they all share the same port and settings (I pass -Davant.hornetq.host=<the same server> to each of them)? But when I try this, messages are occasionally getting 'lost'. The exact same code works fine if I configure my nodes to use clustered local queues again.

Is this expected behaviour? I know, for example, it is no-no to configure multiple nodes to use the same journal file. Do I need to configure the multiple nodes to use different Netty ports too? I should mention I have not changed the acceptor/connector config in hornetq-configuration.xml (the doc doesn't say to), is this conflicting?

<connectors>
      
      <connector name="netty">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
         <param key="host" value="${jboss.bind.address:localhost}"/>
         <param key="port" value="${hornetq.remoting.netty.port:5445}"/>
      </connector>
   </connectors>
   <acceptors>
      
      <acceptor name="netty">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
         <param key="host" value="${jboss.bind.address:localhost}"/>
         <param key="port" value="${hornetq.remoting.netty.port:5445}"/>
      </acceptor>
   </acceptors>

Thanks,

Richard.

16. Re: JMS-based client-side load balancing: doc unclear?

clebert.suconic Jun 15, 2011 7:41 PM (in response to kennardconsulting)

You could have MDBs load balancing on remote.. But your issue is that you have unbalanced resources in your cluster. You have one of you boxes with 4 more times processing power than the others.

I still need to check some details on the message redistribution, but as far as I can think of, you probably should:

- Disable server load balancing of the messages.

* max-hops=0 on the cluster connection

- Control the load balancing of your messages at application level. That means you could either select what connection you want to use, or to make sure your messages are being produced accordingly to the load you need.

- Make sure you balance the MDBs instances accordingly to your resources as you have been doing.

17. Re: JMS-based client-side load balancing: doc unclear?

kennardconsulting Jun 15, 2011 7:55 PM (in response to clebert.suconic)

Clebert,

Having my app balance the messages itself is obviously not ideal. It's not very resilient to new nodes being added/nodes going down/etc. So I'm kind of hoping HornetQ can handle this complicated stuff . I'm happy to wait for https://issues.jboss.org/browse/HORNETQ-721.

In the meantime, consuming off one remote queue is fine. When I try it, all 3 nodes consume the messages as fast as possible - which is great! Except occasionally messages get lost. With respect to the config I described above, should that work? Or do I need to specify different ports per node (not mentioned in 32.5)? Is max-hops=0 required (not mentioned in 32.5)?

Richard.

18. Re: JMS-based client-side load balancing: doc unclear?

clebert.suconic Jun 15, 2011 8:04 PM (in response to kennardconsulting)

I'm not really sure when we will be able to do https://issues.jboss.org/browse/HORNETQ-721

the only issue we are playing with clustering now is the bridge being killed and reconnected. which is the issue I'm currently working at.

It should be really simple to handle this at your application level. Really, really simple IMO.

I was expecting you would be able to handle your producer's load some different way. Like having a producer on each node. I don't really know your business. so it's hard for me to comment on that.

If you were doing a regular web application, you would have producers on each node responding to web requests. That would mean producers on each node, what would avoid any issues.

19. Re: JMS-based client-side load balancing: doc unclear?

kennardconsulting Jun 15, 2011 8:12 PM (in response to clebert.suconic)

Clebert,

Really, really simple IMO.

I'll take you up on that

My app is concerned with batch processing. Either we receive a file over FTP that contains, say, 5000 records to be imported. Or the Web user clicks on a button that says 'go run this batch of 5000 reports'. In either case there is a single trigger (one FTP file arriving, one Web click) that needs to produce 5000 messages. So at the moment the 'file arriving' or 'Web click' is handled by just one node, just one producer, which then produces 5000 messages.

How should I load balance that?

Regards,

Richard.

20. Re: JMS-based client-side load balancing: doc unclear?

clebert.suconic Jun 15, 2011 8:38 PM (in response to kennardconsulting)

If you need uneven load balancing, you will need to create One connection for each node, one session and one producer, and do the logic at your application level.

At this point we won't support uneven load balancing.

And if you think about it... the payload on hornetq to send the messages is really nothing... the issue is that each node will probably need to make a lot of processing for each message.

If I were you I would be dealing with distributing the load myself. (at least until we have the statistical redistribution or redistribution with slow consumers in place).

21. Re: JMS-based client-side load balancing: doc unclear?

kennardconsulting Jun 15, 2011 8:52 PM (in response to clebert.suconic)

Clebert,

One of the key selling points of HornetQ, for me, is its ability to 'discover' nodes and dynamically load balance messages. If I start configuring MDBs per node manually, I lose all that 'discovery' and I have to load balance myself, including when nodes go up/come down. This is not really a viable answer for me.

But like I said, I'm happy to wait for you to support message redistribution.

In the meantime, could you please answer my 'remote queue' configuration issue above?

Richard.

22. Re: JMS-based client-side load balancing: doc unclear?

clebert.suconic Jun 15, 2011 9:38 PM (in response to kennardconsulting)

Two nodes per journal? I don't really understand what mess you're doing there.

You can only have one node per journal.

23. Re: JMS-based client-side load balancing: doc unclear?

kennardconsulting Jun 15, 2011 9:51 PM (in response to clebert.suconic)

I never said I had two nodes per journal! I said the exact opposite: I know it is a no-no.

I'm asking whether there are other no-no's: because the configuration I have up there is losing messages.

24. Re: JMS-based client-side load balancing: doc unclear?

kennardconsulting Jun 16, 2011 8:08 AM (in response to kennardconsulting)

Okay I have made my peace with this whole issue. I have:

1. Kept the queue on just one node

2. Pointed all the MDBs on all nodes to 'consume' from the queue on that one node

3. Made all nodes 'produce' to the queue on that one node

4. Turned off clustering in hornetq-configuration.xml

5. Removed all acceptors/connectors/broadcast-groups/discovery-groups/cluster-connections etc from hornetq-configuration.xml

This works fine for me. I have a Single Point of Failure, but I have the performance I expect (ie. all nodes are kept busy at all times). I can wait for message redistribution (https://issues.jboss.org/browse/HORNETQ-721).

Thanks for everybody's time and suggestions,

Richard.