12 Replies Latest reply on Apr 5, 2005 9:17 AM by leonardosalvatore

    Problem with HAJMS MDB accessing load balanced SLSB.

    mattsh

      Hi,

      I'm having difficulty with load balancing SLSB on a cluster of 2 nodes running JBoss 4.01sp1 (I had the same issue with 3.2.7, upgraded to latest version to see if it was a fixed issue).

      The SLSB is configured to do round robin load balancing and is deployed on 2 nodes.

      I've also got HAJMS configured with a DurableTopic which has a collection of MDB's listening to it.

      When a MDB receives a message it should lookup the SLSB to handle the contents of the message. The idea being that the MDB's are lightweight (since they are only on one node) and SLSB which are clustered handle the processing of the messages.

      The failover for the SLSB's works fine, but for some reason the MDB always connects to the same node to get a SLSB instead of load balancing.

      To test if there is a problem with the SLSB configuration I copied the code that connects to the SLSB out of the MDB and into a client app. When using the client app to access the JBoss cluster both SLSB load balancing and failover work.

      Is load balancing limited to clients connecting from the outside or should load balancing occur between the MDB's & SLSB within the cluster?
      I'm assuming(hoping) it should work there is just some configuration setting somewhere that I've missed.

      I can post the code & configuration files. Decide to wait and see what files people wanted to see instead of a collection of configuration files that nobody cares about.


      Thanks in advanced for any help.

      Matt

        • 1. Re: Problem with HAJMS MDB accessing load balanced SLSB.
          schrouf

          You have to lookup your SLSB remote home interface from within your MDB via a 'special' context ( not a context retrieved via new InitialContext() ) to get a 'marshalled' HA-aware SLSB (home) proxy, and not an optimized local In-JVM-reference.

          Read the javadoc comments of the helper class for more details. Maybe this will help.

          Regards
          Ulf

          /********************************************************************
           * Helper class for JBoss specific lookup of a marshalled naming
           * context required for inter-EAR remote EJB lookups in case of an
           * isolated EAR classloader server configuration
           */
          public class MarshalledContext
          {
           /***************************************
           * Get an initial JNDI context that returns lookup results as marshalled
           * objects instead of simple object references even for in-JVM lookups.
           * <p>
           * Such an initial context is required for inter-EAR remote EJB lookups in
           * case of an isolated EAR classloader server configuration. Usage of the
           * standard server-side InitialContext (created via <code>new InitalContext()
           * </code>) in this situation will cause ClassCastExceptions.
           * <p>
           * As the required local JNDI port might has be changed dynamically by the
           * service binding manager in case of a multi-JBoss-instance server setup,
           * we have to query the actual port number of the local JNDI naming service
           * (default 1099) via JBoss JMX RMI adaptor.
           * </p>
           *
           * @param rInitialContext initial context for RMI adaptor lookup
           */
           public static Context getContext( Context rInitialContext )
          
           throws Exception
           {
           RMIAdaptor rAdapter = (RMIAdaptor) rInitialContext.lookup("jmx/invoker/RMIAdaptor");
           Integer nPort = null;
          
           // get actual port number (default 1099) of the local naming service
           ObjectName rService = new ObjectName("jboss:service=Naming");
          
           nPort = (Integer) rAdapter.getAttribute(rService, "Port");
          
           Properties aProps = new Properties();
          
           aProps.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory");
           aProps.put(Context.PROVIDER_URL, "localhost:" + nPort.toString());
          
           return new InitialContext(aProps);
           }
          
           /***************************************
           * Get an initial HAJNDI context that returns lookup results as marshalled
           * objects instead of simple object references even for in-JVM lookups.
           * <p>
           * <bold>ATTENTION</bold>
           * </p><p>
           * For this to work the local JBoss naming service has to be configured for
           * <bold>CallByValue</bold> semantics, as otherwise HAJNDI to local JNDI
           * delegated lookups would still return by default a simple object references
           * for in-JVM lookups instead of an marshalled object!
           * </p><p>
           * To enable "CallByValue" semantics the following attribute entry has to be
           * added to the naming service mbean definition in 'conf/jboss-service.xml'
           * <code>
           * <mbean code="org.jboss.naming.NamingService" name="jboss:service=Naming">
           * ......
           * <!-- force marshalling of the lookup which is required in case of
           * isolated EAR classloader (see also jboss.j2ee:service=EARDeployer) -->
           * <attribute name="CallByValue">true</attribute>
           * .....
           * </mbean>
           * </code>
           * </p><p>
           * As the required local HAJNDI port might has be changed dynamically by the
           * service binding manager in case of a multi-JBoss-instance server setup,
           * we have to query the actual port number of the local JNDI naming service
           * (default 1100) via JBoss JMX RMI adaptor.
           * </p>
           *
           * @param rInitialContext initial context for RMI adaptor lookup
           */
           public static Context getClusterContext( Context rInitialContext )
          
           throws Exception
           {
           RMIAdaptor rAdapter = (RMIAdaptor) rInitialContext.lookup("jmx/invoker/RMIAdaptor");
           Integer nPort = null;
          
           // get actual port number (default 1100) of the local HAJNDI service
           ObjectName rService = new ObjectName("jboss:service=HAJNDI");
          
           nPort = (Integer) rAdapter.getAttribute(rService, "Port");
          
           Properties aProps = new Properties();
          
           aProps.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory");
           aProps.put(Context.PROVIDER_URL, "localhost:" + nPort.toString());
          
           return new InitialContext(aProps);
           }
          }
          
          


          • 2. Re: Problem with HAJMS MDB accessing load balanced SLSB.
            mattsh

            Thanks for the help.

            I was already specifying the providerUrl="localhost:1100" to connect to HAJNDI but didn't know about changing the 'CallByValue' to 'true' to stop JNDI always connecting to the local JVM object.

            I followed the instructions in the javadoc and changed the naming service attribute 'CallByValue' to true.
            But this did not fix the problem.

            I've included some of my configuration files in the hope that any errors will be spotted.

            jboss-service.xml

            <mbean code="org.jboss.naming.NamingService"
             name="jboss:service=Naming"
             xmbean-dd="resource:xmdesc/NamingService-xmbean.xml">
            
             <!-- The call by value mode. true if all lookups are unmarshalled using
             the caller's TCL, false if in VM lookups return the value by reference.
             -->
             <attribute name="CallByValue">true</attribute>
             ...
            </mbean>
            


            jboss.xml
            <jboss>
            
             <enterprise-beans>
             <session>
             <ejb-name>MessagingManager</ejb-name>
             <jndi-name>ejb/hp/notification/MessagingManager</jndi-name>
             <local-jndi-name>ejb/hp/notification/MessagingManagerLocal</local-jndi-name>
             <clustered>true</clustered>
             <cluster-config>
             <partition-name>HPNEPartition</partition-name>
             <home-load-balance-policy>org.jboss.ha.framework.interfaces.RoundRobin</home-load-balance-policy>
             <bean-load-balance-policy>org.jboss.ha.framework.interfaces.RoundRobin</bean-load-balance-policy>
             </cluster-config>
            
             </session>
             </enterprise-beans>
            
             <resource-managers>
             </resource-managers>
            </jboss>
            


            ejb-jar.xml
            <ejb-jar >
             <enterprise-beans>
             <session >
             <ejb-name>MessagingManager</ejb-name>
             <home>com.mycomp.notification.ejb.core.interfaces.MessagingManagerHome</home>
             <remote>com.mycomp.notification.ejb.core.interfaces.MessagingManager</remote>
             <local-home>com.mycomp.notification.ejb.core.interfaces.MessagingManagerLocalHome</local-home>
             <local>com.mycomp.notification.ejb.core.interfaces.MessagingManagerLocal</local>
             <ejb-class>com.mycomp.notification.ejb.core.MessagingManagerSession</ejb-class>
             <session-type>Stateless</session-type>
             <transaction-type>Container</transaction-type>
             </session>
             </enterprise-beans>
            
             <assembly-descriptor >
             <method-permission >
             <unchecked/>
             <method>
             <ejb-name>MessagingManager</ejb-name>
             <method-name>*</method-name>
             </method>
             </method-permission>
            
             <container-transaction >
             <method >
             <ejb-name>MessagingManager</ejb-name>
             <method-name>*</method-name>
             </method>
             <trans-attribute>Required</trans-attribute>
             </container-transaction>
            
             </assembly-descriptor>
            </ejb-jar>
            


            my code in mdb to access SLSB
            Properties prop = new Properties();
            prop.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory");
            prop.put(Context.PROVIDER_URL, "localhost:1100");
            Context context = new InitialContext(prop);
            MessagingManagerHome messagingManagerHome = (MessagingManagerHome) context.lookup("ejb/hp/notification/MessagingManager");
            MessagingManager messagingManager = messagingManagerHome.create();
            




            Can anyone see anything wrong with my configuration?

            Regards

            Matt

            • 3. Re: Problem with HAJMS MDB accessing load balanced SLSB.
              leonardosalvatore

              Watching in your MDB code your are lookuping always on HAJndi ()

              prop.put(Context.PROVIDER_URL, "localhost:1100");

              So local JNDI names will not discovered.

              So you should lookup in local JNDI too.
              ObjectName rService = new ObjectName("jboss:service=HAJNDI");
              ObjectName rService = new ObjectName("jboss:service=Naming");

              And retreive the correct port Adapter.getAttribute(rService, "Port"); via RMI.

              • 4. Re: Problem with HAJMS MDB accessing load balanced SLSB.
                schrouf

                I do not see any obvious problems within your configuration, so I think you have to do some JBoss code debugging. Step into both calls to

                MessagingManagerHome messagingManagerHome = (MessagingManagerHome) context.lookup("ejb/hp/notification/MessagingManager");

                and

                MessagingManager messagingManager = messagingManagerHome.create();

                First check, if you are getting a marshalled object reference (CallByValue) by your call to context.lookup().

                That step into the MessagingMangerHom.create() and and MessagingManger. code and look for some HA-aware JBoss internal proxy code sequences....

                • 5. Re: Problem with HAJMS MDB accessing load balanced SLSB.
                  schrouf

                  So local JNDI names will not discovered.


                  Not exactly, HAJNDI always delegates lookups to local JNDI first. When the object cannot be found within the local JNDI a cluster-wide lookup will be started afterwards.

                  • 6. Re: Problem with HAJMS MDB accessing load balanced SLSB.
                    schrouf

                    Just another thought on your problem

                    When a MDB receives a message it should lookup the SLSB to handle the contents of the message. The idea being that the MDB's are lightweight (since they are only on one node) and SLSB which are clustered handle the processing of the messages.

                    The failover for the SLSB's works fine, but for some reason the MDB always connects to the same node to get a SLSB instead of load balancing.



                    Why do you restrict the MDB's to one node ?

                    Deploying them on ALL the same nodes as your SLSB and delegation to a local SLBS from within onMessage() would also loadbalance your message processing !

                    Furthermore you would gain transactional safety, as message reception (MDB) and processing (SLSB) could be handled within the same local transaction and could be consistently be rolled back in case of a failure.

                    Keep in mind that JBoss transaction manager does not support distributed transactions, so a remote call to a SLSB on another cluster-node DOES NOT take part in the MDB transaction !

                    Just a thought :-)

                    Regards
                    Ulf


                    • 7. Re: Problem with HAJMS MDB accessing load balanced SLSB.
                      mattsh

                       


                      Why do you restrict the MDB's to one node ?

                      Deploying them on ALL the same nodes as your SLSB and delegation to a local SLBS from within onMessage() would also loadbalance your message processing !


                      The MDB's are restricted to one node because they listen to a HAJMS DurableTopic if I deploy them to multiple nodes I get JMS exceptions because of duplicate client id's.

                       javax.jms.InvalidClientIDException: This client id 'blah' is already registered
                      


                      Unfortunately if I use different client ids on each node the MDB on both nodes process the message, which leads to duplicate requests.

                      I decided to deploy the MDB's on one node. The plan was then to remove most of the logic out of the MDB into SLSB which are load balanced across the cluster.

                      Unless is there some way to have MDB's on multiple nodes listening to a HAJMS DurableTopic? I think Distributed Destinations might supported this but it won't be available to JBossMessaging is complete in JBoss5.

                      Thanks for the warning about the transaction manager not supporting distributed transactions, will have to think about a way round that problem :-(

                      Regards

                      Matt

                      • 8. Re: Problem with HAJMS MDB accessing load balanced SLSB.
                        leonardosalvatore

                        If you try to run HAJMS with a different shared database installed only on one member of cluster, you will have a round robin load balancing


                        Some words from: http://www.jboss.org/wiki/Wiki.jsp?page=JBossMQHA


                        Load-balancing message processing with MDBs is a one of the best ways to distribute load across a cluster because the node with the least activity will process the message (as opposed to a round-robin load-balancing algorithm which could continue to place load on an already overloaded node).

                        To run HAJMS with a different shared database, you will need to replace the hsqldb-jdbc2-service.xml file with one tuned to the specific database. For example if you use MySQL the file is mysql-jdbc2-service.xml. Configuration files for a number of RDBMS are bundled with the JBoss distribution. They can be found under docs/examples/jms.


                        I resolved all HAJMS problem with this documents in my cluster.



                        • 9. Re: Problem with HAJMS MDB accessing load balanced SLSB.
                          schrouf

                          I decided to deploy the MDB's on one node. The plan was then to remove most of the logic out of the MDB into SLSB which are load balanced across the cluster.


                          Ok, what about the following two-step approch:

                          The MDB with a durable subscription to your JMS topic (restricted to one node due to the unique client ID requirement) simply forwards each received message to an 'internal' processing queue (let's call this MDB message forwarder).

                          These messages then can be processed (without duplicates) by a second MDB () deployed on ALL nodes (let's call these MDB's message processors).

                          You would gain perfect loadbalacing AND transactional safety at the cost of an additonal JMS message send operation.

                          As the processing is anyway asynchronous this shouldn not be a real problem in most cases :-)

                          Regards
                          Ulf

                          • 10. Re: Problem with HAJMS MDB accessing load balanced SLSB.
                            leonardosalvatore

                            Good if messages forwarding is not a problem for bandwith.


                            I'm trying the way of share DB.
                            I'm setting up 2 Jboss cluster + 1 DbMysql.

                            The Load balancing is granted by rounde robin for master of cluster.
                            The message queue/topic are stored on external db , JMS deployed singleton with Mysql as default datasource.

                            So MDB are deployed on each node but only temporary masterJboss will be retrieve the message from DB.

                            For message sending/posting obviously no problem.

                            This allow to drop HSQL.I try this on a single machine and it's great!

                            • 11. Re: Problem with HAJMS MDB accessing load balanced SLSB.
                              mattsh

                               

                              "schrouf" wrote:
                              I decided to deploy the MDB's on one node. The plan was then to remove most of the logic out of the MDB into SLSB which are load balanced across the cluster.


                              Ok, what about the following two-step approch:

                              The MDB with a durable subscription to your JMS topic (restricted to one node due to the unique client ID requirement) simply forwards each received message to an 'internal' processing queue (let's call this MDB message forwarder).

                              These messages then can be processed (without duplicates) by a second MDB () deployed on ALL nodes (let's call these MDB's message processors).

                              You would gain perfect loadbalacing AND transactional safety at the cost of an additonal JMS message send operation.

                              As the processing is anyway asynchronous this shouldn not be a real problem in most cases :-)

                              Regards
                              Ulf


                              I had considered this approach but was trying to use just the one set of MDB's listening to a DurableTopic.

                              Anyway decided to use this two step approach of a topic with mdb's forwarding the message onto a queue, etc.

                              Which I now have working with load balancing, failover,etc.

                              Thanks for all your help.

                              Matt

                              • 12. Re: Problem with HAJMS MDB accessing load balanced SLSB.
                                leonardosalvatore

                                I finished to stand up my cluster in this way:

                                Two Machine Jboss and one Mysql.

                                I deployed cluster Entity,MDB and a JMX class able to control and monitor my application. (All these in a single SAR)
                                All persistency is on Mysql and the queue is on cluster master (Roundrobin) and allow load balancing for MDB and Entity.

                                Creation of Entity and message sending/receiving is now balanced with a real test with thousands of messages.