8 Replies Latest reply on Aug 14, 2013 7:41 AM by rhusar

    Client discovery of new nodes for clustered stateless EJBs

    wesleydstrickland

      We are running with JBoss 7.1.3 Final release and are using stateless replicated EJBs on the different nodes within a cluster.  We are using standalone-ha.xml to bring up the clusters, not the domain method.  We see where the nodes do join the cluster successfully in the JBoss server logs.

       

      What we would like to know is if there is a way for the clients to discover new nodes that are added to the cluster dynamically without specifically spelling out the node IPs and ports in the client connection parameters?  We have seen documentation that says the clients should need only one node in the list to be up and running and then it will get a new topology from the Infinispan cluster process and can see new nodes and access them.  In practice, however, we have not seen the client discover new nodes.  We have to manually add any new nodes to the client configuration and restart the client for any EJB calls to hit the new server nodes.

       

      Our client configuration currently looks like this:

       

      endpoint.name=ejb

       

      remote.connectionprovider.create.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT=false

      remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false

       

      remote.connections=one,two

             

      remote.connection.one.port=4447

      remote.connection.one.host=hp-server-1

       

      remote.connection.one.connect.options.org.xnio.Options.SSL_ENABLED=false

      remote.connection.one.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=true

      remote.connection.one.connect.options.org.xnio.Options.SASL_DISALLOWED_MECHANISMS=JBOSS-LOCAL-USER

      remote.connection.one.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT=false

       

      remote.connection.two.port=4447

      remote.connection.two.host=hp-server-2

       

      remote.connection.two.connect.options.org.xnio.Options.SSL_ENABLED=false

      remote.connection.two.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=true

      remote.connection.two.connect.options.org.xnio.Options.SASL_DISALLOWED_MECHANISMS=JBOSS-LOCAL-USER

      remote.connection.two.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT=false

       

      Any help would be greatly appreciated!

      -Wes

        • 1. Re: Client discovery of new nodes for clustered stateless EJBs
          wdfink

          You need to add the cluster parameter. See this jboss-ejb-client.properties from the ejb-clientes quickstart.

           

          Add the cluster with connection parameter and credentials like this:

          remote.clusters=ejb

          remote.cluster.ejb.connect.options.org.xnio.Options.*=<same properties as above>

          remote.cluster.ejb.username=<if needed>

          remote.cluster.ejb.password=<if needed>

          • 2. Re: Client discovery of new nodes for clustered stateless EJBs
            wesleydstrickland

            Thanks for that info Wolf, but I had also tried that as well without any luck.  Using Wireshark with a filter to only look at the network traffic coming back from the JBoss server node to the client, the only multicast packet that I see that I suspect might be the cluster information being multicast back to the client is in this following image:

             

             

            Capture.PNG

             

            I don't know if this is really the information and the client isn't processing it, or if the server is just not sending it back to the client.  We are using standalone-ha.xml and our inifinspan <subsystem> is configured like this:

             

            <subsystem xmlns="urn:jboss:domain:infinispan:1.3">

                        <cache-container name="cluster" aliases="ha-partition" default-cache="default">

                            <transport lock-timeout="60000"/>

                            <replicated-cache name="default" mode="SYNC" batching="true">

                                <locking isolation="REPEATABLE_READ"/>

                            </replicated-cache>

                        </cache-container>

                        <cache-container name="web" aliases="standard-session-cache" default-cache="repl" module="org.jboss.as.clustering.web.infinispan">

                            <transport lock-timeout="60000"/>

                            <replicated-cache name="repl" mode="ASYNC" batching="true">

                                <file-store/>

                            </replicated-cache>

                            <replicated-cache name="sso" mode="SYNC" batching="true"/>

                            <distributed-cache name="dist" mode="ASYNC" batching="true" l1-lifespan="0">

                                <file-store/>

                            </distributed-cache>

                        </cache-container>

                        <cache-container name="ejb" aliases="sfsb sfsb-cache" default-cache="repl" module="org.jboss.as.clustering.ejb3.infinispan">

                            <transport lock-timeout="60000"/>

                            <replicated-cache name="repl" mode="ASYNC" batching="true">

                                <eviction strategy="LRU" max-entries="10000"/>

                                <file-store/>

                            </replicated-cache>

                            <!--

                              ~  Clustered cache used internally by EJB subsytem for managing the client-mapping(s) of

                              ~                 the socketbinding referenced by the EJB remoting connector

                              -->

                            <replicated-cache name="remote-connector-client-mappings" mode="SYNC" batching="true"/>

                            <distributed-cache name="dist" mode="ASYNC" batching="true" l1-lifespan="0">

                                <eviction strategy="LRU" max-entries="10000"/>

                                <file-store/>

                            </distributed-cache>

                        </cache-container>

                        <cache-container name="hibernate" default-cache="local-query" module="org.jboss.as.jpa.hibernate:4">

                            <transport lock-timeout="60000"/>

                            <local-cache name="local-query">

                                <transaction mode="NONE"/>

                                <eviction strategy="LRU" max-entries="10000"/>

                                <expiration max-idle="100000"/>

                            </local-cache>

                            <invalidation-cache name="entity" mode="SYNC">

                                <transaction mode="NON_XA"/>

                                <eviction strategy="LRU" max-entries="10000"/>

                                <expiration max-idle="100000"/>

                            </invalidation-cache>

                            <replicated-cache name="timestamps" mode="ASYNC">

                                <transaction mode="NONE"/>

                                <eviction strategy="NONE"/>

                            </replicated-cache>

                        </cache-container>

                    </subsystem>

             

            Is there some other configuration I should check or any other missing client or standalone-ha.xml configurations or subsystems I should check?

             

            The second server node is recognized and does join the cluster.  Here is a snippet from the JBoss server log:

             

            12:23:55,603 INFO  [stdout] (ServerService Thread Pool -- 63)
            12:23:55,603 INFO  [stdout] (ServerService Thread Pool -- 63) ------------------
            -------------------------------------------------
            12:23:55,603 INFO  [stdout] (ServerService Thread Pool -- 63) GMS: address=node1
            /ejb, cluster=ejb, physical address=160.207.178.64:55200
            12:23:55,619 INFO  [stdout] (ServerService Thread Pool -- 63) ------------------
            -------------------------------------------------
            12:23:57,897 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport]
            (ServerService Thread Pool -- 63) ISPN000078: Starting JGroups Channel
            12:23:57,897 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport]
            (ServerService Thread Pool -- 63) ISPN000094: Received new cluster view: [node1/
            ejb|0] [node1/ejb]
            12:23:57,897 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport]
            (ServerService Thread Pool -- 63) ISPN000079: Cache local address is node1/ejb,
            physical addresses are [160.207.178.64:55200]
            12:23:57,912 INFO  [org.infinispan.factories.GlobalComponentRegistry] (ServerSer
            vice Thread Pool -- 63) ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.
            7.Final

            12:23:58,037 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Po
            ol -- 63) JBAS010281: Started remote-connector-client-mappings cache from ejb co
            ntainer
            12:23:58,037 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Po
            ol -- 64) JBAS010281: Started repl cache from ejb container
            12:23:58,053 INFO  [org.jboss.as.clustering] (MSC service thread 1-7) JBAS010238
            : Number of cluster members: 1

            12:23:58,162 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015961: Http mana
            gement interface listening on http://127.0.0.1:4850/management
            12:23:58,162 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin con
            sole listening on http://127.0.0.1:4850
            12:23:58,162 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS
            7.1.3.Final "Arges" started in 11420ms - Started 358 of 519 services (161 servic
            es are passive or on-demand)
            12:24:09,769 INFO  [org.jboss.as.clustering] (Incoming-1,null) JBAS010225: New c
            luster view for partition ejb (id: 1, delta: 1, merge: false) : [node1/ejb, node
            2/ejb]
            12:24:09,769 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport]
            (Incoming-1,null) ISPN000094: Received new cluster view: [node1/ejb|1] [node1/ej
            b, node2/ejb]

             

            Thanks for you help!

            -Wes

            • 3. Re: Client discovery of new nodes for clustered stateless EJBs
              rhusar
              We are running with JBoss 7.1.3 Final release and are using stateless replicated EJBs on the different nodes within a cluster.

               

              Stateless EJBs are not replicated. There is no state to replicate.

               

               

              What we would like to know is if there is a way for the clients to discover new nodes that are added to the cluster dynamically without specifically spelling out the node IPs and ports in the client connection parameters?

               

              No, its not possible to dynamically lookup the nodes (e.g. via multicast) at the moment.

               

               

              In practice, however, we have not seen the client discover new nodes.

               

              Wolf's advice is right. You need to "start" (e.g. make a first call) the client to receive the new topology before failover to new nodes is possible. Otherwise you may be hitting a bug.

              • 4. Re: Client discovery of new nodes for clustered stateless EJBs
                wesleydstrickland

                Thanks for your response Radoslav.  I didn't mean session replication for our stateless EJBs, I just meant we had deployed the exact same EAR and the exact same code for the EJBs on all of our server nodes.

                 

                I had thought that once I was connected successfully to one node listed under "remote.connections", that I would get back the topology from the server and then if new nodes joined the cluster, the client would do some sort of Random Round Robin and would be able to call the same EJBs on the new server nodes without making the user explicitly add them to the list of nodes in "remote.connections".  But I take it the topology information sent back to the client is not enough, and each node in the cluster must be listed under "remote.connections" or the client does not know about them?

                • 5. Re: Client discovery of new nodes for clustered stateless EJBs
                  rhusar

                  But I take it the topology information sent back to the client is not enough, and each node in the cluster must be listed under "remote.connections" or the client does not know about them?

                  Let me sum it up again: once the client is connected to one of the cluster nodes, it will receive the topology of the whole cluster. Therefore you only need the knowledge of at least one cluster node (of course, preferably multiple to achieve HA).

                  that I would get back the topology from the server and then if new nodes joined the cluster

                   

                  I need to check but IIRC cluster changes are not propagated eagerly -- you need to make an invocation to receive the new topology.

                  the client would do some sort of Random Round Robin and would be able to call the same EJBs

                   

                  The client prefers already connected nodes. Strict random/round robin makes no sense: imagine 1000 node cluster -- would it make sense for every client after 1000 invocation make/keep 1000 connections?

                  • 6. Re: Client discovery of new nodes for clustered stateless EJBs
                    wesleydstrickland

                    Thanks for your reply Radoslav.  I think I was misunderstanding what the topolgy information that was sent back to the client was used for.  My customer was trying to achieve HA without telling the clients about the new cluster nodes that were dynamically joining the cluster.  They were hoping that that topology information was all the client would need to discover and make EJB calls to new nodes without adding them to the client list of connections.

                     

                    Thanks for you help!

                    -Wes

                    • 7. Re: Client discovery of new nodes for clustered stateless EJBs
                      jaikiran

                      Radoslav Husar wrote:

                      that I would get back the topology from the server and then if new nodes joined the cluster

                       

                      I need to check but IIRC cluster changes are not propagated eagerly -- you need to make an invocation to receive the new topology.

                      Actually, no EJB invocation is required for the cluster topology propagation. The propagation starts happening eagerly as soon as the client enrolls a connection with the EJB client context. The propagation keeps happening in the background.

                      • 8. Re: Client discovery of new nodes for clustered stateless EJBs
                        rhusar

                        Actually, no EJB invocation is required for the cluster topology propagation. The propagation starts happening eagerly as soon as the client enrolls a connection with the EJB client context. The propagation keeps happening in the background.

                        That's perfect, thanks Jaikiran. So the initial address is only needed until the connection is established, then the topology is received and the invocations are HA (all until client is restarted).