6 Replies Latest reply on May 23, 2012 11:17 AM by jaikiran pai

    Configuring Multiple Clusters

    Graeme Wallace Newbie

      Running into issues around having one cluster make an EJB call to another cluster.

       

      Do I have to 'name' the clusters differently ? It seems that if my remote cluster is called 'ejb', and i have a local cluster called 'ejb' then i end up with conflicts.

       

      For Example:

       

      Cluster #1

      on machine 1 i deploy a clustered EJB named 'LocationModule' on cluster 'ejb'

      on machine 2 i deploy a clustered EJB named 'LocationModule' on cluster 'ejb'

       

      Cluster #2

      on machine 3 i deploy a clustered EJB named 'RelayModule' which wants to make a call to 'LocationModule' on cluster #1

      on machine 4 ditto

       

      When i make a call to the 'RelayModule' EJB - i can see from the information passed to my RoundRobinClusterNodeSelector that all the cluster hosts

      from cluster #2 have been added to the ejb cluster for cluster #1. When my cluster node selector picks a cluster #2 machine, of course the EJB call fails

      because LocationModule is only on the cluster #1 machines.

       

      I was thinking that the cluster 'name' was insignificant and that IP and multicast would control who was in what cluster like JBoss 4 used to do.

       

      Is this a bug ? Or if not - do cluster names have to be unique across my network ? And if so - do you just change the infinispan name in the config file ?

        • 1. Re: Configuring Multiple Clusters
          jaikiran pai Master

          Graeme Wallace wrote:

           

           

           

           

          When i make a call to the 'RelayModule' EJB -

          Where are you making that call from? Standalone client? What do the jboss-ejb-client.properties look like? Which initial connect nodes does it declare?

          • 2. Re: Configuring Multiple Clusters
            Graeme Wallace Newbie

            THe call is being made from a web application - its configured up to talk to cluster #2 - that part works ok. It's configured to talk to a cluster called EJB and the initial connection is to machine #3

             

            ITs when cluster #2 is trying to call an EJB on cluster #1 that things go pear shaped. Machine #3 ends up in the list of hosts passed to the cluster node selector.

            • 3. Re: Configuring Multiple Clusters
              jaikiran pai Master

              Okay, I'll have to take a look at this tomorrow. It's a bit too late in the night here and all that talk of cluster#1, machine#2, cluster#2, machine#3 isn't giving me a clear picture of what's going on I'll come back to this tomorrow morning.

              • 4. Re: Configuring Multiple Clusters
                jkreska Newbie

                jaikiran,

                 

                Graeme asked me to see if I could reproduce the problem with a simple example.  I was able to reproduce it by having a clustered ejb call a remote clustered ejb.

                 

                In the example I have attached, should be able to just run ant to build it, the jars dir will have 2 ears.  The cluster-server ear and relay ear.  There is also a ClientApp which connects to the relay ear.  My tests had 2 servers in the cluster-server cluster and 1 server in the relay cluster.  Depending on order of ejbs returned from the cluster the call might work the first couple of times but then will fail.

                 

                In the log below jb7-5 and jb7-7 are the cluster-server members and jb7-8 is the relay-server.

                 

                17:17:27,811 INFO  [org.jboss.ejb.client] (pool-6-thread-1) JBoss EJB Client version 1.0.10.Final
                17:17:33,478 INFO  [com.farecompare.jboss7example.relay.util.RoundRobinClusterNodeSelector] (EJB default - 2) RoundRobin::selectNode ejb ConnectedNodes: [jb7-7, jb7-8, jb7-5] AvailableNodes: [jb7-7, jb7-8, jb7-5]
                17:17:33,479 INFO  [com.farecompare.jboss7example.relay.util.RoundRobinClusterNodeSelector] (EJB default - 2) RoundRobin::selectNode chose jb7-7
                17:17:37,637 INFO  [com.farecompare.jboss7example.relay.util.RoundRobinClusterNodeSelector] (EJB default - 3) RoundRobin::selectNode ejb ConnectedNodes: [jb7-7, jb7-8, jb7-5] AvailableNodes: [jb7-7, jb7-8, jb7-5]
                17:17:37,637 INFO  [com.farecompare.jboss7example.relay.util.RoundRobinClusterNodeSelector] (EJB default - 3) RoundRobin::selectNode chose jb7-8
                17:17:37,639 ERROR [org.jboss.as.ejb3.invocation] (EJB default - 3) JBAS014134: EJB Invocation failed on component RelayModuleSessionBean for method public abstract java.lang.String com.farecompare.jboss7example.relay.api.RelayModuleAPI.getNameForCityCode(java.lang.String): javax.ejb.EJBException: java.lang.IllegalArgumentException: JBAS014155: Unknown deployment - app name: locationmodule module name: farecompare-locationmodule-ejb distinct name: 
                        at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:165) [jboss-as-ejb3-7.1.2.Final.jar:7.1.2.Final]
                        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:229) [jboss-as-ejb3-7.1.2.Final.jar:7.1.2.Final]
                        at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:303) [jboss-as-ejb3-7.1.2.Final.jar:7.1.2.Final]
                        at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:189) [jboss-as-ejb3-7.1.2.Final.jar:7.1.2.Final]
                        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
                        at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:80) [jboss-as-ejb3-7.1.2.Final.jar:7.1.2.Final]
                        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
                        at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.1.2.Final.jar:7.1.2.Final]
                        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
                        at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:42) [jboss-as-ejb3-7.1.2.Final.jar:7.1.2.Final]
                        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
                        at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.1.2.Final.jar:7.1.2.Final]
                        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
                        at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.1.2.Final.jar:7.1.2.Final]
                        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
                        at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:43) [jboss-as-ejb3-7.1.2.Final.jar:7.1.2.Final]
                        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
                        at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.1.2.Final.jar:7.1.2.Final]
                        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
                        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
                        at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee-7.1.2.Final.jar:7.1.2.Final]
                        at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:299) [jboss-as-ejb3-7.1.2.Final.jar:7.1.2.Final]
                        at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:67) [jboss-as-ejb3-7.1.2.Final.jar:7.1.2.Final]
                        at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:199) [jboss-as-ejb3-7.1.2.Final.jar:7.1.2.Final]
                        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_04]
                        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_04]
                        at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_04]
                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_04]
                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_04]
                        at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_04]
                        at org.jboss.threads.JBossThread.run(JBossThread.java:122)
                Caused by: java.lang.IllegalArgumentException: JBAS014155: Unknown deployment - app name: locationmodule module name: farecompare-locationmodule-ejb distinct name: 
                        at org.jboss.as.ejb3.remote.LocalEjbReceiver.findBean(LocalEjbReceiver.java:318) [jboss-as-ejb3-7.1.2.Final.jar:7.1.2.Final]
                        at org.jboss.as.ejb3.remote.LocalEjbReceiver.processInvocation(LocalEjbReceiver.java:146) [jboss-as-ejb3-7.1.2.Final.jar:7.1.2.Final]
                        at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:179) [jboss-ejb-client-1.0.10.Final.jar:1.0.10.Final]
                        at org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:43) [jboss-ejb-client-1.0.10.Final.jar:1.0.10.Final]
                        at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:181) [jboss-ejb-client-1.0.10.Final.jar:1.0.10.Final]
                        at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:125) [jboss-ejb-client-1.0.10.Final.jar:1.0.10.Final]
                        at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:181) [jboss-ejb-client-1.0.10.Final.jar:1.0.10.Final]
                        at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136) [jboss-ejb-client-1.0.10.Final.jar:1.0.10.Final]
                        at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121) [jboss-ejb-client-1.0.10.Final.jar:1.0.10.Final]
                        at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104) [jboss-ejb-client-1.0.10.Final.jar:1.0.10.Final]
                        at $Proxy31.getCityName(Unknown Source) at com.farecompare.jboss7example.relay.impl.RelayModule.getNameForCityCode(Unknown Source) [farecompare-relaymodule-impl.jar:]
                        at com.farecompare.jboss7example.relay.ejb.RelayModuleSessionBean.getNameForCityCode(Unknown Source) [farecompare-relaymodule-ejb.jar:]
                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_04]
                        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_04]
                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_04]
                        at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_04]
                        at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee-7.1.2.Final.jar:7.1.2.Final]
                        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
                        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
                        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:36) [jboss-as-ee-7.1.2.Final.jar:7.1.2.Final]
                        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
                        at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3-7.1.2.Final.jar:7.1.2.Final]
                        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
                        at org.jboss.as.jpa.interceptor.SBInvocationInterceptor.processInvocation(SBInvocationInterceptor.java:47)
                        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
                        at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
                        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
                        at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
                        at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee-7.1.2.Final.jar:7.1.2.Final]
                        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
                        at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:51) [jboss-as-ejb3-7.1.2.Final.jar:7.1.2.Final]
                        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final.jar:1.1.1.Final]
                        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:227) [jboss-as-ejb3-7.1.2.Final.jar:7.1.2.Final]
                        ... 29 more
                
                • 5. Re: Configuring Multiple Clusters
                  Graeme Wallace Newbie

                  Jaikiran - i see you've been busy today...Any chance you can take a look ?

                  • 6. Re: Configuring Multiple Clusters
                    jaikiran pai Master

                    I did take a look So here's how it works - The EJB client API identifies a cluster by a cluster identifier. That's all from the client side of things. The server side @Clustered SFSB is backed by a clustered-cache (see the stateful element in the EJB3 subsystem xml):

                     

                    <stateful default-access-timeout="5000" cache-ref="simple" clustered-cache-ref="clustered"/>
                    

                     

                    The clustered cache inturn has a passivation store. In case of clustered beans, it's a infinispan backed passivation store:

                     

                    <caches>
                    ..
                                    <cache name="clustered" passivation-store-ref="infinispan" aliases="StatefulTreeCache"/>
                    
                    ...
                    

                    The "infinispan" passivation store is backed by a cache-container:

                     

                    <passivation-stores>
                    ...
                                    <cluster-passivation-store name="infinispan" cache-container="ejb"/>
                    

                     

                    Notice the name "ejb" for that cache-container. This cache-container finally is configured in the Infinispan subsystem:

                     

                    <subsystem xmlns="urn:jboss:domain:infinispan:1.2" default-cache-container="cluster">
                    ...            
                                <cache-container name="ejb" aliases="sfsb sfsb-cache" default-cache="repl">
                    ...
                    

                    So in short, this is where the cluster name "ejb" comes from and that's what is used to identify the cluster in the EJB client API. So if you have a client which can/will interact with multiple nodes belonging to different clusters then you'll have to make sure that the cluster name (effectively cache-container names are unique).


                    So in your case, for cluster#2 you'll have to create a copy of "ejb" cache-container and use it in the cluster-passivation-store. Once you do those changes, on the client side, you have to use this new cluster name for configurations of your remote cluster connections in jboss-ejb-client.properties/jboss-ejb-client.xml.

                     

                    I think we might be able to make this simpler by maybe using an alias cluster name for reference in the passivation store, so that you don't have to recreate a copy of the cache-container. But for now, that's not possible. I'll check with Paul.