4 Replies Latest reply on Sep 2, 2004 5:31 PM by chalakanth

    How to minimize the traffic in a cluster.

    chalakanth
    chalakanth Sep 2, 2004 5:31 PM

      This is a newbie question I am afraid. Also this maybe related solely to JGroups.

      Our company has just started to use JBoss Clusters (3.2.3 on Windows XP). We put all our developers machines on one cluster - that is about 20 machines. We have noticed that the network traffic between the machines in the cluster is so huge that our network is slowing down considerably.

      We were wondering if there was some way to reduce this traffic without giving up the cluster.

      For instance, perhaps we can do without auto-discovery of membership. Is it possible to define a static list of members?

      Since we are not doing anything very sophisticated with our cluster (not clustering EJBs or JSP sessions), we want to test a really minimal JGroups configuration (shown below). This assumes there is a static way to define cluster membership. Can anybody see any problems with this? Any comments will be greatly appreciated.


      <attribute name="PartitionConfig">
      <Config>
      <UDP mcast_addr="228.1.2.3" mcast_port="45506"
      ip_ttl="1"
      ip_mcast="true"
      mcast_send_buf_size="150000"
      mcast_recv_buf_size="80000"
      ucast_send_buf_size="150000"
      ucast_recv_buf_size="80000"
      loopback="false" />

      <pbcast.NAKACK
      gc_lag="50"
      retransmit_timeout="300,600,1200,2400,4800"
      up_thread="true" down_thread="true" />
      </Config>
      </attribute>


      thanks,

      chalakanth

      • 3011 Views
      • Tags:
        • 1. cannot list on any port in range 0-123
          chalakanth
          chalakanth Sep 2, 2004 5:31 PM (in response to chalakanth)

          Hi, I posted this question previously but forgot to include my cluster-service.xml. I'm using JBoss 4.0.0 on a Red Hat Linux machine and am having trouble with JGroups when I start JBoss. Below is startup log in debug mode and then my cluster-service.xml. Please help me. Thank you very much.

          ......
          23:40:31,465 DEBUG [ServiceController] Creating dependent components for: jboss:service=invoker,type=jrmpha dependents are: []
          23:40:31,465 DEBUG [ServiceController] Creating service jboss.cache:service=InvalidationBridge,type=JavaGroups
          23:40:31,465 DEBUG [ServiceController] Ignoring create request for service: jboss.cache:service=InvalidationBridge,type=JavaGroups
          23:40:31,467 DEBUG [MainDeployer] Done with create step of deploying cluster-service.xml
          23:40:31,467 DEBUG [MainDeployer] Begin deployment start file:/opt/jboss-4.0.0/server/default/deploy/cluster-service.xml
          23:40:31,467 DEBUG [SARDeployer] Deploying SAR, start step: url file:/opt/jboss-4.0.0/server/default/deploy/cluster-service.xml
          23:40:31,467 DEBUG [ServiceController] starting service jboss:service=DefaultPartition
          23:40:31,467 DEBUG [ClusterPartition] Starting jboss:service=DefaultPartition
          23:40:31,491 DEBUG [ClusterPartition] Starting ClusterPartition: DefaultPartition
          23:40:31,542 ERROR [JChannel] exception: java.lang.Exception: exception caused by UDP.start(): java.lang.Exception: UDP.createSocket
          s(): cannot list on any port in range 0-123:40:31,545 ERROR [ClusterPartition] Starting failed jboss:service=DefaultPartition
          ChannelException: java.lang.Exception: exception caused by UDP.start(): java.lang.Exception: UDP.createSockets(): cannot list on any
          port in range 0-1 at org.jgroups.JChannel.connect(JChannel.java:328)
          at org.jboss.ha.framework.server.ClusterPartition.startService(ClusterPartition.java:297)
          at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:271)
          at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:221)
          at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:324)
          at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
          at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
          at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
          at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:242)
          at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:642)
          at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:242)
          at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:642)
          at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:891)
          at $Proxy0.start(Unknown Source)
          at org.jboss.system.ServiceController.start(ServiceController.java:416)
          at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:324)
          at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
          at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
          at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
          at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:242)
          at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:642)
          at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
          at $Proxy4.start(Unknown Source)
          at org.jboss.deployment.SARDeployer.start(SARDeployer.java:261)
          at org.jboss.deployment.MainDeployer.start(MainDeployer.java:935)
          at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:746)
          at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:709)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:324)
          at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
          at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
          at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:119)
          at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
          at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:131)
          at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
          at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:242)
          at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:642)
          at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
          at $Proxy8.deploy(Unknown Source)
          at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:305)
          at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:481)
          at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:305)
          at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:481)
          at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:204)
          at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:277)
          at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:271)
          at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:221)
          at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:324)
          at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
          at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
          at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
          at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:242)
          at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:642)
          at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:891)
          at $Proxy0.start(Unknown Source)
          at org.jboss.system.ServiceController.start(ServiceController.java:416)
          at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:324)
          at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
          at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
          at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
          at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:242)
          at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:642)
          at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
          at $Proxy4.start(Unknown Source)
          at org.jboss.deployment.SARDeployer.start(SARDeployer.java:261)
          at org.jboss.deployment.MainDeployer.start(MainDeployer.java:935)
          at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:746)
          at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:709)
          at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:693)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:324)
          at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
          at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
          at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:119)
          at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
          at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:131)
          at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
          at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:242)
          at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:642)
          at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
          at $Proxy5.deploy(Unknown Source)
          at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:396)
          at org.jboss.system.server.ServerImpl.start(ServerImpl.java:293)
          at org.jboss.Main.boot(Main.java:151)
          at org.jboss.Main$1.run(Main.java:405)
          at java.lang.Thread.run(Thread.java:534)
          23:40:31,550 WARN [ServiceController] Problem starting service jboss:service=DefaultPartition


          ------------- cluster-service.xml -------------------

          <?xml version="1.0" encoding="UTF-8"?>

<!-- ===================================================================== -->
<!-- -->
<!-- Sample Clustering Service Configuration -->
<!-- -->
<!-- ===================================================================== -->

<server>

 <classpath codebase="lib" archives="jbossha.jar"/>

 <!-- ==================================================================== -->
 <!-- Cluster Partition: defines cluster -->
 <!-- ==================================================================== -->

 <mbean code="org.jboss.ha.framework.server.ClusterPartition"
 name="jboss:service=DefaultPartition">

 <!-- Name of the partition being built -->
 <attribute name="PartitionName">DefaultPartition</attribute>

 <!-- The address used to determine the node name -->
 <attribute name="NodeAddress">${jboss.bind.address}</attribute>

 <!-- Determine if deadlock detection is enabled -->
 <attribute name="DeadlockDetection">False</attribute>

 <!-- Max time (in ms) to wait for state transfer to complete. Increase for large states -->
 <attribute name="StateTransferTimeout">30000</attribute>

 <!-- The JGroups protocol configuration -->
 <attribute name="PartitionConfig">
 <Config>
 <!-- UDP: if you have a multihomed machine,
 set the bind_addr attribute to the appropriate NIC IP address -->
 <!-- UDP: On Windows machines, because of the media sense feature
 being broken with multicast (even after disabling media sense)
 set the loopback attribute to true -->
 <UDP mcast_addr="228.1.2.3" mcast_port="45566"
 ip_ttl="32" ip_mcast="true"
 mcast_send_buf_size="800000" mcast_recv_buf_size="150000"
 ucast_send_buf_size="800000" ucast_recv_buf_size="150000"
 loopback="false" />
 <PING timeout="2000" num_initial_members="3"
 up_thread="true" down_thread="true" />
 <MERGE2 min_interval="10000" max_interval="20000" />
 <FD shun="true" up_thread="true" down_thread="true"
 timeout="2500" max_tries="5" />
 <VERIFY_SUSPECT timeout="3000" num_msgs="3"
 up_thread="true" down_thread="true" />
 <pbcast.NAKACK gc_lag="50" retransmit_timeout="300,600,1200,2400,4800"
 max_xmit_size="8192"
 up_thread="true" down_thread="true" />
 <UNICAST timeout="300,600,1200,2400,4800" window_size="100" min_threshold="10"
 down_thread="true" />
 <pbcast.STABLE desired_avg_gossip="20000"
 up_thread="true" down_thread="true" />
 <FRAG frag_size="8192"
 down_thread="true" up_thread="true" />
 <pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
 shun="true" print_local_addr="true" />
 <pbcast.STATE_TRANSFER up_thread="true" down_thread="true" />
 </Config>
 </attribute>

 </mbean>

 <!-- ==================================================================== -->
 <!-- HA Session State Service for SFSB -->
 <!-- ==================================================================== -->

 <mbean code="org.jboss.ha.hasessionstate.server.HASessionStateService"
 name="jboss:service=HASessionState">
 <depends>jboss:service=DefaultPartition</depends>
 <!-- Name of the partition to which the service is linked -->
 <attribute name="PartitionName">DefaultPartition</attribute>
 <!-- JNDI name under which the service is bound -->
 <attribute name="JndiName">/HASessionState/Default</attribute>
 <!-- Max delay before cleaning unreclaimed state.
 Defaults to 30*60*1000 => 30 minutes -->
 <attribute name="BeanCleaningDelay">0</attribute>
 </mbean>

 <!-- ==================================================================== -->
 <!-- HA JNDI -->
 <!-- ==================================================================== -->

 <mbean code="org.jboss.ha.jndi.HANamingService"
 name="jboss:service=HAJNDI">
 <depends>jboss:service=DefaultPartition</depends>
 <!-- Name of the partition to which the service is linked -->
 <attribute name="PartitionName">DefaultPartition</attribute>
 <!-- bind address of HA JNDI RMI endpoint -->
 <attribute name="BindAddress">${jboss.bind.address}</attribute>
 <!-- RmiPort to be used by the HA-JNDI service
 once bound. 0 => auto. -->
 <attribute name="RmiPort">0</attribute>
 <!-- Port on which the HA-JNDI stub is made available -->
 <attribute name="Port">1100</attribute>
 <!-- Backlog to be used for client-server RMI
 invocations during JNDI queries -->
 <attribute name="Backlog">50</attribute>

 <!-- Multicast Address and Group used for auto-discovery -->
 <attribute name="AutoDiscoveryAddress">230.0.0.4</attribute>
 <attribute name="AutoDiscoveryGroup">1102</attribute>

 <!-- IP Address to which should be bound: the Port, the RmiPort and
 the AutoDiscovery multicast socket. -->
 <!-- Client socket factory to be used for client-server
 RMI invocations during JNDI queries -->
 <!--attribute name="ClientSocketFactory">custom</attribute-->
 <!-- Server socket factory to be used for client-server
 RMI invocations during JNDI queries -->
 <!--attribute name="ServerSocketFactory">custom</attribute-->
 </mbean>

 <mbean code="org.jboss.invocation.jrmp.server.JRMPInvokerHA"
 name="jboss:service=invoker,type=jrmpha">
 <attribute name="ServerAddress">${jboss.bind.address}</attribute>
 <!--
 <attribute name="RMIObjectPort">0</attribute>
 <attribute name="RMIClientSocketFactory">custom</attribute>
 <attribute name="RMIServerSocketFactory">custom</attribute>
 -->
 </mbean>

 <!-- ==================================================================== -->
 <!-- Distributed cache invalidation -->
 <!-- ==================================================================== -->

 <mbean code="org.jboss.cache.invalidation.bridges.JGCacheInvalidationBridge"
 name="jboss.cache:service=InvalidationBridge,type=JavaGroups">
 <depends>jboss:service=DefaultPartition</depends>
 <depends>jboss.cache:service=InvalidationManager</depends>
 <attribute name="InvalidationManager">jboss.cache:service=InvalidationManager</attribute>
 <attribute name="PartitionName">DefaultPartition</attribute>
 <attribute name="BridgeName">DefaultJGBridge</attribute>
 </mbean>

</server>


            • Actions
          • 2. Re: How to minimize the traffic in a cluster.
            belaban
            belaban Sep 2, 2004 6:37 PM (in response to chalakanth)

            If you use the default config, and replace FD with FD_SOCK, then traffic will be *null* when there is no data between the nodes (e.g. replication messages).
            Isn't it the case that you're seeing lots of trafic because you are making a lot of changes, thus replicating a lot ?
            Bela

              • Actions
            • 3. Re: How to minimize the traffic in a cluster.
              chalakanth
              chalakanth Sep 3, 2004 9:47 AM (in response to chalakanth)

              Thanks for the reply. We'll try using FD_SOCK first.

              However, we use sticky sessions, and we do not use clustered EJBs. Would there still be a lot of replication traffic?

                • Actions
              • 4. Re: How to minimize the traffic in a cluster.
                nickman Sep 3, 2004 1:41 PM (in response to chalakanth)

                The amount of session replication data that moves back and forth will depend on the replication configuration (SYNC/ASYNC/Interval etc.) and the amount of data that gets changed and how offen.

                We deployed an app in a cluster of three servers using SYNC and a legacy architecture that required a large amount of session state in the HTTP session. We were able to significantly reduce the payload size of the transfers by making all the session resident objects externalizable and hand coding the serlialization. It's quite a bit of work but it paid off.

                Based on the invocation statistics in the web console, the ClusteredHttpSession EJB had averages of no more than 1 ms. for any method, so we're happy with it. (And unfortunately/fortunately, we have had a few production occurences where we were able to verify that it worked first hand........:) )

                  • Actions