5 Replies Latest reply on Aug 30, 2005 1:34 PM by Jadoo

    JBoss clustering gives errors on starting up new server

    Jadoo Newbie

      Hope some of you have had this issue earlier and can respond to it.
      I have two scenarios and they are related:
      Scenario 1
      ========
      Deployment description
      ----------------------------
      I have a JBoss installation on two Linux boxes ,say lin-1 ,lin-2 ;both on the same network.
      On boxes lin-1 and lin-2, i have setup four Jboss servers/domains (of type-all i.e. cluster-aware).Four different applications app-A,app-B,app-C,app-D have been installed on each server on lin-1 box.The same deployment is repeated on lin-2 box.
      If you have understood,the idea is that app-A will be clustered between its setup on lin-1 and lin-2, so will app-B,app-C and app-D.So ideally I should have 4 JBoss-clusters of two members each[(lin-1/app-A + lin-2/app-A );
      (lin-1/app-B + lin2-app-B) and so on] deployed between the two boxes lin-1 and lin-2.
      The clustering document at JBoss website says that by default every server that comes up on the network is part of the the cluster called "DefaultPartition". So instead of 4 clusters or 2 members each , I am getting a single cluster with 8 members in it all belonging to the "DefaultPartition".
      Question is : How do i split them to form four JBoss-clusters of two memenrs each.If you are suggesting the way of changing the mcast_addr in cluster_service.xml file, then please detail on the following:
      1. What value can the mcast_addr element take?
      2. How do i ensure multicast is enabled on my network.
      3. Do I leave the PartitionName element 's value to "DefaultPartition" itself.

      Scenario 2
      ========
      Deployment description
      ----------------------------
      The same as the Scenario 1 .In addition , I have a new linux box on the network running a JBoss installation, lin-3 .Now on this JBoss installation , I have setup four new apps app-P ,app-Q,app-R and app-S again running with server config. all-type server(cluster aware).
      So this makes the total no. of JBoss servers 12. Here is a strange occurence.Until the 11th server,all server come up fine.However ,when I start up the 12th server, the startup log goes only until the Mbean config of Cluster and shows the No. of members as "11" and then hangs .The application ear-files on this server does not get deployed and hence the server i assume does not come up fully.The server is running since i can see it when i do a ps -ef |grep
      The same is the result for any new server coming up after the 11th server as shown in the log below which was when i brought up the 13th one.

      -------------------------------------------------------
      GMS: address is mpapp1-d:33766 (additional data: 18 bytes)
      -------------------------------------------------------
      16:28:22,234 INFO [DefaultPartition] Number of cluster members: 13
      16:28:22,234 INFO [DefaultPartition] Other members: 12
      16:28:22,234 INFO [DefaultPartition] Fetching state (will wait for 30000 milliseconds):
      16:28:22,234 INFO [DefaultPartition] New cluster view for partition DefaultPartition: 265 ([10.120.102.91:1099, 10.120.102.91:1199, 10.120.102.94:1099, 10.120.102.94:1199, 10.120.102.97:1399, 10.120.102.97:1099, 10.120.102.103:1099, 10.120.102.103:1199, 10.120.102.106:1099, 10.120.102.106:1199, 10.120.102.91:1299, 10.120.102.97:1899, 10.120.102.91:1399] delta: 0)
      16:28:22,241 INFO [DefaultPartition] I am (null) received membershipChanged event:
      16:28:22,241 INFO [DefaultPartition] Dead members: 0 ([])
      16:28:22,241 INFO [DefaultPartition] New Members : 0 ([])
      16:28:22,241 INFO [DefaultPartition] All Members : 13 ([10.120.102.91:1099, 10.120.102.91:1199, 10.120.102.94:1099, 10.120.102.94:1199, 10.120.102.97:1399, 10.120.102.97:1099, 10.120.102.103:1099, 10.120.102.103:1199, 10.120.102.106:1099, 10.120.102.106:1199, 10.120.102.91:1299, 10.120.102.97:1899, 10.120.102.91:1399])

      Now , I tried to work around in that if i rename the PartitionName element's value in the cluster-service.xml to something other than "${jboss.partition.name:DefaultPartition}" to any string and then restart the server. This time the server comes up ,deploys my ear-file and i can launch the application thorugh the browser.However,towards the end of the server log , I can see messages like :

      15:12:47,924 ERROR [HANamingService] Could not start on port 1100
      java.net.BindException: Address already in use
      at java.net.PlainSocketImpl.socketBind(Native Method)
      at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:331)
      at java.net.ServerSocket.bind(ServerSocket.java:318)
      at java.net.ServerSocket.(ServerSocket.java:185)
      at org.jboss.ha.jndi.DetachedHANamingService.startService(DetachedHANamingService.java:223)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:272)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:222) at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:324)
      .....
      .....
      .....
      followed by


      I Depend On: jboss:service=DefaultPartition
      jboss.cache:service=InvalidationManager

      Depends On Me: javax.naming.NameNotFoundException: DefaultPartition not bound
      ObjectName: jboss:service=HAJNDI
      state: FAILED
      I Depend On: jboss:service=DefaultPartition

      Depends On Me: java.lang.NullPointerException
      ObjectName: jboss:service=HASessionState
      state: FAILED
      I Depend On: jboss:service=DefaultPartition

      Depends On Me: javax.naming.NameNotFoundException: DefaultPartition not bound
      ObjectName: jboss.j2ee:jndiName=clustering/HTTPSession,plugin=pool,service=EJB
      state: CREATED
      I Depend On:
      Depends On Me:
      ObjectName: jboss.j2ee:jndiName=clustering/HTTPSession,service=EJB
      state: FAILED
      I Depend On: jboss:service=DefaultPartition
      jboss:service=invoker,type=jrmp

      At the end of this procedure, my server is up as I mentioned but then I guess there should be a more graceful way of doing this.

      Questions
      -------------
      1.Is there a limit on the no. of nodes that can form a cluster?and if yes, is it 11?
      2. Some of my applications do not need cluster.How can i disable it in the config file.
      3. Once my clusters are all deployed is there a stub I can use to check the cluster info. or is it available on the management console?

      Thanks in advance,
      goks

      P.S:I can provide the logs just in case

        • 1. Re: JBoss clustering gives errors on starting up new server
          David Webb Newbie

          You need to use unique multcast addresses and ports for each application.

          For Jboss (cluster-service.xml) I use

          228.1.2.40:44566 - App A
          228.1.2.41:44567 - App B
          228.1.2.42:44568 - App C

          You also need to take the same consideration for the tc5-cluster-service.xml file if you are using session replication within the Tomcat module, where I use

          230.1.2.40:44576 - App A
          230.1.2.41:44577 - App B
          230.1.2.42:44578 - App C

          The answers to your last 2 questions are:

          1) Based on your log messages your cluster is aware of its nodes, there fore multicast is enabled...although I think it always is, but I could be wrong
          2) The partition name can be left alone given the above changes.

          I use the following currently in a production environment and it works :)

          Enjoy.

          • 2. Re: JBoss clustering gives errors on starting up new server
            Jadoo Newbie

             

            "davewebb" wrote:
            You need to use unique multcast addresses and ports for each application.

            For Jboss (cluster-service.xml) I use

            228.1.2.40:44566 - App A
            228.1.2.41:44567 - App B
            228.1.2.42:44568 - App C

            You also need to take the same consideration for the tc5-cluster-service.xml file if you are using session replication within the Tomcat module, where I use

            230.1.2.40:44576 - App A
            230.1.2.41:44577 - App B
            230.1.2.42:44578 - App C

            The answers to your last 2 questions are:

            1) Based on your log messages your cluster is aware of its nodes, there fore multicast is enabled...although I think it always is, but I could be wrong
            2) The partition name can be left alone given the above changes.

            I use the following currently in a production environment and it works :)

            Enjoy.

            Hi dave
            Thanks for the fix.I tried it and it works fine in that now my server comes up as a different cluster.So I guess the key is to set the mcast IP and port no. to different one.
            But now after going past the cluster , in the bootup log I can see that some other service is failing,"InvocationLayer" to be precise.
            Here is the log:
            13:51:50,392 INFO [C] Bound to JNDI name: queue/C
            13:51:50,393 INFO [D] Bound to JNDI name: queue/D
            13:51:50,395 INFO [ex] Bound to JNDI name: queue/ex
            13:51:50,418 INFO [testTopic] Bound to JNDI name: topic/testTopic
            13:51:50,420 INFO [securedTopic] Bound to JNDI name: topic/securedTopic
            13:51:50,423 INFO [testDurableTopic] Bound to JNDI name: topic/testDurableTopic
            13:51:50,425 INFO [testQueue] Bound to JNDI name: queue/testQueue
            13:51:50,477 ERROR [OILServerILService] Starting failed jboss.mq:service=InvocationLayer,type=OIL
            java.net.BindException: Address already in use
             at java.net.PlainSocketImpl.socketBind(Native Method)
             at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:331)
             at java.net.ServerSocket.bind(ServerSocket.java:318) at java.net.ServerSocket.<init>(ServerSocket.java:185) at javax.net.DefaultServerSocketFactory.createServerSocket(DashoA12275)
             at org.jboss.mq.il.oil.OILServerILService.startService(OILServerILService.java:578)
             at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:272)
             at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:222)
             at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
             at java.lang.reflect.Method.invoke(Method.java:324)
             at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:144)
             at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
             at org.jboss.mx.server.Invocation.invoke(Invocation.java:72) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
             at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:642) at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:891)
             at $Proxy0.start(Unknown Source) at org.jboss.system.ServiceController.start(ServiceController.java:416)
             at org.jboss.system.ServiceController.start(ServiceController.java:438) at org.jboss.system.ServiceController.start(ServiceController.java:438)
             at org.jboss.system.ServiceController.start(ServiceController.java:438) at org.jboss.system.ServiceController.start(ServiceController.java:438)
             at org.jboss.system.ServiceController.start(ServiceController.java:438)
            
             at org.jboss.system.ServiceController.start(ServiceController.java:438)
             at org.jboss.system.ServiceController.start(ServiceController.java:438)
             at org.jboss.system.ServiceController.start(ServiceController.java:438)
            
            and then finally we get
            13:51:53,489 ERROR [URLDeploymentScanner] Incomplete Deployment listing:
            MBeans waiting for other MBeans:
            ObjectName: jboss.mq:service=InvocationLayer,type=OIL
             state: FAILED
             I Depend On: jboss.mq:service=Invoker
            
             Depends On Me: java.net.BindException: Address already in use
            ObjectName: jboss.mq:service=InvocationLayer,type=UIL2
             state: FAILED
             I Depend On: jboss.mq:service=Invoker
            
             Depends On Me: java.net.BindException: Address already in use
            
            MBEANS THAT ARE THE ROOT CAUSE OF THE PROBLEM:
            ObjectName: jboss.mq:service=InvocationLayer,type=OIL
             state: FAILED
             I Depend On: jboss.mq:service=Invoker
            
             Depends On Me: java.net.BindException: Address already in use
            ObjectName: jboss.mq:service=InvocationLayer,type=UIL2
             state: FAILED
             I Depend On: jboss.mq:service=Invoker
            
             Depends On Me: java.net.BindException: Address already in use
            
            13:51:53,616 INFO [Http11Protocol] Starting Coyote HTTP/1.1 on http-0.0.0.0-8380
            13:51:53,714 INFO [ChannelSocket] JK2: ajp13 listening on /0.0.0.0:8309
            13:51:53,718 INFO [JkMain] Jk running ID=0 time=1/28 config=null
            13:51:53,739 INFO [Server] JBoss (MX MicroKernel) [4.0.1sp1 (build: CVSTag=JBoss_4_0_1_SP1 date=200502160314)] Started in 22s:907ms
            

            Although it says "port already in use " , i checked the bindings file and ports do not conflict with other servers.
            However as you can see towards the end of the log, the ear gets deployed and application comes up fine.
            Just wondering why JMS service is failing.
            Since my application does not need to use JMS ,I tried commenting put that particular section in my bindings.xml file.But that seems to have no effect.
            Any guesses what might be causing the JMS service failure?

            Regards


            • 3. Re: JBoss clustering gives errors on starting up new server
              Jerry Gauthier Apprentice

              Not sure why you're having problems with the OIL & UIL2 ports but the bindings are probably the ones defined in ..\deploy-hasingleton\jms\oil-service.xml and ..\deploy-hasingleton\jms\uil2-service.xml.

              • 4. Re: JBoss clustering gives errors on starting up new server
                prasad joshi Newbie

                For the BindException: Address already in use.

                close messengers such as yahoo messenger etc running on ur system and then start your Jboss...




                prasad

                • 5. Re: JBoss clustering gives errors on starting up new server
                  Jadoo Newbie

                   

                  "JerryGauth" wrote:
                  Not sure why you're having problems with the OIL & UIL2 ports but the bindings are probably the ones defined in ..\deploy-hasingleton\jms\oil-service.xml and ..\deploy-hasingleton\jms\uil2-service.xml.


                  Thanks Jerry,it was a conflict on this port .
                  How do i disable a service (eg:JMS ) in my server, just commenting out the JMS section in jboss-service.xml does not seem to work as there might be dependencies.

                  TIA
                  goks