7 Replies Latest reply on Apr 23, 2007 11:35 AM by tnine

    JBOSS CLUSTERING ON VIRTUAL MACHINE (LINUX)

    sonic1

      We have an issue it is a little odd, we are running two instances of JBOSS clustering on two Red Hat Linux instrances, using VM ware (on same machine), this is for testing and replicating a live environment that has two teirs (fw in between)

      We are having trouble starting jboss in a clustered configuration, as it
      needs to have this, the errors seems to relate to not being able to send
      (broadcast?) to UDP sockets. The error is below

      I am pretty sure it isn't the firewall, but some permissions error. We can run unclustered, but JBOSS wont start in clustered mode


      PS this is the error we recieve
      17:59:54,869 ERROR [UDP] exception sending msg: java.lang.Exception:
      dest=/229.1.2.3:45551 (83 bytes)
      java.io.IOException: Operation not permitted
      at java.net.PlainDatagramSocketImpl.send(Native Method)
      at java.net.DatagramSocket.send(DatagramSocket.java:612)
      at org.jgroups.protocols.UDP._send(UDP.java:316)
      at org.jgroups.protocols.UDP.sendToAllMembers(UDP.java:286)

        • 1. Re: JBOSS CLUSTERING ON VIRTUAL MACHINE (LINUX)
          genman

          What are the ports you have configured? It's coming from the JVM.

          • 2. Re: JBOSS CLUSTERING ON VIRTUAL MACHINE (LINUX)
            sonic1

            The ports are the default ones, just using 45531 or so. The JVM is showing the error but only happens in a Virtual Machine instance so seems lower

            • 3. Re: JBOSS CLUSTERING ON VIRTUAL MACHINE (LINUX)
              dimitris

              AFAIR, there is some problem with jgroups/UCD multicasting when run inside VMWare.

              • 4. Re: JBOSS CLUSTERING ON VIRTUAL MACHINE (LINUX)
                sonic1

                thanks is there any more details or links about this

                • 5. Re: JBOSS CLUSTERING ON VIRTUAL MACHINE (LINUX)
                  tnine

                  I'm also having this problem. I'm running RHEL 4 in a VMware instance. I have 2 nodes that are running in a virtual network on my PC. I'm letting the nodes get their IP's via DHCP. I have the following common configuration

                  JDK 1.5.0_11-b03
                  JBoss 4.0.5
                  Server Config: all

                  IPs: 192.168.109.128, 192.168.109.130.

                  The nodes can communicate, I can ping back and forth, as well as SSH between nodes. I assumed my problems were due to a fire walled port, so I disabled all fire walling by using "system-config-securitylevel-tui" and setting it to disabled. However, this has not resolved the issue. Is this a problem with vmware virtual networking and NAT? I've also tried bridged, and I have not had any success. This is a big stumbling block for us as we're migrating to a VMWare infrastructure on blade servers. Are there any diagnostic utilities I can run to help find the issue?

                  Thanks,
                  Todd

                  • 6. Re: JBOSS CLUSTERING ON VIRTUAL MACHINE (LINUX)
                    tnine

                    Ok, I have made some changes. I've changed both nodes to have static IPs. Node 1 has IP 192.168.109.10 and node 2 has IP 192.168.109.11

                    From node 1 if I execute "ping -L 224.0.0.1" I receive a response from 192.168.109.11 (node2) and vice versa. At the network level my multicast is working, but my response time varies from .02 ms to 4 ms. Could this be a cause of my problems?

                    Thanks,
                    Todd

                    • 7. Re: JBOSS CLUSTERING ON VIRTUAL MACHINE (LINUX)
                      tnine

                      Bump......

                      I have tried all of the following and I'm thorougly stumped. Any help would be appreciated.

                      1. Make sure that I have multicast working. I followed the instructions here

                      http://kbase.redhat.com/faq/FAQ_45_4865.shtm

                      using "ping 224.0.0.1" I receive the following.

                      64 bytes from 192.168.109.11: icmp_seq=0 ttl=64 time=0.966 ms
                      64 bytes from 192.168.109.10: icmp_seq=0 ttl=64 time=4.78 ms (DUP!)
                      64 bytes from 192.168.109.11: icmp_seq=1 ttl=64 time=1.45 ms
                      64 bytes from 192.168.109.10: icmp_seq=1 ttl=64 time=0.882 ms (DUP!)
                      


                      I have the following from "iptables --list"
                      iptables --list
                      Chain INPUT (policy ACCEPT)
                      target prot opt source destination
                      
                      Chain FORWARD (policy ACCEPT)
                      target prot opt source destination
                      
                      Chain OUTPUT (policy ACCEPT)
                      target prot opt source destination
                      


                      So at my network level I should be good. I don't have any firewall, and multicast is enabled. I then created the following text.xml and placed it on both node 1 and node 2.

                      <Config>
                       <UDP mcast_addr="228.1.2.3" mcast_port="45566"
                       ip_ttl="8" ip_mcast="false"
                       mcast_recv_buf_size="2000000" mcast_send_buf_size="640000"
                       ucast_recv_buf_size="2000000" ucast_send_buf_size="640000"
                       loopback="true"
                       />
                       <PING timeout="2000" num_initial_members="3"
                       up_thread="true" down_thread="true"/>
                       <MERGE2 min_interval="10000" max_interval="20000"/>
                       <FD_SOCK down_thread="false" up_thread="false"/>
                       <FD shun="true" up_thread="true" down_thread="true"
                       timeout="10000" max_tries="5"/>
                       <VERIFY_SUSPECT timeout="3000" num_msgs="3"
                       up_thread="true" down_thread="true"/>
                       <pbcast.NAKACK gc_lag="50" retransmit_timeout="300,600,1200,2400,4800"
                       max_xmit_size="8192"
                       up_thread="true" down_thread="true"/>
                       <UNICAST timeout="300,600,1200,2400,4800" window_size="100" min_threshold="10"
                       down_thread="true"/>
                       <pbcast.STABLE desired_avg_gossip="20000" max_bytes="400000"
                       up_thread="true" down_thread="true"/>
                       <FRAG frag_size="8192"
                       down_thread="true" up_thread="true"/>
                       <pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
                       shun="true" print_local_addr="true"/>
                       <pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>
                       </Config>
                      



                      I then start each node with the following command

                      java -Djava.net.preferIPv4Stack=true -cp lib/concurrent.jar:server/all/lib/jgroups.jar:server/all/lib/commons-logging.jar org.jgroups.demos.ViewD
                      emo -props test.xml
                      


                      I get the following output for node 1
                      Apr 23, 2007 11:31:12 AM org.jgroups.protocols.UDP createSockets
                      INFO: unicast sockets will use interface 192.168.109.10
                      Apr 23, 2007 11:31:12 AM org.jgroups.protocols.UDP createSockets
                      INFO: socket information:
                      local_addr=prodjboss01:32779, mcast_addr=null, bind_addr=/192.168.109.10, ttl=8
                      sock: bound to 192.168.109.10:32779, receive buffer size=131071, send buffer size=131071
                      
                      -------------------------------------------------------
                      GMS: address is prodjboss01:32779
                      -------------------------------------------------------
                      ** New view: [prodjboss01:32779|0] [prodjboss01:32779]
                      


                      This is node 2

                      Apr 23, 2007 11:31:46 AM org.jgroups.protocols.UDP createSockets
                      INFO: unicast sockets will use interface 192.168.109.11
                      Apr 23, 2007 11:31:46 AM org.jgroups.protocols.UDP createSockets
                      INFO: socket information:
                      local_addr=prodjboss02:32784, mcast_addr=null, bind_addr=/192.168.109.11, ttl=8
                      sock: bound to 192.168.109.11:32784, receive buffer size=131071, send buffer size=131071
                      
                      -------------------------------------------------------
                      GMS: address is prodjboss02:32784
                      -------------------------------------------------------
                      ** New view: [prodjboss02:32784|0] [prodjboss02:32784]
                      


                      I'm utterly stumped. I've followed all of the diagnostics I can think of, and I still am not having any luck connecting. The diagnostics tell me that the systems aren't communicating, but they don't tell me a starting point to work with. I've tried to telnet to the port that it says its listening on, but I can't connect to the socket. Can anyone give me a hand?

                      Thanks,
                      Todd