JBoss 6.1 cluster with TCP

[kbielecki]

I've read:

http://community.jboss.org/thread/164797 

http://community.jboss.org/thread/174934?wwparam=1324543958

and do everythink, but my jboss instances don't see each other.

The same on each server:

<stack name="tcp"

        <config>

            <TCP

                 singleton_name="tcp"

                 bind_port="${jboss.jgroups.tcp.tcp_port:7600}"

                 tcp_nodelay="true"

                 loopback="true"

                 recv_buf_size="20000000"

                 send_buf_size="640000"

                 discard_incompatible_packets="true"

                 max_bundle_size="64000"

                 max_bundle_timeout="30"

                 use_incoming_packet_handler="true"

                 enable_bundling="false"

                 use_send_queues="false"

                 sock_conn_timeout="300"

                 skip_suspected_members="true"

                 timer.num_threads="12"

                 enable_diagnostics="${jboss.jgroups.enable_diagnostics:true}"

                 diagnostics_addr="${jboss.jgroups.diagnostics_addr:224.0.75.75}"

                 diagnostics_port="${jboss.jgroups.diagnostics_port:7500}"

                   thread_pool.enabled="true"

                   thread_pool.min_threads="20"

                   thread_pool.max_threads="200"

                   thread_pool.keep_alive_time="5000"

                   thread_pool.queue_enabled="true"

                   thread_pool.queue_max_size="1000"

                   thread_pool.rejection_policy="discard"

                 oob_thread_pool.enabled="true"

                   oob_thread_pool.min_threads="20"

                   oob_thread_pool.max_threads="200"

                   oob_thread_pool.keep_alive_time="1000"

                   oob_thread_pool.queue_enabled="false"

                   oob_thread_pool.rejection_policy="discard"/>

              <!-- Alternative 1: multicast-based automatic discovery. -->  

            <!--<MPING timeout="3000"

                   num_initial_members="3"

                   mcast_addr="${jboss.jgroups.tcp.mping_mcast_addr,jboss.partition.udpGroup:230.11.11.11}"

                   mcast_port="${jboss.jgroups.tcp.mping_mcast_port:45700}"

                   ip_ttl="${jgroups.udp.ip_ttl:2}"/>           

             Alternative 2: non multicast-based replacement for MPING. Requires a static configuration

                 of *all* possible cluster members.-->

            <TCPPING timeout="3000"

                     initial_hosts="${jboss.jgroups.tcp.tcpping.initial_hosts:server1[7600],server2[7600]}"

                     port_range="1"

                     num_initial_members="3"/>

            <MERGE2 max_interval="100000" min_interval="20000"/>

            <FD_SOCK start_port="${jboss.jgroups.tcp.fd_sock_port:57600}"/>

            <FD timeout="6000" max_tries="5"/>

            <VERIFY_SUSPECT timeout="1500"/>

            <BARRIER/>

            <pbcast.NAKACK use_mcast_xmit="false" gc_lag="0"

                           retransmit_timeout="300,600,1200,2400,4800"

                           discard_delivered_msgs="true"/>

            <UNICAST timeout="300,600,1200,2400,3600"/>

            <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"

                           max_bytes="400000"/>           

            <VIEW_SYNC avg_send_interval="10000"/>              

            <pbcast.GMS print_local_addr="true" join_timeout="3000"

                        view_bundling="true"

                        view_ack_collection_timeout="5000"

                        resume_task_timeout="7500"/>

            <UFC max_credits="2000000" ignore_synchronous_response="true"/>

            <MFC max_credits="2000000" ignore_synchronous_response="true"/>

            <FRAG2 frag_size="60000"/>

            <pbcast.STREAMING_STATE_TRANSFER/>

            <!--pbcast.STATE_TRANSFER/-->

            <pbcast.FLUSH timeout="0" start_flush_timeout="10000"/>

        </config>

    </stack>

and I use to run:

./run.sh -c all -b server1 -g newPart -Djboss.default.jgroups.stack=tcp  -Djboss.messaging.ServerPeerID=2 -Djboss.service.binding.set=ports-01 -Djgroups.bind_addr=server1

and second:

./run.sh -c all -b server2 -g newPart -Djboss.default.jgroups.stack=tcp -Djboss.messaging.ServerPeerID=3 -Djboss.service.binding.set=ports-01 -Djgroups.bind_addr=server2

I also have i consol log:

14:27:09,044 INFO  [ClusterPartition] Initializing partition newPart

14:27:09,248 INFO  [TransactionManagerFactory] Using a batchMode transaction manager

14:27:10,366 INFO  [JGroupsTransport] Starting JGroups Channel

14:27:10,422 INFO  [JChannel] JGroups version: 2.11.0.GA

14:27:10,945 WARN  [Configurator] TCP property skip_suspected_members was deprecated and is ignored

14:27:10,946 WARN  [Configurator] TCP property use_incoming_packet_handler was deprecated and is ignored

14:27:10,947 WARN  [Configurator] TCP property timer.num_threads was deprecated and is ignored

What I've done wrong?

[rhusar]

You need to pass correct property value for: jboss.jgroups.tcp.tcpping.initial_hosts

(You see, you are shifting ports, but you are discovering members at two ports 7600 both open at the same host, which is not possible).

[kbielecki]

Yes, I have to change this, but I think it is not enough. Cluster don't want to start at jboss 6.0 or 6.1. Instances don't see each other. I tried at jboss 5.1 and everythink was working.

Part log from start jbosss 6.0:

For me its strange:

11:30:43,998 INFO  [STDOUT] GMS: address=server1:1199, cluster=newPart-HAPartition, physical address=server1:7700

11:30:43,999 INFO  [STDOUT] -------------------------------------------------------------------

11:30:47,123 INFO  [JGroupsTransport] Received new cluster view: [server1:1199|0] [server1:1199]

11:30:47,134 INFO  [JGroupsTransport] Cache local address is server1:1199, physical addresses are [server1:7700]

I set 7600 and 7601...

Full log:

bash-3.00$ ./run.sh -c all -b server1 -g newPart -Djboss.default.jgroups.stack=tcp  -Djboss.messaging.ServerPeerID=2 -Djboss.service.binding.set=ports-01 -Djgroups.bind_addr=server1

=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /export/home/vo/jboss/jboss-6.0.0.Final

  JAVA: /export/home/vo/Instal/jdk1.6.0_21/bin/java

  JAVA_OPTS: -server -Xms128m -Xmx512m -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dprogram.name=run.sh -Djava.library.path=/export/home/vo/jboss/jboss-6.0.0.Final/bin/native/lib

  CLASSPATH: /export/home/vo/jboss/jboss-6.0.0.Final/bin/run.jar

=========================================================================

11:29:15,167 INFO  [AbstractJBossASServerBase] Server Configuration:

        JBOSS_HOME URL: file:/export/home/vo/jboss/jboss-6.0.0.Final/

        Bootstrap: $JBOSS_HOME/server/all/conf/bootstrap.xml

        Common Base: $JBOSS_HOME/common/

        Common Library: $JBOSS_HOME/common/lib/

        Server Name: all

        Server Base: $JBOSS_HOME/server/

        Server Library: $JBOSS_HOME/server/all/lib/

        Server Config: $JBOSS_HOME/server/all/conf/

        Server Home: $JBOSS_HOME/server/all/

        Server Data: $JBOSS_HOME/server/all/data/

        Server Log: $JBOSS_HOME/server/all/log/

        Server Temp: $JBOSS_HOME/server/all/tmp/

11:29:15,183 INFO  [AbstractServer] Starting: JBossAS [6.0.0.Final "Neo"]

11:29:23,895 INFO  [ServerInfo] Java version: 1.6.0_21,Sun Microsystems Inc.

11:29:23,896 INFO  [ServerInfo] Java Runtime: Java(TM) SE Runtime Environment (build 1.6.0_21-b06)

11:29:23,896 INFO  [ServerInfo] Java VM: Java HotSpot(TM) Server VM 17.0-b16,Sun Microsystems Inc.

11:29:23,897 INFO  [ServerInfo] OS-System: SunOS 5.10,sparc

11:29:23,899 INFO  [ServerInfo] VM arguments: -Xms128m -Xmx512m -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dprogram.name=run.sh -Djava.library.path=/export/home/vo/jboss/jboss-6.0.0.Final/bin/native/lib:/export/home/vo/Apache2_fs/lib:/export/home/vo/Apache2_fs/lib/iconv/lib:/usr/lib:/pkg/lib:/usr/sfw/lib: -Djava.endorsed.dirs=/export/home/vo/jboss/jboss-6.0.0.Final/lib/endorsed

11:29:24,098 INFO  [JMXKernel] Legacy JMX core initialized

11:29:50,770 INFO  [AbstractServerConfig] JBoss Web Services - Stack CXF Server 3.4.1.GA

11:29:54,745 INFO  [JSFImplManagementDeployer] Initialized 3 JSF configurations: [Mojarra-1.2, MyFaces-2.0, Mojarra-2.0]

11:30:10,021 WARNING [FileConfigurationParser] AIO wasn't located on this platform, it will fall back to using pure Java NIO. If your platform is Linux, install LibAIO to enable the AIO journal

11:30:23,442 INFO  [SnmpAgentService] SNMP agent going active

11:30:26,334 WARNING [FileConfigurationParser] AIO wasn't located on this platform, it will fall back to using pure Java NIO. If your platform is Linux, install LibAIO to enable the AIO journal

11:30:27,067 WARN  [config] jacorb.home unset! Will use '.'

11:30:27,068 WARN  [config] File ./jacorb.properties for configuration jacorb not found

11:30:27,072 INFO  [config] configuration jacorb loaded from classpath

11:30:27,334 INFO  [orb] Property "jacorb.hashtable_class" is set to: java.util.Hashtable

11:30:27,413 WARN  [codeset] Warning - unknown codeset (ASCII) - defaulting to ISO-8859-1

11:30:27,443 WARN  [config] jacorb.home unset! Will use '.'

11:30:27,444 WARN  [config] File ./jacorb.properties for configuration jacorb not found

11:30:27,448 INFO  [config] configuration jacorb loaded from classpath

11:30:27,450 INFO  [singleton] created ORBSingleton

11:30:27,841 INFO  [interceptors] InterceptorManager started with 2 Server Interceptors, 2 Client Interceptors and 4 IOR Interceptors

11:30:28,327 INFO  [orb] ORB run

11:30:28,954 INFO  [JMXConnector] starting JMXConnector on host server1:1190

11:30:29,673 INFO  [MailService] Mail Service bound to java:/Mail

11:30:29,739 INFO  [poa] oid:

72 6F 6F 74                                                 root

object is activated

11:30:29,960 INFO  [CorbaNamingService] CORBA Naming Started

11:30:32,408 INFO  [DefaultCacheContainerRegistry] Binding hibernate cache container to java:CacheManager/entity

11:30:33,793 INFO  [UnifiedInvokerHA] Service name is jboss:service=invoker,type=unifiedha

11:30:36,385 INFO  [HornetQServerImpl] live server is starting..

11:30:36,693 INFO  [JournalStorageManager] Using NIO Journal

11:30:36,789 WARNING [HornetQServerImpl] Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this.

11:30:40,082 INFO  [NettyAcceptor] Started Netty Acceptor version 3.2.1.Final-r2319 server1:5545 for CORE protocol

11:30:40,119 INFO  [NettyAcceptor] Started Netty Acceptor version 3.2.1.Final-r2319 server1:5555 for CORE protocol

11:30:40,141 INFO  [HornetQServerImpl] HornetQ Server version 2.1.2.Final (Colmeia, 120) started

11:30:40,424 INFO  [WebService] Using RMI server codebase: http://server1:8183/

11:30:40,512 INFO  [ClusterPartition] Initializing partition newPart

11:30:40,766 INFO  [TransactionManagerFactory] Using a batchMode transaction manager

11:30:42,067 INFO  [JGroupsTransport] Starting JGroups Channel

11:30:42,133 INFO  [JChannel] JGroups version: 2.11.0.GA

11:30:42,767 WARN  [Configurator] TCP property skip_suspected_members was deprecated and is ignored

11:30:42,768 WARN  [Configurator] TCP property use_incoming_packet_handler was deprecated and is ignored

11:30:42,769 WARN  [Configurator] TCP property timer.num_threads was deprecated and is ignored

11:30:43,996 INFO  [STDOUT]

11:30:43,997 INFO  [STDOUT] -------------------------------------------------------------------

11:30:43,998 INFO  [STDOUT] GMS: address=server1:1199, cluster=newPart-HAPartition, physical address=server1:7700

11:30:43,999 INFO  [STDOUT] -------------------------------------------------------------------

11:30:47,123 INFO  [JGroupsTransport] Received new cluster view: [server1:1199|0] [server1:1199]

11:30:47,134 INFO  [JGroupsTransport] Cache local address is server1:1199, physical addresses are [server1:7700]

11:30:47,136 INFO  [GlobalComponentRegistry] Infinispan version: Infinispan 'Ursus' 4.2.0.FINAL

11:30:47,304 INFO  [ComponentsJmxRegistration] Could not register object with name: org.infinispan:type=Cache,name="distributed-state(repl_sync)",manager="ha-partition",component=Cache

11:30:47,307 INFO  [CacheJmxRegistration] MBeans were successfully registered to the platform mbean server.

11:30:47,308 INFO  [ComponentRegistry] Infinispan version: Infinispan 'Ursus' 4.2.0.FINAL

11:30:47,325 INFO  [DefaultCacheContainerFactory] Started "distributed-state" cache from "ha-partition" container

11:30:47,395 INFO  [newPart] Number of cluster members: 1

11:30:47,398 INFO  [newPart] Fetching initial service state (will wait for 30000 milliseconds for each service):

11:30:48,200 INFO  [HANamingService] Started HAJNDI bootstrap; jnpPort=1200, backlog=50, bindAddress=/server1

11:30:48,246 INFO  [DetachedHANamingService$AutomaticDiscovery] Listening on /server1:1102, group=230.0.0.4, HA-JNDI address=server1:1200

11:30:48,260 INFO  [TransactionManagerFactory] Using a batchMode transaction manager

11:30:48,333 INFO  [ComponentsJmxRegistration] Could not register object with name: org.infinispan:type=Cache,name="distributed-tree(repl_sync)",manager="ha-partition",component=Cache

11:30:48,336 INFO  [CacheJmxRegistration] MBeans were successfully registered to the platform mbean server.

11:30:48,336 INFO  [ComponentRegistry] Infinispan version: Infinispan 'Ursus' 4.2.0.FINAL

11:30:48,337 INFO  [DefaultCacheContainerFactory] Started "distributed-tree" cache from "ha-partition" container

11:30:52,137 INFO  [jbossatx] ARJUNA-32010 JBossTS Recovery Service (tag: JBOSSTS_4_14_0_Final) - JBoss Inc.

11:30:52,162 INFO  [arjuna] ARJUNA-12324 Start RecoveryActivators

11:30:52,251 INFO  [arjuna] ARJUNA-12296 ExpiredEntryMonitor running at Fri, 23 Dec 2011 11:30:52

11:30:52,569 INFO  [arjuna] ARJUNA-12310 Recovery manager listening on endpoint server1:4812

11:30:52,570 INFO  [arjuna] ARJUNA-12344 RecoveryManagerImple is ready on port 4812

11:30:52,575 INFO  [jbossatx] ARJUNA-32013 Starting transaction recovery manager

11:30:52,626 INFO  [arjuna] ARJUNA-12163 Starting service com.arjuna.ats.arjuna.recovery.ActionStatusService on port 4813

11:30:52,664 INFO  [arjuna] ARJUNA-12337 TransactionStatusManagerItem host: server1 port: 4813

11:30:52,682 INFO  [arjuna] ARJUNA-12170 TransactionStatusManager started on port 4813 and host server1 with service com.arjuna.ats.arjuna.recovery.ActionStatusService

11:30:52,832 INFO  [jbossatx] ARJUNA-32017 JBossTS Transaction Service (JTA version - tag: JBOSSTS_4_14_0_Final) - JBoss Inc.

11:30:53,013 INFO  [arjuna] ARJUNA-12202 registering bean jboss.jta:type=ObjectStore.

11:30:53,942 INFO  [AprLifecycleListener] The Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /export/home/vo/jboss/jboss-6.0.0.Final/bin/native/lib:/export/home/vo/Apache2_fs/lib:/export/home/vo/Apache2_fs/lib/iconv/lib:/usr/lib:/pkg/lib:/usr/sfw/lib:

11:30:54,632 INFO  [HAModClusterService$ClusteredModClusterService] Initializing mod_cluster 1.1.0.Final

11:30:54,696 INFO  [TomcatDeployment] deploy, ctxPath=/invoker

11:30:56,272 INFO  [RARDeployment] Required license terms exist, view vfs:/export/home/vo/jboss/jboss-6.0.0.Final/server/all/deploy/jboss-local-jdbc.rar/META-INF/ra.xml

11:30:56,337 INFO  [RARDeployment] Required license terms exist, view vfs:/export/home/vo/jboss/jboss-6.0.0.Final/server/all/deploy/jboss-xa-jdbc.rar/META-INF/ra.xml

11:30:56,389 INFO  [RARDeployment] Required license terms exist, view vfs:/export/home/vo/jboss/jboss-6.0.0.Final/server/all/deploy/jms-ra.rar/META-INF/ra.xml

11:30:56,463 INFO  [HornetQResourceAdapter] HornetQ resource adaptor started

11:30:56,504 INFO  [RARDeployment] Required license terms exist, view vfs:/export/home/vo/jboss/jboss-6.0.0.Final/server/all/deploy/mail-ra.rar/META-INF/ra.xml

11:30:56,568 INFO  [RARDeployment] Required license terms exist, view vfs:/export/home/vo/jboss/jboss-6.0.0.Final/server/all/deploy/quartz-ra.rar/META-INF/ra.xml

11:30:56,878 INFO  [SimpleThreadPool] Job execution threads will use class loader of thread: Thread-2

11:30:57,076 INFO  [SchedulerSignalerImpl] Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl

11:30:57,077 INFO  [QuartzScheduler] Quartz Scheduler v.1.8.3 created.

11:30:57,089 INFO  [RAMJobStore] RAMJobStore initialized.

11:30:57,098 INFO  [QuartzScheduler] Scheduler meta-data: Quartz Scheduler (v1.8.3) 'JBossQuartzScheduler' with instanceId 'NON_CLUSTERED'

  Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.

  NOT STARTED.

  Currently in standby mode.

  Number of jobs executed: 0

  Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.

  Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.

11:30:57,099 INFO  [StdSchedulerFactory] Quartz scheduler 'JBossQuartzScheduler' initialized from an externally opened InputStream.

11:30:57,100 INFO  [StdSchedulerFactory] Quartz scheduler version: 1.8.3

11:30:57,101 INFO  [QuartzScheduler] Scheduler JBossQuartzScheduler_$_NON_CLUSTERED started.

11:30:58,847 INFO  [ConnectionFactoryBindingService] Bound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=DefaultDS' to JNDI name 'java:DefaultDS'

11:30:59,692 INFO  [TomcatDeployment] deploy, ctxPath=/juddi

11:30:59,784 INFO  [RegistryServlet] Loading jUDDI configuration.

11:30:59,811 INFO  [RegistryServlet] Resources loaded from: /WEB-INF/juddi.properties

11:30:59,812 INFO  [RegistryServlet] Initializing jUDDI components.

11:31:00,742 INFO  [ConnectionFactoryBindingService] Bound ConnectionManager 'jboss.jca:service=ConnectionFactoryBinding,name=JmsXA' to JNDI name 'java:JmsXA'

11:31:01,391 INFO  [xnio] XNIO Version 2.1.0.CR2

11:31:01,438 INFO  [nio] XNIO NIO Implementation Version 2.1.0.CR2

11:31:02,287 INFO  [remoting] JBoss Remoting version 3.1.0.Beta2

11:31:02,683 INFO  [TomcatDeployment] deploy, ctxPath=/

11:31:02,854 INFO  [service] Removing bootstrap log handlers

11:31:03,534 INFO  [org.apache.coyote.http11.Http11Protocol] Starting Coyote HTTP/1.1 on http-server1-8180

11:31:03,548 INFO  [org.apache.coyote.ajp.AjpProtocol] Starting Coyote AJP/1.3 on ajp-server1-8109

11:31:03,562 INFO  [org.jboss.bootstrap.impl.base.server.AbstractServer] JBossAS [6.0.0.Final "Neo"] Started in 1m:48s:356ms

[rhusar]

Hm, if it works in AS5 and not in AS6 it might as well be a regression or just a change in how stuff is configured.

Nevertheless, I dont like the concept of port shifting (-Djboss.service.binding.set=ports-01) are you still using it? If yes can you try to bind to different hosts (ie. localhost and your ip adress on the network for instance)?

Ideally you could switch to AS7 ;-)