clustering/JMS problem among machines not using same partiti
Hello, all. We are using jboss-4.2.3.GA on some machines with CentOS 4.4 and some with CentOS 5. For development purposes we do not like to cluster so we all change our mcast_port in cluster-service.xml to something unique. That was fine for a time. Then we realized that even though JBoss was not clustering, JMS was. I read http://jboss.org/community/docs/DOC-12460 and consequently started using unique partition names and jboss.partition.udpGroup addresses. This LOOKS like it is working, but does not.
Here is the start of my JBoss log on a machine called "cars" at 150.102.65.10:
================================================================================ JBoss Bootstrap Environment JBOSS_HOME: /usr/jboss-4.2.3.GA JAVA: JAVA_OPTS: -server -Xmx512m -XX:PermSize=96m -XX:MaxPermSize=128m -Dprogram.name=jboss CLASSPATH: /usr/jboss-4.2.3.GA/bin/run.jar:/usr/java/jdk1.5.0_02/lib/tools.jar CMD_START: /usr/java/jdk1.5.0_02/bin/java -server -Xmx512m -XX:PermSize=96m -XX:MaxPermSize=128m -Dprogram.name=jboss -classpath /usr/jboss-4.2.3.GA/bin/run.jar:/usr/java/jdk1.5.0_02/lib/tools.jar org.jboss.Main -c appServer -b cars -Djava.security.manager=java.lang.SecurityManager -Djava.security.policy=/usr/jboss-4.2.3.GA/server/appServer/conf/server.policy -Djava.rmi.server.codebase="file:/usr/jboss-4.2.3.GA/server/appServer/lib/KDCSservices.jar file:/usr/jboss-4.2.3.GA/server/appServer/lib/kdcsclient.jar" -Djava.rmi.server.hostname=cars -Dkdcs.hostname=cars -Djboss.partition.name=workstation_32_development_Rochester -Djboss.partition.udpGroup=228.1.2.10 -Djboss.platform.mbeanserver -Djava.endorsed.dirs=/usr/jboss-4.2.3.GA/lib/endorsed ================================================================================ 10:11:13,954 INFO [JChannel] JGroups version: 2.4.1 SP-4 10:11:14,661 INFO [STDOUT] ------------------------------------------------------- GMS: address is 150.102.65.10:52299 ------------------------------------------------------- 10:11:16,684 INFO [TreeCache] viewAccepted(): [150.102.65.10:52299|0] [150.102.65.10:52299] 10:11:16,709 INFO [TreeCache] TreeCache local address is 150.102.65.10:52299 10:11:16,709 INFO [TreeCache] State could not be retrieved (we are the first member in group) 10:11:16,709 INFO [TreeCache] parseConfig(): PojoCacheConfig is empty 10:11:20,919 INFO [NativeServerConfig] JBoss Web Services - Native 10:11:20,919 INFO [NativeServerConfig] jbossws-3.0.1-native-2.0.4.GA (build=200803312044) 10:11:22,094 INFO [SnmpAgentService] SNMP agent going active 10:11:22,610 INFO [JChannel] JGroups version: 2.4.1 SP-4 10:11:22,759 INFO [workstation_32_development_Rochester] Initializing 10:11:22,809 INFO [STDOUT] ------------------------------------------------------- GMS: address is 150.102.65.10:52303 ------------------------------------------------------- 10:11:24,817 INFO [workstation_32_development_Rochester] Number of cluster members: 1 10:11:24,818 INFO [workstation_32_development_Rochester] Other members: 0 10:11:24,818 INFO [workstation_32_development_Rochester] Fetching state (will wait for 30000 milliseconds): 10:11:24,818 INFO [workstation_32_development_Rochester] State could not be retrieved (we are the first member in group) 10:11:24,862 INFO [HANamingService] Started ha-jndi bootstrap jnpPort=1100, backlog=50, bindAddress=cars/150.102.65.10 10:11:24,868 INFO [DetachedHANamingService$AutomaticDiscovery] Listening on cars/150.102.65.10:1102, group=228.1.2.10, HA-JNDI address=150.102.65.10:1100 10:11:26,171 INFO [TreeCache] No transaction manager lookup class has been defined. Transactions cannot be used 10:11:26,336 INFO [JChannel] JGroups version: 2.4.1 SP-4 10:11:26,481 INFO [STDOUT] ------------------------------------------------------- GMS: address is 150.102.65.10:52309 ------------------------------------------------------- 10:11:28,485 INFO [TreeCache] viewAccepted(): [150.102.65.10:52309|0] [150.102.65.10:52309] 10:11:28,486 INFO [TreeCache] TreeCache local address is 150.102.65.10:52309 10:11:28,640 INFO [JChannel] JGroups version: 2.4.1 SP-4 10:11:28,790 INFO [STDOUT] ------------------------------------------------------- GMS: address is 150.102.65.10:52312 ------------------------------------------------------- 10:11:30,794 INFO [TreeCache] viewAccepted(): [150.102.65.10:52312|0] [150.102.65.10:52312] 10:11:30,796 INFO [TreeCache] TreeCache local address is 150.102.65.10:52312
Looks great, right? cars is the only machine in the cluster with the partition name workstation_32_development_Rochester. Before when we all were using DefaultPartition, and only had unique mcast_ports, many other ip addresses showed up in the list following "[TreeCache] viewAccepted():"
Then, for unknown reasons it tries to access another machine called "wizard":
10:12:52,894 INFO [QuartzJob] startSingleton(): Start QuartzJob Singleton 10:12:57,500 ERROR [QuartzJob] *********************Exception Thrown********************* 10:12:57,501 ERROR [QuartzJob] createJobAndTrigger(): null 10:12:57,501 ERROR [QuartzJob] *********************End of Exception********************* 10:12:57,501 ERROR [STDERR] javax.naming.CommunicationException [Root exception is java.rmi.RemoteException: Service unavailable.; nested exception is: java.rmi.ConnectIOException: Exception creating connection to: wizard; nested exception is: java.net.NoRouteToHostException: No route to host] 10:12:57,502 ERROR [STDERR] at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:780) 10:12:57,502 ERROR [STDERR] at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627) 10:12:57,502 ERROR [STDERR] at javax.naming.InitialContext.lookup(InitialContext.java:351) 10:12:57,502 ERROR [STDERR] at kdcs.timers.QuartzJob.jobInvoiceHandlerJob(QuartzJob.java:797) 10:12:57,502 ERROR [STDERR] at kdcs.timers.QuartzJob.createJobAndTrigger(QuartzJob.java:849) 10:12:57,502 ERROR [STDERR] at kdcs.timers.QuartzJob.startSingleton(QuartzJob.java:390) 10:12:57,502 ERROR [STDERR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 10:12:57,502 ERROR [STDERR] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 10:12:57,503 ERROR [STDERR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 10:12:57,503 ERROR [STDERR] at java.lang.reflect.Method.invoke(Method.java:585) 10:12:57,503 ERROR [STDERR] at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) 10:12:57,503 ERROR [STDERR] at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) 10:12:57,503 ERROR [STDERR] at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) 10:12:57,503 ERROR [STDERR] at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) 10:12:57,503 ERROR [STDERR] at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) 10:12:57,503 ERROR [STDERR] at org.jboss.ha.singleton.HASingletonController.invokeSingletonMBeanMethod(HASingletonController.java:207) 10:12:57,503 ERROR [STDERR] at org.jboss.ha.singleton.HASingletonController.startSingleton(HASingletonController.java:144) 10:12:57,503 ERROR [STDERR] at org.jboss.ha.singleton.HASingletonSupport.startNewMaster(HASingletonSupport.java:272) 10:12:57,503 ERROR [STDERR] at org.jboss.ha.singleton.HASingletonSupport.makeThisNodeMaster(HASingletonSupport.java:254) 10:12:57,503 ERROR [STDERR] at org.jboss.ha.singleton.HASingletonSupport.partitionTopologyChanged(HASingletonSupport.java:196) 10:12:57,503 ERROR [STDERR] at org.jboss.ha.jmx.HAServiceMBeanSupport$1.replicantsChanged(HAServiceMBeanSupport.java:247) 10:12:57,503 ERROR [STDERR] at org.jboss.ha.framework.server.DistributedReplicantManagerImpl.notifyKeyListeners(DistributedReplicantManagerImpl.java:846) 10:12:57,504 ERROR [STDERR] at org.jboss.ha.framework.server.DistributedReplicantManagerImpl.add(DistributedReplicantManagerImpl.java:409) 10:12:57,504 ERROR [STDERR] at org.jboss.ha.jmx.HAServiceMBeanSupport.registerDRMListener(HAServiceMBeanSupport.java:255) 10:12:57,504 ERROR [STDERR] at org.jboss.ha.jmx.HAServiceMBeanSupport.startService(HAServiceMBeanSupport.java:177) 10:12:57,504 ERROR [STDERR] at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289) 10:12:57,504 ERROR [STDERR] at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245) 10:12:57,504 ERROR [STDERR] at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) 10:12:57,504 ERROR [STDERR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 10:12:57,504 ERROR [STDERR] at java.lang.reflect.Method.invoke(Method.java:585) 10:12:57,504 ERROR [STDERR] at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) 10:12:57,504 ERROR [STDERR] at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) 10:12:57,504 ERROR [STDERR] at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) 10:12:57,504 ERROR [STDERR] at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
wizard used to be in cars' /etc/hosts/ file, but I commented it out, so it makes sense that it cannot create a connection.
What I do not understand is, why is it trying to connect to wizard?????
I have searched and searched again and there is no reference to wizard in any config files nor is cars still trying to use DefaultPartition!
Any help would be GREATLY appreciated!!!
Thanks,
Josephine
