clustered MDB problem during failover
I have successfully configured two JBOSS nodes to cluster.
I have a simple EJB3 MessageDriven Bean (it just sleeps for some time).
It is succesfully deployes to farm directory. And starts then messages come to the JMS queue.
I'm using HAJNDI.
Problem cames then I kill master JBOSS node (which handles JMS queues)
MDB can't restart on second server. As I can see JMS server restarts on second node, but MDB can't bind on it.
Following error appear in log file:
--------------------------------------
2005-12-01 10:05:20,976 INFO [org.jboss.web.tomcat.tc5.TomcatDeployer] deploy, ctxPath=/jbossmq-httpil, warUrl=.../deploy-hasingleton/jms/jbossmq-httpil.sar/jbossmq-httpil.war/
2005-12-01 10:05:21,211 INFO [org.apache.catalina.startup.ContextConfig] Missing application web.xml, using defaults only StandardEngine[jboss.web].StandardHost[localhost].StandardContext[/jbossmq-httpil]
2005-12-01 10:05:21,739 INFO [org.jboss.mq.server.jmx.Queue.A] Bound to JNDI name: queue/A
2005-12-01 10:05:21,744 INFO [org.jboss.mq.server.jmx.Queue.B] Bound to JNDI name: queue/B
2005-12-01 10:05:21,749 INFO [org.jboss.mq.server.jmx.Queue.C] Bound to JNDI name: queue/C
2005-12-01 10:05:21,764 INFO [org.jboss.mq.server.jmx.Queue.D] Bound to JNDI name: queue/D
2005-12-01 10:05:21,769 INFO [org.jboss.mq.server.jmx.Queue.ex] Bound to JNDI name: queue/ex
2005-12-01 10:05:21,793 INFO [org.jboss.mq.server.jmx.Topic.testTopic] Bound to JNDI name: topic/testTopic
2005-12-01 10:05:21,798 INFO [org.jboss.mq.server.jmx.Topic.securedTopic] Bound to JNDI name: topic/securedTopic
2005-12-01 10:05:21,803 INFO [org.jboss.mq.server.jmx.Topic.testDurableTopic] Bound to JNDI name: topic/testDurableTopic
2005-12-01 10:05:21,808 INFO [org.jboss.mq.server.jmx.Queue.testQueue] Bound to JNDI name: queue/testQueue
2005-12-01 10:05:21,828 INFO [org.jboss.mq.il.uil2.UILServerILService] JBossMQ UIL service available at : /0.0.0.0:8093
2005-12-01 10:05:21,837 INFO [org.jboss.mq.server.jmx.Queue.DLQ] Bound to JNDI name: queue/DLQ
2005-12-01 10:05:21,980 INFO [org.jboss.ha.framework.interfaces.HAPartition.lifecycle.CludevPartition] New cluster view for partition CludevPartition (id: 2, delta: -1) : [10.100.204.101:1099]
2005-12-01 10:05:21,993 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.CludevPartition] I am (10.100.204.101:1099) received membershipChanged event:
2005-12-01 10:05:21,993 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.CludevPartition] Dead members: 1 ([10.100.204.102:1099])
2005-12-01 10:05:21,993 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.CludevPartition] New Members : 0 ([])
2005-12-01 10:05:21,993 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.CludevPartition] All Members : 1 ([10.100.204.101:1099])
2005-12-01 10:05:22,269 WARN [org.jboss.mq.Connection] Connection failure, use javax.jms.Connection.setExceptionListener() to handle this error and reconnect
org.jboss.mq.SpyJMSException: Exiting on IOE; - nested throwable: (java.io.EOFException)
at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
at org.jboss.mq.Connection.asynchFailure(Connection.java:437)
at org.jboss.mq.il.uil2.UILClientILService.asynchFailure(UILClientILService.java:156)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleStop(SocketManager.java:413)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:345)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2670)
at java.io.ObjectInputStream.readByte(ObjectInputStream.java:864)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:290)
... 1 more
2005-12-01 10:05:22,270 WARN [org.jboss.ejb3.mdb.MDB] JMS provider failure detected:
org.jboss.mq.SpyJMSException: Exiting on IOE; - nested throwable: (java.io.EOFException)
at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
at org.jboss.mq.Connection.asynchFailure(Connection.java:437)
at org.jboss.mq.il.uil2.UILClientILService.asynchFailure(UILClientILService.java:156)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleStop(SocketManager.java:413)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:345)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2670)
at java.io.ObjectInputStream.readByte(ObjectInputStream.java:864)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:290)
... 1 more
2005-12-01 10:05:22,281 INFO [org.jboss.ejb3.mdb.MDB] Trying to reconnect to JMS provider
2005-12-01 10:05:32,283 DEBUG [org.jboss.ejb3.mdb.MDB] unset exception listener
2005-12-01 10:05:32,283 ERROR [org.jboss.ejb3.mdb.MDB] Could not stop JMS connection
org.jboss.mq.SpyJMSException: Cannot disable the connection with the JMS server; - nested throwable: (java.io.IOException: Client is not connected)
at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
at org.jboss.mq.SpyJMSException.rethrowAsJMSException(SpyJMSException.java:51)
at org.jboss.mq.Connection.doStop(Connection.java:1235)
at org.jboss.mq.Connection.stop(Connection.java:696)
at org.jboss.ejb3.mdb.MDB.innerStop(MDB.java:961)
at org.jboss.ejb3.mdb.MDB$ExceptionListenerImpl.onException(MDB.java:1185)
at org.jboss.mq.Connection$ExceptionListenerRunnable.run(Connection.java:1366)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.IOException: Client is not connected
at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:238)
at org.jboss.mq.il.uil2.SocketManager.sendMessage(SocketManager.java:206)
at org.jboss.mq.il.uil2.UILServerIL.setEnabled(UILServerIL.java:197)
at org.jboss.mq.Connection.doStop(Connection.java:1231)
... 5 more
2005-12-01 10:05:32,285 DEBUG [org.jboss.ejb3.mdb.DLQHandler] Destroy requested before stop, calling stop now
2005-12-01 10:05:32,285 DEBUG [org.jboss.ejb3.mdb.DLQHandler] Stopping DLQHandler
2005-12-01 10:05:32,285 WARN [org.jboss.ejb3.mdb.DLQHandler] Stopping failed DLQHandler
org.jboss.mq.SpyJMSException: Cannot disable the connection with the JMS server; - nested throwable: (java.io.IOException: Client is not connected)
at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
at org.jboss.mq.SpyJMSException.rethrowAsJMSException(SpyJMSException.java:51)
at org.jboss.mq.Connection.doStop(Connection.java:1235)
at org.jboss.mq.Connection.stop(Connection.java:696)
at org.jboss.ejb3.mdb.DLQHandler.stopService(DLQHandler.java:159)
at org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:300)
at org.jboss.system.ServiceMBeanSupport.stop(ServiceMBeanSupport.java:191)
at org.jboss.system.ServiceMBeanSupport.jbossInternalDestroy(ServiceMBeanSupport.java:323)
at org.jboss.system.ServiceMBeanSupport.destroy(ServiceMBeanSupport.java:206)
at org.jboss.ejb3.mdb.MDB.destroy(MDB.java:1067)
at org.jboss.ejb3.mdb.MDB$ExceptionListenerImpl.onException(MDB.java:1186)
at org.jboss.mq.Connection$ExceptionListenerRunnable.run(Connection.java:1366)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.IOException: Client is not connected
at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:238)
at org.jboss.mq.il.uil2.SocketManager.sendMessage(SocketManager.java:206)
at org.jboss.mq.il.uil2.UILServerIL.setEnabled(UILServerIL.java:197)
at org.jboss.mq.Connection.doStop(Connection.java:1231)
... 10 more
2005-12-01 10:05:32,294 DEBUG [org.jboss.ejb3.mdb.DLQHandler] Destroying DLQHandler
2005-12-01 10:05:32,294 DEBUG [org.jboss.ejb3.mdb.DLQHandler] Destroyed DLQHandler
2005-12-01 10:05:32,294 DEBUG [org.jboss.ejb3.tx.UserTransactionImpl] new UserTx: org.jboss.ejb3.tx.UserTransactionImpl@1a0bb6f
2005-12-01 10:05:32,294 ERROR [org.jboss.ejb3.mdb.MDB] Reconnect failed: JMS provider failure detected:
javax.naming.NameAlreadyBoundException
at org.jnp.server.NamingServer.bind(NamingServer.java:129)
at org.jnp.interfaces.NamingContext.bind(NamingContext.java:551)
at org.jnp.interfaces.NamingContext.bind(NamingContext.java:516)
at org.jboss.util.naming.Util.bind(Util.java:87)
at org.jboss.util.naming.Util.bind(Util.java:74)
at org.jboss.ejb3.EJBContainer.resolveInjectors(EJBContainer.java:479)
at org.jboss.ejb3.EJBContainer.initializePool(EJBContainer.java:421)
at org.jboss.ejb3.EJBContainer.start(EJBContainer.java:374)
at org.jboss.ejb3.mdb.MDB.start(MDB.java:223)
at org.jboss.ejb3.mdb.MDB$ExceptionListenerImpl.onException(MDB.java:1187)
at org.jboss.mq.Connection$ExceptionListenerRunnable.run(Connection.java:1366)
at java.lang.Thread.run(Thread.java:595)
2005-12-01 10:05:32,300 INFO [org.jboss.ejb3.mdb.MDB] Trying to reconnect to JMS provider
2005-12-01 10:05:42,306 DEBUG [org.jboss.ejb3.tx.UserTransactionImpl] new UserTx: org.jboss.ejb3.tx.UserTransactionImpl@112731f
2005-12-01 10:05:42,307 ERROR [org.jboss.ejb3.mdb.MDB] Reconnect failed: JMS provider failure detected:
javax.naming.NameAlreadyBoundException
at org.jnp.server.NamingServer.bind(NamingServer.java:129)
at org.jnp.interfaces.NamingContext.bind(NamingContext.java:551)
at org.jnp.interfaces.NamingContext.bind(NamingContext.java:516)
at org.jboss.util.naming.Util.bind(Util.java:87)
at org.jboss.util.naming.Util.bind(Util.java:74)
at org.jboss.ejb3.EJBContainer.resolveInjectors(EJBContainer.java:479)
at org.jboss.ejb3.EJBContainer.initializePool(EJBContainer.java:421)
at org.jboss.ejb3.EJBContainer.start(EJBContainer.java:374)
at org.jboss.ejb3.mdb.MDB.start(MDB.java:223)
at org.jboss.ejb3.mdb.MDB$ExceptionListenerImpl.onException(MDB.java:1187)
at org.jboss.mq.Connection$ExceptionListenerRunnable.run(Connection.java:1366)
at java.lang.Thread.run(Thread.java:595)
2005-12-01 10:05:42,319 INFO [org.jboss.ejb3.mdb.MDB] Trying to reconnect to JMS provider