Problem connecting to Topic
Hi,
We are having a problem connecting to a distributed JMS topic. The details are mentioned below. Any help would be much appreciated.
Many Thanks.
We installed the JBoss Messaging version (JBoss 4.2.2.GA + JBM 1.4.0SP3) on both our staging servers and the clustering itself seems to be working fine. When I enabled DEBUG log I see the following messages every 10 seconds on both the servers :
2008-04-04 10:41:30,993 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 10.101.109.179:7800 (own address=10.101.109.178:7800)
2008-04-04 10:41:30,993 DEBUG [org.jgroups.protocols.FD] received ack from 10.101.109.179:7800
The client to the JMS Server is a WAR deployed in another JBoss instance (JBoss 4.0.2). On the JMS Server we have setup a distributed Queue and a Distributed Topic. Job requests are sent as messages to the Queue and to kill any jobs messages are sent to the Topic. After consuming a message from the Queue a job is executed by a thread provided by a thread pool which runs the job contained in that message. This thread first registers itself with the Shutdown Topic as a Message Listener to check periodically during the execution of the job for any shutdown messages (with a selector) sent for this job and if there is one, the job stops itself by throwing a Runtime Exception. However the problem is when registering to the Topic there is an error from the JMS classes as a result of which the thread cannot register as a subscriber to the topic. And this seems to be working fine for the first few jobs after restarting the service but the problem appears after the servers stayed idle for a while.
This is the stack trace we see in our application (JMS Client) logs:
2008-04-04 10:19:58,792 ERROR [WIZARD_VERIFY#20220] [WizardVerifyOperation] Exception while executing Job 20220
org.jboss.jms.exception.MessagingJMSException: A failure has occurred during processing of the request. Please consult the server logs for more details. SessionEndpoint[sw2-16fpjmef-1-dx3silef-y0jrzy-y413t2a] createConsumerDelegate [fx2-sb2qjmef-1-dx3silef-y0jrzy-y413t2a]
at org.jboss.messaging.util.ExceptionUtil.handleJMSInvocation(ExceptionUtil.java:72)
at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegate(ServerSessionEndpoint.java:258)
at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$createConsumerDelegate$aop(SessionAdvised.java:94)
at org.jboss.jms.server.endpoint.advised.SessionAdvised$createConsumerDelegate_8721389917985689973.invokeNext(SessionAdvised$createConsumerDelegate_8721389917985689973.java)
at org.jboss.jms.server.container.SecurityAspect.handleCreateConsumerDelegate(SecurityAspect.java:123)
at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
at org.jboss.jms.server.endpoint.advised.SessionAdvised$createConsumerDelegate_8721389917985689973.invokeNext(SessionAdvised$createConsumerDelegate_8721389917985689973.java)
at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
at org.jboss.jms.server.endpoint.advised.SessionAdvised$createConsumerDelegate_8721389917985689973.invokeNext(SessionAdvised$createConsumerDelegate_8721389917985689973.java)
at org.jboss.jms.server.endpoint.advised.SessionAdvised.createConsumerDelegate(SessionAdvised.java)
at org.jboss.jms.wireformat.SessionCreateConsumerDelegateRequest.serverInvoke(SessionCreateConsumerDelegateRequest.java:100)
at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:795)
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:573)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:163)
at org.jboss.remoting.Client.invoke(Client.java:1634)
at org.jboss.remoting.Client.invoke(Client.java:548)
at org.jboss.remoting.Client.invoke(Client.java:536)
at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$createConsumerDelegate$aop(ClientSessionDelegate.java:245)
at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_8721389917985689973.invokeNext(ClientSessionDelegate$createConsumerDelegate_8721389917985689973.java)
at org.jboss.jms.client.container.StateCreationAspect.handleCreateConsumerDelegate(StateCreationAspect.java:148)
at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect30.invoke(StateCreationAspect30.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_8721389917985689973.invokeNext(ClientSessionDelegate$createConsumerDelegate_8721389917985689973.java)
at org.jboss.jms.client.container.ConsumerAspect.handleCreateConsumerDelegate(ConsumerAspect.java:73)
at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect29.invoke(ConsumerAspect29.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_8721389917985689973.invokeNext(ClientSessionDelegate$createConsumerDelegate_8721389917985689973.java)
at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_8721389917985689973.invokeNext(ClientSessionDelegate$createConsumerDelegate_8721389917985689973.java)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_8721389917985689973.invokeNext(ClientSessionDelegate$createConsumerDelegate_8721389917985689973.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate.createConsumerDelegate(ClientSessionDelegate.java)
at org.jboss.jms.client.JBossSession.createConsumer(JBossSession.java:237)
at org.jboss.jms.client.JBossSession.createConsumer(JBossSession.java:220)
at com.aviseurope.rm.pmech.asynchjob.spi.AbstractSchedulableOperation.registerWithShutdownTopic(AbstractSchedulableOperation.java:185)
at com.aviseurope.rm.pmech.asynchjob.spi.AbstractSchedulableOperation.run(AbstractSchedulableOperation.java:103)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
And this is the tack trace we see in the JMS Server logs:
2008-04-04 10:19:49,667 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 10.101.109.179:7800 (own address=10.101.109.178:7800)
2008-04-04 10:19:49,667 DEBUG [org.jgroups.protocols.FD] received ack from 10.101.109.179:7800
2008-04-04 10:19:52,448 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 10.101.109.179:3410 (own address=10.101.109.178:3940)
2008-04-04 10:19:52,448 DEBUG [org.jgroups.protocols.FD] received ack from 10.101.109.179:3410
2008-04-04 10:19:54,932 DEBUG [org.jgroups.protocols.MERGE2] initial_mbrs=[[own_addr=10.101.109.178:7900, coord_addr=10.101.109.178:7900, is_server=true]]
2008-04-04 10:19:55,526 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 10.101.109.179:3445 (own address=10.101.109.178:3972)
2008-04-04 10:19:55,526 DEBUG [org.jgroups.protocols.FD] received ack from 10.101.109.179:3445
2008-04-04 10:19:58,792 ERROR [org.jboss.messaging.util.ExceptionUtil] SessionEndpoint[sw2-16fpjmef-1-dx3silef-y0jrzy-y413t2a] createConsumerDelegate [fx2-sb2qjmef-1-dx3silef-y0jrzy-y413t2a]
java.lang.IllegalStateException: org.jboss.messaging.core.impl.postoffice.GroupMember@511196c0 response not received from 10.101.109.179:3445 - there may be others
at org.jboss.messaging.core.impl.postoffice.GroupMember.multicastControl(GroupMember.java:253)
at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.internalAddBinding(MessagingPostOffice.java:1886)
at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.addBinding(MessagingPostOffice.java:444)
at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegateInternal(ServerSessionEndpoint.java:1885)
at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegate(ServerSessionEndpoint.java:252)
at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$createConsumerDelegate$aop(SessionAdvised.java:94)
at org.jboss.jms.server.endpoint.advised.SessionAdvised$createConsumerDelegate_8721389917985689973.invokeNext(SessionAdvised$createConsumerDelegate_8721389917985689973.java)
at org.jboss.jms.server.container.SecurityAspect.handleCreateConsumerDelegate(SecurityAspect.java:123)
at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
at org.jboss.jms.server.endpoint.advised.SessionAdvised$createConsumerDelegate_8721389917985689973.invokeNext(SessionAdvised$createConsumerDelegate_8721389917985689973.java)
at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
at org.jboss.jms.server.endpoint.advised.SessionAdvised$createConsumerDelegate_8721389917985689973.invokeNext(SessionAdvised$createConsumerDelegate_8721389917985689973.java)
at org.jboss.jms.server.endpoint.advised.SessionAdvised.createConsumerDelegate(SessionAdvised.java)
at org.jboss.jms.wireformat.SessionCreateConsumerDelegateRequest.serverInvoke(SessionCreateConsumerDelegateRequest.java:100)
at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:795)
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:573)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
2008-04-04 10:19:58,807 DEBUG [org.jboss.jms.server.connectionmanager.SimpleConnectionManager] unregistered connection ConnectionEndpoint[rw2-l5fpjmef-1-dx3silef-y0jrzy-y413t2a] with remoting session ID ...-femjpf4q-3vk
2008-04-04 10:19:58,807 DEBUG [org.jboss.jms.server.remoting.JMSServerInvocationHandler] removing callback handler ServerInvokerCallbackHandler[a2t314y-m8t55z-feliwgow-1-femjpf4q-3vk+a2t314y-m8t55z-feliwgow-1-femjpf55-3vn]
2008-04-04 10:19:58,807 DEBUG [org.jboss.remoting.InvokerRegistry] removed SocketClientInvoker[793565b0, bisocket://10.101.109.178:1240646676] from registry
2008-04-04 10:19:58,807 DEBUG [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[793565b0, bisocket://10.101.109.178:1240646676] disconnecting ...
2008-04-04 10:19:59,667 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 10.101.109.179:7800 (own address=10.101.109.178:7800)
2008-04-04 10:19:59,667 DEBUG [org.jgroups.protocols.FD] received ack from 10.101.109.179:7800