Need help with JB 5.1 Cluster - HA singleton JMS problem

[psaunders300]

Hi all,

I am in the process of creating a prototype cluster using JB 5.1 by migrating an existing JB 4.0.5 application.  I am doing this on my local Windows XP machine and most everything is working except for the JMS configuration. I have a two node cluster with an Apache web server running mod_jk doing the load balancing.

The application I am migrating has a series of jobs in a workflow that must be single threaded.  In 4.0.5 we are accomplishing this by deploying the MDB and queue definition file to the deploy-hasingleton folder and putting a singelton config name into the deployment descriptor (jboss.xml) (i.e. "<configuration-name>Singleton Message Driven Bean</configuration-name>"), which creates only one instance of the MDB.

When I migrated the application to JB 5.1, I configured the application in the same way, namely the queue definition file and the MDB are deployed to the deploy-hasingleton folder. The servers start up fine but I am having a problem on the second node of the cluster not being able to write messages to the queue.  The sendMessage() method is failing with a JMSException - [org.jboss.messaging.util.ExceptionUtil] (ajp-localhost%2F127.0.0.1-8109-1) SessionEndpoint[ka-iavw1u5g-1-yw7o1u5g-2csb7u-q32s2a] send [la-eovw1u5g-1-yw7o1u5g-2csb7u-q32s2a]
javax.jms.JMSException: Failed to route Reference[20750790429212672]:RELIABLE to ppmRmrSingleThreadedQueue).  The full stack trace is listed below.

2010-02-18 16:15:36,158 ERROR [org.jboss.messaging.util.ExceptionUtil] (ajp-localhost%2F127.0.0.1-8109-1) SessionEndpoint[ka-iavw1u5g-1-yw7o1u5g-2csb7u-q32s2a] send [la-eovw1u5g-1-yw7o1u5g-2csb7u-q32s2a]
javax.jms.JMSException: Failed to route Reference[20750790429212672]:RELIABLE to ppmRmrSingleThreadedQueue
     at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:757)
     at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:399)
     at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
     at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeTarget(SessionAdvised$send_7280680627620114891.java)
     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
     at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:597)
     at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
     at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
     at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
     at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
     at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:891)
     at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:106)
     at org.jboss.remoting.Client.invoke(Client.java:1724)
     at org.jboss.remoting.Client.invoke(Client.java:629)
     at org.jboss.remoting.Client.invoke(Client.java:617)
     at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
     at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
     at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:499)
     at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeTarget(ClientSessionDelegate$send_6145266547759487588.java)
     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
     at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:661)
     at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect_z_handleSend_4018505.invoke(SessionAspect_z_handleSend_4018505.java)
     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
     at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
     at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
     at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:269)
     at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect_z_handleSend_4018505.invoke(ProducerAspect_z_handleSend_4018505.java)
     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
     at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
     at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
     at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
     at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
     at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
     at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)
     at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
     at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136)
     at com.arbitron.ppmrmr.service.imp.PpmRmrQueueSender.sendToQueue(PpmRmrQueueSender.java:89)
     at com.arbitron.ppmrmr.web.businessdelegate.RunPpmRmrBusinessDelegate.startProcessing(RunPpmRmrBusinessDelegate.java:141)
     at com.arbitron.ppmrmr.web.action.PpmRmrRunAction.executeLogic(PpmRmrRunAction.java:116)
     at com.arbitron.ppmrmr.web.action.BaseAction.execute(BaseAction.java:69)
     at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421)
     at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:226)
     at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164)
     at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:415)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
     at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
     at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)
     at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
     at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
     at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
     at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
     at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:436)
     at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:384)
     at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
     at java.lang.Thread.run(Thread.java:619)

I have tried to search the forums to find a similar problem, but to no avail.  If anyone has suggestions as to what the problem may be, I would greatly appreciate it!!

Thanks,

Paul

[brian.stansberry]

Probably the simplest solution is to deploy the queue on all nodes. It's the MDB that needs to be an HASingleton. Deploying the queue only on the HASingleton master was something you had to do in 4.x because the JBossMQ JMS server would only be running on the HASingleton master. JBoss Messaging used in AS 5 doesn't work that way; a JMS server is running on all nodes in the cluster.

[psaunders300]

Hi Brian,

Thank you for the reply.  I tried what you suggested and now the message gets written to the queue successfully, but just sits in the queue, it is never picked up by the MDB.  Any suggestions?

Thanks again,

Paul

[brian.stansberry]

Make sure the queue configuration has a clustered attribute, set to true. See http://labs.jboss.com/file-access/default/members/jbossmessaging/freezone/docs/userguide-1.4.5.GA/html/c_configuration.html

[psaunders300]

Brian,

Adding the "Clustered" attribute to the queue worked!  The messages are being consumed by the singleton MDB on the first node instance and all is well.

Thank you so much for the help with this, it is appreciated!!

Paul

[dimitarn]

Hi

I have the oposite problem. I have 2 jboss insance on different machines.

I want to deploy my application on both but i don't want to be in cluster. When i deploy the ear in the server/all/deploy dir all works but they run in cluster. When i deploy the ear in the default directory my queues don't work i.e. the MDB does not seems to work.

This is how i define queue in xml

[CODE]

<mbean code="org.jboss.jms.server.destination.QueueService"
        name="jboss.messaging.destination:service=Queue,name=MyDurableQueue"
        xmbean-dd="xmdesc/Queue-xmbean.xml">
        <attribute name="JNDIName">queue/purchaseDurableQueue
        </attribute>
        <depends optional-attribute-name="ServerPeer">
            jboss.messaging:service=ServerPeer</depends>
        <depends>jboss.messaging:service=PostOffice
        </depends>
        <attribute name="Clustered">true</attribute>
    </mbean>

[/CODE]

When i deploy the ear in default dir i change the clustered to false, because it throws exception.

and the MDB

[CODE]

@MessageDriven(activationConfig = {
    @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),
    @ActivationConfigProperty(propertyName = "destination", propertyValue = "queue/MyDurableQueue") })
public class PurchaseMDB implements MessageListener {

...

}

[/CODE]

Also is there way to run the app in all dir without to be in cluster mode?

[brian.stansberry]

What does "all works but they run in cluster" mean? You need to be specific about what it is you want versus what you are seeing. The "all" config just provides some capabilities that don't exist in default; whether those capabilities are actually used mostly depends on how you configure your application.

Why is running in default with the clustered attribute set to false a problem? If it's problem that implies you want some feature available in "all" but not clustering. In that case, does running in all with the clustered attribute set to false work?

[dimitarn]

I mean that i have deployed my app in all folder where all services are configured. But today i wanted to do custom failover mechanisam using jms, i implemented some logic and replicated the jboss As with the ear(deployed in all/deploy) (also the configuration of the queue was clustered=true) on 2 servers(machines). Start 1st wit run.sh -c all bind IP1 and the same with IP2. Then they detect themselfs in a cluster. I don't want that! So i deployed the ear on both servers in default/deploy and change the configuration of the queue to clustered=false, bacause it trhows exception if it is true(in the default). And the queue does not work or the MDB does not reveive any message. And there is no exception.

Also i have some statful beans that i keep in a jboss cache and thistrange error is trown every time when i call my service. THIS happans only in the default deploy

[CODE]

y] javax.ejb.NoSuchEJBException: Could not find stateful bean: 5c4o22y-gc0pmq-g6268i8b-1-g626akad-lw
18:37:23,997 ERROR [ManagerBean] reload() ERROR: Could not find stateful bean: 5c4o22y-gc0pmq-g6268i8b-1-g626akad-lw
18:37:23,998 ERROR [STDERR] javax.ejb.NoSuchEJBException: Could not find stateful bean: 5c4o22y-gc0pmq-g6268i8b-1-g626akad-lw
18:37:23,998 ERROR [STDERR]     at org.jboss.ejb3.cache.simple.SimpleStatefulCache.get(SimpleStatefulCache.java:479)
18:37:23,998 ERROR [STDERR]     at org.jboss.ejb3.cache.simple.SimpleStatefulCache.get(SimpleStatefulCache.java:430)

[/CODE]

[dimitarn]

Is there way to eclude cluster service in the all configuration?

[brian.stansberry]

dimitarn wrote:

Is there way to eclude cluster service in the all configuration?

See "Removing Clustering" in http://community.jboss.org/docs/DOC-13278

If you have other questions about your application, please open a different thread, as this sounds only tangentially related to the original post.