Connection pool exhausted messages after undeploy

[jkemp101]

I have a simple test bed that I am using to test failure and hot deployment scenarios. I have two services (A&B) in their own .esb files. Service A has a single action that is a static router which sends messages to Service B. Service B is basically the helloworld quickstart. The test client sends a message to Service A using deliverAsync of ServiceInvoker. Under normal conditions everything works fine. The message is returned to the client as expected.

The test case I am having problems with is when I undeploy Service B and run the test client. It fails as expected (although I actually have to end task on a process to totally kill it). The server starts printing

13:37:12,652 INFO [JmsConnectionPool] The connection pool was exhausted. Waiting 1 second before trying again..

every second. The Service A reply queue has 40 messages in it. Redeploying Service B’s esb does not bring things back to normal. I have to undeploy A and restart the server. Should the esb be able to recover from this test case? I am using JMS for all messaging.

Thanks.

[kurtstam]

So let me get this straight. Service A (in A.esb) send messages to Services B (in B.esb). Let's say service B listens to queue/B. When you undeploy B.esb is queue/B is still there? If so, there should be no problem, it should simply fill up queue/B.

If queue/B is undeployed with B.esb, and B's EPR is removed from the registry, then Service A has no EPR to deliver the messages to and it should start sending them to the redelivery service.

Which scenario are you in? What errors do you see?
It looks like there is a connection is leaking when am exception is thrown or something.

--K

[jkemp101]

You will have to bare with me, I am a little new to the ESB experience. I posted some config files at http://cpp.capwin.org/joetest/app/ESBTest.zip so you can see what I have.

I think I am in the second situation you listed. The queue goes away with the esb. While the test client is running I get this exception

14:46:32,014 INFO [ServiceInvoker] Unresponsive EPR: JMSEpr [ PortReference < <wsa:Address jms://192.168.50.242/queue/QSPnP_Request_esb/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.initial : org.jnp.interfaces.NamingContextFactory/>, <wsa:ReferenceProperties jbossesb:java.naming.provider.url : jnp://127.0.0.1:1099/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.url.pkgs : org.jboss.naming:org.jnp.interfaces/>, <wsa:ReferenceProperties jbossesb:destination-type : queue/>, <wsa:ReferenceProperties jbossesb:specification-version : 1.1/>, <wsa:ReferenceProperties jbossesb:connection-factory : ConnectionFactory/>, <wsa:ReferenceProperties jbossesb:persistent : true/>, <wsa:ReferenceProperties jbossesb:acknowledge-mode : 1/>, <wsa:ReferenceProperties jbossesb:type : urn:jboss/esb/epr/type/jms/> > ] for message: header: [ To: JMSEpr [ PortReference < <wsa:Address jms://192.168.50.242/queue/QM_Request_esb/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.initial : org.jnp.interfaces.NamingContextFactory/>, <wsa:ReferenceProperties jbossesb:java.naming.provider.url : jnp://127.0.0.1:1099/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.url.pkgs : org.jboss.naming:org.jnp.interfaces/>, <wsa:ReferenceProperties jbossesb:destination-type : queue/>, <wsa:ReferenceProperties jbossesb:specification-version : 1.1/>, <wsa:ReferenceProperties jbossesb:connection-factory : ConnectionFactory/>, <wsa:ReferenceProperties jbossesb:persistent : true/>, <wsa:ReferenceProperties jbossesb:acknowledge-mode : 1/> > ] ReplyTo: JMSEpr [ PortReference < <wsa:Address jms://localhost/queue/QM_Request_esb_reply/>, <wsa:ReferenceProperties jbossesb:java.naming.provider.url : jnp://127.0.0.1:1099/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.initial : org.jnp.interfaces.NamingContextFactory/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.url.pkgs : org.jboss.naming:org.jnp.interfaces/>, <wsa:ReferenceProperties jbossesb:destination-type : queue/>, <wsa:ReferenceProperties jbossesb:specification-version : 1.1/>, <wsa:ReferenceProperties jbossesb:connection-factory : ConnectionFactory/>, <wsa:ReferenceProperties jbossesb:message-selector : jbossESBresponseUUID='dc65fbce-8fcd-4360-b892-ff60fb11c606'/>, <wsa:ReferenceProperties jbossesb:persistent : true/>, <wsa:ReferenceProperties jbossesb:acknowledge-mode : 1/>, <wsa:ReferenceProperties jbossesb:type : urn:jboss/esb/epr/type/jms/> > ] MessageID: ID:JBM-64000 RelatesTo: jms:correlationID#ece1515f-df87-4320-8904-1d5fac48300b ]
14:46:32,155 ERROR [ExceptionUtil] SessionEndpoint[b4-qjhm4n9f-1-py4i4n9f-1qqgb-q6e1o4c5] createQueue [c4-vjhm4n9f-1-py4i4n9f-1qqgb-q6e1o4c5]
javax.jms.JMSException: There is no administratively defined queue with name:queue/QSPnP_Request_esb
 at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createQueue(ServerSessionEndpoint.java:282)
 at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$createQueue$aop(SessionAdvised.java:98)
 at org.jboss.jms.server.endpoint.advised.SessionAdvised$createQueue_6431069199924553036.invokeNext(SessionAdvised$createQueue_6431069199924553036.java)
 at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
 at org.jboss.jms.server.endpoint.advised.SessionAdvised$createQueue_6431069199924553036.invokeNext(SessionAdvised$createQueue_6431069199924553036.java)
 at org.jboss.jms.server.endpoint.advised.SessionAdvised.createQueue(SessionAdvised.java)
 at org.jboss.jms.wireformat.SessionCreateQueueRequest.serverInvoke(SessionCreateQueueRequest.java:74)
 at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:144)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:771)
 at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
 at org.jboss.remoting.Client.invoke(Client.java:1634)
 at org.jboss.remoting.Client.invoke(Client.java:548)
 at org.jboss.remoting.Client.invoke(Client.java:536)
 at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:186)
 at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:157)
 at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$createQueue$aop(ClientSessionDelegate.java:283)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$createQueue_6431069199924553036.invokeNext(ClientSessionDelegate$createQueue_6431069199924553036.java)
 at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:91)
 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$createQueue_6431069199924553036.invokeNext(ClientSessionDelegate$createQueue_6431069199924553036.java)
 at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$createQueue_6431069199924553036.invokeNext(ClientSessionDelegate$createQueue_6431069199924553036.java)
 at org.jboss.jms.client.delegate.ClientSessionDelegate.createQueue(ClientSessionDelegate.java)
 at org.jboss.jms.client.JBossSession.createQueue(JBossSession.java:250)
 at org.jboss.internal.soa.esb.couriers.JmsCourier.createMessageProducer(JmsCourier.java:327)
 at org.jboss.internal.soa.esb.couriers.JmsCourier.deliver(JmsCourier.java:189)
 at org.jboss.internal.soa.esb.couriers.TwoWayCourierImpl.deliver(TwoWayCourierImpl.java:188)
 at org.jboss.soa.esb.client.ServiceInvoker$EPRInvoker.attemptDelivery(ServiceInvoker.java:437)
 at org.jboss.soa.esb.client.ServiceInvoker$EPRInvoker.access$200(ServiceInvoker.java:370)
 at org.jboss.soa.esb.client.ServiceInvoker.post(ServiceInvoker.java:259)
 at org.jboss.soa.esb.client.ServiceInvoker.deliverAsync(ServiceInvoker.java:184)
 at org.jboss.internal.soa.esb.persistence.format.db.DBMessageStoreImpl.redeliver(DBMessageStoreImpl.java:328)
 at org.jboss.soa.esb.actions.MessageRedeliverer.process(MessageRedeliverer.java:74)
 at org.jboss.soa.esb.listeners.message.ActionProcessingPipeline.process(ActionProcessingPipeline.java:316)
 at org.jboss.soa.esb.listeners.ScheduleListener.onSchedule(ScheduleListener.java:121)
 at org.jboss.soa.esb.schedule.ScheduleProvider$ESBScheduledJob.execute(ScheduleProvider.java:217)
 at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
 at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)

and then after I end task on the test client I get the connection pool exhausted with exceptions every minute or so.

[kurtstam]

The stacktrace tells me that the Registry still has stale EPRs which are causing the redeliverService to try to redeliver to a non existing endpoint. Try cleaning out the registry and try running the test again. By default the redeliverService will run every 5 min forever (see the jbossesb.esb jboss-esb.xml). On a clean undeploy the endpoint should get removed from the Registry.

That is the only problem I see here. If you get that solved you should print the pool exhaustion stacktrace.

--Kurt

p.s. the easiest way to browse the registry is using the uddibrowser:
http://anonsvn.labs.jboss.com/labs/jbossesb/branches/JBESB_4_2_1_GA_CP/product/docs/services/Registry.pdf

[jkemp101]

That seems to have fixed the problem. However, it looks like the undeploy of the esb is not removing it from the registry. Using the UDDI browser of RHDS shows the service is still published. Does it just remove the binding template but leaves the service definition in the registry, is this ok?

I am now running into some performance issues. I changed the test client to submit 100 messages in a for loop. The esb process goes to 50% CPU use (really %100 of 1 core) and the messages are taking about 1 second for each. I end the client test process in the middle and the esb process remains at the 50% for a few minutes. I think undeploying the services helps the cpu usage to come back down to 0. It doesn't seem to recover from these types of failures very gracefully.