How does InVMCourier work?

[lightbody]

Apologies, but I'm totally new to JBoss ESB. I'm an ex-Java open source guy though (OpenSymphony, Struts 2, etc), so hopefully I still can learn quickly Anyway, these days I do product management at New Relic and I'm helping a customer get better visibility in to their JBoss ESB app using our product. Our thread profiler indicates this partial stack trace as to where a bunch of time is being sent:

35.0%  - org.jboss.soa.esb.client.ServiceInvoker.deliverSync

35.0%  - org.jboss.soa.esb.client.ServiceInvoker.post

35.0%  - org.jboss.soa.esb.client.ServiceInvoker$EPRInvoker.access$200

35.0%  - org.jboss.soa.esb.client.ServiceInvoker$EPRInvoker.attemptDelivery

35.0%  - org.jboss.internal.soa.esb.couriers.TwoWayCourierImpl.pickup

35.0%  - org.jboss.internal.soa.esb.couriers.TwoWayCourierImpl.pickup

35.0%  - org.jboss.internal.soa.esb.couriers.InVMCourier.pickup

35.0%  - org.jboss.internal.soa.esb.couriers.transport.InVMTemporaryTransport.pickup

35.0%  - org.jboss.internal.soa.esb.couriers.transport.InVMTemporaryTransport$InVMEntry.pickup

35.0%  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await

35.0%  - java.util.concurrent.locks.LockSupport.parkNanos

35.0%  - sun.misc.Unsafe.park

Without knowing the details of JBoss ESB or the customer's app, I'm guessing that what is happening here is the receiving service is inside the JVM (thus the name "InVMCourier") and there is not an actual HTTP POST taking place. Rather, some other thread is picking up the work and this profile simply shows off that 35% of the time we're waiting for that thread. What I can't figure out when browsing through the code is where that other thread pool is instantiated. It's important because this is where I'd like to add additional probes to find out the root cause of the slowdown. I've poked around ServiceInvoker, TwoWayCourierImpl, InVMCourier, InVMTemporaryTransport, etc but not seeing anything obvious like an ExecutorService.

Any tips? Thanks!

Patrick

PS: If we can make some initial progress here, my hope is to use this work as the start of official JBoss ESB support for New Relic. Woot!

[tcunning]

First off, sorry for such a delayed reply on this question.

InVMCourier is used in the delivery of an ESB message to an InVM endpoint (links below explain InVM transport).    The pickup method is pretty simple - it tries to pickup an InVM message, or waits a specified timeout :

http://anonsvn.jboss.org/repos/labs/labs/jbossesb/trunk/product/rosetta/src/org/jboss/internal/soa/esb/couriers/InVMCourier.java

I'm not sure what the customer is doing in the rest of the app - but I think the stack you've posted is showing an idle InVM thread waiting to pick up a message (see sun.misc.Unsafe.park).

http://docs.jboss.org/jbossesb/docs/4.12/manuals/html/Programmers_Guide/index.html#sect-SOA_ESB_Programmers_Guide-What_is_a_Service-InVM_Transport

[kconner]

To expand on Tom's reply and your initial question.

As Tom says the stack trace is showing that your client is invoking a synchronous endpoint and, having sent the original request, is now waiting to receive a response.  The message will be picked up by a thread that is associated with the target service (MessageAwareListener) which will drive the action pipeline to its conclusion.  The last service invocation in the chain can return a response to this invoker using the temporary endpoint that was created when the request was made.  This will cause the client thread to resume and retrieve the response.

[lightbody]

Thanks for confirming my suspicion. MessageAwareListener is also very helpful as I can see the actual thread activity, which I was not finding before. I'll report back if I'm having trouble.