InVMCourier high cpu while idle (if more than 3 esb packages

[andyredhead]

Hi,

I'm are using JBoss ESB 4.5.GA and JBoss AS 4.3 and the jvm is 1.5.

I see the same behaviour on both development Intel XP machines and pre-production Solaris/Opteron servers.

My scenario is that a number of regional business units (12 to be precise) use FTP to upload CSV update files.

There are 4 types of updates, each of which has a different file format and so is handled by a different process.

Each line of an uploaded file is converted into an ESB message and forwarded on to a "validation" esb service (over an invm transport).

There is an esb deployment (expanded .esb directory structure) per business unit.

So, there are

12 business units
4 FTP listeners per business unit
4 validation services per business unit
1 error processing service per business unit

Which means that there are 108 services of which 48 are FTP listeners.

The machines are dedicated to running the ESB, so there are no other applications running.

The problem is that even with the FTP listeners triggering once every 1000 seconds and with no files under the FTP server (so the server is idle), the CPU is at 100%

Using a combination of the CPU usage and thread dump mbeans from the JBoss JMX console I had a look at what the server is doing, the top 10 threads (and probably most of the top 100) are showing as follows (only typed out the top 3):

Thread-121 53531 (AR - CPU Time ms)

Thread: Thread-121 :
priority:5, demon:false, threadId:324,
org.jboss.internal.soa.esb.couriers.InVMCourier.pickup(InVMCourier.java:277)
threadState:TIMED_WAITING, lockName:[e-mail](Native Method)
org.jboss.internal.soa.esb.couriers.TwoWayCourierImpl.pickup(TwoWayCourierImpl.java:228)
org.jboss.internal.soa.esb.couriers.TwoWayCourierImpl.pickup(TwoWayCourierImpl.java:204)
org.jboss.soa.esb.listeners.message.MessageAwareListener.waitForEventAndProcess(MessageAwareListener.java:302)
org.jboss.soa.esb.listeners.message.MessageAwareListener.doRun(MessageAwareListener.java:258)
org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle.run(AbstractThreadedManagedLifecycle.java:115)
java.lang.Thread.run(Thread.java:595)


Thread-40 53046 (AR - CPU Time ms)

Thread: Thread-40 : priority:5, demon:false, threadId:152,
threadState:TIMED_WAITING, lockName:[e-mail](Native Method)
org.jboss.internal.soa.esb.couriers.InVMCourier.pickup(InVMCourier.java:277)
org.jboss.internal.soa.esb.couriers.TwoWayCourierImpl.pickup(TwoWayCourierImpl.java:228)
org.jboss.internal.soa.esb.couriers.TwoWayCourierImpl.pickup(TwoWayCourierImpl.java:204)
org.jboss.soa.esb.listeners.message.MessageAwareListener.waitForEventAndProcess(MessageAwareListener.java:302)
org.jboss.soa.esb.listeners.message.MessageAwareListener.doRun(MessageAwareListener.java:258)
org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle.run(AbstractThreadedManagedLifecycle.java:115)
java.lang.Thread.run(Thread.java:595)


Thread-105 52953 (AR - CPU Time ms)

Thread: Thread-105 : priority:5, demon:false, threadId:282,
threadState:RUNNABLE, lockName:nullorg.jboss.internal.soa.esb.couriers.InVMCourier.pickup(InVMCourier.java:274)
org.jboss.internal.soa.esb.couriers.TwoWayCourierImpl.pickup(TwoWayCourierImpl.java:228)
org.jboss.internal.soa.esb.couriers.TwoWayCourierImpl.pickup(TwoWayCourierImpl.java:204)
org.jboss.soa.esb.listeners.message.MessageAwareListener.waitForEventAndProcess(MessageAwareListener.java:302)
org.jboss.soa.esb.listeners.message.MessageAwareListener.doRun(MessageAwareListener.java:258)
org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle.run(AbstractThreadedManagedLifecycle.java:115)
java.lang.Thread.run(Thread.java:595)

So lots of clock cycles going in and around the area of InVMCourier.pickup

The really strange thing is if I only deploy 3 of the the esb packages, everything is ok (cpu usage is idling). As soon as I deploy the forth, the cpu is maxed and stays maxed...


I've had a look around to see if there are any obvious "pooling" config settings that might affect the behaviour but to no avail.

I've currently run out of ideas and would greatly appreciate any pointers :)

Cheers,

Andy

[kconner]

This has recently come up through the support portal and has been fixed in the platform release, we will shortly pull it across into trunk.

The CP issue is https://jira.jboss.org/jira/browse/JBESB-2603

Kev

[andyredhead]

Excellent news :)

Thanks you.