High Thread count and OutOfMemory exception

[mohitt]

Hi,

I am working on a proof of concept project as part of evaluation of JBossESB. I believe either Jboss Application Server or JBossESB listener process is not releasing threads that it created to service the action pipeline.

Originally I had a more complex design but as part of troubleshooting this problem I have simplified the design quite a bit. Basically, it is a work management application. Certain external event would fire an event that would invoke an action (DistributorAction) that would feed the 'bus' with several work items (in WorkQueue). These work items would in turn invoke action class (AlloterAction) to find an available worker and if one is found, assigns the work to that worker by sending a message to a corresponding queue. This AlloterAction waits till a worker becomes available. After the worker is done with the work, it sends an 'available-again' type of a message back to the bus resulting in another action (RegisterWorkerAction) that makes this worker available for the next waiting AlloterAction. Hopefully this explanation of the design gives the necessary context for the problem.

Now, the problem - if I flood the WorkQueue with say 400 work items, by the time all of them are processed, the thread count of the listener process is already very high. The maxThread property for AlloterAction is 20, and for the others it is under 5. After couple of such 'high volume' tests, the system runs out of any more threads to allocate and eventually throws OutOfMemory exceptions.

Here is the jbossesb.xml.

 <?xml version="1.0" encoding="UTF-8" ?>
- <jbossesb xmlns="http://anonsvn.labs.jboss.com/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb-1.0.xsd" parameterReloadSecs="5">
- <providers>
- <jms-provider name="JBossMQ" connection-factory="ConnectionFactory" jndi-context-factory="org.jnp.interfaces.NamingContextFactory" jndi-URL="localhost">
- <jms-bus busid="GwChannel">
 <jms-message-filter dest-type="QUEUE" dest-name="queue/NonESBMessages" />
 </jms-bus>
- <jms-bus busid="EsbChannel">
 <jms-message-filter dest-type="QUEUE" dest-name="queue/MessageRouterQueue" />
 </jms-bus>
- <jms-bus busid="IncomingWorkChannel">
 <jms-message-filter dest-type="QUEUE" dest-name="queue/WorkTrigger" />
 </jms-bus>
- <jms-bus busid="WorkAllocationChannel">
 <jms-message-filter dest-type="QUEUE" dest-name="queue/WorkQueue" />
 </jms-bus>
- <jms-bus busid="WorkersChannel">
 <jms-message-filter dest-type="QUEUE" dest-name="queue/NewWorker" />
 </jms-bus>
+ <!-- <jms-bus busid="WorkDoneChannel">
 <jms-message-filter
 dest-type="QUEUE"
 dest-name="queue/WorkDone"
 />
 </jms-bus>

 <jms-bus busid="MonitorsChannel">
 <jms-message-filter
 dest-type="QUEUE"
 dest-name="queue/MonitorTrigger"
 />
 </jms-bus>

 -->
 </jms-provider>
 </providers>
- <services>
- <!-- ESB CBR Service
 -->
- <service category="CBRServicesESB" name="WorkCBRServiceESB" description="ESB Listener">
- <listeners>
- <!-- Gateway
 -->
 <jms-listener name="WorkGateway" busidref="GwChannel" maxThreads="4" is-gateway="true" />
 <jms-listener name="XPathContentBasedRouter" busidref="EsbChannel" maxThreads="4" />
 </listeners>
- <actions>
- <action class="org.jboss.soa.esb.actions.ContentBasedRouter" name="ContentBasedRouter">
 <property name="ruleSet" value="WorkESBRules-XPath.drl" />
 <property name="ruleLanguage" value="XPathLanguage.dsl" />
 <property name="ruleReload" value="true" />
- <property name="destinations">
 <route-to destination-name="newWork" service-category="WorkCoordination" service-name="WorkDistributionService" />
 <route-to destination-name="workDone" service-category="WorkStatus" service-name="UpdateStatusService" />
 <route-to destination-name="moreWorker" service-category="WorkerManagement" service-name="WorkerRegistrationService" />
 </property>
 </action>
 </actions>
 </service>
- <!-- Distribute Work
 -->
- <service category="WorkCoordination" name="WorkDistributionService" description="Work Distribution Service">
- <listeners>
 <jms-listener name="NewWorkListner" busidref="IncomingWorkChannel" maxThreads="3" />
 </listeners>
- <actions>
 <action name="distributeWorkAction" class="workManagement.DistributorAction" process="displayMessage,distributeWork" />
 </actions>
 </service>
- <!-- Register Worker Service
 -->
- <service category="WorkerManagement" name="WorkerRegistrationService" description="Register Worker Available Service">
- <listeners>
 <jms-listener name="WorkerRegistrationListner" busidref="WorkersChannel" maxThreads="4" />
 </listeners>
- <actions>
 <action name="registerWorkerAction" class="workManagement.WorkerRegistrationAction" process="registerWorker" />
 </actions>
 </service>
- <!-- Allot Work Service
 -->
- <service category="WorkAllocation" name="WorkAllocationService" description="Allocate Work Service">
- <listeners>
 <jms-listener name="WorkAllocationListner" busidref="WorkAllocationChannel" maxThreads="20" />
 </listeners>
- <actions>
 <action name="workAllocationAction" class="workManagement.AlloterAction" process="displayMessage,allocateWork" />
 </actions>
 </service>
- <!-- Monitor Work Status
 -->
+ <!-- <service
 category="WorkMonitor"
 name="monitorWorkService"
 description="Monitor Work Service">
 <listeners>
 <jms-listener
 name="WorkMonitorListner"
 busidref="MonitorsChannel"
 maxThreads="4"
 />
 </listeners>
 <actions>
 <action name="displayMessageAction"
 class="workManagement.MonitorAction"
 process="displayMessage"
 />
 </actions>
 </service>

 -->
- <!-- Update Work Status
 -->
+ <!-- <service
 category="WorkStatus"
 name="UpdateStatusService"
 description="Update Work Status Service">
 <listeners>
 <jms-listener
 name="WorkStatusListner"
 busidref="WorkDoneChannel"
 maxThreads="4"
 />
 </listeners>
 <actions>
 <action name="displayMessageAction"
 class="workManagement.WorkDoneAction"
 process="updateWorkStatus"
 />
 </actions>
 </service>

 -->
 </services>
 </jbossesb>

I have experienced the same result on Windows as well as Linux servers.

I am using JBoss Application Server 4.0.5.GA.

Any help or suggestion would be highly appreciated.

Cheers,
Mohit

[kurtstam]

Hi Mohitt,

What version of jbossesb are you using (trunk?), and which JMS provider (JBoss Messaging?) and what version of it?

I've seen this in the past, and it should be relatively easy to find and fix hopefully :).

In the meantime can you set the max thread count to 1 for each of the actions and do the same test? If there is a leak it should still run of threads and memory.

thx,

--Kurt

[mohitt]

Thanks for a prompt reply Kurt.

I am using JBossESB 4.0 GA. The JMS provider is JBoss MQ, exact version I am not sure of, but it is the one that comes bundled with JBossAS 4.0.5.GA.

I did try the same test with maxThread set to 1 for all the actions - the result is the same.

Thanks,
Mohit

[kurtstam]

Some critical fixes where applied after 4.0 GA and where not back-ported. Can you try 4.2 MR2 or get a snapshot of the trunk (probably the better option). We are close to releasing 4.2 GA (a few weeks out). So we're putting all our efforts into that.

If you can I would advise using the jbossesb-server, since it's all preconfigured. Lateron you can also deploy to JBossAS to get ejb3 and WS features.

Note that the trunk has JBoss Messaging 1.3 and 4.2 MR2 has JBoss Messaging 1.2.

Cheers,

--Kurt