Unable to create native thread

[hendra_netm]

Hi guys,

Currently, I am using JBoss AS 4.2.2 and JBoss Messaging 1.4 SP3.
When I put the messaging server under load, the number of the threads were increasing until I got "Out of Memory: Unable to create native thread"

2008-02-29 20:41:10,267 [ERROR] [WorkerThread#26[172.21.101.9:52414]] server.endpoint.ServerSessionEndpoint: Failed to prompt delivery
java.lang.OutOfMemoryError: unable to create new native thread
 at java.lang.Thread.start0(Native Method)
 at java.lang.Thread.start(Thread.java:597)
 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor.restart(QueuedExecutor.java:142)
 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor.execute(QueuedExecutor.java:157)
 at org.jboss.jms.server.endpoint.ServerSessionEndpoint.promptDelivery(ServerSessionEndpoint.java:1539)
 at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.promptDelivery(ServerConsumerEndpoint.java:643)
 at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.changeRate(ServerConsumerEndpoint.java:420)
 at org.jboss.jms.server.endpoint.advised.ConsumerAdvised.org$jboss$jms$server$endpoint$advised$ConsumerAdvised$changeRate$aop(ConsumerAdvised.java:69)
 at org.jboss.jms.server.endpoint.advised.ConsumerAdvised$changeRate_N952316153687074823.invokeNext(ConsumerAdvised$changeRate_N952316153687074823.java)
 at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
 at org.jboss.jms.server.endpoint.advised.ConsumerAdvised$changeRate_N952316153687074823.invokeNext(ConsumerAdvised$changeRate_N952316153687074823.java)
 at org.jboss.jms.server.endpoint.advised.ConsumerAdvised.changeRate(ConsumerAdvised.java)
 at org.jboss.jms.wireformat.ConsumerChangeRateRequest.serverInvoke(ConsumerChangeRateRequest.java:71)
 at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:795)
 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:573)
 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)

When I dump the threads, I found many waiting threads.

Thread: Thread-5097 : priority:5, demon:false, threadId:20461, threadState:WAITING, lockName:java.lang.Object@469ace34

 java.lang.Object.wait(Native Method)
 java.lang.Object.wait(Object.java:485)
 EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
 EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
 java.lang.Thread.run(Thread.java:619)

In several hours, the number of this kind of threads can be more than 3000 threads, and they seem waiting forever.


Do you have any idea, why this can be happened? Does my application forget to free the threads? In what occasion the waiting threads are created and how to notify the threads so they are not waiting anymore?

My application is using client acknowledge. When a message is coming, it will be sent to another queue, before acknowledged.

Thank you for your time.

Best Regards,
Hendra

[timfox]

I would check if you are creating a lot of sessions somewhere and not closing them.

Common causes would be creating a connection, session, sending a message etc from inside an ejb, but not using the JCA managed connection factory at java:/JmsXA

[hendra_netm]

Hi Tim,

Thank you for your respond.

Yes, I created many sessions. I am going to check them now. May be I forget to close them.

Btw, there is an occasion where the memory allocated for the threads is freed. This is happened daily, as if, there is a cron job in jboss server.

I said that this is a cron job, and not a normal garbage collection, since "the cleaner" works at certain time everyday. I have read the manual in jboss server and jboss messaging, but don't find anything about this.

Do you know something about it? I think if I can configure "the cleaner" to run every 12 hours, my application will work fine.

Thank you in advance.

Best Regards,
Hendra