OutOfMemoryError: unable to create new native thread

[gquintana]

Hello,

I get an exception java.lang.OutOfMemoryError: unable
to create new native thread under heavy conditions. I tried to increase Java memory size with -Xmx1024M option but it didn't help.

Here is the stack trace:

2006-02-21 10:22:34,272 INFO [STDOUT] java.lang.OutOfMemoryError: unable
to create new native thread
2006-02-21 10:22:34,273 INFO [STDOUT] at java.lang.Thread.start(Native
Method)
2006-02-21 10:22:34,273 INFO [STDOUT] at
sun.rmi.transport.tcp.TCPChannel.free(TCPChannel.java:322)
2006-02-21 10:22:34,273 INFO [STDOUT] at
sun.rmi.server.UnicastRef.free(UnicastRef.java:402)
2006-02-21 10:22:34,273 INFO [STDOUT] at
sun.rmi.server.UnicastRef.done(UnicastRef.java:419)
2006-02-21 10:22:34,273 INFO [STDOUT] at
sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
2006-02-21 10:22:34,273 INFO [STDOUT] at
sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:328)
2006-02-21 10:22:34,273 INFO [STDOUT] at
sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:144)
2006-02-21 10:22:34,273 INFO [STDOUT] at
sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:539)
2006-02-21 10:22:34,273 INFO [STDOUT] at
java.lang.Thread.run(Thread.java:534)
2006-02-21 10:22:37,097 ERROR [org.jboss.ejb.plugins.LogInterceptor]
Unexpected Error in method: public abstract void
fr.gtietransport.myrabel.serveur.gestionUtilisateurs.bean.UtilisateursFacade.signalerPresence(fr.gtietransport.myrabel.serveur.gestionUtilisateurs.bo.UtilisateurBo)
 throws
fr.gtietransport.lugdunum.serveur.biz.BusinessException,java.rmi.RemoteException
java.lang.OutOfMemoryError: unable to create new native thread
 at java.lang.Thread.start(Native Method)
 at java.util.Timer.<init>(Timer.java:95)
 at org.jboss.ejb.txtimer.TimerImpl.startInTx(TimerImpl.java:328)
 at org.jboss.ejb.txtimer.TimerImpl.startTimer(TimerImpl.java:108)
 at
org.jboss.ejb.txtimer.TimerServiceImpl.createTimer(TimerServiceImpl.java:207)
 at
org.jboss.ejb.txtimer.TimerServiceImpl.createTimer(TimerServiceImpl.java:121)
 at
org.jboss.ejb.StatelessSessionEnterpriseContext$TimerServiceWrapper.createTimer(StatelessSessionEnterpriseContext.java:255)
 at
fr.gtietransport.myrabel.serveur.gestionUtilisateurs.bean.UtilisateursFacadeBean.signalerPresence(UtilisateursFacadeBean.java:168)
 at sun.reflect.GeneratedMethodAccessor464.invoke(Unknown Source)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:324)
 at org.jboss.invocation.Invocation.performCall(Invocation.java:345)
 at
org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:214)

I am using:

• 
  
• JBoss 4.0.1SP1
  
• Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_08-b03)
  
• Red Hat Enterprise Linux 3 (Kernel 2.4.21-32.ELsmp)
  
  
  Is it the bug JBAS-1369 described in Jira (http://www.jboss.com/index.html?module=bb&op=viewtopic&t=65857)?
  
  I didn't do the LD_ASSUME_KERNEL thing described in http://wiki.jboss.org/wiki/Wiki.jsp?page=LinuxThreadingModel, can it be related to my problem?
  
  Thanks for your help,
  Cheers,
  Gérald

[starksm64]

http://wiki.jboss.org/wiki/Wiki.jsp?page=OutOfMemoryExceptions

[gquintana]

I am experimenting the LD_ASSUME_KERNEL thing since yesterday, by now I didn't get an OutOfMemoryException.

However, after one day in production, I see in the jboss.system:type=ServerInfo bean that the ActiveThreadCount over 1400! Is it normal? Is it related to the TimerService.createTimer call?

I will try the -XX:MaxPermSize=128m thing as soon as I can.

[dimitris]

You should be able to see the running threads through the jmx console

jboss.system:type=ServerInfo --> listThreadDump

In the current implementation, whenever you TimerService.createTimer(), this results internally in a new java.util.Timer object which runs in each own thread.

[gquintana]

With the listThreadDump operation I see that the majority of threads are under the Thread Group called RMI Runtime.

As you can see it in the stack trace, it is an RMI call that triggers timer creation. Is it related?

[gquintana]

After 2 days in production, the ActiveThreadCount is over 3300! OutOfMemoryException is getting closer!

How can I determine what (which RMI call) is creating so many threads?

[gquintana]

I tried to -Xss128k in the Java VM options as explained in http://wiki.jboss.org/wiki/Wiki.jsp?page=OutOfMemoryExceptionWhenCannotCreateThread but it doesn't help.

I have run JBoss with JProfiler (profiling tool by EJ-Technology) and I saw 100 threads created in an hour. Threads appear then stay in a "Waiting" state and never end nor disappear. I have the feeling (I don't have any proof) that starting a Timer creates a new thread, but canceling the timer doesn't destroy this thread.

What can I do?

[gquintana]

I have identified the portion of code which is creating lots of threads, my application calls it frequently, when I comment it my problem disappears...:

TimerService timerService = getSessionContext().getTimerService();
Iterator timerIter = timerService.getTimers().iterator();
while(timerIter.hasNext()) {
 Timer timer = (Timer) timerIter.next();
 if (timer.getInfo().equals(timerInfo)) {
 timer.cancel();
 }
}
timerService.createTimer(120000l, timerInfo);

The aim of this piece of code is to re-schedule a timer. What's wrong with it?

[dimitris]

It doesn't seem wrong.

How many tasks do you schedule/cancle?
What is the timerInfo Serializable you use?
Do you keep somewhere a reference to the Timer?

I'd say you need to profile your application to see why the cancelled Timers are not garbage collected.

[gquintana]

How many tasks do you schedule/cancle?

About 5 timers per minute

What is the timerInfo Serializable you use?

A simple String measuring about 20 characters.

Do you keep somewhere a reference to the Timer?

No, that's why I am searching the timer I want to cancel with that loop.

Is my problem related to the bug http://jira.jboss.com/jira/browse/JBAS-1690?

[gquintana]

I'd say you need to profile your application to see why the cancelled Timers are not garbage collected.

I think it's not a problem of "Timers not being garbage collected", I think it's rather a problem of "Threads not ending" (see previous post threads are stuck in "Waiting" state).

[dimitris]

Yes, it probably is JBAS-1690. You can try the 4.0.0RC1 release and see if that fixes that.

This release has more than 10 fixes related to EJB Timers.

[dimitris]

meant 4.0.4RC1

[gquintana]

I am going to test whether this problem occurs with release 4.0.4RC1 in order to be sure it is the same.

By the way, I can not use a release candidate for production.
Was this bug present in releases earlier that 4.0.1SP1? Is it worth testing release 4.0.1?
Is there a workaround?
Is 4.0.4final scheduled?

Thanks for your help and advice Dimitris

[dimitris]

> Was this bug present in releases earlier that 4.0.1SP1?
Most probably yes

> Is it worth testing release 4.0.1?
No

> Is there a workaround?
Maybe *not* having an active transaction in the method that cancels the timers?

>
Is 4.0.4 final scheduled?
17th/March, give or take