Memory Leak using custom JSP tags on JBoss as7

[tc7]

environment:

JDK1.7.0_01

JBoss 7.0.2.Finale num

Struts v1.3.8

Redhat 5.4 x64

We've been running on good old JBoss 4 (v4.2.3.GA currrently) for as long as I can remember with no issues regarding memory.

However since upgrading to v7.0.2.Final I've noticed an accumulation of org.apache.jasper.runtime.PerThreadTagHandlerPool$PerThreadData instances on each request thread.

With a few clicks through our application, threads accumulate up to a few hundred thread local instances - depending on which request thread(s) are activated.

The result is heap memory blowout. I started using -Xmx512m, then -Xmx1024m, then -Xmx2048m.

With a heap of 2g and after about 1/2 a day's usage with 50-60 users memory began to top out (~95% used) and was not recaimed by garbage collection.

Using jvisualvm I was able to check the number of PerThreadTagHandlerPool instances - in all there were 66000 odd.

In contrast on a machine running the same code base, but JBoss 4 (and for several days), the instance count was around 1400.

After clicking around with a single user on my dev machine I was able to create almost 1400 PerThreadTagHandlerPool instances in a few minutes (several hundred per request thread). The numbers do not appear to reduce over time, on user logout, or on garbage collection. See attachment (note thread number, eg: 8080-1 for request thread 1).

I've tried disabling tag pooling using "-Dorg.apache.jasper.runtime.JspFactoryImpl.USE_POOL=false", but to no avail.

Has anyone else noticed similar behaviour and/or have any suggestions?

[jaikiran]

Try disabling tag pooling in the web subsystem's jsp-configuration:

<subsystem xmlns="urn:jboss:domain:web:1.0" default-virtual-server="default-host">
...
            <configuration>
                <jsp-configuration tag-pooling="false"/>
            </configuration>
...           

[tc7]

Thank you Jaikiran - so far this is looking really good.

Is this likely to impact performance? We make fairly extensive use of custom tags in our application.

Also, any idea whether this will be updated in the documentation (i.e. the documentation currently refers to the Apache doco using java runtime parameters as above).

http://docs.jboss.org/jbossweb/7.0.x/jasper-howto.html

Especially the Hints section (didn't work for me):

Observations

The extent of ThreadLocal elements on completion of request thread processing is limited to:

com.myapp.dal.transaction.Transaction$TransactionThreadData (1 instance),

java.lang.Boolean (1 instance),

java.lang.String (1 instance),

java.lang.ref.SoftReference (2 instances),

java.text.SimpleDateFormat (1 instance),

java.util.ArrayDeque (1 instance),

java.util.regex.Pattern (1 instance),

org.apache.catalina.core.ApplicationContext$DispatchData (1 instance),

org.apache.jasper.runtime.JspFactoryImpl$PageContextPool (1 instance),

org.jboss.logmanager.NDC$Stack (1 instance)

Which looks much better than before, eg:

...

org.apache.jasper.runtime.PerThreadTagHandlerPool$PerThreadData (156 instances),

[tc7]

We've been running with this configuration for a week or so now - and glad to say the memory leak workaround does the trick.

I'll mark the original question as "answered" (or workaround given!).

thanks again Jaikiran.

[treacledog]

I can confirm that this fix solved a similar issue on our system too. We're running AS 7.1.0.Final.

More evidence is that the offending class has been marked as deprecated, for causing memory leaks. http://tomcat.apache.org/tomcat-7.0-doc/api/org/apache/jasper/runtime/PerThreadTagHandlerPool.html

One thing to note is that we had to delete the tmp directory so that the jsps are recompiled without the memory leak. Just changing the configuration didn't seem to be enough to get the jsps recompiled.

[martin.ball]

I have also been investigating this and the underlying issue is in the reuse() method in PerThreadTagHandlerPool, which does not call instanceManager.destroyInstance(handler). The instance manager which is implemented by org.jboss.as.web.deployment.WebInjectionContainer contains a map which holds references to the tag instances and as these are never removed by reuse() the map just grows over time. I have tested a patched version where the reuse method  calls instanceManager.destroyInstance(handler) and this does appear to fix the issue. I am going to raise an issue against jbossweb as the issue still appears to be there in trunk.

I also saw the deprecation in the tomcat 7 version of PerThreadTagHandlerPool which states "Use of ThreadLocals is likely to trigger memory leaks"  and having come across an issue in the apache bugzilla (https://issues.apache.org/bugzilla/show_bug.cgi?id=43790) where there is discussion around use of thread locals and memory leaks and Mark Thomas states "The PerThreadTagHandlerPool needs to be deprecated" and then later does so (see http://svn.apache.org/viewvc?view=revision&revision=1137815) it looks like the deprecation is to do with the use of threadlocal rather than this issue. This does not change the fact that there is a leak because of the lack of a call to instanceManager.destroyInstance(handler), it's just worth knowing that it might not be the reason the class was deprecated.

It is also worth looking at the commit history of TagHandlerPool.java (http://anonsvn.jboss.org/repos/jbossweb/tags/JBOSSWEB_7_0_16_FINAL/java/org/apache/jasper/runtime/TagHandlerPool.java), we can see that it did once call instanceManager.destroyInstance(handler),  but it was removed in revision 1697 with the commit message "I don't understand why reuse calls predestroy (get does not call post construct if the tag is not instantiated). Drop that." So it looks like this class would also have this issue if it was used it instead of the PerThreadTagHandlerPool unless this change were reverted.

[martin.ball]

Raised issue JBWEB-242.

[timba]

  Hi there,

  None of the workaround listed here worked for me!

  I tried:

1. update jbossweb to 7.0.17
2. disable the tag-pooling
3. disable (default) and enable org.apache.jasper.Constants.USE_INSTANCE_MANAGER_FOR_TAGS
4. increase to 100 org.apache.jasper.Constants.MAX_POOL_SIZE
5. disable org.apache.jasper.runtime.JspFactoryImpl.USE_POOL

  I keep getting the count of org.apache.jasper.runtime.PerThreadTagHandlerPool$PerThreadData very high, in a few minutes, less than 5, I can reach 50K instances, and even after a while or after several runs of GC the number does not decrease.

  I'm able to reproduce this in my local machine and I have almost 100% sure that is what is causing my production servers to exhaustion.

  I'm running JBoss 7.1.1 (with jbossweb-7.0.13) and vanilla Liferay 6.1.20 on top of it.

   Any hints?

[martin.ball]

Have you tried the running the tagLeakTest war which is attached to JBWEB-242? If you run it on your on jboss 7.1.1 with jbossweb-7.0.17 you should see that the leak does not occur.

It could be that you are suffering from https://issues.jboss.org/browse/JBWEB-269 - "Memory Leak when Custom Tags Throw Runtime Exception". Which is a similar issue to JBWEB-242, but only occurs if you have tags that are throwing exceptions.

Hope that's helpful.

[timba]

  Hi Martin,

  There was a misunderstood from my side regarding each class was leaking. After reading other bug reports I verified that the classes that are leaking are implementation of interface javax.servlet.jsp.tagext.Tag.

[rwolosker]

O problema é que essa configuração de desabilitar o pool das tags não está funcionando. Existe um BUG na versão 7.1.1 final

<configuration>
<jsp-configuration development="true" tag-pooling="false"/>
</configuration>

Esta configuração não adianta nada, ele não está lendo.

  private void tagPooling(ServletContext servletContext){
    try{
      Context context=application.getInitialContext().copy();
      AppReflection reflection=application.getBean("appReflection",context);
      Object object;
      
      object=reflection.get(servletContext,"context");
      object=reflection.get(object,"context");
      object=reflection.get(object,"children");
      object=((ConcurrentHashMap<Object,Object>)object).get("jsp");
      object=reflection.get(object,"parameters");
      ((HashMap<Object,Object>)object).put("enablePooling","false");
    }
    catch(Exception e){
      throw new RuntimeException(e);
    }
  }

Para desabilitar o tag-pooling definitivamente, é necessário fazer uma intervenção com uma classe que implemente ServletContextListener

Eu já tinha reportado antes este problema.