13 Replies Latest reply on Jan 20, 2015 7:18 PM by Marc Marc

    Out Of Memory PermGen - ClassLoader Leak

    Marc Marc Newbie

      Every time I deploy, it is increasing the PermGen space by 8mb ~ as shown in picture. I have increased the permgen to 512 but would rather fix it than just prolonging it.

       

      PKTdjLA.png

       

      Running:

       

      AS.7.1.1.Final

      JBPM 5.4.0.Final

      Hibernate with mysql driver

      Spring 4.1.3.Release

       

      Here is a few views from Eclipse MAT of a heapdump (Not the same as the above picture). The last picture is looking at cxf bus without weak references.

       

      ylmqo2u.png

      A4i8IGs.png

      LRN4po4.png

        • 1. Re: Out Of Memory PermGen - ClassLoader Leak
          Tomaz Cerar Master

          AFAIR this was fixed long time ago.

          Try using newer version of app server.

          WildFly 8.2 being the latest.

          1 of 1 people found this helpful
          • 2. Re: Out Of Memory PermGen - ClassLoader Leak
            Marc Marc Newbie

            I do remember reading that, and was first thing to try

             

            Thanks, will let you know how it goes.

            • 3. Re: Out Of Memory PermGen - ClassLoader Leak
              Marc Marc Newbie

              Tried using AS 7.4.0.Final-redhat-19.

               

              Problem still persisted.

              • 4. Re: Out Of Memory PermGen - ClassLoader Leak
                Tomaz Cerar Master

                So EAP 6.3.0.

                Hmm, that is odd, any chance you can provide a app that can reproduce this (can be private)

                Last time I was checking we did leak about 120-200K per redeployment but noting like 8M which is too much.

                • 5. Re: Out Of Memory PermGen - ClassLoader Leak
                  Wolf-Dieter Fink Master

                  It might be application specific. If you have references from other deployments/modules to this application which are not undeployed the JVM might keep references to classes.

                  I've not seen this in EAP6.3 (but not using JBPM and Spring).

                   

                  Is it possible to try Wildfly?

                  If you have a valid EAP subscription you might open a support case for this.

                  • 6. Re: Out Of Memory PermGen - ClassLoader Leak
                    Marc Marc Newbie

                    Sure!

                     

                    I will sort it out tomorrow and send your way !

                    • 7. Re: Out Of Memory PermGen - ClassLoader Leak
                      Marc Marc Newbie

                      Yea, that's what I was thinking that was something to do my implementation.

                       

                      Should be able to give it a try! Will ask supervisor.

                      • 8. Re: Out Of Memory PermGen - ClassLoader Leak
                        Tomaz Cerar Master

                        Problem with your deployment is that you bundle bunch of duplicate jars & duplicate classes that are also part of application server. And some of this then keep memory references and cause your prem gen leak.

                         

                        you should remove most of the jars in your deployment as they are probably there by mistake as result of wrongly configured maven dependencies, for all dependancies that are part of app server and you only need them for compile time you should use <scope>provided</scope in maven.

                         

                        so lets take a look at the list of jars:

                         

                        activation-1.1.jar <-- javax.activation, part of JDK & app server, remove

                        antlr-2.7.6.jar <-- probably wrongly got in as transitive dependancy to hibernate

                        antlr-3.3.jar <-- same story

                        antlr-runtime-3.1.1.jar <-- same story

                        aopalliance-1.0.jar <-- definitly not needed

                        apacheds-all-1.5.5.jar <-- not sure where you got this from if you don't really use need it and is part of some transitive dependancy remove it

                        asm-3.3.jar <-- probably part of hibernate dependency

                        avalon-framework-4.1.3.jar <-- probably not needed

                        bsh-1.3.0.jar <-- transitive dependency as well.

                        btm-2.1.4.jar

                        cglib-nodep-2.2.jar <-- really old dependancy of maven i asume

                        commons-beanutils-1.8.2.jar

                        commons-collections-3.2.1.jar

                        commons-compress-1.0.jar

                        commons-dbcp-1.4.jar

                        commons-digester-2.0.jar

                        commons-exec-1.0.1.jar

                        commons-io-1.4.jar

                        commons-lang-2.4.jar

                        commons-logging-1.1.jar <-- remove, as it will just cause problems as it has known memory leak issues

                        commons-net-2.0.jar

                        commons-pool-1.5.5.jar  <-- if you don't directly use any of the commons-* jars remove them

                        cxf-api-2.4.4.jar <-- your probably should bundling CXF as it is already part of application server. remove it.

                        cxf-common-utilities-2.4.4.jar

                        cxf-rt-bindings-soap-2.4.4.jar

                        cxf-rt-bindings-xml-2.4.4.jar

                        cxf-rt-core-2.4.4.jar

                        cxf-rt-databinding-jaxb-2.4.4.jar

                        cxf-rt-frontend-jaxws-2.4.4.jar

                        cxf-rt-frontend-simple-2.4.4.jar

                        cxf-rt-transports-common-2.4.4.jar

                        cxf-rt-transports-http-2.4.4.jar

                        cxf-rt-ws-addr-2.4.4.jar

                        cxf-tools-common-2.4.4.jar

                        drools-compiler-5.5.0.Final.jar

                        drools-core-5.5.0.Final.jar

                        drools-persistence-jpa-5.5.0.Final.jar

                        ecj-3.5.1.jar <-- eclipse java compliler? really? you this is part of app server as well

                        ehcache-core-2.3.0.jar

                        el-api-1.0.jar <-- old version expression language api, part of app server and can cause problems with memory leaks

                        geronimo-javamail_1.4_spec-1.7.1.jar <-- part of app server, remove

                        hamcrest-core-1.3.jar <-- par of junit, testing framework which you probably don't need at runtime, scope=test

                        hibernate-commons-annotations-4.0.1.Final.jar

                        hibernate-core-4.0.1.Final.jar

                        hibernate-ehcache-4.0.1.Final.jar

                        hibernate-entitymanager-4.0.1.Final.jar <-- all hibernate-* jars shouldn't be here, as they are part of server itself, scope=provided

                        hibernate-jpa-2.0-api-1.0.1.Final.jar <-- jpa API, definitely needs to be scope=provided

                        hibernate-jpa-2.1-api-1.0.0.Final.jar <-- two version of JPA api, good thing anything works, scope=provided

                        hornetq-core-2.2.10.Final.jar <-- part of app server, don't include it as part of deployment

                        javassist-3.14.0-GA.jar <-- definitely shouldn't be here, probably dependency of hibernate,

                        javassist-3.4.GA.jar <-- same as above

                        jaxb-impl-2.1.13.jar <-- remove, scope=provided, this is part of jdk & app server

                        jaxb-xjc-2.1.13.jar <-- part of app server, scope=provided

                        jboss-el-1.0_02.CR2.jar <-- another EL jar? remove

                        jboss-logging-3.1.0.CR2.jar <-- scope=provided

                        jboss-transaction-api_1.1_spec-1.0.0.Final.jar part of app server, scope=provided

                        jbpm-bam-5.4.0.Final.jar

                        jbpm-bpmn2-5.4.0.Final.jar

                        jbpm-flow-5.4.0.Final.jar

                        jbpm-flow-builder-5.4.0.Final.jar

                        jbpm-human-task-core-5.4.0.Final.jar

                        jbpm-human-task-hornetq-5.4.0.Final.jar

                        jbpm-persistence-jpa-5.4.0.Final.jar

                        jbpm-workitems-5.4.0.Final.jar

                        jdom-1.0.jar <-- really? probably dep of hibernate

                        jna-4.1.0.jar <-- not sure if you need it but keep if you do

                        jna-platform-4.1.0.jar

                        jndi-1.2.1.jar <-- what is this? jndi is part of jdk probably shouldn't be part of runtime, scope=provided

                        jsch.agentproxy.connector-factory-0.0.7.jar

                        jsch.agentproxy.core-0.0.7.jar

                        jsch.agentproxy.pageant-0.0.7.jar

                        jsch.agentproxy.sshagent-0.0.7.jar

                        jsch.agentproxy.svnkit-trilead-ssh2-0.0.7.jar

                        jsch.agentproxy.usocket-jna-0.0.7.jar

                        jsch.agentproxy.usocket-nc-0.0.7.jar

                        jstl-1.1.2.jar <-- part of server, remove, scope=provided

                        jta-1.1.jar <-- second jta jar on classpath, remove, scope = provided

                        junit-4.11.jar <-- this is needed for testing, scope = provided

                        knowledge-api-5.5.0.Final.jar

                        knowledge-internal-api-5.5.0.Final.jar

                        log4j-1.2.14.jar <-- log4j? remove, part of app server, scope=provided

                        logkit-1.0.1.jar <-- another logging lib, remove

                        mail-1.4.jar <-- javamail jar again, scope=provided

                        mina-core-2.0.0-RC1.jar

                        mvel2-2.1.3.Final.jar

                        mysql-connector-java-5.1.18.jar <-- jdbc driver doesn't belong into deployment, scope = provided

                        neethi-3.0.2.jar

                        netty-3.2.0.Final.jar

                        persistence-api-1.0.jar <-- third JPA jar on classpath, remove

                        platform-3.4.0.jar

                        protobuf-java-2.4.1.jar

                        rome-1.0.jar

                        sequence-library-1.0.2.jar

                        serializer-2.7.1.jar <-- part of jdk & app server, remove

                        servlet-api-2.5.jar <-- absolutely remove, i would say this is one of bigger reasons for memory leak, scope = provided

                        shared-asn1-0.9.15.jar

                        shared-ldap-0.9.15.jar

                        shared-ldap-constants-0.9.15.jar

                        slf4j-api-1.7.7.jar <-- another logging lib, remove, it is part of app sever, scope = provided

                        slf4j-jdk14-1.7.7.jar <-- same as above

                        slf4j-log4j12-1.7.7.jar <-- same as above

                        smack-3.1.0.jar

                        spring-aop-4.1.3.RELEASE.jar

                        spring-beans-4.1.3.RELEASE.jar

                        spring-context-4.1.3.RELEASE.jar

                        spring-context-support-4.1.3.RELEASE.jar

                        spring-core-4.1.3.RELEASE.jar

                        spring-expression-4.1.3.RELEASE.jar

                        spring-jdbc-4.1.3.RELEASE.jar

                        spring-orm-4.1.3.RELEASE.jar

                        spring-tx-4.1.3.RELEASE.jar

                        spring-web-4.1.3.RELEASE.jar

                        spring-webmvc-4.1.3.RELEASE.jar

                        sqljet-1.1.10.jar

                        standard-1.1.2.jar <-- another server provided lib, scope = provided

                        stax2-api-3.1.1.jar <-- jdk & server provided, scope = provided

                        stringtemplate-3.2.jar

                        svnkit-1.8.6.jar

                        trilead-ssh2-1.0.0-build217.jar

                        woodstox-core-asl-4.1.1.jar

                        wsdl4j-1.6.2.jar <-- part of server, remove

                        xalan-2.7.1.jar <-- jdk & server, remove

                        xml-apis-1.0.b2.jar <-- same as above

                        xml-resolver-1.2.jar <-- same as above

                        xmlschema-core-2.0.2.jar <-- part of the server

                        xpp3_min-1.1.4c.jar <-- same as above

                        xstream-1.3.1.jar <-- same as above

                         

                         

                        when you do remove all this, your war will also be considerably smaller and faster to deploy.

                         

                        i would recommend you running maven dependency tree  http://maven.apache.org/plugins/maven-dependency-plugin/tree-mojo.html

                        to see from where all you get bunch of this jars brought in as transitive dependancy

                        just run mvn dependency:tree and examine the output.

                         

                        i also recommend reading http://maven.apache.org/plugins/maven-war-plugin/examples/skinny-wars.html

                        1 of 1 people found this helpful
                        • 9. Re: Out Of Memory PermGen - ClassLoader Leak
                          Marc Marc Newbie

                          Hey Tomaz,

                           

                          Thanks for going through all that!

                           

                          Reduced the size considerably. However, it did not stop the class loader leak.

                           

                          Therefore, decided to see what would happen if I did not deploy my app, and instead redeployed drools. The same issue was apparent as shown in the bump on the top right graph.

                           

                          BT7KDdZ.png

                           

                          7eooc5y.png

                          LSyLiuQ.png

                          pHVnLPt.png

                          • 10. Re: Out Of Memory PermGen - ClassLoader Leak
                            Tomaz Cerar Master

                            DeploymentUnitImpl would be the result of memory leak.

                             

                            but, can you take a look what is holding references to DeploymentUnitImpl as this will show why it didn't get cleaned up.

                            • 11. Re: Out Of Memory PermGen - ClassLoader Leak
                              Marc Marc Newbie

                              Hey Tomaz,

                               

                              Sorry about the late reply was a long weekend here

                               

                              It seems that the same object type is responsible.

                               

                              WUx2zLk.png

                              • 12. Re: Out Of Memory PermGen - ClassLoader Leak
                                Tomaz Cerar Master

                                This would indicate leak in InitFacesContext that has reference in ThreadLocal aka the most fun mem leaks to debug

                                • 13. Re: Out Of Memory PermGen - ClassLoader Leak
                                  Marc Marc Newbie

                                  Hey Tomaz,

                                   

                                  How would you suggest I approach this?