3 Replies Latest reply on Jun 16, 2005 2:12 PM by Bob OMalley

    Big Memory Leak when Clustering

    Bob OMalley Newbie

      I've set up clustering in JBoss 4.0.1 per the documents and we have tested successfully in staging. The day we first tried this in production, a memory leak occurred and crashed the system within 8 minutes ! The original release used JDK 1.5.0_02.

      I built the environment by using the "default" server deployment and added Clustering by adding to it jbossha-httpsession.sar, tc5-cluster-service.xml, httpha-invoker.sar jbossha.jar and cluster-service.xml files in the deploy folder and changing configuration files as documented.

      We are only needing the HTTPSession clustering. Do I really need all the other stuff listed above ?

      To get over the memory leak, I first rolled back to JDK 1.4, which seemed to slow the leak down to 1 hour instead of 8 minutes. Then I tried removing the reliance on farm deployment which had no affect. Finally I removed the clustering modules (and replaced httpha-invoker with http-invoker). That relieved the problem -- no more fast leak (although over the course of a week we will leak slowly -- perhaps thats our own application).

      I do believe some of our java modules that are added to httpSession are not serializable while most are. The occurance of the "not serializable" error is minimal when clustering is turned on.

      Has anyone else had this problem ? Any ideas ?

        • 1. Re: Big Memory Leak when Clustering
          Bob OMalley Newbie

          forgot to mention that to deploy clustered I also included additional jar files in the server/deployment/lib folder: jboss-cache.jar, jgroups.jar and jboss-remoting.jar.

          So, if Clustering now uses jgroups, does that mean it is not using a lick of ejb under the covers? If our application has no EJBs, do I really need the httpha-invoker then or does jgroups use that ?

          • 2. Re: Big Memory Leak when Clustering
            Ben Wang Master

            You can safely remove httpha if you want. I'd recommend you try out 4.0.2 as there have fixes for http session replication.

            In terms of memory leak though, it can cause by the frequent exceptions handling whereas the exceptions can come from your "non-serialzable" objects or from internal ones.

            BTW, if your attributes are not serializable, you should not put them into http session.

            -Ben

            • 3. Re: Big Memory Leak when Clustering
              Bob OMalley Newbie

              Thank you for your response!

              Alright, I will make all the objects placed on httpSession serializable. Coming from a commercial app server environment, we were spoiled by it's ability to simply igore these objects on the session, not forwarding them to other cluster member nodes.

              So, if we are throwing errors in production that might go to the Console, running in Windows 2003 as a Service, and redirecting the "out" and "err" to a file, are we accumulating memory simply because of the console output ? If so is there a way to flush this periodically?

              Or, is it that errors thrown up this high into the console have components that get added to some collection and therefore won't get garbage collected ?

              Also, not that it should be our first line of defence, but to prevent excessive console messages, is there a way to catch the errors thrown from httpSession Clustering ?

              Bob