1 2 Previous Next 17 Replies Latest reply on Jul 8, 2009 2:03 PM by Pratim Mukherjee

    JBoss slowing down

    Pratim Mukherjee Newbie

      We are using JBM 1.4.2 GA SP1 with JBoss 4.2 GA. We are hosting our jboss servers on Solaris 10 (4 processors /8 GB memory). We do not persist messages more than 30-40 at a time. We also use JMS transactions for message rollback within session beans. Some of the message consumers are local MDBs and some of them are remote custom JMS clients. The remote clients are beyond a firewall and we have to refresh the connections between the client and JBM every 30 minutes.
      We are noticing that the JBoss server is gradually slowing down as more messages are processed and after about 3-4 weeks it tries to consume more than 100% of the cpu resources and crashes. We have checked with JProbe that there are no memory leak and also tuned the JVM with these parameters -Xms2048m -Xmx2048m -server -Dsun.rmi.dgc.client.gcInterval=1800000
      -Dsun.rmi.dgc.server.gcInterval=1800000 -XX:MaxPermSize=768m -XX:+UseParallelGC -X
      X:ParallelGCThreads=4 -XX:NewRatio=1 -XX:SurvivorRatio=12 -XX:+UseLargePages -verb
      ose:gc -XX:+PrintGCDetails"

      We also see this kind of messages before crash
      javax.jms.JMSException: Could not create a session: org.jboss.resource.JBossResourceExcept
      ion: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (java.lang.IllegalStateException: Transaction TransactionImple < ac,
      BasicAction: ad7d148:b3f9:49ff9638:c7b9 status: ActionStatus.ABORTED > is not active STATUS_ROLLEDBACK)
      2009-05-13 05:08:51,229 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - second pass <Wed, 13 May 2009 05:08:51>
      2009-05-13 05:08:51,367 ERROR [org.jboss.ejb.plugins.LogInterceptor] TransactionRolledbackLocalException in method: public abstract java.lang.String ebond.co
      mmon.XMLCommunicationHandlerLocal.putToWaitingQueue(java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String), causedBy:
      java.lang.IllegalStateException: [com.arjuna.ats.internal.jta.transaction.arjunacore.inactive] [com.arjuna.ats.internal.jta.transaction.arjunacore.inactive]
      The transaction is not active!

      Any help would be appreciated.

        • 1. Re: JBoss slowing down
          Pratim Mukherjee Newbie

          forgot to mention that we use two Jboss servers in cluster and JBoss messaging database in Oracle 10g.

          • 2. Re: JBoss slowing down
            Clebert Suconic Master

            You should produce a Stack trace of your server when things are slowing down.

            Add -XX:+PrintClassHistogram to your VM Options. and call kill -3 when the process is getting slow (before it crashed).

            -XX:+PrintClassHistogram or kill -3 wouldn't hurt your production system. But if you can replicate this on a test environment it's better.

            This link will provide you some extra information:


            http://www.jboss.org/community/wiki/StackTrace

            You are probably running out of memory. I used to work with Solaris years ago, and running out of memory would certainly crash the VM on Solaris.

            • 3. Re: JBoss slowing down
              Clebert Suconic Master

               

              "pratimkm" wrote:
              forgot to mention that we use two Jboss servers in cluster and JBoss messaging database in Oracle 10g.



              Right...
              The ClassHistogram would be able to tell you where your memory is going to.

              • 4. Re: JBoss slowing down
                Pratim Mukherjee Newbie

                thanks for the suggestion,let me try that.

                • 5. Re: JBoss slowing down
                  Pratim Mukherjee Newbie

                  Clebert,

                  I tried with the jmap -histo option which gives me the below results while the production server is running on a normal day. It doesn't look like anything consuming a huge memory compared to the allocated 2GB memory. But our server has become slow already. It takes almost 10 minutes to shut down,however starting the server again is normal. We have found out that after few days if we bring down one of the jboss in cluster and try to start again it cannot join the cluster. or the running one doesn't recognize the one being started.

                  Size Count Class description
                  -------------------------------------------------------
                  16301352 159950 char[]
                  14233496 15189 int[]
                  11675784 92177 java.lang.Object[]
                  11550160 18668 byte[]
                  11352408 90303 * ConstMethodKlass
                  6506832 90303 * MethodKlass
                  5362616 128417 * SymbolKlass
                  4816200 8454 * ConstantPoolKlass
                  4815816 200659 java.lang.String
                  3883456 121358 java.util.TreeMap$Entry
                  3808368 49192 short[]
                  3568064 8454 * InstanceKlassKlass
                  2897120 7281 * ConstantPoolCacheKlass
                  1658216 17595 java.util.HashMap$Entry[]
                  1488640 18608 java.lang.reflect.Method
                  1215312 50638 java.util.HashMap$Entry
                  1094040 27351 java.util.TreeMap

                  • 6. Re: JBoss slowing down
                    Clebert Suconic Master

                    At this point It's impossible to say what it is. You need to keep looking down the list.. for JBoss/JGroups related objects.

                    It could be anything.. from JGroups retransmissions because of the network to real leakages.

                    Also.. look at the stack traces on the dump itself.

                    • 7. Re: JBoss slowing down
                      Clebert Suconic Master

                      Anything.. I mean... *anything*.


                      you can't even say it is a JBM until you fully analyze the histogram.

                      • 8. Re: JBoss slowing down
                        Pratim Mukherjee Newbie

                        Thanks for replying. Let us analyze more on the stack trace and the histogram and will update you.

                        • 9. Re: JBoss slowing down
                          Pratim Mukherjee Newbie

                          Clebert,

                          We did the stack trace and found out that the number of WorkerThread for the clients beyond a firewall are steadily increasing by 1 or 2 everyday.It's shwoing up on jconsole as WorkerThread#17[10.**.2*.1**:34128] or similar.When it's reaching about 30-35 it's bringing down the server on it's knees.I was trying to find out if we close the connections properly after each 30 mins refresh period that I mentioned in the first post. We found that the code is correct.
                          Just for curiosity I tried to find out what happens with standalone clients. So we made a simple standalone client(non-firewall) and connected to jboss and did a stack trace through jconsole while doing load testing. We found out that the worker threads remain even after the client disconnects. Is it normal ?

                          • 10. Re: JBoss slowing down
                            Pratim Mukherjee Newbie

                            Our JBoss server in production became very slow so we had to bounce it. But we were able to get some important information. We found this kind of messages in our log file
                            May 29, 01:25:41 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id ad7d147:95eb:4a1f69f5:7f4 invoked while multiple threads active within it.
                            May 29, 01:25:41 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action ad7d147:95eb:4a1f69f5:7f4 aborting with 1 threads active!

                            Also we found out that "JMS SessionPool " thread went very high along with the Worker Thread. Please help.

                            Thanks,
                            Pratim

                            • 11. Re: JBoss slowing down
                              Tim Fox Master

                              Perhaps you are creating a huge number of sessions in your code?

                              • 12. Re: JBoss slowing down
                                Yong Hao Gao Master

                                Can I ask what Rmoting version are you using? I ask this because I saw you mentioned the ServerThreads were leaking, which might be a remoting version issue.

                                • 13. Re: JBoss slowing down
                                  Pratim Mukherjee Newbie

                                  Tim,

                                  The number of client session created by remote clients are constant at 9 as they use a connection pool. However we have some local components on the jboss server which opens and closes connections while they put messages to the queues.We are using the java:/JmsXA connection factory. Should we be using the connection pooling mecahnism for local components as well?

                                  • 14. Re: JBoss slowing down
                                    Pratim Mukherjee Newbie

                                    Howard,
                                    Nice to see your reply again. As you suggested earlier we are using the jboss-remoting version 2.2.2.SP11

                                    Thanks,
                                    Pratim

                                    1 2 Previous Next