3 Replies Latest reply on Apr 30, 2010 3:51 PM by Rafael Soares (Tuelho)

    Auto-calculation of baselines failure

    Rafael Soares (Tuelho) Novice

      Hello all!

       

      I observed on JON Server's log that always it tries make Auto-calculations Job the process failure. Sounds like the transactions is aborted during the process...

       

      010-04-19 03:01:04,174 INFO  [org.rhq.enterprise.server.measurement.MeasurementBaselineManagerBean] Calculating auto baselines
      2010-04-19 03:01:04,175 INFO  [org.rhq.enterprise.server.measurement.MeasurementBaselineManagerBean] Deleting baselines computations older than Fri Apr 16 03:01:04 GMT-03:00 2010
      2010-04-19 03:01:04,175 INFO  [org.rhq.enterprise.server.measurement.MeasurementBaselineManagerBean] Inserting new baselines using last 7 days of 1H data
      2010-04-19 03:01:04,203 INFO  [org.rhq.enterprise.server.measurement.MeasurementBaselineManagerBean] Removed [0] old baselines - they will now be recalculated (28)ms
      2010-04-19 03:12:59,451 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id -53e2fdfb:be4e:4bcbcbf6:1eaff invoked while multiple threads active within it.
      2010-04-19 03:12:59,454 INFO  [org.rhq.enterprise.server.common.TransactionInterruptInterceptor] Transaction [-53e2fdfb:be4e:4bcbcbf6:1eaff] is aborting with active thread [RHQScheduler_Worker-4]. interrupting=[false]
      java.lang.Throwable: STACK TRACE OF ACTIVE THREAD IN TERMINATING TRANSACTION
          at java.net.SocketInputStream.socketRead0(Native Method)
          at java.net.SocketInputStream.read(SocketInputStream.java:129)
          at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:135)
          at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:104)
          at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
          at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:259)

      ...


      action -53e2fdfb:be4e:4bcbcbf6:1eaff
      2010-04-19 03:17:52,996 ERROR [org.rhq.enterprise.server.measurement.MeasurementBaselineManagerBean] Failed to auto-calculate baselines
      java.lang.IllegalStateException: [com.arjuna.ats.internal.jta.transaction.arjunacore.inactive] [com.arjuna.ats.internal.jta.transaction.arjunacore.inactive] The transaction is not active!
          at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1379)
          at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:135)
          at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:87)
          at org.jboss.aspects.tx.TxPolicy.endTransaction(TxPolicy.java:175)


      ...

       

      I attached a piece of Server Log with the full StackTrace.

       

      I suspect that may be some type of timeout on Transaction.

       

      Thanks!

        • 1. Re: Auto-calculation of baselines failure
          mazz Master

          Looking at the time stamps of some of those log messages - 03:01:04  - 03:12:59 - the calculation took longer than 10 minutes.

           

          First, what version are you running? I believe we made some changes in this area of autocalculation of baselines that fixed this kind of thing.

           

          Second, is it possible your Postgres box just doesn't have enough resources (CPU, RAM) to support the size of your RHQ inventory? How many resources do you have in inventory right now?

          • 2. Re: Auto-calculation of baselines failure
            Rafael Soares (Tuelho) Novice

            I'm using:


            JON Server env.:

            JON version: 2.3.1.GA
            build number: 10566


            Java Sun HotSpot 1.6.0_14-b08

            OS: Debian 5

            4 CPUs

            16 GB RAM


            Postgres env.:

            PostgreSQL 8.3.7

            $ uname -a
            Linux localhost.localdomain 2.6.18-164.el5xen #1 SMP Tue Aug 18 15:59:52 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

             

            About the Postgres server resources I saw that it the mem. is not sufficient


            top - 11:43:21 up 12:24,  1 user,  load average: 0.00, 0.00, 0.02
            Tasks: 107 total,   2 running, 105 sleeping,   0 stopped,   0 zombie
            Cpu0  :  3.9%us,  0.8%sy,  0.0%ni, 93.6%id,  0.8%wa,  0.0%hi,  0.3%si,  0.5%st
            Cpu1  :  0.2%us,  0.0%sy,  0.0%ni, 99.6%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
            Mem:   4194304k total,  4103068k used,    91236k free,    34356k buffers

            Swap:  2096472k total,     2232k used,  2094240k free,  3732432k cached


            But we follow the Postgres tuning recomendation [1].


            Today our inventory has:
            Platforms ( 29 ) | Servers ( 340 )  | Services ( 5864 )  | Compatible Groups ( 20 )  | Mixed Groups ( 0 )  | Group Definitions ( 8 )

             

            We'll try scale up the Postgres Server.

             

            I saw that JTA's timeout for Jon Server is 600ms. Is there any problem to increase this value?

            _________________

            [1] http://www.rhq-project.org/display/JOPR2/PostgreSQL#PostgreSQL-PostgreSQLPreparation

            • 3. Re: Auto-calculation of baselines failure
              Rafael Soares (Tuelho) Novice

              About this issue...

               

              I decided to grow the Transaction timeout on JON Server. After this change sounds like the problem was resolved!

               

              I've changed on $JON_HOME/jbossas/server/default/conf/jboss-service.xml

               

              <!-- JBoss Transactions JTA -->
                 <mbean code="com.arjuna.ats.jbossatx.jta.TransactionManagerService"
                    name="jboss:service=TransactionManager">
                    <attribute name="TransactionTimeout">1200</attribute> <!-- chaged -->
                    <!-- attribute name="TransactionTimeout">600</attribute --> <!-- RHQ -->
                    <attribute name="ObjectStoreDir">${jboss.server.data.dir}/tx-object-store</attribute>
                    <attribute name="StatisticsEnabled">true</attribute>
                 </mbean>

               

              Regards!