Auto-calculation of baselines failure

[rafaelcba]

Hello all!

I observed on JON Server's log that always it tries make Auto-calculations Job the process failure. Sounds like the transactions is aborted during the process...

010-04-19 03:01:04,174 INFO  [org.rhq.enterprise.server.measurement.MeasurementBaselineManagerBean] Calculating auto baselines
2010-04-19 03:01:04,175 INFO  [org.rhq.enterprise.server.measurement.MeasurementBaselineManagerBean] Deleting baselines computations older than Fri Apr 16 03:01:04 GMT-03:00 2010
2010-04-19 03:01:04,175 INFO  [org.rhq.enterprise.server.measurement.MeasurementBaselineManagerBean] Inserting new baselines using last 7 days of 1H data
2010-04-19 03:01:04,203 INFO  [org.rhq.enterprise.server.measurement.MeasurementBaselineManagerBean] Removed [0] old baselines - they will now be recalculated (28)ms
2010-04-19 03:12:59,451 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id -53e2fdfb:be4e:4bcbcbf6:1eaff invoked while multiple threads active within it.
2010-04-19 03:12:59,454 INFO  [org.rhq.enterprise.server.common.TransactionInterruptInterceptor] Transaction [-53e2fdfb:be4e:4bcbcbf6:1eaff] is aborting with active thread [RHQScheduler_Worker-4]. interrupting=[false]
java.lang.Throwable: STACK TRACE OF ACTIVE THREAD IN TERMINATING TRANSACTION
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:135)
    at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:104)
    at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
    at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:259)
...
action -53e2fdfb:be4e:4bcbcbf6:1eaff
2010-04-19 03:17:52,996 ERROR [org.rhq.enterprise.server.measurement.MeasurementBaselineManagerBean] Failed to auto-calculate baselines
java.lang.IllegalStateException: [com.arjuna.ats.internal.jta.transaction.arjunacore.inactive] [com.arjuna.ats.internal.jta.transaction.arjunacore.inactive] The transaction is not active!
    at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1379)
    at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:135)
    at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:87)
    at org.jboss.aspects.tx.TxPolicy.endTransaction(TxPolicy.java:175)
...

I attached a piece of Server Log with the full StackTrace.

I suspect that may be some type of timeout on Transaction.

Thanks!

[mazz]

Looking at the time stamps of some of those log messages - 03:01:04  - 03:12:59 - the calculation took longer than 10 minutes.

First, what version are you running? I believe we made some changes in this area of autocalculation of baselines that fixed this kind of thing.

Second, is it possible your Postgres box just doesn't have enough resources (CPU, RAM) to support the size of your RHQ inventory? How many resources do you have in inventory right now?

[rafaelcba]

I'm using:

JON Server env.:

JON version: 2.3.1.GA
build number: 10566

Java Sun HotSpot 1.6.0_14-b08

OS: Debian 5

4 CPUs

16 GB RAM

Postgres env.:

PostgreSQL 8.3.7

$ uname -a
Linux localhost.localdomain 2.6.18-164.el5xen #1 SMP Tue Aug 18 15:59:52 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

About the Postgres server resources I saw that it the mem. is not sufficient

top - 11:43:21 up 12:24,  1 user,  load average: 0.00, 0.00, 0.02
Tasks: 107 total,   2 running, 105 sleeping,   0 stopped,   0 zombie
Cpu0  :  3.9%us,  0.8%sy,  0.0%ni, 93.6%id,  0.8%wa,  0.0%hi,  0.3%si,  0.5%st
Cpu1  :  0.2%us,  0.0%sy,  0.0%ni, 99.6%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   4194304k total,  4103068k used,    91236k free,    34356k buffers

Swap:  2096472k total,     2232k used,  2094240k free,  3732432k cached

But we follow the Postgres tuning recomendation [1].

Today our inventory has:
Platforms ( 29 ) | Servers ( 340 )  | Services ( 5864 )  | Compatible Groups ( 20 )  | Mixed Groups ( 0 )  | Group Definitions ( 8 )

We'll try scale up the Postgres Server.

I saw that JTA's timeout for Jon Server is 600ms. Is there any problem to increase this value?

_________________

[1] http://www.rhq-project.org/display/JOPR2/PostgreSQL#PostgreSQL-PostgreSQLPreparation

[rafaelcba]

About this issue...

I decided to grow the Transaction timeout on JON Server. After this change sounds like the problem was resolved!

I've changed on $JON_HOME/jbossas/server/default/conf/jboss-service.xml

<!-- JBoss Transactions JTA -->
   <mbean code="com.arjuna.ats.jbossatx.jta.TransactionManagerService"
      name="jboss:service=TransactionManager">
      <attribute name="TransactionTimeout">1200</attribute> <!-- chaged -->
      <!-- attribute name="TransactionTimeout">600</attribute --> <!-- RHQ -->
      <attribute name="ObjectStoreDir">${jboss.server.data.dir}/tx-object-store</attribute>
      <attribute name="StatisticsEnabled">true</attribute>
   </mbean>

Regards!