[f28b1edf]

... and another one

java.lang.Throwable: STACK TRACE OF ACTIVE THREAD IN TERMINATING TRANSACTION
 at java.lang.Object.wait(Native Method)
 at org.jboss.ejb.plugins.lock.NonReentrantLock.acquireReentrant(NonReentrantLock.java:159)
 at org.jboss.ejb.plugins.lock.NonReentrantLock.attempt(NonReentrantLock.java:182)
 at org.jboss.ejb.plugins.EntityReentranceInterceptor.invoke(EntityReentranceInterceptor.java:94)
 at org.jboss.ejb.plugins.EntityInstanceInterceptor.invoke(EntityInstanceInterceptor.java:278)
 at org.jboss.ejb.plugins.EntityLockInterceptor.invoke(EntityLockInterceptor.java:104)
 at org.jboss.ejb.plugins.EntityCreationInterceptor.invoke(EntityCreationInterceptor.java:76)
 at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
 at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
 at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
 at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
 at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)
 at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
 at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
 at org.jboss.ejb.EntityContainer.internalInvoke(EntityContainer.java:527)
 at org.jboss.ejb.Container.invoke(Container.java:960)
 at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:430)
 at org.jboss.ejb.plugins.local.EntityProxy.invoke(EntityProxy.java:65)
 at ...

I have to apologize for my first comment, we use both ejb2 and ejb3. This issue relates only to ejb2 in our project (i think).

[f28b1edf]

Hello again, our problem seems to be solved... In our application, there was an entity (value object?) with self reference. There wasn't deadlock, it worked well. But in heavy traffic, it failed with "aborting with 1 threads active", always on the same line of code (thank you, mazz!). Well, i decided that on that heavy requested page we don't need that information (the self reference) and i made light entity without the self reference. Now it seems everything works well, even in heavy traffic. I have some suspicion: https://jira.jboss.org/jira/browse/JBAS-4177 together with lazy evaluation. But i don't understand it well.
This is not a complete solution, but for our project is it enough.
Thank you all for your help and patience :)

[tarekahf]

We're facing similar problem and error messages. This is related to Adobe LiveCycle ES4 Server installation using turnkey setup. It is based on jboss and MySql.

We noticed if the end user clicks the button to render PDF form by merging XML data with xfa XDP template (which eventually invokes the LC render API) , if he clicks the button several times like 14 times in one minute, then after some time, we see tens of Abort of action errors in the log and eventually the jboss server must be restarted.

Any idea if this repatitive action in a short period can cause the server to crash? How we can prevent this?

Tarek

[mmusgrov]

Could you provide more context for your issue. Is it still

WARN : [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id 7f000002:a108:48b40e06:282b06e invoked while multiple threads active within it.

Which versions of the software are you running. The original report dates back to 2008, have you tried with any of the more recent versions?

[tarekahf]

This problem happens when someone starts clicking the button or link and if there is no response, he clicks again, and again until it brings the server down. We can build logic to prevent multiple repetitive clicks but we cannot guarantee that we can cover all cases. We want to try from Adobe LiveCycle ES4/JBoss side if we can do anything to at lease allow the server to resume without the need to restart.

I will provide the following details from boot.log and server.log in C:\...\Adobe LiveCycle ES4\jboss\server\lc_turnkey\log

===

16:57:14,978 INFO  [ServerImpl] Release ID: JBoss [EAP] 5.1.0 (build: SVNTag=JBPAPP_5_1_0 date=201009150028)

16:57:14,978 DEBUG [ServerImpl] Using config: org.jboss.bootstrap.BaseServerConfig@5f989f84

16:57:23,262 DEBUG [BaseClassLoader] Created BaseClassLoader@4d74f02c{vfsfile:/C:/Adobe/Adobe%20LiveCycle%20ES4/jboss/server/lc_turnkey/conf/} with policy VFSClassLoaderPolicy@3844006e{name=vfsfile:/C:/Adobe/Adobe%20LiveCycle%20ES4/jboss/server/lc_turnkey/conf/ domain=null roots=[FileHandler@2021151389[path= context=file:/C:/Adobe/Adobe%20LiveCycle%20ES4/jboss/server/lc_turnkey/conf/ real=file:/C:/Adobe/Adobe%20LiveCycle%20ES4/jboss/server/lc_turnkey/conf/]]  delegates=null exported=[, bindingservice.beans.META-INF, bootstrap, xmdesc, bindingservice.beans.jboss-bindingservice.jar.org.jboss.services.binding.managed, bindingservice.beans.jboss-bindingservice.jar.org.jboss.services.binding.impl, bindingservice.beans.jboss-bindingservice.jar.META-INF, props, bindingservice.beans.jboss-bindingservice.jar.org.jboss.services.binding] <IMPORT-ALL>NON_EMPTY}

16:57:23,262 DEBUG [BaseClassLoaderDomain] ClassLoaderDomain@7f60c4b0{DefaultDomain} registerClassLoader BaseClassLoader@4d74f02c{vfsfile:/C:/Adobe/Adobe%20LiveCycle%20ES4/jboss/server/lc_turnkey/conf/}

16:57:23,277 INFO  [ServerInfo] Java version: 1.6.0_31,Sun Microsystems Inc.

16:57:23,277 INFO  [ServerInfo] Java Runtime: Java(TM) SE Runtime Environment (build 1.6.0_31-b05)

16:57:23,277 INFO  [ServerInfo] Java VM: Java HotSpot(TM) 64-Bit Server VM 20.6-b01,Sun Microsystems Inc.

16:57:23,277 INFO  [ServerInfo] OS-System: Windows Server 2008 R2 6.1,amd64

16:57:23,277 INFO  [ServerInfo] VM arguments: -Xrs -Xms12288m -Xmx24576m -XX:PermSize=512m -XX:MaxPermSize=1024m -XX:+UseCompressedOops -Dadobeidp.serverName=server1 -Dfile.encoding=utf8 -Djava.net.preferIPv4Stack=true -DentityExpansionLimit=10000 -XX:+HeapDumpOnOutOfMemoryError -Dcom.adobe.xmlform.bmc.POOL_MAX=6 -Dcom.adobe.xmlform.bmc.MAXIMUM_REUSE_COUNT=10000 -Dsun.lang.ClassLoader.allowArraySyntax=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dorg.jboss.resolver.warning=true -Dlc.um.csrffilter.disabled=true -Dprogram.name=run.bat -Djava.endorsed.dirs=C:\Adobe\Adobe LiveCycle ES4\jboss\lib\endorsed

===

The following errors in the server.log file indicates that a crash is imminent:

=====

2019-11-15 11:22:49,552 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (Thread-13) [com.arjuna.ats.arjuna.coordinator.TransactionReaper_18] - TransactionReaper::check timeout for TX a640c49:c3ee:5dcc31df:147c90 in state  RUN

2019-11-15 11:22:49,552 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (Thread-14) [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id a640c49:c3ee:5dcc31df:147c90 invoked while multiple threads active within it.

2019-11-15 11:22:49,552 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (Thread-14) [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action a640c49:c3ee:5dcc31df:147c90 aborting with 1 threads active!

2019-11-15 11:22:50,052 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (Thread-13) [com.arjuna.ats.arjuna.coordinator.TransactionReaper_18] - TransactionReaper::check timeout for TX a640c49:c3ee:5dcc31df:147c90 in state  CANCEL

2019-11-15 11:22:50,052 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (Thread-14) [com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator_4] TwoPhaseCoordinator.afterCompletion - returned failure for com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple@7270e204

2019-11-15 16:26:08,899 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (WorkerThread#20[10.100.12.74:48733]) [com.arjuna.ats.arjuna.coordinator.BasicAction_40] - Abort called on already aborted atomic action a640c49:c3ee:5dcc31df:17f177

2019-11-15 16:26:08,899 ERROR [com.adobe.idp.dsc.transaction.impl.ejb.adapter.EjbTransactionBMTAdapterBean] (WorkerThread#20[10.100.12.74:48733]) The current transaction has been marked for rollback.  This means one of three things; 1) This transaction has timed-out (the timeout period was set to [180(sec)]180, while the actual transaction took [188(sec)]), 2) An unhandled exception occurred when calling another service (please check the logs for more detail), or 3) This is a JTA transaction and a service has explicitly marked this transaction for rollback

javax.transaction.RollbackException: [com.arjuna.ats.internal.jta.transaction.arjunacore.inactive] [com.arjuna.ats.internal.jta.transaction.arjunacore.inactive] The transaction is not active!

at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1422)

at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:137)

=====

In the last log, I saw 200+ references to the above errors and the server had to be restarted.

This is a legacy setup and it's hard to make major changes such as upgrade since it's a turnkey installation. We have duplicated the setup to a test environment with all components, and we can try to change the configuration. I think we can play with garbage collection parameters or timeout settings to see if this helps. I need some guidance.

I appreciate your help.

Tarek

[ochaloup]

Hi tarekahf ,

I think you would need to investigate deeper about the application functionality.

As explained above the transaction timeout error means that it was not possible to finish the transactional work in the defined time. After the timeout elapsed, the work is about to be aborted.

The timeouted transaction is usually a side effect. The transactional errors say about some trouble in the system but it's not the root issue.

First, you need to define the transaction timeout in way it fits your application.

As an example, let's assume

- or you know the work managed in the transaction takes a minute but in peaks it's fine to take 3 minutes then you should define the timeout bigger than 3 minutes

- or you know that 3 minutes means some issues and you don't want to burden the system with unfinished work then it should be less than 3 minutes

- or ...

Second, you need to find the reason why the transactional work takes so long. That depends on your workload.

I would assume as a blind shot(!) following - you say the user click at the button and a document is generated from template. There is probably some background job that's started when the button is clicked. As the document generator has probably some capacity and it can't process than X number of jobs concurrently. If the user click 14 times it could be that multiple jobs starts to generate the document. It may knock out the system.

I would consider to check and

- to defend the button against the multiple execution. If the button is clicked it should be verified if the job was already started for the user and only if not then a new job execution (a new transaction) is started

- the application that generates the documents could be more resilient. It should be capable to refuse the jobs which has no capacity for. Multiple jobs should not be knocking the system down.

- there could be added some queue. All the submitted jobs (from the button click) will go to the queue. The generator would then take job by job from the queue and the capacity of the generator won't be overpassed.

- check what the transaction timeout invokes in the application. It should be a call to a XAResource rollback. The XAResource could be defined as the document generator(?, just guessing). Then the rollback functionality could be written in a wrong way and it should be checked if all resources (all the generated document references) are cleaned when rollback happens.

Ondra