TransactionRollBack error!

[mcevikce]

We have a two servers running on cluster. Server that is not receiving the actual request throws the following exception. It seems like this is happening when it is trying to replicate cache entry:


[09-05-2006 10:32:34.868] [53831715] [org.apache.jk.common.ChannelSocket] [INFO ] [TP-Processor2] connection timeout reached
[09-05-2006 10:32:51.108] [53847955] [org.apache.jk.common.ChannelSocket] [INFO ] [TP-Processor4] connection timeout reached
[09-05-2006 10:34:54.735] [53971582] [org.jgroups.blocks.RpcDispatcher] [ERROR] [Thread-11] exception=java.lang.IllegalArgumentException: java.lang.ClassNotFoundException: No ClassLoaders found for: com.bfm.app.viewserver.treedefinition.RptDefCacheInfo
[09-05-2006 10:34:54.781] [53971628] [org.jgroups.blocks.RpcDispatcher] [ERROR] [Thread-11] exception=java.lang.IllegalArgumentException: java.lang.ClassNotFoundException: No ClassLoaders found for: com.bfm.app.viewserver.treedefinition.ReportDefinition
[09-05-2006 10:34:54.831] [53971678] [org.jboss.cache.interceptors.TxInterceptor] [ERROR] [Thread-41] Transaction has a null transaction entry - beforeCompletion() will fail.
[09-05-2006 10:34:54.831] [53971678] [org.jboss.cache.interceptors.PessimisticLockInterceptor] [ERROR] [Thread-41] entry for transaction GlobalTransaction:<172.16.52.31:45290>:340 not found (transaction has possibly already been rolled back)
[09-05-2006 10:34:54.832] [53971679] [org.jboss.cache.interceptors.TxInterceptor] [ERROR] [Thread-41] Commit/rollback failed.
org.jboss.tm.JBossRollbackException: Unable to commit, tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=nycpasv002/12967, BranchQual=, localId=12967] status=STATUS_NO_TRANSACTION; - nested throwable: (java.lang.IllegalStateException: cannot find transaction entry for GlobalTransaction:<172.16.52.31:45290>:340)
at org.jboss.tm.TransactionImpl.commit(TransactionImpl.java:354)
at org.jboss.cache.interceptors.TxInterceptor.handlePessimisticPrepare(TxInterceptor.java:447)
at org.jboss.cache.interceptors.TxInterceptor.handleRemotePrepare(TxInterceptor.java:280)
at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:110)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:67)
at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:181)
at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:4804)
at org.jboss.cache.TreeCache._replicate(TreeCache.java:4366)
at sun.reflect.GeneratedMethodAccessor202.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jgroups.blocks.MethodCall.invoke(MethodCall.java:286)
at org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:236)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:618)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:515)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:326)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUp(MessageDispatcher.java:734)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.access$300(MessageDispatcher.java:566)
at org.jgroups.blocks.MessageDispatcher$1.run(MessageDispatcher.java:703)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.IllegalStateException: cannot find transaction entry for GlobalTransaction:<172.16.52.31:45290>:340
at org.jboss.cache.interceptors.TxInterceptor$RemoteSynchronizationHandler.beforeCompletion(TxInterceptor.java:921)
at org.jboss.cache.interceptors.OrderedSynchronizationHandler.beforeCompletion(OrderedSynchronizationHandler.java:72)
at org.jboss.tm.TransactionImpl.doBeforeCompletion(TransactionImpl.java:1473)
at org.jboss.tm.TransactionImpl.beforePrepare(TransactionImpl.java:1092)
at org.jboss.tm.TransactionImpl.commit(TransactionImpl.java:306)
... 19 more
[09-05-2006 10:34:54.833] [53971680] [org.jboss.cache.interceptors.TxInterceptor] [INFO ] [Thread-41] Attempting anotehr rollback
[09-05-2006 10:34:54.833] [53971680] [org.jboss.cache.interceptors.TxInterceptor] [ERROR] [Thread-41] Unable to rollback
java.lang.IllegalStateException: Transaction has terminated TransactionImpl:XidImpl[FormatId=257, GlobalId=nycpasv002/12967, BranchQual=, localId=12967]
at org.jboss.tm.TransactionImpl.lock(TransactionImpl.java:816)
at org.jboss.tm.TransactionImpl.rollback(TransactionImpl.java:377)
at org.jboss.cache.interceptors.TxInterceptor.handlePessimisticPrepare(TxInterceptor.java:459)
at org.jboss.cache.interceptors.TxInterceptor.handleRemotePrepare(TxInterceptor.java:280)
at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:110)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:67)
at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:181)
at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:4804)
at org.jboss.cache.TreeCache._replicate(TreeCache.java:4366)
at sun.reflect.GeneratedMethodAccessor202.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jgroups.blocks.MethodCall.invoke(MethodCall.java:286)
at org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:236)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:618)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:515)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:326)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUp(MessageDispatcher.java:734)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.access$300(MessageDispatcher.java:566)
at org.jgroups.blocks.MessageDispatcher$1.run(MessageDispatcher.java:703)
at java.lang.Thread.run(Thread.java:595)
[09-05-2006 10:34:55.125] [53971972] [org.jboss.cache.interceptors.TxInterceptor] [ERROR] [Thread-41] Transaction has a null transaction entry - beforeCompletion() will fail.
[09-05-2006 10:34:55.125] [53971972] [org.jboss.cache.interceptors.PessimisticLockInterceptor] [ERROR] [Thread-41] entry for transaction GlobalTransaction:<172.16.52.31:45290>:341 not found (transaction has possibly already been rolled back)
[09-05-2006 10:34:55.126] [53971973] [org.jboss.cache.interceptors.TxInterceptor] [ERROR] [Thread-41] Commit/rollback failed.
org.jboss.tm.JBossRollbackException: Unable to commit, tx=TransactionImpl:XidImpl[FormatId=257, GlobalId=nycpasv002/12968, BranchQual=, localId=12968] status=STATUS_NO_TRANSACTION; - nested throwable: (java.lang.IllegalStateException: cannot find transaction e

[manik]

Which version of JBC is this?

[mcevikce]

I am rinnig with JBossCache 1.3.0SP1.

[manik]

What's your replication mode?

[mcevikce]

Replication mode is REPL_ASYNC.


TreeCache.xml:


[
<?xml version="1.0" encoding="UTF-8"?>
<!-- ===================================================================== -->
<!-- -->
<!-- Sample Clustering Service Configuration -->
<!-- -->
<!-- ===================================================================== -->





<!-- ==================================================================== -->
<!-- Cluster Partition: defines cluster -->
<!-- ==================================================================== -->



<!-- Name of the partition being built -->
${jboss.partition.name}

<!-- The address used to determine the node name -->
${jboss.bind.address}

<!-- Determine if deadlock detection is enabled -->
False

<!-- Max time (in ms) to wait for state transfer to complete. Increase for large states -->
30000

<!-- The JGroups protocol configuration -->

<!--
The default UDP stack:
- If you have a multihomed machine, set the UDP protocol's bind_addr attribute to the
appropriate NIC IP address, e.g bind_addr="192.168.0.2".
- On Windows machines, because of the media sense feature being broken with multicast
(even after disabling media sense) set the UDP protocol's loopback attribute to true
-->

<UDP mcast_addr="228.1.2.3" mcast_port="45566"
ip_ttl="8" ip_mcast="true"
mcast_send_buf_size="800000" mcast_recv_buf_size="150000"
ucast_send_buf_size="800000" ucast_recv_buf_size="150000"
loopback="false"/>
<PING timeout="2000" num_initial_members="3"
up_thread="true" down_thread="true"/>
<MERGE2 min_interval="10000" max_interval="20000"/>
<FD shun="true" up_thread="true" down_thread="true"
timeout="2500" max_tries="5"/>
<VERIFY_SUSPECT timeout="3000" num_msgs="3"
up_thread="true" down_thread="true"/>
<pbcast.NAKACK gc_lag="50" retransmit_timeout="300,600,1200,2400,4800"
max_xmit_size="8192"
up_thread="true" down_thread="true"/>
<UNICAST timeout="300,600,1200,2400,4800" window_size="100" min_threshold="10"
down_thread="true"/>
<pbcast.STABLE desired_avg_gossip="20000"
up_thread="true" down_thread="true"/>
<FRAG frag_size="8192"
down_thread="true" up_thread="true"/>
<pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
shun="true" print_local_addr="true"/>
<pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>


<!-- Alternate TCP stack: customize it for your environment, change bind_addr and initial_hosts -->
<!--

<TCP bind_addr="thishost" start_port="7800" loopback="true"/>
<TCPPING initial_hosts="thishost[7800],otherhost[7800]" port_range="3" timeout="3500"
num_initial_members="3" up_thread="true" down_thread="true"/>
<MERGE2 min_interval="5000" max_interval="10000"/>
<FD shun="true" timeout="2500" max_tries="5" up_thread="true" down_thread="true" />
<VERIFY_SUSPECT timeout="1500" down_thread="false" up_thread="false" />
<pbcast.NAKACK down_thread="true" up_thread="true" gc_lag="100"
retransmit_timeout="3000"/>
<pbcast.STABLE desired_avg_gossip="20000" down_thread="false" up_thread="false" />
<pbcast.GMS join_timeout="5000" join_retry_timeout="2000" shun="false"
print_local_addr="true" down_thread="true" up_thread="true"/>
<pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>

-->




<!-- ==================================================================== -->
<!-- HA Session State Service for SFSB -->
<!-- ==================================================================== -->


jboss:service=${jboss.partition.name}
<!-- Name of the partition to which the service is linked -->
${jboss.partition.name}
<!-- JNDI name under which the service is bound -->
/HASessionState/Default
<!-- Max delay before cleaning unreclaimed state.
Defaults to 30*60*1000 => 30 minutes -->
0


<!-- ==================================================================== -->
<!-- HA JNDI -->
<!-- ==================================================================== -->


jboss:service=${jboss.partition.name}
<!-- Name of the partition to which the service is linked -->
${jboss.partition.name}
<!-- Bind address of bootstrap and HA-JNDI RMI endpoints -->
${jboss.bind.address}
<!-- Port on which the HA-JNDI stub is made available -->
${myserver.service.HAJNDI.port}
<!-- Accept backlog of the bootstrap socket -->
50
<!-- The thread pool service used to control the bootstrap and
auto discovery lookups -->
<depends optional-attribute-name="LookupPool"
proxy-type="attribute">jboss.system:service=ThreadPool

<!-- A flag to disable the auto discovery via multicast -->
false
<!-- Set the auto-discovery bootstrap multicast bind address. If not
specified and a BindAddress is specified, the BindAddress will be used. -->
${jboss.bind.address}
<!-- Multicast Address and group port used for auto-discovery -->
230.0.0.4
1102
<!-- The TTL (time-to-live) for autodiscovery IP multicast packets -->
16

<!-- RmiPort to be used by the HA-JNDI service once bound. 0 => auto. -->
0
<!-- Client socket factory to be used for client-server
RMI invocations during JNDI queries
custom
-->
<!-- Server socket factory to be used for client-server
RMI invocations during JNDI queries
custom
-->



${jboss.bind.address}

<!-- 0
custom
custom
-->


<!-- the JRMPInvokerHA creates a thread per request. This implementation uses a pool of threads -->

1
300
300
60000
${jboss.bind.address}
${myserver.service.invoker-pooledha.port}
${jboss.bind.address}
0
false
<depends optional-attribute-name="TransactionManagerService">jboss:service=TransactionManager


<!-- ==================================================================== -->

<!-- ==================================================================== -->
<!-- Distributed cache invalidation -->
<!-- ==================================================================== -->

<!--
jboss:service=${jboss.partition.name}
jboss.cache:service=InvalidationManager
jboss.cache:service=InvalidationManager
${jboss.partition.name}
DefaultJGBridge
-->


]

[mcevikce]

I assume this occurs when server tries to replicate cahe entry on the cluster server. In my treecache replication ASYNC and TransactionManagerLookupClass is org.jboss.cache.JBossTransactionManagerLookup.

The odd part to this problem is that if I start 2 servers using the same configuration on a another environment (different physical boxes) the I do not see these exceptions. They only occur on our production environment.

[manik]

Yes, this is a known issue - it is to do with a race condition.

I suggest downloading and tring 1.3.0.SP2 (released earlier today) which fixes this, among other things.

Cheers,
Manik