Wildfly + Cluster (full_ha) + HTTPD (mod_cluster) - All ajax calls generate exception

[ehamerski]

Hello,

I have a wildfly (9.0.2) cluster with two nodes. I'm using httpd / mod_cluster (1.3.1) for load balance. The tests have found an error when calling ajax requests. Have you ever run into this problem?

2017-10-25 10:45:32,596 ERROR [io.undertow.request] (default task-102) Undertow request failed HttpServerExchange{ GET /app/UserAjax.do}: org.infinispan.commons.CacheException: javax.transaction.HeuristicRollbackException

                at org.wildfly.clustering.ee.infinispan.ActiveTransactionBatch.close(ActiveTransactionBatch.java:50)

                at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:78)

                at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:765)

                at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:548)

                at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:329)

…

                at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261)

                at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80)

                at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172)

                at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)

                at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774)

                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                at java.lang.Thread.run(Thread.java:745)

Caused by: javax.transaction.HeuristicRollbackException

                at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:398)

                at org.infinispan.transaction.tm.DummyTransaction.commitResources(DummyTransaction.java:413)

                at org.infinispan.transaction.tm.DummyTransaction.runCommit(DummyTransaction.java:303)

                at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:104)

                at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:73)

                at org.wildfly.clustering.ee.infinispan.ActiveTransactionBatch.close(ActiveTransactionBatch.java:48)

                ... 12 more

Caused by: javax.transaction.xa.XAException

                at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:213)

                at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159)

                at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:112)

                at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:367)

                ... 17 more

Caused by: org.infinispan.commons.CacheException: java.lang.RuntimeException: Failure to marshal argument(s)

                at org.infinispan.commons.util.Util.rewrapAsCacheException(Util.java:657)

                at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:161)

                at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.broadcastRemoteCommands(CommandAwareRpcDispatcher.java:279)

                at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:542)

                at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:287)

                at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:222)

                at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:206)

                at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)

…

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)

                at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)

                at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123)

                at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)

                at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:102)

                at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)

                at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:96)

                at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:91)

                at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)

                at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:36)

                at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)

                at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:138)

                at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:125)

                at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)

…

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)

                at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)

                at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123)

                at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:42)

                at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)

                at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:200)

                at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:88)

                at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)

…

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)

                at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)

                at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123)

                at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)

…

                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)

                at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)

                at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)

                at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:123)

                at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:123)

                at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)

                at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:157)

                ... 19 more

Caused by: java.lang.RuntimeException: Failure to marshal argument(s)

                at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:417)

                at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java:466)

                at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:152)

                ... 63 more

Caused by: java.io.NotActiveException: writeFields() may only be called when the fields have not yet been written

                at org.jboss.marshalling.river.RiverObjectOutputStream.defaultWriteObject(RiverObjectOutputStream.java:162)

                at org.apache.commons.beanutils.DynaProperty.writeObject(DynaProperty.java:282)

                at sun.reflect.GeneratedMethodAccessor81.invoke(Unknown Source)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                at java.lang.reflect.Method.invoke(Method.java:498)

                at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271)

                at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976)

                at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)

                at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:778)

                at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)

                at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)

                at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)

                at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1032)

                at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:988)

                at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:967)

                at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)

                at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)

                at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)

                at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)

                at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)

                at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:344)

                at java.util.HashMap.internalWriteEntries(HashMap.java:1785)

                at java.util.HashMap.writeObject(HashMap.java:1362)

                at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                at java.lang.reflect.Method.invoke(Method.java:498)

                at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:271)

                at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:976)

                at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:854)

                at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)

                at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)

                at org.wildfly.clustering.marshalling.SimpleMarshalledValue.getBytes(SimpleMarshalledValue.java:77)

                at org.wildfly.clustering.marshalling.SimpleMarshalledValue.writeExternal(SimpleMarshalledValue.java:150)

                at org.wildfly.clustering.infinispan.spi.io.ExternalizableExternalizer.writeObject(ExternalizableExternalizer.java:46)

                at org.wildfly.clustering.infinispan.spi.io.ExternalizableExternalizer.writeObject(ExternalizableExternalizer.java:37)

                at org.infinispan.marshall.core.ExternalizerTable$ForeignExternalizerAdapter.writeObject(ExternalizerTable.java:491)

                at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)

                at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)

                at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)

                at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:58)

                at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeObject(ReplicableCommandExternalizer.java:43)

                at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeObject(ReplicableCommandExternalizer.java:31)

                at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:442)

                at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)

                at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)

                at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)

                at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:58)

                at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:124)

                at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:108)

                at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:65)

                at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:442)

                at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)

                at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)

                at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)

                at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:72)

                at org.infinispan.marshall.core.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:77)

                at org.infinispan.commons.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:41)

                at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:85)

                at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:23)

                at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:415)

                ... 65 more

Caused by: an exception which occurred:

                in field properties

                in field dynaClass

                in object java.util.HashMap@97f44329

                in object org.wildfly.clustering.marshalling.SimpleMarshalledValue@97f44329

                in object org.infinispan.commands.write.ReplaceCommand@b38c590d

                in object org.infinispan.commands.tx.PrepareCommand@3b24dd0

[pferraro]

Can you reproduce this issue with the latest release?  (i.e. WildFly 11.0.0.Final)

[pferraro]

Actually - this sounds familiar.  Can you tell which object type in the session is problematic?  You can likely workaround this issue by providing a custom Externalizer for this object.

[lukaszracon]

This is a serialization issue of DynaProperty from Bean Utils package (BEANUTILS-417):

Caused by: an exception which occurred:

                in field properties

                in field dynaClass

                in object java.util.HashMap@97f44329

Same issue happens on Wildfly 10. I resolved it by patching a library – see JIRA for patch:

[BEANUTILS-417] java.io.StreamCorruptedException: defaultWriteObject or writeFields not called on the sender's side - AS…

Similar serialization issue: [JBMAR-174] Can't serialize org.dom4j.Document using JBoss Marshalling API java.io.NotActiveException: writeFields() - J…

[ehamerski]

The client does not provide access to the source code and he does not agree in updating the application server right now.

I have made some changes in the full-ha/infinispan cache configuration and now the application seems to be working fine.

The web cache was defined as (to solve infinispan lock timeout and broken pipe exceptions, based on HTTP Session Replication - org.infinispan.util.concurrent.TimeoutException):

          <cache-container name="web" default-cache="repl" module="org.wildfly.clustering.web.infinispan">

                    <transport lock-timeout="60000"/>

                    <replicated-cache name="repl" mode="ASYNC">

                        <locking isolation="READ_COMMITTED"/>

                        <transaction mode="BATCH" locking="OPTIMISTIC"/>

                        <file-store/>

                    </replicated-cache>

                </cache-container>

The web cache was changed to:

               <cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">

                    <transport lock-timeout="60000"/>

                    <distributed-cache name="dist" mode="ASYNC">

                        <locking isolation="READ_COMMITTED"/>

                        <transaction mode="BATCH" locking="OPTIMISTIC"/>

                        <file-store/>

                    </distributed-cache>

                </cache-container>

Basically I changed the configuration back from replicated-cache to distributed-cache and kept isolation="READ_COMMITTED" and locking="OPTIMISTIC".

Any thoughts?

Thank you for your reply.