Wildfly 10 support for optimistic web cache

[matlach]

Hello everyone,

I'm currently trying to setup the distributed web session cache to be optimistic in Wildfly 10.0.0.CR5 as following:

{code}

            <cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">

                <transport lock-timeout="60000"/>

                <distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">

                    <locking locking="OPTIMISTIC" isolation="READ_COMMITTED"/>

                    <transaction mode="BATCH"/>

                </distributed-cache>

            </cache-container>

{code}

But each time, I'm getting "Explicit locking is not allowed with optimistic caches!" :

{code}

Caused by: org.infinispan.InvalidCacheUsageException: Explicit locking is not allowed with optimistic caches!

          at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitLockControlCommand(OptimisticLockingInterceptor.java:142) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:157) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:215) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitLockControlCommand(TransactionSynchronizerInterceptor.java:78) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:238) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:102) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:81) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:163) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:110) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.cache.impl.CacheImpl.lock(CacheImpl.java:828) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.cache.impl.DecoratedCache.lock(DecoratedCache.java:136) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.lock(AbstractDelegatingAdvancedCache.java:177) ~[infinispan-core-8.1.0.Final.jar!/:8.1.0.Final]

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.remove(InfinispanSessionMetaDataFactory.java:124) ~[?:?]

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.remove(InfinispanSessionMetaDataFactory.java:39) ~[?:?]

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.remove(InfinispanSessionFactory.java:89) ~[?:?]

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.remove(InfinispanSessionFactory.java:40) ~[?:?]

          at org.wildfly.clustering.web.infinispan.session.InfinispanSession.invalidate(InfinispanSession.java:67) ~[?:?]

          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager$SchedulableSession.invalidate(InfinispanSessionManager.java:439) ~[?:?]

          at org.wildfly.clustering.web.undertow.session.DistributableSession.invalidate(DistributableSession.java:181) ~[?:?]

          at io.undertow.servlet.spec.HttpSessionImpl.invalidate(HttpSessionImpl.java:199) ~[undertow-servlet-1.3.12.Final.jar!/:1.3.12.Final]

...

{code}

That said, this configuration was working fine in either EAP 6.1 and Wildfly 8.2.1.Final with [WFLY-5331] ConcurrentModificationException in InfinispanSessionManager.findListeners - JBoss Issue Tracker backported.

I know there was some debates wheter if optimistic session cache should be a thing or not according to the specs:

HTTP Session Replication - org.infinispan.util.concurrent.TimeoutException

Servlet 3.0 final specification under section 7.7.1 and 7.7.2

Sadly, the application I'm currently trying to migrate to Wildfly 10.0.0.CR5+ is dependent of that optimistic behavior and I was wondering if that clustering configuration was still supported or not?

If it should, then I'll raise a ticket.

Else, if this is by design, is there any best pattern to follow when dealing with pessimistic web session cache? Would moving attribute to @SessionScoped bean would help, or under the hood, SessionScoped bean implementation just piggyback on the session attributes?

Thanks,

[pferraro]

The pessimistic locking of a session was put in place as a workaround for [ISPN-6007] Cache.get(...) using Flag.FORCE_WRITE_LOCK should retry on OutdatedTopologyException - JBoss Issue Tracker.

Admittedly, that workaround was a bit sloppy, as the explicit lock should only occur when using a cache that supports pessimistic locking.  I've opened [WFLY-5998] ISPN-6007 workaround fails for caches using optimistic locking - JBoss Issue Tracker.

[matlach]

Thanks Paul, I'll follow closely WFLY-5998 development.

[pferraro]

I've submitted a fix here: https://github.com/wildfly/wildfly/pull/8576

[pferraro]

matlach This pull request was merged and will be included in WF 10.0.0.Final.

[matlach]

Hi Paul,

I've tested with WildFly-latest-master Changes [Jenkins] Build #2163 and using the same configuration as mentioned (OPTIMISTIC + READ_COMMITTED + BATCH) in my initial post and I've got the following exception:

{code}

14:51:13,441 WARN  [org.infinispan.transaction.tm.DummyTransaction] (default task-7) ISPN000112: exception while committing: javax.transaction.xa.XAException

  at org.infinispan.transaction.xa.TransactionXaAdapter.getLocalTransactionAndValidateImpl(TransactionXaAdapter.java:242)

  at org.infinispan.transaction.xa.TransactionXaAdapter.getLocalTransactionAndValidate(TransactionXaAdapter.java:235)

  at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:103)

  at org.infinispan.transaction.tm.DummyTransaction.finishResource(DummyTransaction.java:389)

  at org.infinispan.transaction.tm.DummyTransaction.commitResources(DummyTransaction.java:435)

  at org.infinispan.transaction.tm.DummyTransaction.runCommit(DummyTransaction.java:314)

  at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:105)

  at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:73)

  at org.wildfly.clustering.ee.infinispan.ActiveTransactionBatch.close(ActiveTransactionBatch.java:48)

  at org.wildfly.clustering.web.undertow.session.DistributableSession.requestDone(DistributableSession.java:77)

  at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:768)

  at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:557)

  at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:331)

  at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)

  at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

  at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)

  at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)

  at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)

  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

  at java.lang.Thread.run(Thread.java:745)

{code}

I've also tried (REPETABLE_READ + OPTIMISTIC + BATCH) :

{code}

            <cache-container name="web" module="org.wildfly.clustering.web.infinispan" default-cache="dist">

                <transport lock-timeout="60000"/>

                <distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">

                    <locking isolation="READ_COMMITTED"/>

                    <transaction locking="OPTIMISTIC" mode="BATCH"/>

                </distributed-cache>

            </cache-container>

{code}

but I'm also getting the same previously mentionned stack trace.

Is there another build number I should try and/or raise another ticket?

Also, something new, if I set ip_ttl in jgroups to 0 (doing so I can prevent spamming the company network) I'm now getting during startup:

{code}

14:49:14,349 ERROR [org.jgroups.protocols.UDP] (MSC service thread 1-4) failed setting ip_ttl: java.lang.reflect.InvocationTargetException

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  at java.lang.reflect.Method.invoke(Method.java:497)

  at org.jgroups.protocols.UDP.setTimeToLive(UDP.java:339)

  at org.jgroups.protocols.UDP.createSockets(UDP.java:368)

  at org.jgroups.protocols.UDP.start(UDP.java:270)

  at org.jgroups.stack.ProtocolStack.startStack(ProtocolStack.java:965)

  at org.jgroups.JChannel.startStack(JChannel.java:890)

  at org.jgroups.JChannel._preConnect(JChannel.java:553)

  at org.jgroups.JChannel.connect(JChannel.java:288)

  at org.jgroups.JChannel.connect(JChannel.java:279)

  at org.wildfly.clustering.jgroups.spi.service.ChannelConnectorBuilder.start(ChannelConnectorBuilder.java:94)

  at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)

  at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)

  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

  at java.lang.Thread.run(Thread.java:745)

Caused by: java.io.IOException: Method not implemented!

  at java.net.DualStackPlainDatagramSocketImpl.setTimeToLive(DualStackPlainDatagramSocketImpl.java:236)

  ... 18 more

{code}

I guess the later is ongoing development?

[pferraro]

I was able to reproduce your 1st issue and am looking into it.

As far as your 2nd issue, this is a known issue on Windows, see: https://issues.jboss.org/browse/JGRP-1970

[matlach]

Hi pferraro, thanks for the answer,

Just wondering, will JGroups 3.6.7.Final be included in Wildfly 10 Final release? else should I enter an enhancement request to do so?

[pferraro]

[WFLY-5981] Upgrade JGroups to 3.6.7.Final - JBoss Issue Tracker

But probably not in time for 10.0.0.Final as this upgrade requires some changes to wildfly-core, which have not yet been merged.  See:

WFCORE-1292 SocketBindingManager.createMulticast/DatagramSocket(...) variants needed for unbound sockets by pferraro · P…

I expect this will be merged after the 10.0.0.Final release, and be included in a subsequent bug fix release.

[rhusar]

I have created [WFLY-6109] Using OPTIMISTIC locking results in "ISPN000112: exception while committing: javax.transaction.xa.XAExceptio… to keep track of this issue. Looks like an Infinispan issue.

The transaction in the exception is however indeed committed so no data might be lost in some scenarios.

[rhusar]

The underlying issue in Infinispan has been identified as [ISPN-6146] XA transactions already committed in prepare as XA_RDONLY fail on commit with "ISPN000112: exception while c… please follow it for updates, a pull request is already submitted.