13 Replies Latest reply on Jul 10, 2018 5:18 AM by galder.zamarreno

org.infinispan.util.concurrent.TimeoutException infinispan 9.3.0.Final

pkolanda Jul 4, 2018 1:25 AM

Hi Team,

Cluster is formed and after some point of time, we hit this bellow exception: Three node cluster. No split brain happended. All the three servers are connected all the time.

I manged to get the thead dump via jstack. Attached here.

Let us know if any information is need. We will keep the setup running.

Infinispan 9 config.

-----------------------

<?xml version="1.0" encoding="UTF-8"?>

<stack-file name="tcp" path="configuration/xncjgroups.xml"/>

</jgroups>

<cache-container default-cache="default">

<replicated-cache name="default" statistics="true" mode="ASYNC" >

</replicated-cache>

<replicated-cache name="transactional-type" statistics="true" mode="SYNC" remote-timeout="200000" >

<locking

isolation="READ_COMMITTED" striping="false" acquire-timeout="200000"

<transaction

locking="OPTIMISTIC"

auto-commit="true"

complete-timeout="180000"

mode="FULL_XA"

notifications="true"

stop-timeout="30000"

transaction-manager-lookup="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup"/>

</replicated-cache>

</cache-container>

</infinispan>

jgroup.xml

-----------------------------

<config xmlns="urn:org:jgroups"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups-4.0.xsd">

<TCP bind_addr="${jgroups.tcp.address}"

bind_port="12000" port_range="10"

recv_buf_size="20000000"

send_buf_size="640000"

bundler_type="no-bundler"

enable_diagnostics="true"

thread_naming_pattern="cl"

thread_pool.min_threads="10"

thread_pool.max_threads="32"

thread_pool.keep_alive_time="5000"

<VERIFY_SUSPECT timeout="500"/>

<pbcast.NAKACK2 use_mcast_xmit="false"

discard_delivered_msgs="true" />

<pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"

max_bytes="10m"/>

<pbcast.GMS print_local_addr="true" join_timeout="${jgroups.join_timeout:5000}"

max_bundling_time="30"

view_bundling="true"/>

<MFC max_credits="2M"

min_threshold="0.4"/>

</config>

Exception:

2018-07-03 17:24:12.040 EDT [timeout-thread--p3-t1] ERROR org.infinispan.interceptors.impl.InvocationContextInterceptor - ISPN000136: Error executing command CommitCommand, writing keys [NX|FDO211622EJ]

org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 10574 from rhel64-vm-prod-nxapi-3-28598

at org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167) ~[na:na]

at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87) ~[na:na]

at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22) ~[na:na]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_131]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_131]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_131]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]

at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]

2018-07-03 17:24:12.062 EDT [ConnectionEvent Thread] ERROR org.opendaylight.controller.logging.bridge.internal.UncaughtExceptionPolicy - Thread Thread[ConnectionEvent Thread,5,main] died because of an uncaught exception

java.lang.StackOverflowError: null

at ch.qos.logback.classic.spi.ThrowableProxy.<init>(ThrowableProxy.java:54) ~[logback-classic-1.0.9.jar:na]