JBossDeveloper

After invoking CacheManager.stop, CacheManager.start  and then CacheManager.getCache, below are the debug logs with JGroups/ISPN implementation clearly indicating that the DefaultCacheManager did stop & start the cache but throws exception in getCache regardless ...

WARN  [11:39:23,362] [remote-thread-0,201][Node] - Begin restarting the cache on Node: 201; as it will be evicted fr        om the cluster once suspected.

   1242 DEBUG [11:39:23,362] [remote-thread-0,201][DefaultCacheManager] - Stopping cache manager AggregationEngine on 201-95        1(201)

   1243 DEBUG [11:39:23,369] [remote-thread-0,201][CacheImpl] - Stopping cache ___defaultcache on 201-951(201)

   1244 DEBUG [11:39:23,371] [remote-thread-0,201][TransactionTable] - Wait for on-going transactions to finish for 30 secon        ds.

   1245 DEBUG [11:39:23,371] [remote-thread-0,201][TransactionTable] - All transactions terminated

   1246 DEBUG [11:39:23,371] [remote-thread-0,201][LocalTopologyManagerImpl] - Node 201-951(201) leaving cache ___defaultcac        he

   1247 DEBUG [11:39:23,378] [remote-thread-0,201][BackupReceiverRepositoryImpl] - Processing cache stop: EventImpl{type=CAC        HE_STOPPED, newMembers=null, oldMembers=null, localAddress=null, viewId=0, subgroupsMerged=null, mergeView=false}. C        ache name: '___defaultcache'

   1248 INFO  [11:39:23,379] [remote-thread-0,201][JGroupsTransport] - ISPN000080: Disconnecting and closing JGroups Channel

   1249 DEBUG [11:39:23,382] [remote-thread-0,201][GMS] - 201-951(201): sending LEAVE request to 101-42786(101)

   1250 DEBUG [11:39:23,885] [remote-thread-0,201][UDP] - multicast socket closed

   1251 DEBUG [11:39:23,885] [multicast receiver,AggregationEngine,201-951(201)][UDP] - receiver socket is closed, exception        =java.net.SocketException: Socket closed

   1252 DEBUG [11:39:23,886] [multicast receiver,AggregationEngine,201-951(201)][UDP] - multicast receiver thread terminated   1253 DEBUG [11:39:23,887] [unicast receiver,AggregationEngine,201-951(201)][UDP] - receiver socket is closed, exception=j        ava.net.SocketException: Socket closed

   1254 DEBUG [11:39:23,887] [unicast receiver,AggregationEngine,201-951(201)][UDP] - unicast receiver thread terminated

   1255 DEBUG [11:39:23,887] [remote-thread-0,201][UDP] - closing sockets and stopping threads

   1256 INFO  [11:39:23,891] [remote-thread-0,201][JGroupsTransport] - ISPN000082: Stopping the RpcDispatcher

   1257 DEBUG [11:39:23,891] [remote-thread-3,201][AbstractJBossMarshaller] - Using JBoss Marshalling

   1258 DEBUG [11:39:23,891] [remote-thread-0,201][DefaultCacheManager] - Started cache manager AggregationEngine on 201

   1259 ERROR [11:39:23,892] [remote-thread-0,201][InboundInvocationHandlerImpl] - Exception executing command

   1260 java.lang.IllegalStateException: Cache container has been stopped and cannot be reused. Recreate the cache container        .

   1261         at org.infinispan.manager.DefaultCacheManager.assertIsNotTerminated(DefaultCacheManager.java:935)

   1262         at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:535)

   1263         at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:517)

   1264         at com.jpmorgan.lri.cs.ae.distexec.Node.restartCache(Node.java:249)

   1265         at com.jpmorgan.lri.cs.ae.distexec.exec.command.CommandExec.call(CommandExec.java:268)

   1266         at com.jpmorgan.lri.cs.ae.distexec.exec.command.CommandExec.call(CommandExec.java:28)

   1267         at org.infinispan.commands.read.DistributedExecuteCommand.perform(DistributedExecuteCommand.java:121)

   1268         at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122        )

   1269         at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:68)

   1270         at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:194)

   1271         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

Ups, by bad. After CacheManager.stop, you'd need to recreate the CacheManager to be able to re-use it. Sorry for the confusion.

Hi Mircea,

Please note that we have been able to resolve the SUSPECT_EXCEPTION issue by stopping the cache & then recreating the CacheManager. But there were other changes required under MergeView event to update the topology on Master node with the new address being assigned to the suspected node whose cache was re-created to allow re-submission of our request (without re-starting the cluster). We now have a remedy for this issue and a big THANKS to both you & Bela for helping us out on this thread.

As an aside on the 7.0 ISPN bug fix to silence the SUSPECT_EXCEPTION, does this fix mean that the JGroups FD_ALL/VERIFY_SUSPECT protocols will NOT evict the suspected node(s) (inspite of timeout value being exceeded) leaving the cluster in it's original state thereby making SUSPECT_EXCEPTION deprecated? OR does this fix allow ISPN to NOT throw SUSPECT_EXCEPTION back to the caller but JGroups would still evict suspected node(s) from the cluster? If the former; that's a true solution to this issue. Otherwise the latter still leaves the cluster impaired requiring cache re-creation to allow suspected node(s) to join back to the cluster.

This all happens at the Infinispan level; the operation of JGroups doesn't change: it will continue to expell suspected members and form a new view.