getting new infinispan exceptions in 8.1.0

[rituraj]

we are moving from 8.0.0 to 8.1.0 ...we are getting some new exceptions which was not there on 8.0.0...we need a confirmation about why they are coming ...

we are running 8.1.0 in domain mode

there are 2 clusters and each of them has 3 nodes distributed on 3 boxes

so

box1 : node1,node2 --master

box2 : node3 node4 --slave1

box3: node5 node6  --slave2

cluster 1 : (node1 , node3 , node5)

cluster 2 : (node2 , node4 , node6)

we were testing the session replication and for that we shutdown node1 of master which was in cluster 1 and we got below exceptions ...can someone explain why are they coming ..?

[Server:node1] 04:14:04,034 WARN  [org.infinispan.topology.CacheTopologyControlCommand] (remote-thread-5) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=dist, type=REBALANCE_CONFIRM, sender=slave1:node3/cluster1, joinInfo=null, topologyId=6, currentCH=null, pendingCH=null, throwable=null, viewId=2}: org.infinispan.commons.CacheException: Received invalid rebalance confirmation from slave1:node3/cluster1 for cache dist, we don't have a rebalance in progress

[Server:node1]  at org.infinispan.topology.ClusterTopologyManagerImpl.handleRebalanceCompleted(ClusterTopologyManagerImpl.java:190)

[Server:node1]  at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:147)

[Server:node1]  at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124)

[Server:node1]  at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:270)

[Server:node1]  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]

[Server:node1]  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]

[Server:node1]  at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]

[Server:node1]

[Server:node1] 04:14:04,036 WARN  [org.infinispan.topology.CacheTopologyControlCommand] (remote-thread-4) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=dist, type=REBALANCE_CONFIRM, sender=slave2:node5/cluster1, joinInfo=null, topologyId=6, currentCH=null, pendingCH=null, throwable=null, viewId=2}: org.infinispan.commons.CacheException: Received invalid rebalance confirmation from slave2:node5/cluster1 for cache dist, we don't have a rebalance in progress

[Server:node1]  at org.infinispan.topology.ClusterTopologyManagerImpl.handleRebalanceCompleted(ClusterTopologyManagerImpl.java:190)

[Server:node1]  at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:147)

[Server:node1]  at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124)

[Server:node1]  at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:270)

[Server:node1]  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]

[Server:node1]  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]

[Server:node1]  at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]

[Server:node1]

[Server:node1] 04:14:04,036 WARN  [org.infinispan.topology.CacheTopologyControlCommand] (remote-thread-3) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=default-host/myapp, type=REBALANCE_CONFIRM, sender=slave2:node5/cluster1, joinInfo=null, topologyId=6, currentCH=null, pendingCH=null, throwable=null, viewId=2}: org.infinispan.commons.CacheException: Received invalid rebalance confirmation from slave2:node5/cluster1 for cache default-host/myapp, we don't have a rebalance in progress

[Server:node1]  at org.infinispan.topology.ClusterTopologyManagerImpl.handleRebalanceCompleted(ClusterTopologyManagerImpl.java:190)

[Server:node1]  at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:147)

[Server:node1]  at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124)

[Server:node1]  at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:270)

[Server:node1]  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]

[Server:node1]  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]

[Server:node1]  at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]

[Server:node1]

[Server:node1] 04:14:04,036 WARN  [org.infinispan.topology.CacheTopologyControlCommand] (remote-thread-0) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=default-host/myapp, type=REBALANCE_CONFIRM, sender=slave1:node3/cluster1, joinInfo=null, topologyId=6, currentCH=null, pendingCH=null, throwable=null, viewId=2}: org.infinispan.commons.CacheException: Received invalid rebalance confirmation from slave1:node3/cluster1 for cache default-host/myapp, we don't have a rebalance in progress

[Server:node1]  at org.infinispan.topology.ClusterTopologyManagerImpl.handleRebalanceCompleted(ClusterTopologyManagerImpl.java:190)

[Server:node1]  at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:147)

[Server:node1]  at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124)

[Server:node1]  at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:270)

[Server:node1]  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]

[Server:node1]  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]

[Server:node1]  at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]

-Thanks

[rhusar]

What you are seeing is probably [ISPN-3791] Silence "Received invalid rebalance confirmation from NodeX" exceptions - JBoss Issue Tracker, take a look at the explanation there.

Note that these are logged "only" as warning, do you see this resulting in concrete problem?

[rituraj]

Thanks a lot Radoslav for the quick response ....we are seeing a issue to the other node (where the session is now) in the cluster so when we shutdown node1 ...session replicates to another node in the cluster lets take it as node2 on slave ..i can see the jsession is same for node2 as that was of node1 but once the site uploads completely on node2 and we hit on few of the links ....we are getting below error right now (it may be a code issue as well...but right now we are investigating it ...)

[Server:node2] 04:17:33,188 ERROR [io.undertow.servlet.request] (default task-1) UT015005: Error invoking method requestDestroyed on listener class org.springframework.web.context.request.RequestContextListener: java.lang.ClassCastException: com.opensymphony.clickstream.Clickstream cannot be cast to java.lang.String

[Server:node2]  at org.jboss.marshalling.ModularClassResolver.resolveClass(ModularClassResolver.java:99)

[Server:node2]  at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:943)

[Server:node2]  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:310)

[Server:node2]  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

[Server:node2]  at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1712)

[Server:node2]  at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1628)

[Server:node2]  at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1269)

[Server:node2]  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)

[Server:node2]  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

[Server:node2]  at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1712)

[Server:node2]  at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1628)

[Server:node2]  at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1269)

[Server:node2]  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)

[Server:node2]  at org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:149)

[Server:node2]  at org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:135)

[Server:node2]  at org.jboss.marshalling.MarshallerObjectInputStream.readObjectOverride(MarshallerObjectInputStream.java:53)

[Server:node2]  at org.jboss.marshalling.river.RiverObjectInputStream.readObjectOverride(RiverObjectInputStream.java:307)

[Server:node2]  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:364) [rt.jar:1.7.0_51]

[Server:node2]  at java.util.concurrent.ConcurrentHashMap.readObject(ConcurrentHashMap.java:1574) [rt.jar:1.7.0_51]

[Server:node2]  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51]

[Server:node2]  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51]

[Server:node2]  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51]

[Server:node2]  at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51]

[Server:node2]  at org.wildfly.clustering.web.infinispan.session.MarshalledValueSessionAttributeMarshaller.read(MarshalledValueSessionAttributeMarshaller.java:46)

[Server:node2]  at org.wildfly.clustering.web.infinispan.session.MarshalledValueSessionAttributeMarshaller.read(MarshalledValueSessionAttributeMarshaller.java:33)

[Server:node2]  at org.wildfly.clustering.web.infinispan.session.coarse.CoarseImmutableSessionAttributes.getAttributes(CoarseImmutableSessionAttributes.java:46)

[Server:node2]  at org.wildfly.clustering.web.infinispan.session.coarse.CoarseImmutableSessionAttributes.getAttributeNames(CoarseImmutableSessionAttributes.java:51)

[Server:node2]  at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findListeners(InfinispanSessionManager.java:320)

[Server:node2]  at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.triggerPostActivationEvents(InfinispanSessionManager.java:309)

[Server:node2]  at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:164)

[Server:node2]  at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:115)

[Server:node2]  at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:677) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

[Server:node2]  at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:353) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

[Server:node2] Caused by: an exception which occurred:

[Server:node2]  in field appAttributes

[Server:node2]  in object of type com.myorg.app.domain.Part

[Server:node2]  in field part

[Server:node2]  in object of type com.myorg.app.domain.LineItem

[Server:node2]  in object of type java.util.concurrent.ConcurrentHashMap

[Server:node2]  in field cartItems

[Server:node2]  in object of type com.myorg.app.domain.Cart

[Server:node2]  in map value at index [1] of size [8]

[Server:node2]

[Server:node2] 04:17:33,199 ERROR [io.undertow.request] (default task-1) Blocking request failed HttpServerExchange{ GET /myapp/error.jsp}: java.lang.RuntimeException: java.lang.ClassCastException: com.opensymphony.clickstream.Clickstream cannot be cast to java.lang.String

Caused by: java.lang.ClassCastException: com.opensymphony.clickstream.Clickstream cannot be cast to java.lang.String

[Server:node2]  at org.jboss.marshalling.ModularClassResolver.resolveClass(ModularClassResolver.java:99)

[Server:node2]  at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:943)

[Server:node2]  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:310)

[Server:node2]  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

[Server:node2]  at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1712)

2)is there a workaround for this bug ..?

3) we are also seeing a new WARN message as 03:53:59,058 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 30) JBAS010153: Node identifier property is set to the default value. Please make sure it is unique.

Thanks

[rhusar]

Caused by: java.lang.ClassCastException: com.opensymphony.clickstream.Clickstream cannot be cast to java.lang.String

Looks like a serialization problem in your code. Also, make sure to update all deployments in the cluster where sessions are replicated. If you change what the session data looks like the old version of the application will not be able do deserialize the session data, and vice versa.

3) we are also seeing a new WARN message as 03:53:59,058 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 30) JBAS010153: Node identifier property is set to the default value. Please make sure it is unique.

See this thread: Re: JBAS010153: Node identifier property is set to the default value. Please make sure it is unique.

[rituraj]

Thanks Radoslav  for the quick response  even i felt the same for the stacktrace but now we are sure...we are looking to resolve the code issue ...

just to confirm once more ....this will keep away the WARN mssg...?

we are also seeing a new WARN message as 03:53:59,058 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 30) JBAS010153: Node identifier property is set to the default value. Please make sure it is unique.

See this thread: Re: JBAS010153: Node identifier property is set to the default value. Please make sure it is unique.

<subsystem xmlns="urn:jboss:domain:transactions:2.0">

   <core-environment node-identifier="${jboss.node.name}">...</core-environment>

</subsystem>

Thanks

Rituraj

[pferraro]

just to confirm once more ....this will keep away the WARN mssg...?

<subsystem xmlns="urn:jboss:domain:transactions:2.0">

   <core-environment node-identifier="${jboss.node.name}">...</core-environment>

</subsystem>

Yes.

[pferraro]

FYI - I've opened [WFLY-3502] Create pluggable strategy for dealing with serialization incompatiblities - JBoss Issue Tracker to allow users the option to configure this behavior.

[rituraj]

Paul Ferraro wrote:

just to confirm once more ....this will keep away the WARN mssg...?

<subsystem xmlns="urn:jboss:domain:transactions:2.0">

   <core-environment node-identifier="${jboss.node.name}">...</core-environment>

</subsystem>

Yes.

we are getting the below error while applying the change

10:46:02,920 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 97) JBAS014613: Operation ("write-attribute") failed - address: ([("subsystem" => "transactions")]) - failure description: "JBAS014703: '${Project_name}_dc:${configuration_name}-${hostname -s}w1' is an invalid value for parameter node-identifier. Values must have a maximum length of 23 characters"

so our application_server_name is like : ${configuration_name}-${hostname -s}w1
name of domain_controller name is : ${Project_name}_dc

so the overall node name becomes ${Project_name}_dc:${configuration_name}-${hostname -s}w1' (why does it adds the domain_controller name as well in the node name..?)

1)  while creating the node name wildfly adds the domain_controller_name : application-server_name and this is making it  > 23 characters allowed

    is there a way to increase the number of characters...?
2) also if i don't pass any name for domain controller in master-host.xml it picks the node name as  domain-name of server  : ${configuration_name}-${hostname -s}w1'....
    which we dont want ...
3) same instance-id is also used in undertow subsystem and i have passed the same values there as well ....it is not a problem there ....

4) what/How would it affect if we let it be as it is ...??

Thanks

-RIturaj