Preloading Exception when using infinispan as RemoteCacheStore for JBoss As 7.2.0
Hi
I got a problem while I was trying to use standalone infinispan data grid as a remote cache store of Jboss AS(7.2.0 Final) in order to avoid web session replication.
Here's my ideal architecture:
My nodes infomation:
Jboss As Node address:172.17.254.230
Jboss As Version: 7.2.0.Final
Infinispan Node1 address:172.17.254.231
Infinispan Node2 address:172.17.254.232
Infinispan Data Grid version: 5.3.0.Final
and also my Infinispan Standalone Server version: 5.2.1.Final
When I put some data into cache infinispan data grid:
infinispan@/home/infinispan/infinispan-server-5.3.0.Final/bin>ispn-cli.sh -c remoting://172.17.254.232:9999 Mar 1, 2013 2:43:31 PM org.xnio.Xnio <clinit> INFO: XNIO Version 3.0.7.GA Mar 1, 2013 2:43:31 PM org.xnio.nio.NioXnio <clinit> INFO: XNIO NIO Implementation Version 3.0.7.GA Mar 1, 2013 2:43:31 PM org.jboss.remoting3.EndpointImpl <clinit> INFO: JBoss Remoting version 3.2.14.GA [remoting://172.17.254.232:9999/clustered/]> cache usertable [remoting://172.17.254.232:9999/clustered/usertable]> put key1 aaa [remoting://172.17.254.232:9999/clustered/usertable]> put key2 bbb [remoting://172.17.254.232:9999/clustered/usertable]> put key3 ccc [remoting://172.17.254.232:9999/clustered/usertable]> get key1 aaa [remoting://172.17.254.232:9999/clustered/usertable]> get key2 bbb [remoting://172.17.254.232:9999/clustered/usertable]> get key3 ccc [remoting://172.17.254.232:9999/clustered/usertable]>
and then I started a Jboss AS instance, got an error while preloading cache data from remote cache store.
it seems that the exception was found on infinispan and passed to jboss as.
here's my Jboss AS log:
========================================================================= JBoss Bootstrap Environment JBOSS_HOME: /home/infinispan/jbossas/jboss-as-7.2.0.Final JAVA: /usr/java/jdk1.6.0_45/bin/java JAVA_OPTS: -server -XX:+UseCompressedOops -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true ========================================================================= 13:12:01,060 INFO [org.jboss.modules] (main) JBoss Modules version 1.2.0.CR1 13:12:01,391 INFO [org.jboss.msc] (main) JBoss MSC version 1.0.4.GA 13:12:01,491 INFO [org.jboss.as] (MSC service thread 1-1) JBAS015899: JBoss AS 7.2.0.Final "Janus" starting 13:12:04,521 INFO [org.xnio] (MSC service thread 1-3) XNIO Version 3.0.7.GA 13:12:04,549 INFO [org.xnio.nio] (MSC service thread 1-3) XNIO NIO Implementation Version 3.0.7.GA 13:12:04,551 INFO [org.jboss.as.server] (Controller Boot Thread) JBAS015888: Creating http management service using socket-binding (management-http) 13:12:04,589 INFO [org.jboss.remoting] (MSC service thread 1-3) JBoss Remoting version 3.2.14.GA 13:12:04,703 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 36) JBAS010280: Activating Infinispan subsystem. 13:12:04,706 INFO [org.jboss.as.jacorb] (ServerService Thread Pool -- 37) JBAS016300: Activating JacORB Subsystem 13:12:04,740 INFO [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 42) JBAS010260: Activating JGroups subsystem. 13:12:04,804 INFO [org.jboss.as.security] (ServerService Thread Pool -- 54) JBAS013171: Activating Security Subsystem 13:12:04,848 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 49) JBAS011800: Activating Naming Subsystem 13:12:04,861 INFO [org.jboss.as.jsf] (ServerService Thread Pool -- 45) JBAS012605: Activated the following JSF Implementations: [main, 1.2] 13:12:04,951 INFO [org.jboss.as.webservices] (ServerService Thread Pool -- 58) JBAS015537: Activating WebServices Extension 13:12:05,083 INFO [org.jboss.as.security] (MSC service thread 1-3) JBAS013170: Current PicketBox version=4.0.15.Final 13:12:05,085 INFO [org.jboss.as.connector.logging] (MSC service thread 1-1) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.15.Final) 13:12:05,135 INFO [org.jboss.as.naming] (MSC service thread 1-1) JBAS011802: Starting Naming Service 13:12:05,492 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 32) JBAS010403: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3) 13:12:05,620 INFO [org.jboss.jaxr] (MSC service thread 1-1) JBAS014000: Started JAXR subsystem, binding JAXR connection factory into JNDI as: java:jboss/jaxr/ConnectionFactory 13:12:05,642 INFO [org.jboss.as.mail.extension] (MSC service thread 1-1) JBAS015400: Bound mail session [java:jboss/mail/Default] 13:12:06,112 INFO [org.jboss.ws.common.management] (MSC service thread 1-3) JBWS022052: Starting JBoss Web Services - Stack CXF Server 4.1.3.Final 13:12:06,296 INFO [org.apache.coyote.http11] (MSC service thread 1-4) JBWEB003001: Coyote HTTP/1.1 initializing on : http-/172.17.254.230:8080 13:12:06,298 INFO [org.apache.coyote.ajp] (MSC service thread 1-2) JBWEB003046: Starting Coyote AJP/1.3 on ajp-/172.17.254.230:8009 13:12:06,327 INFO [org.apache.coyote.http11] (MSC service thread 1-4) JBWEB003000: Coyote HTTP/1.1 starting on: http-/172.17.254.230:8080 13:12:06,661 INFO [org.jboss.modcluster.ModClusterService] (ServerService Thread Pool -- 60) Initializing mod_cluster 1.2.3.Final 13:12:06,792 INFO [org.jboss.modcluster.advertise.impl.AdvertiseListenerImpl] (ServerService Thread Pool -- 60) Listening to proxy advertisements on 224.0.1.105:23,364 13:12:07,009 INFO [org.jboss.as.jacorb] (MSC service thread 1-1) JBAS016330: CORBA ORB Service started 13:12:07,037 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 36) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated. 13:12:07,072 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 36) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated. 13:12:07,764 INFO [org.jboss.as.server.deployment.scanner] (MSC service thread 1-3) JBAS015012: Started FileSystemDeploymentService for directory /home/infinispan/jbossas/jboss-as-7.2.0.Final/standalone/deployments 13:12:07,816 INFO [org.jboss.as.jacorb] (MSC service thread 1-4) JBAS016328: CORBA Naming Service started 13:12:07,893 INFO [org.jboss.as.remoting] (MSC service thread 1-4) JBAS017100: Listening on 172.17.254.230:4447 13:12:07,893 INFO [org.jboss.as.remoting] (MSC service thread 1-2) JBAS017100: Listening on 172.17.254.230:9999 13:12:08,070 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS] 13:12:08,845 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated. 13:12:08,848 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-3) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated. 13:12:08,859 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated. 13:12:08,865 INFO [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-3) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated. 13:12:08,949 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 60) ISPN000078: Starting JGroups Channel 13:12:08,978 INFO [stdout] (ServerService Thread Pool -- 60) 13:12:08,978 INFO [stdout] (ServerService Thread Pool -- 60) ------------------------------------------------------------------- 13:12:08,979 INFO [stdout] (ServerService Thread Pool -- 60) GMS: address=slave07/web, cluster=web, physical address=172.17.254.230:55200 13:12:08,979 INFO [stdout] (ServerService Thread Pool -- 60) ------------------------------------------------------------------- 13:12:11,017 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 60) ISPN000094: Received new cluster view: [slave07/web|0] [slave07/web] 13:12:11,235 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 60) ISPN000079: Cache local address is slave07/web, physical addresses are [172.17.254.230:55200] 13:12:11,263 INFO [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 60) ISPN000128: Infinispan version: Infinispan 'Delirium' 5.2.1.Final 13:12:11,381 INFO [org.infinispan.factories.TransactionManagerFactory] (ServerService Thread Pool -- 60) ISPN000161: Using a batchMode transaction manager 13:12:11,628 INFO [org.infinispan.client.hotrod.RemoteCacheManager] (ServerService Thread Pool -- 60) ISPN004021: Infinispan version: Infinispan 'Delirium' 5.2.1.Final 13:12:11,632 INFO [org.infinispan.client.hotrod.RemoteCacheManager] (ServerService Thread Pool -- 60) ISPN004021: Infinispan version: Infinispan 'Delirium' 5.2.1.Final 13:12:11,855 INFO [org.infinispan.client.hotrod.impl.protocol.Codec12] (ServerService Thread Pool -- 60) ISPN004006: /172.17.254.231:11222 sent new topology view (id=2) containing 2 addresses: [/172.17.254.232:11222, /172.17.254.231:11222] 13:12:11,910 INFO [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 60) ISPN000031: MBeans were successfully registered to the platform MBean server. 13:12:11,928 WARN [org.infinispan.client.hotrod.impl.protocol.Codec12] (ServerService Thread Pool -- 60) ISPN004005: Error received from the server: java.lang.ClassCastException: java.lang.String cannot be cast to [B 13:12:11,932 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 60) MSC00001: Failed to start service jboss.infinispan.web.repl: org.jboss.msc.service.StartException in service jboss.infinispan.web.repl: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.loaders.CacheLoaderManagerImpl.preload() on object of type CacheLoaderManagerImpl at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87) [jboss-as-clustering-common-7.2.0.Final.jar:7.2.0.Final] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45] at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.0.Final.jar:2.1.0.Final] Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.loaders.CacheLoaderManagerImpl.preload() on object of type CacheLoaderManagerImpl at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205) at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:883) at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:654) at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:643) at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:546) at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:199) at org.infinispan.CacheImpl.start(CacheImpl.java:559) at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:686) at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:649) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:545) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:559) at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:109) at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:100) at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78) at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82) [jboss-as-clustering-common-7.2.0.Final.jar:7.2.0.Final] ... 4 more Caused by: org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for message id[3] returned server error (status=0x85): java.lang.ClassCastException: java.lang.String cannot be cast to [B at org.infinispan.client.hotrod.impl.protocol.Codec10.checkForErrorsInResponseStatus(Codec10.java:157) at org.infinispan.client.hotrod.impl.protocol.Codec10.readHeader(Codec10.java:110) at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:78) at org.infinispan.client.hotrod.impl.operations.BulkGetOperation.executeOperation(BulkGetOperation.java:60) at org.infinispan.client.hotrod.impl.operations.BulkGetOperation.executeOperation(BulkGetOperation.java:41) at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:68) at org.infinispan.client.hotrod.impl.RemoteCacheImpl.getBulk(RemoteCacheImpl.java:371) at org.infinispan.client.hotrod.impl.RemoteCacheImpl.getBulk(RemoteCacheImpl.java:363) at org.infinispan.loaders.remote.RemoteCacheStore.loadAll(RemoteCacheStore.java:154) at org.infinispan.loaders.CacheLoaderManagerImpl.loadState(CacheLoaderManagerImpl.java:279) at org.infinispan.loaders.CacheLoaderManagerImpl.preload(CacheLoaderManagerImpl.java:238) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_45] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [rt.jar:1.6.0_45] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [rt.jar:1.6.0_45] at java.lang.reflect.Method.invoke(Method.java:597) [rt.jar:1.6.0_45] at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203) ... 18 more 13:12:12,180 INFO [org.jboss.as.controller] (Controller Boot Thread) JBAS014774: Service status report JBAS014777: Services which failed to start: service jboss.infinispan.web.repl: org.jboss.msc.service.StartException in service jboss.infinispan.web.repl: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.loaders.CacheLoaderManagerImpl.preload() on object of type CacheLoaderManagerImpl 13:12:12,211 INFO [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://172.17.254.230:9990/management 13:12:12,212 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://172.17.254.230:9990 13:12:12,214 ERROR [org.jboss.as] (Controller Boot Thread) JBAS015875: JBoss AS 7.2.0.Final "Janus" started (with errors) in 11982ms - Started 142 of 262 services (1 services failed or missing dependencies, 118 services are passive or on-demand)
and my infinispan log:
10:01:21,433 ERROR [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-32) ISPN005009: Unexpected error before any request parameters read: java.lang.ClassCastException: java.lang.String cannot be cast to [B at org.infinispan.server.hotrod.AbstractEncoder1x$$anonfun$writeResponse$3.apply(AbstractEncoder1x.scala:118) [infinispan-server-hotrod-5.3.0.Final.jar:5.3.0.Final] at org.infinispan.server.hotrod.AbstractEncoder1x$$anonfun$writeResponse$3.apply(AbstractEncoder1x.scala:116) [infinispan-server-hotrod-5.3.0.Final.jar:5.3.0.Final] at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1156) at org.infinispan.server.hotrod.AbstractEncoder1x.writeResponse(AbstractEncoder1x.scala:116) [infinispan-server-hotrod-5.3.0.Final.jar:5.3.0.Final] at org.infinispan.server.hotrod.HotRodEncoder.encode(HotRodEncoder.scala:69) [infinispan-server-hotrod-5.3.0.Final.jar:5.3.0.Final] at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:66) [netty-3.6.5.Final.jar:] at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59) [netty-3.6.5.Final.jar:] at org.jboss.netty.channel.Channels.write(Channels.java:704) [netty-3.6.5.Final.jar:] at org.jboss.netty.channel.Channels.write(Channels.java:671) [netty-3.6.5.Final.jar:] at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248) [netty-3.6.5.Final.jar:] at org.infinispan.server.core.AbstractProtocolDecoder.writeResponse(AbstractProtocolDecoder.scala:183) [infinispan-server-core-5.3.0.Final.jar:5.3.0.Final] at org.infinispan.server.hotrod.HotRodDecoder.customDecodeKey(HotRodDecoder.scala:160) [infinispan-server-hotrod-5.3.0.Final.jar:5.3.0.Final] at org.infinispan.server.core.AbstractProtocolDecoder.decodeKey(AbstractProtocolDecoder.scala:125) [infinispan-server-core-5.3.0.Final.jar:5.3.0.Final] at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:73) [infinispan-server-core-5.3.0.Final.jar:5.3.0.Final] at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:49) [infinispan-server-core-5.3.0.Final.jar:5.3.0.Final] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500) [netty-3.6.5.Final.jar:] at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) [netty-3.6.5.Final.jar:] at org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:394) [infinispan-server-core-5.3.0.Final.jar:5.3.0.Final] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.6.5.Final.jar:] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.6.5.Final.jar:] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.6.5.Final.jar:] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107) [netty-3.6.5.Final.jar:] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) [netty-3.6.5.Final.jar:] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) [netty-3.6.5.Final.jar:] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.6.5.Final.jar:] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_45] at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]
here's my jboss infinispan subsystem config:
...
<cache-container name="web" aliases="standard-session-cache" default-cache="repl" module="org.jboss.as.clustering.web.infinispan">
<transport lock-timeout="60000"/>
<replicated-cache name="repl" mode="ASYNC" batching="true" remote-timeout="30000" start="EAGER">
<remote-store cache="usertable" socket-timeout="60000" tcp-no-delay="true" fetch-state="false" preload="true" purge="false" shared="true">
<remote-server outbound-socket-binding="remote-store-hotrod-server1"/>
<remote-server outbound-socket-binding="remote-store-hotrod-server2"/>
</remote-store>
</replicated-cache>
...
</cache-container>
...
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
...
<outbound-socket-binding name="remote-store-hotrod-server1">
<remote-destination host="172.17.254.231" port="11222"/>
</outbound-socket-binding>
<outbound-socket-binding name="remote-store-hotrod-server2">
<remote-destination host="172.17.254.232" port="11222"/>
</outbound-socket-binding>
</socket-binding-group>
...
then my infinispan config:
... <subsystem xmlns="urn:infinispan:server:core:5.3" default-cache-container="clustered"> <cache-container name="clustered" default-cache="default"> <transport executor="infinispan-transport" lock-timeout="60000"/> ... <distributed-cache name="usertable" mode="SYNC" owners="1" start="EAGER"> </distributed-cache> </cache-container> ... </subsystem> ...
I realized that Jboss AS 7.2.0.Final uses infinispan 5.2.1.Final but my infinispan data grid version is 5.3.0.Final, therefore I changed the infinispan version to 5.2.1.Final and started a infinispan cluster with startServer.sh. Unluckily, I got the same exception.
Here's some documents or discussions I found for reference:
https://community.jboss.org/thread/197216
I looked through the forum and stackoverflow for some clue, ended up with nothing else.
Any ideas or any advice about using RemoteCacheStore are appreciated.
