JBoss 7.1 Hotrod Remote Store Startup Availability
I'm working on setting up a remote store for my infinispan cache, that's been easy enough but I've run into an issue with starting up a JBoss node when there is a hotrod node down. I have multiple hotrod nodes configured and the remote-store references each, my assumption was that if any of the hotrod nodes listed are available the jboss server would be able to get the server topology and proceed with no issues.
Remote store config:
<remote-store cache="web" socket-timeout="60000" shared="true" purge="false">
<remote-server outbound-socket-binding="hotrod-socket-1"/>
<remote-server outbound-socket-binding="hotrod-socket-2"/>
</remote-store>
My issue looks similar to this infinispan/hotrod issue that was fixed in 5.1.0 (the version on 7.1.1 is 5.1.2): https://community.jboss.org/message/628041
Is there a parallel issue with the hotrod remote store and JBoss 7? It seems like this list of hotrod servers should be used to find a running instance and should continue working if there are invalid nodes in the cluster?
Startup Error:
13:45:52,360 ERROR [org.jboss.msc.service.fail] (pool-16-thread-1) MSC00001: Failed to start service jboss.infinispan.web.http-session-local-root: org.jboss.msc.service.StartException in service jboss.infinispan.web.dist: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.loaders.CacheLoaderManagerImpl.start() on object
at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:62)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) [rt.jar:1.6.0_30]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [rt.jar:1.6.0_30]
at java.lang.Thread.run(Unknown Source) [rt.jar:1.6.0_30]
Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.loaders.CacheLoaderManagerImpl.start() on object
at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:238)
at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:882)
at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:637)
at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:626)
at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:173)
at org.infinispan.CacheImpl.start(CacheImpl.java:499)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:626)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:516)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:530)
at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:148)
at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:139)
at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:76)
at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:59)
... 3 more
Caused by: org.infinispan.CacheException: Unable to start cache loaders
at org.infinispan.loaders.CacheLoaderManagerImpl.start(CacheLoaderManagerImpl.java:132)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_30]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) [rt.jar:1.6.0_30]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) [rt.jar:1.6.0_30]
at java.lang.reflect.Method.invoke(Unknown Source) [rt.jar:1.6.0_30]
at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:236)
... 16 more
Caused by: org.infinispan.client.hotrod.exceptions.TransportException:: Could not fetch transport
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:254)
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:143)
at org.infinispan.client.hotrod.RemoteCacheManager.ping(RemoteCacheManager.java:536)
at org.infinispan.client.hotrod.RemoteCacheManager.createRemoteCache(RemoteCacheManager.java:518)
at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:435)
at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:431)
at org.infinispan.loaders.remote.RemoteCacheStore.start(RemoteCacheStore.java:164)
at org.infinispan.loaders.CacheLoaderManagerImpl.start(CacheLoaderManagerImpl.java:129)
... 21 more
Caused by: org.infinispan.client.hotrod.exceptions.TransportException:: Could not connect to server: /10.42.120.30:11222
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:88)
at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:57)
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1212)
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:250)
... 28 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.6.0_30]
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) [rt.jar:1.6.0_30]
at sun.nio.ch.SocketAdaptor.connect(Unknown Source) [rt.jar:1.6.0_30]
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:79)
... 31 more
