2 Replies Latest reply on Sep 29, 2011 6:47 AM by galder.zamarreno

    Problem with hotrod client operation

    jwchoi

      Hello,

       

      I am having a problem with hotrod client operation during making a demo.

      I found similar problem on https://issues.jboss.org/browse/ISPN-488

       

      Please check my scenario and give me a suggestion if I'm wrong;

       

      Test Environment.

      - RHEL 5.5 64bit 5EA( 2 Rod Servers, 2 JBoss AS 5, 1 Web Server)

      - Sun JDK 1.6

        - Infinispan 5.0.0 Final

       

      This demo process works well:

      1. Start two hot rod servers.

      2. Connect servers using JEE sample web app(infinispan.client.hotrod.server_list = 192.168.0.100:11222;192.168.0.101:11222)

      3. Put object to servers.

      4. Make several calls, the calls are being load balanced in round robin fashion.

       

       

      10:25:49,484 INFO  [TcpTransportFactory] ISPN004014: New server added(/192.168.0.100:11222), adding to the pool.

      10:25:49,484 INFO  [TcpTransportFactory] ISPN004014: New server added(/192.168.0.101:11222), adding to the pool.

      10:26:43,453 INFO  [HotRodOperation] ISPN004006: New topology: [/192.168.0.100:11222]

       

      But, below process didn't work :

      1. Start second hot rod server.

      2. Deploy and start web application.

      3. Throw exception when start application :

       

      Exception

      10:57:07,375 ERROR [TcpTransportFactory] ISPN004017: Could not fetch transport

      org.infinispan.client.hotrod.exceptions.TransportException:: Could not connect to server: /192.168.0.100:11222

              at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:84)

              at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:54)

              at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1179)

              at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:243)

              at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:154)

              at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.getTransport(AbstractKeyOperation.java:57)

              at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:60)

              at org.infinispan.client.hotrod.impl.RemoteCacheImpl.get(RemoteCacheImpl.java:320)

              at org.apache.jsp.counter_jsp._jspService(counter_jsp.java:77)

              at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

              at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

              at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:387)

              at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:320)

              at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)

              at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

              at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)

              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)

              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:173)

              at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)

              at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)

       

      Hot Rod client properties is this :

       

      hotrod-client.properties

      infinispan.client.hotrod.transport_factory = org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory

      infinispan.client.hotrod.server_list = 192.168.0.100:11222;192.168.0.101:11222

      infinispan.client.hotrod.marshaller = org.infinispan.marshall.jboss.GenericJBossMarshaller

      infinispan.client.hotrod.async_executor_factory = org.infinispan.client.hotrod.impl.async.DefaultAsyncExecutorFactory

      infinispan.client.hotrod.default_executor_factory.pool_size = 1

      infinispan.client.hotrod.default_executor_factory.queue_size = 10000

      infinispan.client.hotrod.hash_function_impl.1 = org.infinispan.client.hotrod.impl.consistenthash.ConsistentHashV1

      infinispan.client.hotrod.tcp_no_delay = true

      infinispan.client.hotrod.ping_on_startup = true

      infinispan.client.hotrod.request_balancing_strategy = org.infinispan.client.hotrod.impl.transport.tcp.RoundRobinBalancingStrategy

      infinispan.client.hotrod.key_size_estimate = 64

      infinispan.client.hotrod.value_size_estimate = 512

      infinispan.client.hotrod.force_return_values = false

      ## below is connection pooling config

      maxActive=10

      maxTotal = 10

      maxIdle = 100

      testOnBorrow=true

       

       

      It seems that testOnBorrow didn't work.

      Below process didn't work as well :

      1. Start two hot rod servers.

      2. Connect servers using JEE sample web app(infinispan.client.hotrod.server_list = 192.168.0.100:11222;192.168.0.101:21222)

      3. Put object to servers.

      4. Make several calls, the calls are being load balanced in round robin fashion.

      5. Shutdown the first hot rod server.

      6. Call counter web application on JBoss 5

      7. Exception cause like this :

       

      Exception

      10:30:43,453 INFO  [TcpTransportFactory] ISPN004014: New server added(/192.168.0.100:11222), adding to the pool.

      10:30:43,453 INFO  [TcpTransportFactory] ISPN004016: Server not in cluster anymore(/192.168.0.101:11222), removing from the pool.

      10:30:50,515 WARN  [TcpTransport] ISPN004008: Could not connect to server: /192.168.0.100:11222

      java.net.ConnectException: Connection refused: connect

              at sun.nio.ch.Net.connect(Native Method)

              at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500)

              at java.nio.channels.SocketChannel.open(SocketChannel.java:146)

              at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:74)

              at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:54)

              at org.apache.commons.pool.impl.GenericKeyedObjectPool.addObject(GenericKeyedObjectPool.java:1668)

              at org.apache.commons.pool.impl.GenericKeyedObjectPool.ensureMinIdle(GenericKeyedObjectPool.java:2038)

              at org.apache.commons.pool.impl.GenericKeyedObjectPool.ensureMinIdle(GenericKeyedObjectPool.java:2003)

              at org.apache.commons.pool.impl.GenericKeyedObjectPool.access$1600(GenericKeyedObjectPool.java:203)

              at org.apache.commons.pool.impl.GenericKeyedObjectPool$Evictor.run(GenericKeyedObjectPool.java:2282)

              at java.util.TimerThread.mainLoop(Timer.java:512)

              at java.util.TimerThread.run(Timer.java:462)

      10:30:50,453 INFO  [TcpTransportFactory] ISPN004014: New server added(/192.168.0.101:11222), adding to the pool.

      10:30:53,453 INFO  [TcpTransportFactory] ISPN004016: Server not in cluster anymore(/192.168.0.100:11222), removing from the pool.

      10:31:13,515 WARN  [TcpTransport] ISPN004008: Could not connect to server: /192.168.0.100:11222

      java.net.ConnectException: Connection refused: connect

              at sun.nio.ch.Net.connect(Native Method)

              at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:500)

              at java.nio.channels.SocketChannel.open(SocketChannel.java:146)

              at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:74)

              at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:54)

              at org.apache.commons.pool.impl.GenericKeyedObjectPool.addObject(GenericKeyedObjectPool.java:1668)

              at org.apache.commons.pool.impl.GenericKeyedObjectPool.ensureMinIdle(GenericKeyedObjectPool.java:2038)

              at org.apache.commons.pool.impl.GenericKeyedObjectPool.ensureMinIdle(GenericKeyedObjectPool.java:2003)

              at org.apache.commons.pool.impl.GenericKeyedObjectPool.access$1600(GenericKeyedObjectPool.java:203)

              at org.apache.commons.pool.impl.GenericKeyedObjectPool$Evictor.run(GenericKeyedObjectPool.java:2282)

              at java.util.TimerThread.mainLoop(Timer.java:512)

              at java.util.TimerThread.run(Timer.java:462)

       

       

      Initialized Java code snippet here :

      Connection Code

      try {

                        Properties prop = new Properties();

       

                        prop.load(this.getClass().getClassLoader().getResourceAsStream(config));

                        RemoteCacheManager cacheManager = new RemoteCacheManager(prop);

       

                              RemoteCacheContext.setCacheManager(cacheManager);

                          } catch(Exception e1) {

              e1.printStackTrace();

                          } finally {

       

        }

       

       

      This happens after a server shuts down or if all hot rod servers are NOT started.

       

      How to fix this problem?

      Am I wrong about hot rod client configuration?