8 Replies Latest reply on Apr 27, 2015 3:30 PM by humanbeing

    Range of ports infinispan accepts by default: replication timeout happens due to port blocking

    subashsdm

      Hi all,

       

      I came across with this issue, where infinispan cache replication time outs.

      Here is the error log I got. Seems this ports got blocked, and hence time outs. "Replication timeout for blx25ao03-38795"

      So what is the default range of ports that infinispan supports or is it possible to change the range of ports ?

      In fact I looked at the jgroups-tcp and udp xmls. There in  jgroups-tcp.xml I saw an attribute called, port_range="30".(I didn't see such for jgroups-udp.xml).

      If this is this range what is the initial port value. Then  range should be( initvalue - initivalue+30). Isn't it ?

      Here I am assuming 38795 is the port which hangs, according to logs. Correct me if I have read the logs incorrectly.

      I need to identify the udp/tcp port range infinispan accepts to get rid from this issue. Appreciate any help on this?



      org.infinispan.util.concurrent.TimeoutException: Replication timeout for blx25ao03-38795

       

          at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:421)

          at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:100)

          at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:124)

          at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)

          at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:217)

          at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:199)

          at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:193)

          at org.infinispan.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:114)

          at org.infinispan.interceptors.ReplicationInterceptor.visitPutKeyValueCommand(ReplicationInterceptor.java:78)

          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

          at org.infinispan.interceptors.LockingInterceptor.visitPutKeyValueCommand(LockingInterceptor.java:198)

          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)

          at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:57)

          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

          at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:183)

          at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:132)

          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

        • 1. Re: Range of ports infinispan accepts by default: replication timeout happens due to port blocking
          mircea.markus

          Can you post the logs from all the servers?

          What version of ISPN are you using?

          • 2. Re: Range of ports infinispan accepts by default: replication timeout happens due to port blocking
            subashsdm

            Hi Markus,

            Thanks for the quick response. The version of infinispan I use is 4.1.0.CR2.

            Actually the logs are same. This is a scenario which has two identical servers deployed. From each server I am getting this.

             

            After digging deep, the issue seems blocking the mentioned port from a firewall. If I can get to know the port range what infinispan uses, I can try my end opening those ports

            and check whether this issue still reproduces.

            • 3. Re: Range of ports infinispan accepts by default: replication timeout happens due to port blocking
              mircea.markus

              The last Final ISPN release is 5.1.6.FInal, you might want to conisder upgrading to that.

              I doubt it that it is a port issue, as if at that stage ISPN would only do an RPC if it could contact the other members of the cluster.

              The port should be present in the jgroups configuration file.

              • 4. Re: Range of ports infinispan accepts by default: replication timeout happens due to port blocking
                subashsdm

                Hi Markus,

                For the moment I can't move to the new version.

                Following is the jgroups-tcp.xml I used. Can  you plase tell mew how this port range suppose to work ? i..e

                Here port_range =30  bind port 7800.(hence range should be 7800-7830 ?? or Am I doing something wrong ?)

                If so, my problem is how can then it timeouts on port 38795 (I have bold this ort number in the initial error log)

                 

                 

                 

                 

                <config xmlns="urn:org:jgroups"

                        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                        xsi:schemaLocation="urn:org:jgroups file:schema/JGroups-2.8.xsd">

                   <TCP bind_port="7800"

                        loopback="true"

                        port_range="30"

                        recv_buf_size="20000000"

                        send_buf_size="640000"

                        discard_incompatible_packets="true"

                        max_bundle_size="64000"

                        max_bundle_timeout="30"

                        enable_bundling="true"

                        use_send_queues="true"

                        sock_conn_timeout="300"

                        enable_diagnostics="false"

                 

                 

                        thread_pool.enabled="true"

                        thread_pool.min_threads="2"

                        thread_pool.max_threads="30"

                        thread_pool.keep_alive_time="5000"

                        thread_pool.queue_enabled="false"

                        thread_pool.queue_max_size="100"

                        thread_pool.rejection_policy="Discard"

                 

                 

                        oob_thread_pool.enabled="true"

                        oob_thread_pool.min_threads="2"

                        oob_thread_pool.max_threads="30"

                        oob_thread_pool.keep_alive_time="5000"

                        oob_thread_pool.queue_enabled="false"

                        oob_thread_pool.queue_max_size="100"

                        oob_thread_pool.rejection_policy="Discard"       

                         />

                 

                 

                   <!--<TCPPING timeout="3000"-->

                            <!--initial_hosts="localhost[7800],localhost[7801]}"-->

                            <!--port_range="5"-->

                            <!--num_initial_members="3"/>-->

                 

                 

                   <MPING bind_addr="${jgroups.bind_addr:127.0.0.1}" break_on_coord_rsp="true"

                      mcast_addr="${jgroups.udp.mcast_addr:228.6.7.8}" mcast_port="${jgroups.udp.mcast_port:46655}" ip_ttl="${jgroups.udp.ip_ttl:2}"

                      num_initial_members="3"/>

                 

                 

                   <MERGE2 max_interval="30000"

                           min_interval="10000"/>

                   <FD_SOCK/>

                   <FD timeout="3000" max_tries="3"/>

                   <VERIFY_SUSPECT timeout="1500"/>

                   <pbcast.NAKACK

                         use_mcast_xmit="false" gc_lag="0"

                         retransmit_timeout="300,600,1200,2400,4800"

                         discard_delivered_msgs="false"/>

                   <UNICAST timeout="300,600,1200"/>

                   <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"

                                  max_bytes="400000"/>

                   <pbcast.GMS print_local_addr="false" join_timeout="7000" view_bundling="true"/>

                   <FC max_credits="2000000"

                       min_threshold="0.10"/>

                   <FRAG2 frag_size="60000"/>

                   <pbcast.STREAMING_STATE_TRANSFER/>

                   <!-- <pbcast.STATE_TRANSFER/> -->

                   <pbcast.FLUSH timeout="0"/>

                </config>

                • 5. Re: Range of ports infinispan accepts by default: replication timeout happens due to port blocking
                  subashsdm

                  Hi Markus,

                  So as I am reproducing this I wonder if I increase timeouts would help this. If so, what are the possible timeouts that should be increased to overcome this timeout issue (there are multiple timeouts available) ?

                  In other words If I change that  through system properties, what are the properties need to be added . Here [1] are the system props I found. But no timeout related stuff there.

                   

                  [1] - https://community.jboss.org/wiki/SystemProps

                  • 6. Re: Range of ports infinispan accepts by default: replication timeout happens due to port blocking
                    mircea.markus

                    I think it's replTmeout you're after.

                     

                    I've just seen that you posted this question on a different thread as well, please keep related qustions on the same forum thread from now on - this would help other users to get acces to this info.

                    • 7. Re: Range of ports infinispan accepts by default: replication timeout happens due to port blocking
                      subashsdm

                      Hi Markus,

                      Thanks for the response. Will try this and update this thread.

                      • 8. Re: Range of ports infinispan accepts by default: replication timeout happens due to port blocking
                        humanbeing

                        I know this may be a long shot but I have to ask.  Were you able to get past this issue?  If so, how? I'm dealing with it now on wildfly 8.2.0 Final.  Thank you.