10 Replies Latest reply on May 4, 2005 7:04 PM by clebert.suconic

    Connection Timeout into Pooled Invokers

    clebert.suconic

      I have someone running Specj with Pooled invokers at their configuration.

      Right at the beggining of the execution, I'm having this stack-trace:

      java.rmi.ConnectException: Failed to communicate; nested exception is:
       java.net.SocketTimeoutException: Read timed out
       at org.jboss.invocation.pooled.interfaces.PooledInvokerProxy.invoke(PooledInvokerProxy.java:351)
       at org.jboss.invocation.InvokerInterceptor.invokeInvoker(InvokerInterceptor.java:227)
       at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:167)
       at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
       at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:55)
       at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:97)
       at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:86)
       at $Proxy4.scheduleWorkOrder(Unknown Source)
       at org.spec.jappserver.driver.PlannedLine.createVehicle(PlannedLine.java:365)
       at org.spec.jappserver.driver.LargeOrderLine$LrgLine.run(LargeOrderLine.java:288)
      Caused by: java.net.SocketTimeoutException: Read timed out
       at java.net.SocketInputStream.socketRead0(Native Method)
       at java.net.SocketInputStream.read(SocketInputStream.java:129)
       at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
       at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
       at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2200)
       at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2490)
       at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2500)
       at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1267)
       at java.io.ObjectInputStream.readObject(ObjectInputStream.java:339)
       at org.jboss.invocation.pooled.interfaces.PooledInvokerProxy.invoke(PooledInvokerProxy.java:335)
       ... 9 more
      ~
      


      We've tried to increase timeouts and we didn't have any improvements. Here is my current configuration.

      
       <mbean code="org.jboss.invocation.pooled.server.PooledInvoker"
       name="jboss:service=invoker,type=pooled">
       <attribute name="NumAcceptThreads">1</attribute>
       <attribute name="MaxPoolSize">600</attribute>
       <attribute name="ClientMaxPoolSize">600</attribute>
       <attribute name="SocketTimeout">240000</attribute>
       <attribute name="ServerBindAddress">${jboss.bind.address}</attribute>
       <attribute name="ServerBindPort">4445</attribute>
       <attribute name="ClientConnectAddress">${jboss.bind.address}</attribute>
       <attribute name="ClientConnectPort">0</attribute>
       <attribute name="EnableTcpNoDelay">false</attribute>
      
       <depends optional-attribute-name="TransactionManagerService">jboss:service=TransactionManager</depends>
       </mbean>
      
      
      
      This is happening just when all the load is being initialized. So it's kind of everybody starting at the same time.
      
      So, I need to understand if a high initial load would cause this (as I only have one AcceptThread), or if a delay in the application (like query delays) would be causing this.
      
      
      Any help is appreciated.
      
      
      thanks,
      
      
      Clebert Suconic
      


        • 1. Re: Connection Timeout into Pooled Invokers
          clebert.suconic

          oops... sorry...

          My text get inside the code... my bad:


          This is happening just when all the load is being initialized. So it's kind of everybody starting at
          the same time.

          So, I need to understand if a high initial load would cause this (as I only have one AcceptThread),
          or if a delay in the application (like query delays) would be causing this.


          Any help is appreciated.


          thanks,


          Clebert Suconic

          • 2. Re: Connection Timeout into Pooled Invokers

            Moved to the benchmarking forum, although this is not really a development issue.

            • 3. Re: Connection Timeout into Pooled Invokers

               


              java.net.SocketTimeoutException: Read timed out
              at java.net.SocketInputStream.socketRead0(Native Method)
              at java.net.SocketInputStream.read(SocketInputStream.java:129)
              at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
              at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
              at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2200)
              at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2490)
              at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2500)
              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1267)
              at java.io.ObjectInputStream.readObject(ObjectInputStream.java:339)
              at org.jboss.invocation.pooled.interfaces.PooledInvokerProxy.invoke(PooledInvokerProxy.java:


              Means the client was waiting for a response from the server, but got bored
              waiting and closed the connection.

              • 4. Re: Connection Timeout into Pooled Invokers

                The default is 60 seconds and can be changed with

                <attribute name="SocketTimeout">60000</attribute>
                


                60 seconds looks like a stupid timeout when the default transaction timeout is 300 seconds (or five minutes).

                • 5. Re: Connection Timeout into Pooled Invokers

                  Also, if you are using 4.0.x
                  make sure you enable interrupt threads on the transaction manager
                  to stop threads blocking inside crappy DBs that don't detect deadlocks or other
                  resource problems.
                  http://wiki.jboss.org/wiki/Wiki.jsp?page=JBoss4FAQ

                  • 6. Re: Connection Timeout into Pooled Invokers
                    clebert.suconic

                    Thanks...

                    I wasn't sure if the accept thread would take any collateral affects when everybody is starting new threads. I know I could have profiled it myself but I needed a quick answer for now. Thanks for the help.

                    And just one thing about the timeout... I was using 4 minutes at my configuration. I've copied the wrong version. And I will increase it to 5 minutes.


                    Thanks a lot,

                    Clebert

                    • 7. Re: Connection Timeout into Pooled Invokers
                      clebert.suconic

                      Thanks for appointing me the threading blocking.

                      I was having other non related problems that can be related to this.

                      Thank you so much,


                      Clebert

                      • 8. Re: Connection Timeout into Pooled Invokers
                        clebert.suconic

                        Talking about transactions and Pooled invokers,

                        I'm also setting the ClientUserTransaction to use pooled invokers.

                        I have tested and I didn't find any problems until now. I'm just wondering if someone sees any issue on this option.

                        Here is the changed configuration:

                         <mbean code="org.jboss.tm.usertx.server.ClientUserTransactionService"
                         name="jboss:service=ClientUserTransaction"
                         xmbean-dd="resource:xmdesc/ClientUserTransaction-xmbean.xml">
                         <depends>
                         <mbean code="org.jboss.invocation.jrmp.server.JRMPProxyFactory"
                         name="jboss:service=proxyFactory,target=ClientUserTransactionFactory">
                         <attribute name="InvokerName">jboss:service=invoker,type=pooled</attribute> <!-- here -->
                         <attribute name="TargetName">jboss:service=ClientUserTransaction</attribute>
                         <attribute name="JndiName">UserTransactionSessionFactory</attribute>
                         <attribute name="ExportedInterface">org.jboss.tm.usertx.interfaces.UserTransactionSessionFactory</attribute>
                         <attribute name="ClientInterceptors">
                         <interceptors>
                         <interceptor>org.jboss.proxy.ClientMethodInterceptor</interceptor>
                         <interceptor>org.jboss.invocation.InvokerInterceptor</interceptor>
                         </interceptors>
                         </attribute>
                         <depends>jboss:service=invoker,type=pooled</depends> <!-- here -->
                         </mbean>
                         </depends>
                         <depends optional-attribute-name="TxProxyName">
                         <mbean code="org.jboss.invocation.jrmp.server.JRMPProxyFactory"
                         name="jboss:service=proxyFactory,target=ClientUserTransaction">
                         <attribute name="InvokerName">jboss:service=invoker,type=pooled</attribute> <!-- here -->
                         <attribute name="TargetName">jboss:service=ClientUserTransaction</attribute>
                         <attribute name="JndiName"></attribute>
                         <attribute name="ExportedInterface">org.jboss.tm.usertx.interfaces.UserTransactionSession</attribute>
                         <attribute name="ClientInterceptors">
                         <interceptors>
                         <interceptor>org.jboss.proxy.ClientMethodInterceptor</interceptor>
                         <interceptor>org.jboss.invocation.InvokerInterceptor</interceptor>
                         </interceptors>
                         </attribute>
                         <depends>jboss:service=invoker,type=pooled</depends> <!-- here -->
                         </mbean>
                         </depends>
                        
                         </mbean>
                        




                        • 9. Re: Connection Timeout into Pooled Invokers

                          Why would it cause any issue?

                          Pooled = TCP/IP
                          JRMP = RMI

                          The invoker is just mechanism to transport the begin()/commit()
                          event to the server.

                          Now if you needed a feature that required a full RMI implementation (like remote classloading) then it would be an issue.

                          • 10. Re: Connection Timeout into Pooled Invokers
                            clebert.suconic

                            I just wanted to make sure I'm taking the right decision and anyone (you :-) ) would see an issue on this options. (issue regarding performance for example)


                            Thanks again