Connection Timeout into Pooled Invokers

[clebert.suconic]

I have someone running Specj with Pooled invokers at their configuration.

Right at the beggining of the execution, I'm having this stack-trace:

java.rmi.ConnectException: Failed to communicate; nested exception is:
 java.net.SocketTimeoutException: Read timed out
 at org.jboss.invocation.pooled.interfaces.PooledInvokerProxy.invoke(PooledInvokerProxy.java:351)
 at org.jboss.invocation.InvokerInterceptor.invokeInvoker(InvokerInterceptor.java:227)
 at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:167)
 at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
 at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:55)
 at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:97)
 at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:86)
 at $Proxy4.scheduleWorkOrder(Unknown Source)
 at org.spec.jappserver.driver.PlannedLine.createVehicle(PlannedLine.java:365)
 at org.spec.jappserver.driver.LargeOrderLine$LrgLine.run(LargeOrderLine.java:288)
Caused by: java.net.SocketTimeoutException: Read timed out
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
 at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
 at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2200)
 at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2490)
 at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2500)
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1267)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:339)
 at org.jboss.invocation.pooled.interfaces.PooledInvokerProxy.invoke(PooledInvokerProxy.java:335)
 ... 9 more
~

We've tried to increase timeouts and we didn't have any improvements. Here is my current configuration.

 <mbean code="org.jboss.invocation.pooled.server.PooledInvoker"
 name="jboss:service=invoker,type=pooled">
 <attribute name="NumAcceptThreads">1</attribute>
 <attribute name="MaxPoolSize">600</attribute>
 <attribute name="ClientMaxPoolSize">600</attribute>
 <attribute name="SocketTimeout">240000</attribute>
 <attribute name="ServerBindAddress">${jboss.bind.address}</attribute>
 <attribute name="ServerBindPort">4445</attribute>
 <attribute name="ClientConnectAddress">${jboss.bind.address}</attribute>
 <attribute name="ClientConnectPort">0</attribute>
 <attribute name="EnableTcpNoDelay">false</attribute>

 <depends optional-attribute-name="TransactionManagerService">jboss:service=TransactionManager</depends>
 </mbean>



This is happening just when all the load is being initialized. So it's kind of everybody starting at the same time.

So, I need to understand if a high initial load would cause this (as I only have one AcceptThread), or if a delay in the application (like query delays) would be causing this.


Any help is appreciated.


thanks,


Clebert Suconic

[clebert.suconic]

oops... sorry...

My text get inside the code... my bad:


This is happening just when all the load is being initialized. So it's kind of everybody starting at
the same time.

So, I need to understand if a high initial load would cause this (as I only have one AcceptThread),
or if a delay in the application (like query delays) would be causing this.


Any help is appreciated.


thanks,


Clebert Suconic

[adrian.brock]

Moved to the benchmarking forum, although this is not really a development issue.

[adrian.brock]

java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2200)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2490)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2500)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1267)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:339)
at org.jboss.invocation.pooled.interfaces.PooledInvokerProxy.invoke(PooledInvokerProxy.java:

Means the client was waiting for a response from the server, but got bored
waiting and closed the connection.

[adrian.brock]

The default is 60 seconds and can be changed with

<attribute name="SocketTimeout">60000</attribute>

60 seconds looks like a stupid timeout when the default transaction timeout is 300 seconds (or five minutes).

[adrian.brock]

Also, if you are using 4.0.x
make sure you enable interrupt threads on the transaction manager
to stop threads blocking inside crappy DBs that don't detect deadlocks or other
resource problems.
http://wiki.jboss.org/wiki/Wiki.jsp?page=JBoss4FAQ

[clebert.suconic]

Thanks...

I wasn't sure if the accept thread would take any collateral affects when everybody is starting new threads. I know I could have profiled it myself but I needed a quick answer for now. Thanks for the help.

And just one thing about the timeout... I was using 4 minutes at my configuration. I've copied the wrong version. And I will increase it to 5 minutes.


Thanks a lot,

Clebert

[clebert.suconic]

Thanks for appointing me the threading blocking.

I was having other non related problems that can be related to this.

Thank you so much,


Clebert

[clebert.suconic]

Talking about transactions and Pooled invokers,

I'm also setting the ClientUserTransaction to use pooled invokers.

I have tested and I didn't find any problems until now. I'm just wondering if someone sees any issue on this option.

Here is the changed configuration:

 <mbean code="org.jboss.tm.usertx.server.ClientUserTransactionService"
 name="jboss:service=ClientUserTransaction"
 xmbean-dd="resource:xmdesc/ClientUserTransaction-xmbean.xml">
 <depends>
 <mbean code="org.jboss.invocation.jrmp.server.JRMPProxyFactory"
 name="jboss:service=proxyFactory,target=ClientUserTransactionFactory">
 <attribute name="InvokerName">jboss:service=invoker,type=pooled</attribute> <!-- here -->
 <attribute name="TargetName">jboss:service=ClientUserTransaction</attribute>
 <attribute name="JndiName">UserTransactionSessionFactory</attribute>
 <attribute name="ExportedInterface">org.jboss.tm.usertx.interfaces.UserTransactionSessionFactory</attribute>
 <attribute name="ClientInterceptors">
 <interceptors>
 <interceptor>org.jboss.proxy.ClientMethodInterceptor</interceptor>
 <interceptor>org.jboss.invocation.InvokerInterceptor</interceptor>
 </interceptors>
 </attribute>
 <depends>jboss:service=invoker,type=pooled</depends> <!-- here -->
 </mbean>
 </depends>
 <depends optional-attribute-name="TxProxyName">
 <mbean code="org.jboss.invocation.jrmp.server.JRMPProxyFactory"
 name="jboss:service=proxyFactory,target=ClientUserTransaction">
 <attribute name="InvokerName">jboss:service=invoker,type=pooled</attribute> <!-- here -->
 <attribute name="TargetName">jboss:service=ClientUserTransaction</attribute>
 <attribute name="JndiName"></attribute>
 <attribute name="ExportedInterface">org.jboss.tm.usertx.interfaces.UserTransactionSession</attribute>
 <attribute name="ClientInterceptors">
 <interceptors>
 <interceptor>org.jboss.proxy.ClientMethodInterceptor</interceptor>
 <interceptor>org.jboss.invocation.InvokerInterceptor</interceptor>
 </interceptors>
 </attribute>
 <depends>jboss:service=invoker,type=pooled</depends> <!-- here -->
 </mbean>
 </depends>

 </mbean>

[adrian.brock]

Why would it cause any issue?

Pooled = TCP/IP
JRMP = RMI

The invoker is just mechanism to transport the begin()/commit()
event to the server.

Now if you needed a feature that required a full RMI implementation (like remote classloading) then it would be an issue.

[clebert.suconic]

I just wanted to make sure I'm taking the right decision and anyone (you :-) ) would see an issue on this options. (issue regarding performance for example)


Thanks again