JBREM-877: New Socket Connection is being Creat...| JBoss.org Content Archive (Read Only)

15. Re: JBREM-877: New Socket Connection is being Created for Ev

ron_sigal Jan 3, 2009 2:46 AM (in response to ron_sigal)

Hi Vinay,

I'm looking at your thread dumps, and I don't see anything strange from the perspective of Remoting. I see a bunch of Remoting ServerThreads in a WAITING state, which implies they're currently not in use.

The only exception I've noticed is in the server-stack.log file you sent me:

"WorkerThread#21[150.158.73.73:4114]" prio=6 tid=0x2b6c2c00 nid=0x570 in Object.wait() [0x2eaff000..0x2eaffa14]
 java.lang.Thread.State: TIMED_WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x09289300> (a java.lang.Integer)
 at com.barco.cms.openapi.PerspectiveMgtAPIImpl.getEvents(Unknown Source)
 - locked <0x09289300> (a java.lang.Integer)
 at sun.reflect.GeneratedMethodAccessor110.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
 at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
 at com.barco.cms.openapi.PerspectiveMgtAPIImpl.initialSteps(Unknown Source)
 at sun.reflect.GeneratedMethodAccessor84.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:146)
 at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.tx.TxPolicy.invokeInNoTx(TxPolicy.java:66)
 at org.jboss.aspects.tx.TxInterceptor$Never.invoke(TxInterceptor.java:66)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:95)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
 at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:304)
 at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
 at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:908)
 at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:742)
 - locked <0x09a34058> (a org.jboss.remoting.transport.socket.ServerThread)
 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:695)
 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:522)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:230)

which seems to be related to your application code. Could that be where you're experiencing a hang up?

Also, server-stack.log just has 355 threads. That doesn't sound like enough for you to run out of threads.

I'm not getting the picture yet.

16. Re: JBREM-877: New Socket Connection is being Created for Ev

ron_sigal Jan 3, 2009 2:57 AM (in response to ron_sigal)

Hi Clive,

The "invokerDestructionDelay" parameter is processed on the client side, which means your client must have access to the updated version of Remoting. While the server goes to $JBOSS_HOME/server/$CONFIG/lib/default, the client goes to $JBOSS_HOME/client, which the Remoting classes are found in jbossall-client.jar. What you should do is unjar jbossall-client.jar, remove the org.jboss.remoting classes, add the classes from the new jboss-remoting.jar, and jar it back up again. See if that helps.

-Ron

17. Re: JBREM-877: New Socket Connection is being Created for Ev

ron_sigal Jan 4, 2009 1:57 AM (in response to ron_sigal)

I'm continuing the discussion about Vinay Kumar's (vink's) issues on the "Socket Read Timeout on stressing calls for validating produc" thread - http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4199374#4199374.

18. Re: JBREM-877: New Socket Connection is being Created for Ev

kiwi_clive Jan 5, 2009 7:11 AM (in response to ron_sigal)

Hi Ron,

Many thanks for your response. Yes I am being stupid :-)

I'll give that a go.

Cheers,
Clive

19. Re: JBREM-877: New Socket Connection is being Created for Ev

ron_sigal Jan 5, 2009 6:55 PM (in response to ron_sigal)

"kiwi_clive" wrote:

Yes I am being stupid :-)

Noooooo! Not at all.

20. Re: JBREM-877: New Socket Connection is being Created for Ev

kiwi_clive Jan 5, 2009 9:38 PM (in response to ron_sigal)

Hi Ron,

Many thanks for your help. We saw an immediate performance gain, halving response time. This has proved to be the single most effective performance enhancement in our tuning efforts so many thanks for providing the fix.

Most appreciated !

Clive

21. Re: JBREM-877: New Socket Connection is being Created for Ev

ron_sigal Jan 5, 2009 9:45 PM (in response to ron_sigal)

Clive, you made my day. :)

22. Re: JBREM-877: New Socket Connection is being Created for Ev

dschneider Feb 14, 2009 10:41 AM (in response to ron_sigal)

The problem described in this thread would seem to be what we're experiencing, though it's with the HTTP transport used by JBoss Message 1.4.0.SP3. The remoting version on both our server and client is "JBossRemoting Version 2.2.2.SP4 (Bluto)", which we're running under JBoss AS 4.2.2.

Assuming I can get the clients upgraded to the later version of the remoting jar, I'm still a bit confused about where exactly I would set the invokerDestructionDelay. In this case it would seem to be in the JMS server's deploy/jboss-messaging.sar/remoting-http-service.xml file as that's what appears to control how the JMS client's listeners connect back to the server. That file currently has this content:

<server>

 <mbean code="org.jboss.remoting.transport.Connector"
 name="jboss.messaging:service=Connector,transport=http"
 display-name="HTTP Transport Connector">
 <attribute name="Configuration">
 <config>
 <invoker transport="http">
 <!-- There should be no reason to change these parameters - warning!
 Changing them may stop JBoss Messaging working correctly -->
 <attribute name="marshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
 <attribute name="unmarshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
 <attribute name="dataType" isParam="true">jms</attribute>
 <attribute name="serverBindAddress">${jboss.bind.address}</attribute>
 <attribute name="serverBindPort">4460</attribute>
 <attribute name="callbackStore">org.jboss.remoting.callback.BlockingCallbackStore</attribute>
 <!-- End immutable parameters -->

 <!-- The period for polling for messages on the server -->
 <!-- attribute name="callbackPollPeriod" isParam="true">102</attribute -->
 <attribute name="callbackPollPeriod" isParam="true">5003</attribute>

 <!-- The period of sending pings to the server -->
 <attribute name="clientLeasePeriod" isParam="true">10000</attribute>
 </invoker>
 <handlers>
 <handler subsystem="JMS">org.jboss.jms.server.remoting.JMSServerInvocationHandler</handler>
 </handlers>
 </config>
 </attribute>
 </mbean>

</server>

So my guess is that I'd just add the following additional attribute:

<attribute name="invokerDestructionDelay">6000</attribute>

Could someone confirm this is the correct approach?

What also has me confused is that I've already upped the callbackPollPeriod attribute from its default of 102 (ms) to 5003, so I would be expecting connections from the clients only every 5 seconds or so. But what I'm actually seeing is a steady stream of very short-lived socket connections, which results in anywhere from 3-10 ESTABLISHED connections with hundreds in TIME_WAIT just from a single client even when the queue is empty and there's nothing to read. Is there a reason why the poll period of 5 seconds didn't reduce the frequency of the connections?

Thanks,
Dave

23. Re: JBREM-877: New Socket Connection is being Created for Ev

immortalex Feb 19, 2009 12:22 AM (in response to ron_sigal)

I think, you forgot isParam="true".

24. Re: JBREM-877: New Socket Connection is being Created for Ev

ron_sigal Mar 3, 2009 2:58 AM (in response to ron_sigal)

Hi Dave,

"ImmortAlex" wrote:

I think, you forgot isParam="true".

Alex is right. "invokerDestructionDelay" affects the creation of invokers on the client side, and setting "isParam" to "true" causes the "invokerDestructionDelay" parameter to be included in the InvokerLocator, which means it can be seen by the client.

I should tell you, though, that the use of "invokerDestructionDelay" may not help you when you're using the http transport. See the thread "Socket keep alive with client EJB and RMI over HTTP" (http://www.jboss.org/index.html?module=bb&op=viewtopic&t=137758&postdays=0&postorder=asc&start=0) in this forum. Unfortunately, I don't have any good answers right now.

"Dave Schneider" wrote:

What also has me confused is that I've already upped the callbackPollPeriod attribute from its default of 102 (ms) to 5003, so I would be expecting connections from the clients only every 5 seconds or so.

You're using an old version of JBossMessaging which, prior to versions 1.4.1.GA, 1.4.0.SP3.CP04, and 1.4.2.GA, didn't know about blocking/nonblocking mode, so you're seeing the default, nonblocking, behavior. I'm not sure why increasing "callbackPollPeriod" didn't have an effect, but you probably don't want to be using nonblocking mode anyway, and "callbackPollPeriod" is ignored in blocking mode. For that reason, I would suggest upgrading JBM.

Until recently it was necessary for the client code (JBM in this case) to specify the blocking mode when calling org.jboss.remoting.Client.addListener(), but JBREM-1084 "Allow CallbackPoller to access Client and InvokerLocator parameters" will allow org.jboss.remoting.callback.CallbackPoller to get the blocking mode directly from the InvokerLocator if you add the parameter "useAllParams" to the InvokerLocator with the value set to "true"; i.e., add

<attribute name="useAllParams" isParam="true">true</attribute>

to remoting-http-service.xml. That feature doesn't exist in a released version of Remoting yet, but, if you're interested, I could attach a preview jboss-remoting.jar to JBREM-1084.

By the way, here's remoting-http-service.xml from JBossMessaging 1.4.0.SP3_CP07:

<?xml version="1.0" encoding="UTF-8"?>

<!--
 HTTP-based Remoting service deployment descriptor.

 $Id: remoting-http-service.xml 3222 2007-10-20 12:13:57Z timfox $
 -->

<server>

 <mbean code="org.jboss.remoting.transport.Connector"
 name="jboss.messaging:service=Connector,transport=http"
 display-name="HTTP Transport Connector">
 <attribute name="Configuration">
 <config>
 <invoker transport="http">
 <!-- There should be no reason to change these parameters - warning!
 Changing them may stop JBoss Messaging working correctly -->
 <attribute name="marshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
 <attribute name="unmarshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
 <attribute name="dataType" isParam="true">jms</attribute>
 <attribute name="serverBindAddress">${jboss.bind.address}</attribute>
 <attribute name="serverBindPort">4458</attribute>
 <attribute name="callbackStore">org.jboss.remoting.callback.BlockingCallbackStore</attribute>
 <!-- End immutable parameters -->

 <!-- The period of sending pings to the server -->
 <attribute name="clientLeasePeriod" isParam="true">10000</attribute>

 <!-- Set this to true if you want the HTTP transport to block waiting for server->client traffic.
 Or false if you want it to poll for new traffic periodically. Recommended is true -->
 <attribute name="blockingMode" isParam="true">blocking</attribute>

 <!-- Timeout for blocking. Only has relevance if blockingMode = true -->
 <attribute name="blockingTimeout" isParam="true">30000</attribute>

 <!-- The periodicity of polling. Only has relevance if blockingMode = false -->
 <attribute name="callbackPollPeriod" isParam="true">10000</attribute>
 </invoker>
 <handlers>
 <handler subsystem="JMS">org.jboss.jms.server.remoting.JMSServerInvocationHandler</handler>
 </handlers>
 </config>
 </attribute>
 </mbean>

</server>

Using blocking mode should reduce the number of invocations, and therefore reduce the number of open sockets.

-Ron

25. Re: JBREM-877: New Socket Connection is being Created for Ev

dschneider Mar 4, 2009 4:21 PM (in response to ron_sigal)

Thanks a lot, Ron.

I'll try getting our JBM moved up to at least the CP04 level and configure the HTTP remoting for blocking mode and see if that does what I'm looking for.