How to set socket timeouts?

[belcar]

Hi
I have a Web Application that has a Thread polling for incoming messages on a remote JBM 1.4.5 queue. After some time the thread no longer picks up any messages, so I assume the socket connection has timed out.

If I'm correct, you can manage these settings in deploy/jboss-messaging.sar/remoting-bisocket-service.xml. Actually, I'm a bit overwhelmed by the number of switches defined here and it is not clear to me which one I need exactly.

Could someone please hint me on the switch I need?
All values are still default ones.

Thank you in advance

[gaohoward]

Not sure if I understand the situation. Can you clarify how you poll messages? Are you using MDB?

[belcar]

No, I do not use a MDB. The application opens up a single connection and session and manages a thread pool. I've got one daemon thread that polls for incoming messages and dispatches incoming messages to a worker thread obtained from that pool.

That works great, but after X-time messages are no longer being consumed. If I restart the application, it processes all persisted messages on the queue.

Thank you

[belcar]

It was setup this way to make sure traffic flow in 1 direction only.

[gaohoward]

How is your remoting config looks like? It seems to me maybe your remoting ping timeout is too small. If the connection between your message client and server is not good, try to enlarge the ping period.

[belcar]

There is a firewall between both machines, so I enabled those sections:
The configuration looks as follows:

<attribute name="marshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
<attribute name="unmarshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
<attribute name="dataType" isParam="true">jms</attribute>
<attribute name="socket.check_connection" isParam="true">false</attribute>
<attribute name="serverBindAddress">${jboss.bind.address}</attribute>
<attribute name="serverBindPort">4457</attribute>
<attribute name="clientSocketClass" isParam="true">org.jboss.jms.client.remoting.ClientSocketWrapper</attribute>
<attribute name="serverSocketClass">org.jboss.jms.server.remoting.ServerSocketWrapper</attribute>
<attribute name="onewayThreadPool">org.jboss.jms.server.remoting.DirectThreadPool</attribute>

<!-- Firewall settings enabled-->
<attribute name="numberOfCallRetries" isParam="true">5</attribute>
<attribute name="pingFrequency" isParam="true">30000</attribute>
<attribute name="pingWindowFactor" isParam="true">71582</attribute>
<attribute name="generalizeSocketException" isParam="true">true</attribute>

<!-- Now remoting supports socket write timeout configuration. Uncomment this if you need it. -->
<!-- <attribute name="writeTimeout" isParam="true">30000</attribute> -->
<!-- End immutable parameters -->

<attribute name="stopLeaseOnFailure" isParam="true">true</attribute>
<!-- Periodicity of client pings. Server window by default is twice this figure -->
<attribute name="clientLeasePeriod" isParam="true">10000</attribute>
<attribute name="validatorPingPeriod" isParam="true">10000</attribute>
<attribute name="validatorPingTimeout" isParam="true">5000</attribute>

<attribute name="failureDisconnectTimeout" isParam="true">0</attribute>
<attribute name="callbackErrorsAllowed">1</attribute>
<attribute name="registerCallbackListener">false</attribute>
<attribute name="useClientConnectionIdentity" isParam="true">true</attribute>

<attribute name="timeout" isParam="true">0</attribute>

<!-- Number of seconds to wait for a connection in the client pool to become free -->
<attribute name="numberOfRetries" isParam="true">10</attribute>

<!-- Max Number of connections in client pool. This should be significantly higher than
 the max number of sessions/consumers you expect -->
<attribute name="JBM_clientMaxPoolSize" isParam="true">200</attribute>

<!-- The maximum time to wait before timing out on trying to write a message to socket for delivery -->
<attribute name="callbackTimeout">10000</attribute>

<!-- Use these parameters to specify values for binding and connecting control connections to
 work with your firewall/NAT configuration-->
<attribute name="secondaryBindPort">55794</attribute>
<attribute name="secondaryConnectPort">55794</attribute>

Thank you

[gaohoward]

please try to uncomment the 'writeTimeout' and give it a value suitable for your case.

This controls the socket write timeout when the JBM server is trying to deliver messages through a socket that has been closed by the firewall.

[belcar]

I've enabled the setting but it had no effect. The client polling application is still loosing its link. I can build in a reconnect but this is fairly unfancy...

Any ideas on how I could proceed?

Thank you

[gaohoward]

Did you find any exceptions in your log? You can turn on the
TRACE log level to see more details.

If you believe it's a real problem, you can raise a jira and attach logs/tests if any.

Thanks
Howard

[belcar]

Hi Howard

I'm not sure if it is a bug or not but here's the following:

I've TRACED org.jboss.jms and org.jboss.remoting on the polling client.
I can see the client ping the server every 30 seconds.
I see no exceptions in the logs.

However after 10 or 15 minutes this client no longer finds messages on the queue, but the ping remains functioning. If I restart my client process running on a remote jboss 4.2.3, the client perfectly finds all messages it has missed. This client is the only reader.

2009-11-24 16:30:01,952 TRACE [org.jboss.jms.wireformat.JMSWireFormat] Wrote packet
2009-11-24 16:30:01,952 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[a1d92a, bisocket://10.249.6.1:4457] done writing invocation to marshaller
2009-11-24 16:30:01,952 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[a1d92a, bisocket://10.249.6.1:4457] reading version from input stream
2009-11-24 16:30:01,954 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[a1d92a, bisocket://10.249.6.1:4457] read version 22 from input stream
2009-11-24 16:30:01,954 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[a1d92a, bisocket://10.249.6.1:4457] reading response from unmarshaller
2009-11-24 16:30:01,954 TRACE [org.jboss.jms.wireformat.JMSWireFormat] Reading
2009-11-24 16:30:01,954 TRACE [org.jboss.jms.wireformat.JMSWireFormat] Stream is already DataInputStream :)
2009-11-24 16:30:01,954 TRACE [org.jboss.jms.wireformat.JMSWireFormat] Created packet SerializedPacket[null]
2009-11-24 16:30:01,954 TRACE [org.jboss.jms.wireformat.JMSWireFormat] Reading packet
2009-11-24 16:30:01,954 TRACE [org.jboss.jms.wireformat.JMSWireFormat] Read packet
2009-11-24 16:30:01,955 TRACE [org.jboss.jms.wireformat.JMSWireFormat] Returning payload: InvocationResponse[dd09a6, InvocationResponse[1a498b6, true]]
2009-11-24 16:30:01,955 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[a1d92a, bisocket://10.249.6.1:4457] returned NEW ClientSocketWrapper[Socket[addr=/10.249.6.1,port=4457,localport=37299].a6aa31] to pool
2009-11-24 16:30:01,955 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[a1d92a, bisocket://10.249.6.1:4457] released semaphore: 200
2009-11-24 16:30:01,955 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[a1d92a, bisocket://10.249.6.1:4457] received response InvocationResponse[dd09a6, InvocationResponse[1a498b6, true]]
2009-11-24 16:30:01,955 TRACE [org.jboss.remoting.MicroRemoteClientInvoker] SocketClientInvoker[a1d92a, bisocket://10.249.6.1:4457] received InvocationResponse so going to return response's return value of InvocationResponse[1a498b6, true]
2009-11-24 16:30:01,955 TRACE [org.jboss.remoting.LeasePinger] LeasePinger[3j001-jg0dw9-g2et9fcn-1-g2et9kaa-9:SocketClientInvoker[a1d92a, bisocket://10.249.6.1:4457](3j001-jg0dw9-g2et9fcn-1-g2et9ka6-8)] successfully pinged the server
2009-11-24 16:30:01,969 TRACE [org.jboss.remoting.ConnectionValidator] ConnectionValidator[a4effe:SocketClientInvoker[5f8245, bisocket://10.249.6.1:4457], pingPeriod=10000 ms] sending PING tied to lease
2009-11-24 16:30:01,969 TRACE [org.jboss.remoting.ConnectionValidator] pinging, sending InvocationRequest[1b5eaed, self, $PING$] over SocketClientInvoker[5f8245, bisocket://10.249.6.1:4457]
2009-11-24 16:30:01,969 TRACE [org.jboss.remoting.MicroRemoteClientInvoker] SocketClientInvoker[5f8245, bisocket://10.249.6.1:4457](1) invoking InvocationRequest[1b5eaed, self, $PING$]
2009-11-24 16:30:01,969 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[5f8245, bisocket://10.249.6.1:4457] retryCount: 0
2009-11-24 16:30:01,969 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[5f8245, bisocket://10.249.6.1:4457] obtained semaphore: 199
2009-11-24 16:30:01,969 TRACE [org.jboss.jms.client.remoting.ClientSocketWrapper] checking open connection
2009-11-24 16:30:01,969 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[5f8245, bisocket://10.249.6.1:4457] reusing pooled connection: NEW ClientSocketWrapper[Socket[addr=/10.249.6.1,port=4457,localport=49479].fbb5f5]
2009-11-24 16:30:01,969 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[5f8245, bisocket://10.249.6.1:4457] writing version 22 on output stream
2009-11-24 16:30:01,969 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[5f8245, bisocket://10.249.6.1:4457] writing invocation to marshaller
2009-11-24 16:30:01,969 TRACE [org.jboss.jms.wireformat.JMSWireFormat] Writing InvocationRequest[1b5eaed, self, $PING$]
2009-11-24 16:30:01,969 TRACE [org.jboss.jms.wireformat.JMSWireFormat] Stream is a DataOutputStream
2009-11-24 16:30:01,969 TRACE [org.jboss.jms.wireformat.JMSWireFormat] Writing packet: SerializedPacket[InvocationRequest[1b5eaed, self, $PING$]]
2009-11-24 16:30:01,969 TRACE [org.jboss.remoting.transport.socket.TimedOutputStream] org.jboss.remoting.transport.socket.TimedOutputStream@a53502 scheduled org.jboss.remoting.transport.socket.TimedOutputStream$OutputTimerTask@1678aef: 30000
2009-11-24 16:30:01,970 TRACE [org.jboss.jms.wireformat.JMSWireFormat] Wrote packet
2009-11-24 16:30:01,970 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[5f8245, bisocket://10.249.6.1:4457] done writing invocation to marshaller
2009-11-24 16:30:01,970 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[5f8245, bisocket://10.249.6.1:4457] reading version from input stream
2009-11-24 16:30:01,971 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[5f8245, bisocket://10.249.6.1:4457] read version 22 from input stream
2009-11-24 16:30:01,971 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[5f8245, bisocket://10.249.6.1:4457] reading response from unmarshaller
2009-11-24 16:30:01,971 TRACE [org.jboss.jms.wireformat.JMSWireFormat] Reading
2009-11-24 16:30:01,971 TRACE [org.jboss.jms.wireformat.JMSWireFormat] Stream is already DataInputStream :)
2009-11-24 16:30:01,971 TRACE [org.jboss.jms.wireformat.JMSWireFormat] Created packet SerializedPacket[null]
2009-11-24 16:30:01,971 TRACE [org.jboss.jms.wireformat.JMSWireFormat] Reading packet
2009-11-24 16:30:01,971 TRACE [org.jboss.jms.wireformat.JMSWireFormat] Read packet
2009-11-24 16:30:01,971 TRACE [org.jboss.jms.wireformat.JMSWireFormat] Returning payload: InvocationResponse[1fbaf73, true]
2009-11-24 16:30:01,971 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[5f8245, bisocket://10.249.6.1:4457] returned NEW ClientSocketWrapper[Socket[addr=/10.249.6.1,port=4457,localport=49479].fbb5f5] to pool
2009-11-24 16:30:01,971 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[5f8245, bisocket://10.249.6.1:4457] released semaphore: 200
2009-11-24 16:30:01,971 TRACE [org.jboss.remoting.transport.socket.MicroSocketClientInvoker] SocketClientInvoker[5f8245, bisocket://10.249.6.1:4457] received response InvocationResponse[1fbaf73, true]
2009-11-24 16:30:01,971 TRACE [org.jboss.remoting.MicroRemoteClientInvoker] SocketClientInvoker[5f8245, bisocket://10.249.6.1:4457] received InvocationResponse so going to return response's return value of true
2009-11-24 16:30:01,971 TRACE [org.jboss.remoting.ConnectionValidator] ConnectionValidator got successful ping using SocketClientInvoker[5f8245, bisocket://10.249.6.1:4457]

If I would open a JIRA issue (not sure it's a bug though), I guess this is a remoting issue? Advantage is I can add a screenshot of the setup.

Thanks for your help
G.

[belcar]

Howard

I've talked to the guys that manage the 2 firewalls in between and they've disabled all timeouts. After that change the polling client still stops receiving messages after a while. I've also enabled TRACE for jms and remoting on the JBM 1.4.5 server and I can see ping requests/responses.

Maybe, instead of creating my own polling client, just to rule out any threading issue it might be better to implement this as an MDB. Only issue I have with this is with the documentation. It is not updated for EJB3...

http://www.jboss.org/community/wiki/HowDoIConfigureAnMDBToTalkToARemoteQueue

Can you please tell me what you would do & how I can proceed?

Big thanks

[belcar]

I have deployed the same code as a stand-alone Java Application launched from the CLI on that same linux server, and it keeps serving messages, even after a nights sleep.

[gaohoward]

Hi,

If you are using MDB, you can get better help from the AS forum.

Howard