Failure on Control Connection : Messages going into ether

[martin.wickus]

Environment in this case using:
JBoss Messaging 1.4.0.GA
JBoss Remoting 2.2.2.SP1

Seeing the following warning:

Code:
2008-02-18 13:53:43,993 WARN [org.jboss.remoting.transport.bisocket.BisocketServerInvoker] org.jbos
s.remoting.transport.bisocket.BisocketServerInvoker$ControlMonitorTimerTask@72bd77: detected failure
on control connection Thread[control: Socket[addr=/10.140.177.50,port=58029,localport=1654],5,main]
(a3w4x10-9m4crq-fcsan8o9-1-fcsan9pl-9: requesting new control connection
2008-02-18 13:53:43,993 DEBUG [org.jboss.remoting.transport.bisocket.BisocketClientInvoker] getting
secondary locator

This continues for a while then eventually I get exception saying could not connect. I guess at that point I've exhausted the connections in the client pool.

However, before I get the could not connect exception, there is a long period where I can still send JMS messages, yet they never end up on my queue. No JMSException reported either.

My installation was running fine for the last month under this codebase and only recently started showing this problem. I guess the real problem is that something has changed in our network and as a result control connections are experiencing problems. I will investigate that issue our side, but it exposed what looks like a bug in JBoss Messaging / JBoss Remoting.

I've modified my bisocket configuration from config listing 1 to config listing 2 and it seems to be behaving better. I could try updating to a later version of the JBM/JBR libraries... but only want to do so if I can find the Jira marking the issue as closed.

Another user reported the issue is also present in JBM 1.4.0CR2.

Thread here:
http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4130498#4130498

bisocket config listing 1 (old)

<mbean code="org.jboss.remoting.transport.Connector"
 name="jboss.messaging:service=Connector,transport=bisocket"
 display-name="Bisocket Transport Connector">
 <attribute name="Configuration">
 <config>
 <invoker transport="bisocket">
 <attribute name="marshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
 <attribute name="unmarshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
 <attribute name="dataType" isParam="true">jms</attribute>
 <attribute name="socket.check_connection" isParam="true">false</attribute>
 <attribute name="timeout" isParam="true">0</attribute>
 <attribute name="serverBindAddress">${jboss.bind.address}</attribute>
 <attribute name="serverBindPort">${jboss.bisocket.remoting.transport.port}</attribute>
 <attribute name="secondaryBindPort">${jboss.bisocket.remoting.transport.secondary.port}</attribute>
 <attribute name="leasePeriod">10000</attribute>
 <attribute name="clientSocketClass" isParam="true">org.jboss.jms.client.remoting.ClientSocketWrapper</attribute>
 <attribute name="serverSocketClass" isParam="true">org.jboss.jms.server.remoting.ServerSocketWrapper</attribute>
 <attribute name="numberOfRetries" isParam="true">1</attribute>
 <attribute name="numberOfCallRetries" isParam="true">1</attribute>
 <attribute name="clientMaxPoolSize" isParam="true">50</attribute>
 </invoker>
 <handlers>
 <handler subsystem="JMS">org.jboss.jms.server.remoting.JMSServerInvocationHandler</handler>
 </handlers>
 </config>
 </attribute>
 </mbean>

bisocket config listing 2 (updated)

<mbean code="org.jboss.remoting.transport.Connector"
 name="jboss.messaging:service=Connector,transport=bisocket"
 display-name="Bisocket Transport Connector">
 <attribute name="Configuration">
 <config>
 <invoker transport="bisocket">
 <attribute name="marshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
 <attribute name="unmarshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
 <attribute name="dataType" isParam="true">jms</attribute>
 <attribute name="socket.check_connection" isParam="true">false</attribute>
 <attribute name="timeout" isParam="true">0</attribute>
 <attribute name="serverBindAddress">${jboss.bind.address}</attribute>
 <attribute name="serverBindPort">${jboss.bisocket.remoting.transport.port}</attribute>
 <attribute name="secondaryBindPort">${jboss.bisocket.remoting.transport.secondary.port}</attribute>
 <attribute name="clientLeasePeriod" isParam="true">10000</attribute>
 <attribute name="clientSocketClass" isParam="true">org.jboss.jms.client.remoting.ClientSocketWrapper</attribute>
 <attribute name="serverSocketClass" isParam="true">org.jboss.jms.server.remoting.ServerSocketWrapper</attribute>
 <attribute name="numberOfRetries" isParam="true">10</attribute>
 <attribute name="numberOfCallRetries" isParam="true">1</attribute>
 <attribute name="clientMaxPoolSize" isParam="true">1000</attribute>
 <attribute name="pingFrequency" isParam="true">214748364</attribute>
 <attribute name="pingWindowFactor" isParam="true">10</attribute>
 </invoker>
 <handlers>
 <handler subsystem="JMS">org.jboss.jms.server.remoting.JMSServerInvocationHandler</handler>
 </handlers>
 </config>
 </attribute>
 </mbean>

[timfox]

Hi Wickus-

Yes, since the version you are using the default remoting configuration has changed.

I would strongly recommend using the remoting configuration that comes with JBM 1.4.0.SP3.

In particular, there is a known issue with JBoss Remoting control connection pinging, which might cause connections to be prematurely closed.

To remedy this we increased pingFrequency to a very large number:

<attribute name="pingFrequency" isParam="true">214748364</attribute>

This may be causing your problem.

[martin.wickus]

Hi Tim

Thanks for the update. I took a chance last week and did exactly that (config listing 2 from previous forum was modified from JBM 1.4.0.SP3) and after 5 days in production the problem hasn't shown itself again. Looks like the updated config does indeed do the job as you suggest.

PS. The only other change was to make sure I still refer to the older "clientMaxPoolSize" attribute instead of the newer "JBM_clientMaxPoolSize". I may have overcompensated a bit with this figure (1ooo instead of default 200), but haven't had problems.

Regards,
Wickus