[gaohoward]

Hi mclu, I looked at the logs and it seems the remoting LocalClientInvoker is the reason that caused the problem. The reason is you have 6 bridges deployed in one node and each bridge contains one local connection (that is the connection to one of the local queues on node X). I looked up the remoting code a bit but so far I don't understand it. From JBM code i can't see anything that can cause this strange problem. In JBoss Remoting documents there is a parameter called "force_remote", default is false if not specified. Can you try to set it to "true" and try to see if it can make any difference?

[mclu]

I will try it on monday. Currently I try to get my IPMA-Level D certificate and I am back at my client on monday.

I will keep you informend in this thread!

[mclu]

Hi Howard!

I did some tests over the weekend.
I added
true
or
true
to my invoker configuration (see below)

But this results in timeouts while getting connections so bridge or in general JMS does not work then:

Caused by: org.jboss.remoting.ConnectionFailedException: Timed out trying to create control socket
 at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.handleConnect(BisocketClientInvoker.java:272)
 at org.jboss.remoting.MicroRemoteClientInvoker.connect(MicroRemoteClientInvoker.java:263)
 at org.jboss.remoting.Client.connect(Client.java:1528)
 at org.jboss.remoting.Client.connect(Client.java:468)
 at org.jboss.remoting.callback.ServerInvokerCallbackHandler.connect(ServerInvokerCallbackHandler.java:161)
 at org.jboss.remoting.ServerInvoker.getCallbackHandler(ServerInvoker.java:1863)
 at org.jboss.remoting.ServerInvoker.handleInternalInvocation(ServerInvoker.java:1497)
 at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.handleInternalInvocation(BisocketServerInvoker.java:643)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:853)
 at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
 at org.jboss.remoting.Client.invoke(Client.java:1640)
 at org.jboss.remoting.Client.addCallbackListener(Client.java:1709)
 at org.jboss.remoting.Client.addListener(Client.java:927)
 at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:259)

By the way! I am using an SSL Bisocket connector from the the sslbisocket example configuration.

This is my remoting-bisocket-service.xml:

<server>
 <mbean code="org.jboss.remoting.transport.Connector"
 name="jboss.messaging:service=Connector,transport=sslbisocket"
 display-name="SSL Bisocket Transport Connector">
 <attribute name="Configuration">
 <config>
 <invoker transport="sslbisocket">
 <attribute name="marshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
 <attribute name="unmarshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
 <attribute name="dataType" isParam="true">jms</attribute>
 <attribute name="socket.check_connection" isParam="true">false</attribute>
 <attribute name="serverBindAddress">${jboss.bind.address}</attribute>
 <attribute name="serverBindPort">24457</attribute>
 <attribute name="clientSocketClass" isParam="true">org.jboss.jms.client.remoting.ClientSocketWrapper</attribute>
 <attribute name="serverSocketClass">org.jboss.jms.server.remoting.ServerSocketWrapper</attribute>
 <attribute name="serverSocketFactory">jboss.messaging:service=ServerSocketFactory,type=SSL</attribute>
 <attribute name="numberOfCallRetries" isParam="true">1</attribute>
 <attribute name="pingFrequency" isParam="true">214748364</attribute>
 <attribute name="pingWindowFactor" isParam="true">10</attribute>
 <attribute name="onewayThreadPool">org.jboss.jms.server.remoting.DirectThreadPool</attribute>
 <attribute name="stopLeaseOnFailure" isParam="true">true</attribute>
 <attribute name="clientLeasePeriod" isParam="true">10000</attribute>
 <attribute name="timeout" isParam="true">0</attribute>
 <attribute name="numberOfRetries" isParam="true">10</attribute>
 <attribute name="JBM_clientMaxPoolSize" isParam="true">200</attribute>
 <attribute name="callbackTimeout">10000</attribute>
 <!-- <attribute name="force_remote">true</attribute> -->
 </invoker>
 <handlers>
 <handler subsystem="JMS">org.jboss.jms.server.remoting.JMSServerInvocationHandler</handler>
 </handlers>
 </config>
 </attribute>
 <depends>jboss.messaging:service=ServerSocketFactory,type=SSL</depends>
 </mbean>

 <mbean code="org.jboss.remoting.security.SSLServerSocketFactoryService"
 name="jboss.messaging:service=ServerSocketFactory,type=SSL"
 display-name="SSL Server Socket Factory">
 <depends optional-attribute-name="SSLSocketBuilder" proxy-type="attribute">jboss.messaging:service=SocketBuilder,type=SSL</depends>
 </mbean>

 <mbean code="org.jboss.remoting.security.SSLSocketBuilder"
 name="jboss.messaging:service=SocketBuilder,type=SSL"
 display-name="SSL Server Socket Factory Builder">
 <attribute name="UseSSLServerSocketFactory">true</attribute>
 </mbean>

</server>

What should we do now?

[gaohoward]

I need time to investigate. Also can you please post the last timeout issue to remoting forum to see if there is a quick response on this? Thanks.

[ron_sigal]

Hi guys,

I posted an answer on the related Remoting forum.

I don't know what's going on here, but, for what it's worth, I can tell you that Remoting doesn't create leases when the client and server are in the same JVM (unless you set "force_remote" to "true"). Don't know if that's relevant.

Howard, why do you think there's a problem with LocalClientInvoker?

-Ron

[gaohoward]

Hi Ron, I thought about the when a bridge is broken, it will close the connections and retry. Because all the bridges are deployed in same VM, so they may share one instance of LocalClientInvoker. If one of the bridge close the invoker, others also lost it. Is it possible?

[mclu]

Hi Howard.
I think I have to solve this connection Lost issue first.
I have added a Topic in the remoting forum for that too:
http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4219740#4219740

Maybe this influence was no influence... Maybe it was just the side effect of this connection lost error.
I also have tried the forceRemote setting with no effect.

The biggest issue in my situation is that it happens so often but also that the messages are stuck after the reconnect. This has to do with Jira entry
https://jira.jboss.org/jira/browse/JBMESSAGING-1543 I think.

My test is simple:
1. start 2 servers A and B on different machines.
2. deploy a bridge on A which pull messages from B to A
3. unplug network cable --> result connection lost error: my reconnect time is 50 seconds
4. after 50 seconds the reconnect Bridge message occurs
5. add messages to the queue on A. Those are not transfered unless I restart the bridge via jmx or restart the server

[gaohoward]

i'll investigate. Thanks mclu.

[mclu]

Howard.
I did some more tests with this "message stuck after reconnect" problem.

First a correction of my test. In step 5 I said restarting the bridge or the server A solves the problem. But this is not true. Only after I restartet the message sending server B restarts sending the messages to A. So it seams somehow that the messages are "send" to the wrong/old connection...

I also continued testing and I can NOT reproduce it if I use the settings:

<attribute name="force_remote" isParam="true">true</attribute>

But without this force remote its easy to reproduce it in my system. If I disconnect the Network cable longer that 3 Seconds the reconnect works.
But in cases of shorter disconnects like 1 second I see this problem.

One hint maybe:
As you know I have 2 bridges deployed. One for each direction. So the above testbridge BtoA and one AtoB. At this short disconnect I only see Errors on A related to the AtoB Bridge BUT NOT from this other bridge. Maybe this Bridge still thinks that the connection is fine but in the end it isn't because of the short network disconnect?!?

[mclu]

PROBLEMS SOLVED:

1. Message stuck issue
I solved it with my workaround with this force remote setting!
But I think it is related to this error:
https://jira.jboss.org/jira/browse/JBMESSAGING-1456
which is fixed now.

2. Connection error!
I am using the SSLBisocketInvoker and used the example configuration. This example configuration has not changed over the last versions!
As I saw now is that the standard remoting bisocket invoker has changed a lot. For example there are new configurations like:

 <attribute name="validatorPingPeriod" isParam="true">10000</attribute>
 <attribute name="validatorPingTimeout" isParam="true">5000</attribute>

If you miss them the defaults are 2 seconds periode and 1 second timeout.
This one second is too short on utilized systems and slower remote networks!!

Howard, Tim
there are more parameters in the default NON-SSL configuration that have changed.
Can you please validate which one are also applicable on the ssl invoker and then update the ssl configuration example at remoting-sslbisocket-service.xml??

[mclu]

Just found the related Bug for that ping time:

https://jira.jboss.org/jira/browse/JBMESSAGING-1487

But as I said only the non ssl file were updated.

[gaohoward]

Hi mclu,

I'll take a look on the sslbisoeket configure. Thanks for the info.

Howard

[ovi]

I had the same problem which on my side was caused by accessing 2 different data-sources within the same transaction (XA transaction on Oracle database).

E.g. Method X is calling method Y which does an database A insert and calls method Z which does a database B update.

The solution was to annotate method Y and method Z with: @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)

Hope this helps.