long live consumer stops working

[bodrin]

jboss-messaging-1.4.0.SP2 / jboss-4.2.0.GA

I have a single queue consumer which servives cluster restarts like the JMSConnectionExceptionListener : http://wiki.jboss.org/wiki/Wiki.jsp?page=JMSConnectionExceptionListener.

My test is sending 20 messages at a time and the consumer receives them.

The problem is that if I leave it to not send messages for some time, when I go to luchn for example, and then I'm sending the messages, but the consumer does not receive them anymore.
If I check for the messages via jmx-console - I see that they are in the queue.

Every five minutes say I see some repeatable debug messages:

[RMI RenewClean-[centos9.localdomain:1201]] DEBUG sun.rmi.client.ref - RMI RenewClean-[centos9.localdomain:1201]: get connection
[RMI RenewClean-[centos9.localdomain:1201]] DEBUG sun.rmi.transport.tcp - RMI RenewClean-[centos9.localdomain:1201]: create connection
[RMI RenewClean-[centos9.localdomain:1201]] DEBUG sun.rmi.transport.tcp.proxy - RMI RenewClean-[centos9.localdomain:1201]: host: centos9.localdomain, port: 1201
[RMI RenewClean-[centos9.localdomain:1201]] DEBUG sun.rmi.loader - RMI RenewClean-[centos9.localdomain:1201]: name = "java.rmi.dgc.Lease", codebase = "http://centos9.localdomain:8183/"
[RMI RenewClean-[centos9.localdomain:1201]] DEBUG sun.rmi.loader - RMI RenewClean-[centos9.localdomain:1201]: name = "java.rmi.dgc.VMID", codebase = "http://centos9.localdomain:8183/"
[RMI RenewClean-[centos9.localdomain:1201]] DEBUG sun.rmi.loader - RMI RenewClean-[centos9.localdomain:1201]: name = "[B", codebase = ""
[RMI RenewClean-[centos9.localdomain:1201]] DEBUG sun.rmi.loader - RMI RenewClean-[centos9.localdomain:1201]: name = "java.rmi.server.UID", codebase = "http://centos9.localdomain:8183/"
[RMI RenewClean-[centos9.localdomain:1201]] DEBUG sun.rmi.client.ref - RMI RenewClean-[centos9.localdomain:1201]: free connection (reuse = true)
[RMI RenewClean-[centos9.localdomain:1201]] DEBUG sun.rmi.transport.tcp - RMI RenewClean-[centos9.localdomain:1201]: reuse connection
[RMI RenewClean-[centos9.localdomain:1201]] DEBUG sun.rmi.transport.tcp - RMI RenewClean-[centos9.localdomain:1201]: create reaper
[RMI Scheduler(0)] DEBUG sun.rmi.transport.tcp - RMI Scheduler(0): close connection

Here is also the thread dump:

Full thread dump Java HotSpot(TM) Client VM (1.6.0_02-b06 mixed mode):

"Thread-41" daemon prio=6 tid=0x0b468800 nid=0x9fc in Object.wait() [0x0ccff000..0x0ccffc94]
 java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 at java.lang.Object.wait(Object.java:485)
 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
 - locked <0x03961d88> (a java.lang.Object)
 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
 at java.lang.Thread.run(Unknown Source)

"WorkerThread#5[10.58.100.119:2193]" daemon prio=6 tid=0x0b14c400 nid=0x106c runnable [0x0cb6f000..0x0cb6fd14]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(Unknown Source)
 at java.io.BufferedInputStream.fill(Unknown Source)
 at java.io.BufferedInputStream.read(Unknown Source)
 - locked <0x03971e80> (a java.io.BufferedInputStream)
 at java.io.FilterInputStream.read(Unknown Source)
 at org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)
 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)
 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)

"control: Socket[addr=centos9.localdomain/10.58.100.119,port=2193,localport=36069]" daemon prio=6 tid=0x0be29800 nid=0xcf0 runnable [0x0ccaf000..0x0ccafd94]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(Unknown Source)
 at java.net.SocketInputStream.read(Unknown Source)
 at java.io.FilterInputStream.read(Unknown Source)
 at org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlConnectionThread.run(BisocketServerInvoker.java:741)

"control: Socket[addr=centos9.localdomain/10.58.100.119,port=2193,localport=36068]" daemon prio=6 tid=0x0be29400 nid=0xde4 runnable [0x0cc0f000..0x0cc0fa14]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(Unknown Source)
 at java.net.SocketInputStream.read(Unknown Source)
 at java.io.FilterInputStream.read(Unknown Source)
 at org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlConnectionThread.run(BisocketServerInvoker.java:741)

"Thread-36" daemon prio=6 tid=0x0b4e0800 nid=0x1434 in Object.wait() [0x0cc5f000..0x0cc5fb94]
 java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 at java.lang.Object.wait(Object.java:485)
 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
 - locked <0x03949e80> (a java.lang.Object)
 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
 at java.lang.Thread.run(Unknown Source)

"Thread-27" daemon prio=6 tid=0x0b1b3400 nid=0x12c4 in Object.wait() [0x0cd9f000..0x0cd9fc14]
 java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 at java.lang.Object.wait(Object.java:485)
 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
 - locked <0x03901d40> (a java.lang.Object)
 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
 at java.lang.Thread.run(Unknown Source)

"JDWP Transport Listener: dt_socket" daemon prio=6 tid=0x0bf45c00 nid=0x1318 runnable [0x00000000..0x00000000]
 java.lang.Thread.State: RUNNABLE

"WorkerThread#0[10.58.100.119:2706]" daemon prio=6 tid=0x0b47ec00 nid=0x134 runnable [0x0b77f000..0x0b77fd94]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(Unknown Source)
 at java.io.BufferedInputStream.fill(Unknown Source)
 at java.io.BufferedInputStream.read(Unknown Source)
 - locked <0x038f1990> (a java.io.BufferedInputStream)
 at java.io.FilterInputStream.read(Unknown Source)
 at org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)
 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)
 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)

"RMI Scheduler(0)" daemon prio=6 tid=0x0aaf5400 nid=0x1270 waiting on condition [0x0ca7f000..0x0ca7fd94]
 java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for <0x037cc6f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
 at java.util.concurrent.locks.LockSupport.park(Unknown Source)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
 at java.util.concurrent.DelayQueue.take(Unknown Source)
 at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
 at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)

"RMI RenewClean-[centos9.localdomain:1201]" daemon prio=6 tid=0x0aaef800 nid=0xbcc in Object.wait() [0x0c9df000..0x0c9dfa94]
 java.lang.Thread.State: TIMED_WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 at java.lang.ref.ReferenceQueue.remove(Unknown Source)
 - locked <0x037cc7c0> (a java.lang.ref.ReferenceQueue$Lock)
 at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)

Do you have any idea what could be the problem?

[bodrin]

Actually the problem is when the jboss is on machine ther then the client / producer one.

In order to reproduce the problem easy I have created two more examples - long live consumer and long live producer.
You can copy them into the jboss-messaging examples directory and compile/start them via ant (just like the other examples there).

Note that the consumer example will exit only if you hit enter.
So:
1. start the consumer
2. run the producer
- You should observe that the consumer is receiving 10 messages.
3. Leave the consumer to run for a long time (some hours for example).
4. run the producer again
- the consumer does not receive anything?!

The two examples are available via:
http://www.myjavaserver.com/~dobrin/jboss/examples.zip

I think most of the consumer applications run continiously in order to process any incoming messages, but this problem breaks all such scenarios.

[timfox]

Can you first with a more recent version and see if the problem is still there?

JBM 1.4.0.SP3 or 1.4.1 with JBossRemoting 2.2.2.SP4 ?

If problem re-occurs please post a simple test case or simple instructions on how to replicate and someone will take a look.

[bodrin]

Ok. I tried to setup jboss-messaging-1.4.0.SP3 on jboss-4.2.2.GA with jboss-remoting-2.2.2.SP4.

I have done all the steps described in [4.1.1. Automated Installation] for clustered installation with two nodes.

When starting there is a problem with the remoting configuration and the messaging factories failed to start:

16:48:15,281 ERROR [ConnectionFactory] Parameter pingFrequency is not specified in the remoting congiguration
16:48:15,296 ERROR [ExceptionUtil] org.jboss.jms.server.connectionfactory.ConnectionFactory@1c06a6d startService
java.lang.IllegalArgumentException: Failed to deploy connection factory since remoting configuration seems incorrect.
 at org.jboss.jms.server.connectionfactory.ConnectionFactory.startService(ConnectionFactory.java:155)
 at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
 at org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:196)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
 at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
 at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
 at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:995)
 at $Proxy0.start(Unknown Source)
 at org.jboss.system.ServiceController.start(ServiceController.java:417)
 at org.jboss.system.ServiceController.start(ServiceController.java:435)
 at org.jboss.system.ServiceController.start(ServiceController.java:435)
 at org.jboss.system.ServiceController.start(ServiceController.java:435)
 at org.jboss.system.ServiceController.start(ServiceController.java:435)
 at org.jboss.system.ServiceController.start(ServiceController.java:435)
 at org.jboss.system.ServiceController.start(ServiceController.java:435)
 at org.jboss.system.ServiceController.start(ServiceController.java:435)
 at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
 at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
 at $Proxy4.start(Unknown Source)
 at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
 at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
 at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
 at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
 at $Proxy52.start(Unknown Source)
 at org.jboss.deployment.XSLSubDeployer.start(XSLSubDeployer.java:197)
 at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
 at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
 at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
 at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
 at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
 at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
 at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
 at $Proxy10.deploy(Unknown Source)
 at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421)
 at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:634)
 at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263)
 at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:336)
 at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
 at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
 at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
 at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
 at $Proxy0.start(Unknown Source)
 at org.jboss.system.ServiceController.start(ServiceController.java:417)
 at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
 at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
 at $Proxy4.start(Unknown Source)
 at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
 at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
 at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
 at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
 at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:766)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
 at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
 at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
 at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
 at $Proxy5.deploy(Unknown Source)
 at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:482)
 at org.jboss.system.server.ServerImpl.start(ServerImpl.java:362)
 at org.jboss.Main.boot(Main.java:200)
 at org.jboss.Main$1.run(Main.java:508)
 at java.lang.Thread.run(Thread.java:619)

Have you any idea what could be the cause and how to fix this?

[jmesnil]

"bodrin" wrote:
Ok. I tried to setup jboss-messaging-1.4.0.SP3 on jboss-4.2.2.GA with jboss-remoting-2.2.2.SP4.

I have done all the steps described in [4.1.1. Automated Installation] for clustered installation with two nodes.

When starting there is a problem with the remoting configuration and the messaging factories failed to start:

16:48:15,281 ERROR [ConnectionFactory] Parameter pingFrequency is not specified in the remoting congiguration

This means that the pingFrequency parameter is not present in the remoting configuration file (presumably remoting-bisocket-service.xml).

You should check that in that file, you have a line like:

<attribute name="pingFrequency" isParam="true">214748364</attribute>

for the bisocket transport configuration.

[bodrin]

It is present. I haven't modified this configuration file at all, but I get:
> ERROR [ConnectionFactory] Parameter pingFrequency is not specified in the remoting congiguration

I have launched a debuger and these are all the params available at runtime:

clientMaxPoolSize=50
clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper
dataType=jms
marshaller=org.jboss.jms.wireformat.JMSWireFormat
numberOfCallRetries=1
numberOfRetries=1
socket.check_connection=false
timeout=0
unmarshaller=org.jboss.jms.wireformat.JMSWireFormat

The remoting-bisocket-service.xml :

<?xml version="1.0" encoding="UTF-8"?>

<!--
 Standard bisocket-based Remoting service deployment descriptor.

 $Id: remoting-bisocket-service.xml 3409 2007-12-04 21:32:54Z timfox $
 -->

<server>

 <!-- Standard bisocket connector - the bisocket transport only opens connection from client->server
 so can be used with firewalls where only outgoing connections are allowed.
 For examples of HTTP and SSL transports see docs/examples -->
 <mbean code="org.jboss.remoting.transport.Connector"
 name="jboss.messaging:service=Connector,transport=bisocket"
 display-name="Bisocket Transport Connector">
 <attribute name="Configuration">
 <config>
 <invoker transport="bisocket">

 <!-- There should be no reason to change these parameters - warning!
 Changing them may stop JBoss Messaging working correctly -->
 <attribute name="marshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
 <attribute name="unmarshaller" isParam="true">org.jboss.jms.wireformat.JMSWireFormat</attribute>
 <attribute name="dataType" isParam="true">jms</attribute>
 <attribute name="socket.check_connection" isParam="true">false</attribute>
 <attribute name="timeout" isParam="true">0</attribute>
 <attribute name="serverBindAddress">${jboss.bind.address}</attribute>
 <attribute name="serverBindPort">4457</attribute>
 <attribute name="clientSocketClass" isParam="true">org.jboss.jms.client.remoting.ClientSocketWrapper</attribute>
 <attribute name="serverSocketClass">org.jboss.jms.server.remoting.ServerSocketWrapper</attribute>
 <attribute name="numberOfCallRetries" isParam="true">1</attribute>
 <attribute name="pingFrequency" isParam="true">214748364</attribute>
 <attribute name="pingWindowFactor" isParam="true">10</attribute>
 <attribute name="onewayThreadPool">org.jboss.jms.server.remoting.DirectThreadPool</attribute>
 <!-- End immutable parameters -->

 <!-- Periodicity of client pings. Server window by default is twice this figure -->
 <attribute name="clientLeasePeriod" isParam="true">10000</attribute>

 <!-- Number of seconds to wait for a connection in the client pool to become free -->
 <attribute name="numberOfRetries" isParam="true">10</attribute>

 <!-- Max Number of connections in client pool. This should be significantly higher than
 the max number of sessions/consumers you expect -->
 <attribute name="JBM_clientMaxPoolSize" isParam="true">200</attribute>

 <!-- Use these parameters to specify values for binding and connecting control connections to
 work with your firewall/NAT configuration
 <attribute name="secondaryBindPort">xyz</attribute>
 <attribute name="secondaryConnectPort">abc</attribute>
 -->

 </invoker>
 <handlers>
 <handler subsystem="JMS">org.jboss.jms.server.remoting.JMSServerInvocationHandler</handler>
 </handlers>
 </config>
 </attribute>
 </mbean>

</server>

[timfox]

"bodrin" wrote:
Ok. I tried to setup jboss-messaging-1.4.0.SP3 on jboss-4.2.2.GA with jboss-remoting-2.2.2.SP4.

I have done all the steps described in [4.1.1. Automated Installation] for clustered installation with two nodes.

When starting there is a problem with the remoting configuration and the messaging factories failed to start:Have you any idea what could be the cause and how to fix this?

Yes, probably your ServiceBindingManager configuration is out of date:

See

http://labs.jboss.com/file-access/default/members/jbossmessaging/freezone/docs/userguide-1.4.1.Beta1/html/installation.html#install.automated
From the user guide:

You must ensure that the config (like above) is identical to that in remoting-bisocket-service.xml With the exception of the actual serverBindPort which clearly must be different for each ports range. Please note that the default JBoss Messaging service binding manager bindings in sample-bindings.xml shipped with JBAS 4.2.0 may be out of date and you will need to copy the config from remoting-bisocket-service.xml. DO NOT just copy and paste from the above example - copy it from the JBoss Messaging distribution.

[timfox]

If you're using old service binding manager configuration with invalid remoting configuration with 1.3.0 this could explain your timeout issues with that...

[bodrin]

Thanks!

I hope you are right..

[bodrin]

Unfortunatelly I have reproduced the problem with JBM.1.4.0.SP3 on jboss-4.2.2.GA with jboss-remoting-2.2.2.SP4.

The examples are uploaded and available here: http://www.myjavaserver.com/~dobrin/jboss/examples.zip.
Just extract them into the jboss-messaging-1.4.0.SP3\examples directory.
They can be compiled/run via ant like all the other examples provided with the JBM package.

The scenrio is:

1. Start JBOSS on machine A (linux). In my case it is a clustered installation with two nodes, but I start only one of them.
2. Start example long-live-queue-consumer on machine B (windows xp) (use run.but, because and forks and no thread dump could be made)
3. Start example long-live-queue-producer on machine B
result: the long-live-queue-consumer receives successfuly 10 messages
4. Leave long-live-queue-consumer running without sending any messages for a few hours (in my case 3h)
5. Start example long-live-queue-producer on machine B
result: expected to see that long-live-queue-consumer will receive 10 messages, but nothing happens


Here is the long-live-queue-consumer thread dump after step 5. :

31.01.2008 15:35:50 onMessage: delegator->JBossMessage[125446]:PERSISTENT, deliveryId=36
-----------------------------------------
31.01.2008 15:35:50 onMessage: delegator->JBossMessage[125447]:PERSISTENT, deliveryId=37
-----------------------------------------
31.01.2008 15:35:50 onMessage: delegator->JBossMessage[125448]:PERSISTENT, deliveryId=38
-----------------------------------------
31.01.2008 15:35:50 onMessage: delegator->JBossMessage[125449]:PERSISTENT, deliveryId=39
Listening for transport dt_socket at address: 3000
2008-01-31 18:09:09
Full thread dump Java HotSpot(TM) Client VM (1.6.0_02-b06 mixed mode):

"JDWP Transport Listener: dt_socket" daemon prio=6 tid=0x0b1a8800 nid=0xfb0 runnable [0x00000000..0x00000000]
 java.lang.Thread.State: RUNNABLE

"Thread-6" daemon prio=6 tid=0x0b502800 nid=0x1098 in Object.wait() [0x0ba0f000..0x0ba0fa14]
 java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x03024c90> (a java.lang.Object)
 at java.lang.Object.wait(Object.java:485)
 at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
 - locked <0x03024c90> (a java.lang.Object)
 at EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
 at java.lang.Thread.run(Unknown Source)

"WorkerThread#0[10.58.100.119:3457]" daemon prio=6 tid=0x0b198400 nid=0x11a4 runnable [0x0b9bf000..0x0b9bfa94]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(Unknown Source)
 at java.io.BufferedInputStream.fill(Unknown Source)
 at java.io.BufferedInputStream.read(Unknown Source)
 - locked <0x03056358> (a java.io.BufferedInputStream)
 at java.io.FilterInputStream.read(Unknown Source)
 at org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)
 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:510)
 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)

"Timer-2" daemon prio=6 tid=0x0b4ea800 nid=0x11bc in Object.wait() [0x0b90f000..0x0b90fb94]
 java.lang.Thread.State: TIMED_WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x02fdcf60> (a java.util.TaskQueue)
 at java.util.TimerThread.mainLoop(Unknown Source)
 - locked <0x02fdcf60> (a java.util.TaskQueue)
 at java.util.TimerThread.run(Unknown Source)

"control: Socket[addr=centos9.localdomain/10.58.100.119,port=3457,localport=34453]" daemon prio=6 tid=0x0b4ee800 nid=0x14c8 runnable [0x0b8bf000..0x0b8bfc14]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(Unknown Source)
 at java.net.SocketInputStream.read(Unknown Source)
 at java.io.FilterInputStream.read(Unknown Source)
 at org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlConnectionThread.run(BisocketServerInvoker.java:741)

"Timer-1" daemon prio=6 tid=0x0b4ee400 nid=0xec0 in Object.wait() [0x0b86f000..0x0b86fc94]
 java.lang.Thread.State: TIMED_WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x02fc8140> (a java.util.TaskQueue)
 at java.util.TimerThread.mainLoop(Unknown Source)
 - locked <0x02fc8140> (a java.util.TaskQueue)
 at java.util.TimerThread.run(Unknown Source)

"Timer-0" daemon prio=6 tid=0x0b1b8000 nid=0x9b0 in Object.wait() [0x0b81f000..0x0b81fd14]
 java.lang.Thread.State: TIMED_WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x02fbca68> (a java.util.TaskQueue)
 at java.util.TimerThread.mainLoop(Unknown Source)
 - locked <0x02fbca68> (a java.util.TaskQueue)
 at java.util.TimerThread.run(Unknown Source)

"RMI Scheduler(0)" daemon prio=6 tid=0x0b1b5800 nid=0xa08 waiting on condition [0x0b3bf000..0x0b3bfd94]
 java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for <0x02f0f6a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
 at java.util.concurrent.locks.LockSupport.park(Unknown Source)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
 at java.util.concurrent.DelayQueue.take(Unknown Source)
 at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
 at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)

"GC Daemon" daemon prio=2 tid=0x0b1b3c00 nid=0x1394 in Object.wait() [0x0b36f000..0x0b36fa14]
 java.lang.Thread.State: TIMED_WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x02f083c0> (a sun.misc.GC$LatencyLock)
 at sun.misc.GC$Daemon.run(Unknown Source)
 - locked <0x02f083c0> (a sun.misc.GC$LatencyLock)

"RMI RenewClean-[centos9.localdomain:1201]" daemon prio=6 tid=0x0b185c00 nid=0x17cc in Object.wait() [0x0b31f000..0x0b31fa94]
 java.lang.Thread.State: TIMED_WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x02f080c0> (a java.lang.ref.ReferenceQueue$Lock)
 at java.lang.ref.ReferenceQueue.remove(Unknown Source)
 - locked <0x02f080c0> (a java.lang.ref.ReferenceQueue$Lock)
 at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)

"Low Memory Detector" daemon prio=6 tid=0x0aae1c00 nid=0x1200 runnable [0x00000000..0x00000000]
 java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x0aae0400 nid=0x11fc waiting on condition [0x00000000..0x0afdf81c]
 java.lang.Thread.State: RUNNABLE

"JDWP Event Helper Thread" daemon prio=6 tid=0x0aad5400 nid=0xf84 runnable [0x00000000..0x00000000]
 java.lang.Thread.State: RUNNABLE

"Attach Listener" daemon prio=10 tid=0x0aac7000 nid=0x1708 runnable [0x00000000..0x00000000]
 java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x0aac6000 nid=0x5f4 waiting on condition [0x00000000..0x00000000]
 java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=8 tid=0x0aab7800 nid=0xeb4 in Object.wait() [0x0ac2f000..0x0ac2fa94]
 java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x02ea42e0> (a java.lang.ref.ReferenceQueue$Lock)
 at java.lang.ref.ReferenceQueue.remove(Unknown Source)
 - locked <0x02ea42e0> (a java.lang.ref.ReferenceQueue$Lock)
 at java.lang.ref.ReferenceQueue.remove(Unknown Source)
 at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

"Reference Handler" daemon prio=10 tid=0x0aab3400 nid=0xac8 in Object.wait() [0x0abdf000..0x0abdfb14]
 java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x02ea4300> (a java.lang.ref.Reference$Lock)
 at java.lang.Object.wait(Object.java:485)
 at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
 - locked <0x02ea4300> (a java.lang.ref.Reference$Lock)

"main" prio=6 tid=0x002a7000 nid=0x3b8 runnable [0x0090f000..0x0090fe50]
 java.lang.Thread.State: RUNNABLE
 at javax.naming.InitialContext.close(Unknown Source)
 at org.jboss.example.jms.longlivequeue.QueueConsumerExample.example(QueueConsumerExample.java:85)
 at org.jboss.example.jms.common.ExampleSupport.run(ExampleSupport.java:147)
 at org.jboss.example.jms.longlivequeue.QueueConsumerExample.main(QueueConsumerExample.java:121)

"VM Thread" prio=10 tid=0x0aaaec00 nid=0x90c runnable

"VM Periodic Task Thread" prio=10 tid=0x0aaec400 nid=0xd30 waiting on condition

JNI global references: 2974

Heap
 def new generation total 960K, used 186K [0x029a0000, 0x02aa0000, 0x02e80000)
 eden space 896K, 18% used [0x029a0000, 0x029c9570, 0x02a80000)
 from space 64K, 33% used [0x02a80000, 0x02a85668, 0x02a90000)
 to space 64K, 0% used [0x02a90000, 0x02a90000, 0x02aa0000)
 tenured generation total 4096K, used 2034K [0x02e80000, 0x03280000, 0x069a0000)
 the space 4096K, 49% used [0x02e80000, 0x0307cb70, 0x0307cc00, 0x03280000)
 compacting perm gen total 12288K, used 9794K [0x069a0000, 0x075a0000, 0x0a9a0000)
 the space 12288K, 79% used [0x069a0000, 0x073309b0, 0x07330a00, 0x075a0000)
No shared spaces configured.

Any idea how to procees?
How can I enable the jboss tracing, would this help?

I have launched a debugger, but do not know what to do, because there are two threads just blocked on socket read..

org.jboss.remoting.transport.socket.ServerThread.readVersion(ServerThread.java:824)
and
org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlConnectionThread.run(B
isocketServerInvoker.java:741)

[clebert.suconic]

I have executed your examples... and I realized you have several debug options set.

so, few questions for you:

I - Have you used debugging during your tests?

II - Also.. have you seen this sort of message on server?

"16:09:52,005 WARN [SimpleConnectionManager] A problem has been detected with the connection to remote client 3j011-dg4k19-fc3uylzr-1-fc3uymin-4, jmsClientID=b-7omyu3cf-1-rzlyu3cf-91k4gd-110j3. It is possible the client has exited without closing its connection(s) or the network has failed. All connection resources corresponding to that client process will now be removed."

III - If you start another client, Long-live-queue-consumer, would you see messages arriving?

[bodrin]

Here are my answers:

I - Have you used debugging during your tests?

No. I just leave the long-live-consumer to run for a few hours, then check it if it receives the messages. It does not, and then I connect with the debuger to it and try to analyze
its state via suspending all the threads, but withouh any success - I only see two related threads blocked on a socket read operation.

II - Also.. have you seen this sort of message on server?

"16:09:52,005 WARN [SimpleConnectionManager] A problem has been detected with the connection to remote client 3j011-dg4k19-fc3uylzr-1-fc3uymin-4, jmsClientID=b-7omyu3cf-1-rzlyu3cf-91k4gd-110j3. It is possible the client has exited without closing its connection(s) or the network has failed. All connection resources corresponding to that client process will now be removed."

Yes, when checking question III.

I tried to reproduce it again and there is an exception into the JBoss console:

09:48:07,081 INFO [Server] JBoss (MX MicroKernel) [4.2.2.GA (build: SVNTag=JBos

[root@centos9 mpay]#
[root@centos9 mpay]# 15:31:49,131 ERROR [SocketClientInvoker] Got marshalling exception, exiting
java.io.IOException: Connection reset
 at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
 at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
 at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
 at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
 at java.io.DataOutputStream.flush(DataOutputStream.java:106)
 at org.jboss.jms.wireformat.ClientDelivery.write(ClientDelivery.java:93)
 at org.jboss.jms.wireformat.JMSWireFormat.write(JMSWireFormat.java:237)
 at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.versionedWrite(MicroSocketClientInvoker.java:945)
 at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:586)
 at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:418)
 at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
 at org.jboss.remoting.Client.invoke(Client.java:1634)
 at org.jboss.remoting.Client.invoke(Client.java:548)
 at org.jboss.remoting.Client.invokeOneway(Client.java:598)
 at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:826)
 at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:697)
 at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1432)
 at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1344)
 at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:322)
 at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
 at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:582)
 at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
 at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:606)
 at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:505)
 at org.jboss.messaging.core.impl.ChannelSupport$InMemoryCallback.afterCommit(ChannelSupport.java:935)
 at org.jboss.messaging.core.impl.tx.Transaction.commit(Transaction.java:228)
 at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendTransaction(ServerConnectionEndpoint.java:474)
 at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.org$jboss$jms$server$endpoint$advised$ConnectionAdvised$sendTransaction$aop(ConnectionAdvised.java:101)
 at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$sendTransaction_N3268650789275322226.invokeNext(ConnectionAdvised$sendTransaction_N3268650789275322226.java)
 at org.jboss.jms.server.container.SecurityAspect.handleSendTransaction(SecurityAspect.java:195)
 at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
 at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$sendTransaction_N3268650789275322226.invokeNext(ConnectionAdvised$sendTransaction_N3268650789275322226.java)
 at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
 at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$sendTransaction_N3268650789275322226.invokeNext(ConnectionAdvised$sendTransaction_N3268650789275322226.java)
 at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.sendTransaction(ConnectionAdvised.java)
 at org.jboss.jms.wireformat.ConnectionSendTransactionRequest.serverInvoke(ConnectionSendTransactionRequest.java:82)
 at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:795)
 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:573)
 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
15:31:49,136 ERROR [ServerInvokerCallbackHandler] Error handling callback
java.rmi.MarshalException: Failed to communicate. Problem during marshalling/unmarshalling; nested exception is:
 java.io.IOException: Connection reset
 at org.jboss.remoting.transport.socket.SocketClientInvoker.handleException(SocketClientInvoker.java:122)
 at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:669)
 at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:418)
 at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
 at org.jboss.remoting.Client.invoke(Client.java:1634)
 at org.jboss.remoting.Client.invoke(Client.java:548)
 at org.jboss.remoting.Client.invokeOneway(Client.java:598)
 at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:826)
 at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:697)
 at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1432)
 at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1344)
 at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:322)
 at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
 at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:582)
 at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
 at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:606)
 at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:505)
 at org.jboss.messaging.core.impl.ChannelSupport$InMemoryCallback.afterCommit(ChannelSupport.java:935)
 at org.jboss.messaging.core.impl.tx.Transaction.commit(Transaction.java:228)
 at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendTransaction(ServerConnectionEndpoint.java:474)
 at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.org$jboss$jms$server$endpoint$advised$ConnectionAdvised$sendTransaction$aop(ConnectionAdvised.java:101)
 at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$sendTransaction_N3268650789275322226.invokeNext(ConnectionAdvised$sendTransaction_N3268650789275322226.java)
 at org.jboss.jms.server.container.SecurityAspect.handleSendTransaction(SecurityAspect.java:195)
 at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
 at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$sendTransaction_N3268650789275322226.invokeNext(ConnectionAdvised$sendTransaction_N3268650789275322226.java)
 at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
 at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$sendTransaction_N3268650789275322226.invokeNext(ConnectionAdvised$sendTransaction_N3268650789275322226.java)
 at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.sendTransaction(ConnectionAdvised.java)
 at org.jboss.jms.wireformat.ConnectionSendTransactionRequest.serverInvoke(ConnectionSendTransactionRequest.java:82)
 at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:795)
 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:573)
 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
Caused by: java.io.IOException: Connection reset
 at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
 at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
 at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
 at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
 at java.io.DataOutputStream.flush(DataOutputStream.java:106)
 at org.jboss.jms.wireformat.ClientDelivery.write(ClientDelivery.java:93)
 at org.jboss.jms.wireformat.JMSWireFormat.write(JMSWireFormat.java:237)
 at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.versionedWrite(MicroSocketClientInvoker.java:945)
 at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:586)
 ... 35 more
[root@centos9 mpay]#

III - If you start another client, Long-live-queue-consumer, would you see messages arriving?

1. I have started another client and it received 8 messages, not 10 as espected. Two messages are missing
2. I tried to send another 10 messages and the second long-live-consumers receives them all.
3. I stopped the first one, and then a message into the JBoss console appeared:

[root@centos9 mpay]# 15:52:58,801 WARN [SimpleConnectionManager] A problem has been detected with the connection to remote client 4sg1e4f-d8p38v-fc4foojk-1-fc4fop7g-4, jmsClientID=b-tepof4cf-1-kjoof4cf-v83p8d-f4e1gs4. It is possible the client has exited without closing its connection(s) or the network has failed. All connection resources corresponding to that client process will now be removed.

Could you tell me what exactly this means, please?

4.After that I have started again a long-live-consumer and it received the two missing messages.

[clebert.suconic]

Could you tell me what exactly this means, please?

It means your client couldn't send Pings fast enough to the server at some point. Or you have some network problem, or something is making your client to run really slowly. Probably your Heavy debug options are making your client behave this way.


I would suggest you to remove those debug messages... and maybe verify your timeout configurations. Since you are using ServiceBindingManager to a clustered configuration, maybe you didn't update it properly. (Note that there are 4 places you need to replace it on the ServiceBinding).

[clebert.suconic]

"Clebert" wrote:
would suggest you to remove those debug messages

Of course here I meant... "remove those debug options"

[bodrin]

I would suggest you to remove those debug messages... and maybe verify your timeout configurations. Since you are using ServiceBindingManager to a clustered configuration, maybe you didn't update it properly. (Note that there are 4 places you need to replace it on the ServiceBinding).

Ok, I suppose you mean 4 places in case of a cluster with 4 nodes, right?
Because my cluster has 2 nodes and I have already updated this 2 places in $JBOSS_HOME/docs/examples/binding-manager/sample-bindings.xml with the attributes from JBOSS_CONFIG/deploy/jboss-messaging.sar/remoting-bisocket-service.xml.
Is this ok? It is strange that the messaging is using some configuration inside $JBOSS_HOME/docs/examples.

Anyway, this happens (the warning message) when I stopped the first long-live-consumer (the one that should receive the messages, but it does not), so I suppose that this is normal and that it is not related to the problem.

So then it is not because of the debug mode too..

What can I do more to investigate this further?