JBossDeveloper

"rtm133" wrote:
Clebert,

Anything new on the suspected cause you mentioned in a previous posting?

"rtm133" wrote:
I am also able to reproduce it with an unmodified installation of Messaging-1.0.1.SP2 on top of JBoss-4.0.5.GA and a trivial JMS topic subscriber running on the same (WinXP) box.

"rtm333" wrote:

I will be happy to provide the debug logs and/or the source code if deemed helpful.

I will try to replicate the problem using your description...

but if you could just in case attach your modifications here:

http://jira.jboss.org/jira/browse/JBMESSAGING-749

I will try to make it to SP3 that should be out very soon.

rtm333, I've performed the experiment you described.

While it's true that the client sends and invocation into the server every other two seconds (that's the ConnectionValidator, the remoting client uses it to detect failed connections), no new socket is created. The same physical TCP/IP connection is re-used, and the validator seems rather well behaved and working as intended.

I must mention though I am using Remoting 2.2.0.Alpha5, which includes a fair amount of changes in exactly this area, so the bug that you noticed probably has already been fixed.

I am going to close http://jira.jboss.org/jira/browse/JBMESSAGING-749 with "Cant' reproduce bug"

Try 1.0.1.SP4, it will be out soon, and if you see the same behavior again, please reopen the issue

Gentlemen,

We are using JBoss 4.0.5.GA with JBoss Messaging 1.0.1.SP4 and we have encountered this same Exception under heavy load. We don't seem to be losing any messages, however.

2007-03-19 14:49:48,042 ERROR [org.jboss.remoting.transport.socket.ServerThread] Worker thread initialization failure
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor220.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
at org.jboss.remoting.transport.socket.ServerThread.createServerSocketWrapper(ServerThread.java:694)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:358)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:159)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
at java.io.FilterInputStream.read(FilterInputStream.java:90)
at org.jboss.serial.io.JBossObjectInputStream.checkSignature(JBossObjectInputStream.java:110)
at org.jboss.serial.io.JBossObjectInputStream.(JBossObjectInputStream.java:94)
at org.jboss.remoting.serialization.impl.jboss.JBossSerializationManager.createInput(JBossSerializationManager.java:57)
at org.jboss.jms.server.remoting.ServerSocketWrapper.createInputStream(ServerSocketWrapper.java:125)
at org.jboss.jms.client.remoting.ClientSocketWrapper.createStreams(ClientSocketWrapper.java:123)
at org.jboss.jms.client.remoting.ClientSocketWrapper.(ClientSocketWrapper.java:73)
at org.jboss.jms.server.remoting.ServerSocketWrapper.(ServerSocketWrapper.java:67)
... 6 more

Setting the "socket.check_connection" in remoting-service.xml to true seems to solve this problem.

Actually, what I said in the previous post is simply not true. Setting socket.check_connection did have an effect, but not the intended one.

The Exception reported doesn't result in any messages being lost. However, in our system it was one of several types of exceptions coming from JBoss Remoting once we installed JBoss Messaging 1.0.1.SP4. The entire system was unusable. That is, the machine the queues were running on ground to a halt.

We had about 80 queues running on a Solaris/SPARC machine with JBoss 4.0.5.GA and the backing MySQL 5.0.x database installed on the same machine. In addition, we had 40 freestanding java programs each of which establishes and caches a QueueConnection. These programs both read from and send messages to queues. These programs are multithreaded but as far as I can tell individual QueueSessions are not being accessed by more than a single thread. QueueConnections were also being established to/from this JBoss to another, remote JBoss instance. Although this setup worked with JBossMQ, it did not with JBoss Messaging.

What we ended up doing is migrating both JBoss and MySQL to another machine. This was a Linux machine running on an Intel processor. We also installed the latest version of JBoss Remoting, which was 2.2.0.GA. The remoting configuration used is the default.

This seems to work. Our conclusion is that somehow JBoss Remoting with our setup doesn't work well on a Solaris/SPARC machine - given the fact that we have no Solaris expertise in house. We also believe that the upgrade to JBoss Remoting 2.2.0.GA helped.

However, we still have problems and they seem to have to do with JBoss Remoting. In the freestanding programs, we encounter the Exception previously reported and also the following:

2007-03-22 18:39:22,169 ERROR [org.jboss.remoting.transport.socket.SocketClientInvoker] Got marshalling exception, exiting
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at org.jboss.serial.io.JBossObjectInputStream.read(JBossObjectInputStream.java:193)
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:975)
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:567)
at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
at org.jboss.remoting.ConnectionValidator.doCheckConnection(ConnectionValidator.java:126)
at org.jboss.remoting.ConnectionValidator.run(ConnectionValidator.java:195)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
2007-03-22 18:39:22,233 ERROR [org.jboss.jms.client.container.ConnectionAspect] Caught exception from connection
java.lang.Exception: Could not connect to server!
at org.jboss.remoting.ConnectionValidator.run(ConnectionValidator.java:201)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
2007-03-22 18:39:22,234 WARN [com.rdcompany.sars.brokers.QueueFactory] error on queue/730_5050_MtQueue queue/730_5050_MoQueue queue/730_505
0_SrQueue javax.jms.JMSException: Throwable received from underlying connection
2007-03-22 18:39:22,234 WARN [com.rdcompany.sars.brokers.QueueFactory] reconnecting!
2007-03-22 18:39:22,292 INFO [com.rdcompany.sars.brokers.QueueFactory] state changed connected -> reconnect
2007-03-22 18:39:22,292 INFO [com.rdcompany.sars.brokers.QueueFactory] queueConnection.setExceptionListener

As you can see we have functionality that should reestablish the cached QueueConnection. However, sometimes the program appears to hang on the QueueConnection.close(). In which case we have to kill the program and start it again, which is unacceptable.

imaeses, that was a good post.

I can't deny your conclusions about Solaris, but only I can add my observations: I have had no problems with Solaris/Linux machines but Windows.

So, I am not sure whether this problem is related with machines/OSs at all.

However, as it was hitting our production severely, I have had to move to a bit higher version - JBoss 4.0.5.GA (the one you are using) and JBM 1.2.0.GA (which you are not using).

As far as i can see, I have (I think, don't laugh at me if I come back to the same post crying that it didn't work!) surpassed this problem BUT fingers cross!

NOTE: Due to the configuration I'm using, I've to replace the existing jboss-remoting jar with that shipped with JBM (2.2.0.Beta1).

rtm333, you are/were silent for few days, any luck/progress?

Thanks
Madhu

Also, as Ovidiu suggested, as you are still having the issues with SP4, re-open the bug:

http://jira.jboss.org/jira/browse/JBMESSAGING-749

Thanks
Madhu