-
15. Re: JMS Exceptions / Mismatch of JBossSerialization version
rtm333 Jan 15, 2007 2:52 AM (in response to rtm333)Clebert,
Anything new on the suspected cause you mentioned in a previous posting?
Do you need any further information from me?
Thanks. -
16. Re: JMS Exceptions / Mismatch of JBossSerialization version
rtm333 Jan 15, 2007 8:57 AM (in response to rtm333)I am really wondering whether the client side subscriber behaviour of opening a new socket every two seconds is the way it is supposed to work. This leaves a lot of sockets lying around in state TIME_WAIT.
This behaviour is visible in the log excerpt of my posting from Fri Jan 12, 2007 06:59 AM in the current thread.
I am also able to reproduce it with an unmodified installation of Messaging-1.0.1.SP2 on top of JBoss-4.0.5.GA and a trivial JMS topic subscriber running on the same (WinXP) box.
When using a Messaging-1.0.1.GA installation and the appropriate client jar there is just one stable two-way connection between the server and the client as one would expect.
I will be happy to provide the debug logs and/or the source code if deemed helpful. -
17. Re: JMS Exceptions / Mismatch of JBossSerialization version
clebert.suconic Jan 15, 2007 1:14 PM (in response to rtm333)"rtm133" wrote:
Clebert,
Anything new on the suspected cause you mentioned in a previous posting?
I thought this could be related to http://jira.jboss.com/jira/browse/JBREM-649 but I didn't see any clues about it in your logs."rtm133" wrote:
I am also able to reproduce it with an unmodified installation of Messaging-1.0.1.SP2 on top of JBoss-4.0.5.GA and a trivial JMS topic subscriber running on the same (WinXP) box.
If you are able to reproduce this, can we have the testcase? that would be great. -
18. Re: JMS Exceptions / Mismatch of JBossSerialization version
ovidiu.feodorov Jan 16, 2007 4:36 AM (in response to rtm333)"rtm333" wrote:
I will be happy to provide the debug logs and/or the source code if deemed helpful.
Could you please send us the simplest way to reproduce the problem? Or a test case? As soon as I have that, I'll take a look. -
19. Re: JMS Exceptions / Mismatch of JBossSerialization version
rtm333 Jan 16, 2007 7:17 AM (in response to rtm333)Hi,
After some further experimentation I think I finally found the critical part.
I can reproduce the misbehaviour by adding a trivial ExceptionListener to the connection of the topic example included with the Messaging distribution (org.jboss.example.jms.topic.TopicExample):... connection = cf.createConnection(); connection.setExceptionListener(new ExceptionListener() { public void onException(JMSException jmse) { } }); ...
You also have to take care that the subscriber is running for some time (e.g. suppress publishing the message it is waiting for and extending the waiting time to, say, a minute).
During this time you can look at DEBUG output and/or repeatedly run netstat to see that a new connection to the messaging port is opened and closed every two seconds:... [java] 11:59:37,410 DEBUG @main [MessageIdGeneratorFactory] checked out MessageIdGenerator for server.0, reference count is 1 [java] 11:59:38,160 DEBUG @main [JBossSession] attempting to create consumer for destination:JBossTopic[testTopic] [java] 11:59:39,457 DEBUG @Timer-0 [MicroSocketClientInvoker] Setting ClientSocket class name to: org.jboss.jms.client.remoting.ClientSocketWrapper [java] 11:59:39,457 DEBUG @Timer-0 [SocketClientInvoker] Setting SocketClientInvoker::timeout to: 1000 [java] 11:59:39,457 DEBUG @Timer-0 [MicroSocketClientInvoker] Setting ClientSocket class name to: org.jboss.jms.client.remoting.ClientSocketWrapper [java] 11:59:39,457 DEBUG @Timer-0 [SocketClientInvoker] Setting SocketClientInvoker::timeout to: 1000 [java] 11:59:39,472 DEBUG @Timer-0 [MicroRemoteClientInvoker] connect called for: org.jboss.remoting.transport.socket.SocketClientInvoker@15356d5 [java] 11:59:39,472 DEBUG @Timer-0 [MicroRemoteClientInvoker] disconnect called for: org.jboss.remoting.transport.socket.SocketClientInvoker@15356d5 [java] 11:59:41,472 DEBUG @Timer-0 [MicroSocketClientInvoker] Setting ClientSocket class name to: org.jboss.jms.client.remoting.ClientSocketWrapper [java] 11:59:41,472 DEBUG @Timer-0 [SocketClientInvoker] Setting SocketClientInvoker::timeout to: 1000 [java] 11:59:41,472 DEBUG @Timer-0 [MicroSocketClientInvoker] Setting ClientSocket class name to: org.jboss.jms.client.remoting.ClientSocketWrapper [java] 11:59:41,472 DEBUG @Timer-0 [SocketClientInvoker] Setting SocketClientInvoker::timeout to: 1000 [java] 11:59:41,472 DEBUG @Timer-0 [MicroRemoteClientInvoker] connect called for: org.jboss.remoting.transport.socket.SocketClientInvoker@fb6354 [java] 11:59:41,488 DEBUG @Timer-0 [MicroRemoteClientInvoker] disconnect called for: org.jboss.remoting.transport.socket.SocketClientInvoker@fb6354 ...
This is not the initial problem of this thread (JMS Exceptions / Mismatch of JBossSerialization version), but I'm rather certain that it is related to this issue, as it leads to connection timeouts. Anyway, the frequent connection creation is a problem on it's own right. -
20. Re: JMS Exceptions / Mismatch of JBossSerialization version
clebert.suconic Jan 16, 2007 1:49 PM (in response to rtm333)I will try to replicate the problem using your description...
but if you could just in case attach your modifications here:
http://jira.jboss.org/jira/browse/JBMESSAGING-749
I will try to make it to SP3 that should be out very soon. -
21. Re: JMS Exceptions / Mismatch of JBossSerialization version
ovidiu.feodorov Jan 18, 2007 4:07 AM (in response to rtm333)rtm333, I've performed the experiment you described.
While it's true that the client sends and invocation into the server every other two seconds (that's the ConnectionValidator, the remoting client uses it to detect failed connections), no new socket is created. The same physical TCP/IP connection is re-used, and the validator seems rather well behaved and working as intended.
I must mention though I am using Remoting 2.2.0.Alpha5, which includes a fair amount of changes in exactly this area, so the bug that you noticed probably has already been fixed.
I am going to close http://jira.jboss.org/jira/browse/JBMESSAGING-749 with "Cant' reproduce bug"
Try 1.0.1.SP4, it will be out soon, and if you see the same behavior again, please reopen the issue -
22. Re: JMS Exceptions / Mismatch of JBossSerialization version
imaeses Mar 20, 2007 10:14 AM (in response to rtm333)Gentlemen,
We are using JBoss 4.0.5.GA with JBoss Messaging 1.0.1.SP4 and we have encountered this same Exception under heavy load. We don't seem to be losing any messages, however.
2007-03-19 14:49:48,042 ERROR [org.jboss.remoting.transport.socket.ServerThread] Worker thread initialization failure
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor220.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
at org.jboss.remoting.transport.socket.ServerThread.createServerSocketWrapper(ServerThread.java:694)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:358)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:159)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
at java.io.FilterInputStream.read(FilterInputStream.java:90)
at org.jboss.serial.io.JBossObjectInputStream.checkSignature(JBossObjectInputStream.java:110)
at org.jboss.serial.io.JBossObjectInputStream.(JBossObjectInputStream.java:94)
at org.jboss.remoting.serialization.impl.jboss.JBossSerializationManager.createInput(JBossSerializationManager.java:57)
at org.jboss.jms.server.remoting.ServerSocketWrapper.createInputStream(ServerSocketWrapper.java:125)
at org.jboss.jms.client.remoting.ClientSocketWrapper.createStreams(ClientSocketWrapper.java:123)
at org.jboss.jms.client.remoting.ClientSocketWrapper.(ClientSocketWrapper.java:73)
at org.jboss.jms.server.remoting.ServerSocketWrapper.(ServerSocketWrapper.java:67)
... 6 more -
23. Re: JMS Exceptions / Mismatch of JBossSerialization version
imaeses Mar 21, 2007 6:06 PM (in response to rtm333)Setting the "socket.check_connection" in remoting-service.xml to true seems to solve this problem.
-
24. Re: JMS Exceptions / Mismatch of JBossSerialization version
imaeses Mar 23, 2007 6:21 AM (in response to rtm333)Actually, what I said in the previous post is simply not true. Setting socket.check_connection did have an effect, but not the intended one.
The Exception reported doesn't result in any messages being lost. However, in our system it was one of several types of exceptions coming from JBoss Remoting once we installed JBoss Messaging 1.0.1.SP4. The entire system was unusable. That is, the machine the queues were running on ground to a halt.
We had about 80 queues running on a Solaris/SPARC machine with JBoss 4.0.5.GA and the backing MySQL 5.0.x database installed on the same machine. In addition, we had 40 freestanding java programs each of which establishes and caches a QueueConnection. These programs both read from and send messages to queues. These programs are multithreaded but as far as I can tell individual QueueSessions are not being accessed by more than a single thread. QueueConnections were also being established to/from this JBoss to another, remote JBoss instance. Although this setup worked with JBossMQ, it did not with JBoss Messaging.
What we ended up doing is migrating both JBoss and MySQL to another machine. This was a Linux machine running on an Intel processor. We also installed the latest version of JBoss Remoting, which was 2.2.0.GA. The remoting configuration used is the default.
This seems to work. Our conclusion is that somehow JBoss Remoting with our setup doesn't work well on a Solaris/SPARC machine - given the fact that we have no Solaris expertise in house. We also believe that the upgrade to JBoss Remoting 2.2.0.GA helped.
However, we still have problems and they seem to have to do with JBoss Remoting. In the freestanding programs, we encounter the Exception previously reported and also the following:
2007-03-22 18:39:22,169 ERROR [org.jboss.remoting.transport.socket.SocketClientInvoker] Got marshalling exception, exiting
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at org.jboss.serial.io.JBossObjectInputStream.read(JBossObjectInputStream.java:193)
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:975)
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:567)
at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
at org.jboss.remoting.ConnectionValidator.doCheckConnection(ConnectionValidator.java:126)
at org.jboss.remoting.ConnectionValidator.run(ConnectionValidator.java:195)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
2007-03-22 18:39:22,233 ERROR [org.jboss.jms.client.container.ConnectionAspect] Caught exception from connection
java.lang.Exception: Could not connect to server!
at org.jboss.remoting.ConnectionValidator.run(ConnectionValidator.java:201)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
2007-03-22 18:39:22,234 WARN [com.rdcompany.sars.brokers.QueueFactory] error on queue/730_5050_MtQueue queue/730_5050_MoQueue queue/730_505
0_SrQueue javax.jms.JMSException: Throwable received from underlying connection
2007-03-22 18:39:22,234 WARN [com.rdcompany.sars.brokers.QueueFactory] reconnecting!
2007-03-22 18:39:22,292 INFO [com.rdcompany.sars.brokers.QueueFactory] state changed connected -> reconnect
2007-03-22 18:39:22,292 INFO [com.rdcompany.sars.brokers.QueueFactory] queueConnection.setExceptionListener
As you can see we have functionality that should reestablish the cached QueueConnection. However, sometimes the program appears to hang on the QueueConnection.close(). In which case we have to kill the program and start it again, which is unacceptable. -
25. Re: JMS Exceptions / Mismatch of JBossSerialization version
mskonda Mar 23, 2007 6:56 AM (in response to rtm333)imaeses, that was a good post.
I can't deny your conclusions about Solaris, but only I can add my observations: I have had no problems with Solaris/Linux machines but Windows.
So, I am not sure whether this problem is related with machines/OSs at all.
However, as it was hitting our production severely, I have had to move to a bit higher version - JBoss 4.0.5.GA (the one you are using) and JBM 1.2.0.GA (which you are not using).
As far as i can see, I have (I think, don't laugh at me if I come back to the same post crying that it didn't work!) surpassed this problem BUT fingers cross!
NOTE: Due to the configuration I'm using, I've to replace the existing jboss-remoting jar with that shipped with JBM (2.2.0.Beta1).
rtm333, you are/were silent for few days, any luck/progress?
Thanks
Madhu -
26. Re: JMS Exceptions / Mismatch of JBossSerialization version
mskonda Mar 23, 2007 6:58 AM (in response to rtm333)Also, as Ovidiu suggested, as you are still having the issues with SP4, re-open the bug:
http://jira.jboss.org/jira/browse/JBMESSAGING-749
Thanks
Madhu -
27. Re: JMS Exceptions / Mismatch of JBossSerialization version
rtm333 Mar 25, 2007 8:27 AM (in response to rtm333)mskonda,
I'm not sure whether the current discussion is still related to my original problem(s). In fact, we had two issues with SP2 that I described in this thread:
1) "IOException: Mismatch version of JBossSerialization signature" was thrown occasionally on the server side, although it worked most of the time without problems.
2) When an ExceptionListener was set on the client side, a new TCP connection was opened every two seconds by the ConnectionValidator.
I have not observed either issue since upgrading to JBM 1.0.1.SP4. -
28. Re: JMS Exceptions / Mismatch of JBossSerialization version
rtm333 Apr 18, 2007 10:53 AM (in response to rtm333)I have to partially correct my previous statement. Although we did not observe issue 2) since upgrading to JBM 1.0.1.SP4, issue 1) now reoccurred.
It happened again under Solaris 10 as well as under Redhat Linux. As imaeses remarked in an earlier posting it seems to be related to heavy (server) load.
As far as I can tell there two different incarnations of the exception:2007-04-18 13:22:27,632 ERROR [org.jboss.remoting.transport.socket.ServerThread] Worker thread initialization failure java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:494) at org.jboss.remoting.transport.socket.ServerThread.createServerSocketWrapper(ServerThread.java:694) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:358) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:159) Caused by: java.io.IOException: Mismatch version of JBossSerialization signature at org.jboss.serial.io.JBossObjectInputStream.checkSignature(JBossObjectInputStream.java:113) at org.jboss.serial.io.JBossObjectInputStream.<init>(JBossObjectInputStream.java:94) at org.jboss.remoting.serialization.impl.jboss.JBossSerializationManager.createInput(JBossSerializationManager.java:57) at org.jboss.jms.server.remoting.ServerSocketWrapper.createInputStream(ServerSocketWrapper.java:125) at org.jboss.jms.client.remoting.ClientSocketWrapper.createStreams(ClientSocketWrapper.java:123) at org.jboss.jms.client.remoting.ClientSocketWrapper.<init>(ClientSocketWrapper.java:73) at org.jboss.jms.server.remoting.ServerSocketWrapper.<init>(ServerSocketWrapper.java:67) ... 7 more
and2007-04-18 14:22:33,886 ERROR [org.jboss.remoting.transport.socket.ServerThread] Worker thread initialization failure java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedConstructorAccessor80.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:494) at org.jboss.remoting.transport.socket.ServerThread.createServerSocketWrapper(ServerThread.java:694) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:358) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:159) Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:168) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read1(BufferedInputStream.java:256) at java.io.BufferedInputStream.read(BufferedInputStream.java:313) at java.io.FilterInputStream.read(FilterInputStream.java:90) at org.jboss.serial.io.JBossObjectInputStream.checkSignature(JBossObjectInputStream.java:110) at org.jboss.serial.io.JBossObjectInputStream.<init>(JBossObjectInputStream.java:94) at org.jboss.remoting.serialization.impl.jboss.JBossSerializationManager.createInput(JBossSerializationManager.java:57) at org.jboss.jms.server.remoting.ServerSocketWrapper.createInputStream(ServerSocketWrapper.java:125) at org.jboss.jms.client.remoting.ClientSocketWrapper.createStreams(ClientSocketWrapper.java:123) at org.jboss.jms.client.remoting.ClientSocketWrapper.<init>(ClientSocketWrapper.java:73) at org.jboss.jms.server.remoting.ServerSocketWrapper.<init>(ServerSocketWrapper.java:67) ... 6 more
Besides these exceptions I can detect no actual problems caused by this issue. Nevertheless I find it to some extent worrying.