Intermittent communication issue

[mackerman]

Hi, we're seeing intermittent communication errors on our HornetQ JMS Client:

Oct 16, 2012 9:43:33 AM org.hornetq.core.logging.impl.JULLogDelegate warn

WARNING: Connection failure has been detected: Did not receive data from server

for org.hornetq.core.remoting.impl.netty.NettyConnection@f9da4fc[local= /10.2.62

.73:49893, remote=/10.81.224.87:5445] [code=3]

"KEEP_ALIVE"

"KEEP_ALIVE"

"KEEP_ALIVE"

"KEEP_ALIVE"

"KEEP_ALIVE"

Oct 16, 2012 9:46:23 AM org.hornetq.core.logging.impl.JULLogDelegate error

SEVERE: Failed to decode

java.lang.IllegalArgumentException: Invalid type: 0

        at org.hornetq.core.protocol.core.impl.PacketDecoder.decode(PacketDecode

r.java:550)

the KEEP_ALIVE messages are messages that we send peridically to indicate that the communication channel is open and receiving messages.  Every so often we see the exceptions in the logs.

Does anyone have any idea why this might be occuring?

Both Client and Server are built using the same hornet libraries: 2.2.14.Final.  We are also specifying a specific netty version: org.jboss.netty of 3.2.5.Final (which I confirmed from the distribution matches 2.2.14.Final).

Server is running on CentOS 6.3, JRE 1.6, JBoss 7.1.1.Final.

thanks, Mitchell

[clebert.suconic]

I have no idea on what is this Keep Alive? how you send it?

Can you upload hornetQ to a newer version?

git clone git://github.com/hornetq/hornetq.git

cd hornetq

git checkout Branch_2_2_AS7

./build.sh

(or ./mac-build if you are on mac)

replace hornetq-core, hornetq-jms.jar and hornetq-ra.jar under the modules on jboss-as

Also: make sure you're using the sessions correctly (not using them outside of the threading context).

[mackerman]

Thanks Clebert, i've been trying to get to the point where i can try the latest 2.2 version, but i'm buried right now. I'll do so as soon as I can.  The keep alive mechanism I mention is just a simple message with the string "KEEP_ALIVE" in it, that we send every 30 seconds.  That way the client knows the connection is good.  If it doesn't receive the message within a minute or so, it drops the connection and etablishes a new one.

[clebert.suconic]

I had the impression that you were tweaking the protocol with your KeepAlive message.

You should be able to just relay on reconnection mechanisms from HornetQ. I'm a bit puzzled why you're receiving this error:

java.lang.IllegalArgumentException: Invalid type: 0

        at org.hornetq.core.protocol.core.impl.PacketDecoder.decode(PacketDecode

That's why i asked if you were using multi-threading on the session (say one thread sending a message while another thread is receiving.. I mean... going outside of the threading / session boundary). That would break things.

[mackerman]

the keep_alive was added for STOMP clients due to keep alive issues (that we had been discussing on a separate discussion - https://community.jboss.org/message/761770#761770).  The exceptions exhibited here, however, are for a JMS client.  I'm puzzled too, as the same client when connected to another server with identical code does not experience the error.  I'm tempted to blame network errors, but want to rule out other potential problems before heading down that path - i've been wrong about that before.

[clebert.suconic]

Why don't you use this plan:

I - upgrade to a newer version.

II - double check your clients are using the correct version (this could be a cause as well)

III - then check networking issues

[clebert.suconic]

Also: make sure you're not using the same session object simultaneously.