Network switch causes HornetQ to crash

[johnnysoccer]

We are experiencing an odd problem with our Messaging recently. We have HornetQ set up in a WAN environment.

There is 1 central HornetQ server (standalone v2.1.2)

1800 durable subscriptions across 4 topics (450 endpoints)

The network consists of 2 trunks with full failover.

When 1 of the trunks fails over during a high load period, HornetQ CPU rapidly jumps, and the load average will quickly rise to over 4000 on the central server.

The only way we have been able to recover is to turn HornetQ off, and then restart the server.

My suspicion is that this is related to the internal "keep alive" between the client and server attempting to re-establish a connection after the fail-over. I wanted to know if anyone else has experience this type of behavior, or is aware of a setting that we could use that might minimize the impact during a network switch.

We have observed that if the overall message load is light, this same type of switch over does not have the same kind of impact, also if the network connection is lost for more than 30 seconds or so, we do not experience the same problems. It seems to be limited to a nearly instantaneous network switch.

We did have some discussions with some technical guys on the JBoss side of things, more related to AS "stuff" but brought up the situation we were seeing. Their answer was "HornetQ isn't built for the WAN, you should think of switching to A-MQ", so I'd also like to get your thoughts on this response as well.

thanks,

John

[clebert.suconic]

2.1.2 is fairly old... you should probably try a recent build.

And about the statement on "HornetQ isn't built for the WAN"...

That's a statement typical of sales people or Consultant people bringing "definitions" without the proper scientific or technical evaluation. That's definitely not the case and we have plenty of large deploymen cases using HornetQ as a central repository. I don't think anyone from either our support or our engineer would ever make such statement.

Anyone, back to the technical issue: you should try the latest bits as there are many, many fixes beyond 2.1.2...  (Including failover and reconnection).

[clebert.suconic]

Just a statement to debunk the "built for the wan statement".. just look at the SpecJMS Horizontal submissions... the load is way beyond what you're experiencing now.