MDB reconnect fails silently

[michael10]

Hello.

The problem are relating to: https://issues.jboss.org/browse/HORNETQ-981#comment-12723104. But we do not remove any journal. We use the same MDB

Testcase: 2 JBoss-6.0.0 instances. 1st instance with HornetQ-2.1.2. 2nd JBoss instance without HornetQ and with MDB. We killing 1st instance and after startup, the MDB didn't receive messages anymore.

We get an error, the only error, on 1st instance with: HornetQException[errorCode=100 message=Queue jms.queue.testQueue does not exist].

The problem I think is that reconnection mechanism don't wait for the resource (queue) MDB needs. Reconnection aborts when queue is not available and didn't try again. Take a look on class ClientSessionImpl#handleFailover

I also find out that the error message only occures, when a JBoss instance needs more time to boot (our productive JBoss) and therefor more time to deploy his resources. Tests with an fresh JBoss runs always successful.

I also think there are a relation to Deployment ordering on HornetQ destinations and MDBs https://community.jboss.org/message/544222#544222

Please tell me if there are already an fix or is there another way to fix it?

Greetings

[jbertram]

Couple of things...

1. Reproduce this issue with at least HornetQ 2.2.14.
2. Provide logs from the server and client instances.

[michael10]

Hi.

I will check this.

But I hope there are not much configuration changes between 2.1.2 and 2.2.14.

Are there any integration documentation into JBoss-6.

Because the integration of a new HornetQ version into our framework, seems to be a lot of work and we already use it poductive.

[jbertram]

At the end of the day, community support for old versions is simply untenable.  There just isn't enough time or resources.

If you need support and you don't have a strategy to keep your environment on the latest HornetQ release then I recommend you look at an enterprise support contract from Red Hat.  Long term support (including patches) is one the main benefits of such a contract.

[michael10]

Hello, I test it with HornetQ version 2.2.14, but we get the same problem. I'm using the documentation for updating hornetq: https://community.jboss.org/thread/169780

I attach  log files and test programs.

Test case was:

1. starting  jboss1 with HornetQ server

2. starting jboss2. Client consumer is connecting via JNDI

3. starting a producer which sends messages to a testQueue

4. checking if messages received on client jboss2 instance

5. kill JBoss1 and restart it          

6. the producer sends messages again, if jboss1 is available

--> We expected that MDB reconnects to the queue, but it never gets messages.

only if we undeploy consumer, messages are received again

In 5 we get Error:

     HornetQException[errorCode=100 message=Queue jms.queue.testQueue does not exist]

         at org.hornetq.core.server.impl.ServerSessionImpl.createConsumer(ServerSessionImpl.java:339)

         at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:214)

         at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:508)

          ...

before queues are created

Again, we get this problem on our productive environment, which needs more time to deploy all resources. Client reconnection is trying to connect to a queue which are currently not exists and it never tries again.

Greetings

[jbertram]

It appears to me that the JBOSS_SERVER.log you attached is missing some information (i.e. where the "..." entries appear).  Can you confirm this?  If so, please attach the full server log?

[michael10]

Sorry, I cleaned the log. I attach the full log.

[michael10]

Hello, could you please tell me if there are any results and if you can reproduce the problem?

Do you think it is better to open a issue.

Greetings

[jbertram]

I haven't attempted to reproduce the issue.  Feel free to provide an automated (as much as possible) test-case that reproduces the issue.

In the mean-time, can you compress and attach your deploy/hornetq directory? I noticed in your log that you're deploying a file named mpa-hornetq-jms.xml and I'm curious what it contains.  My hunch is that it contains the destination which the MDB is looking for and it isn't being deployed quickly enough because that file is scanned and deployed after the HornetQ broker itself is deployed.

[michael10]

Hello.

Yes in the mpa-hornetq-jms.xml we deploy our queues. But in hornetq-jms.xml is the DLQ defined and in logfile I see that this queue is deployed at the same time as our queues.

2012-10-05 14:24:38,976 INFO  [Thread-10] org.hornetq.core.server.impl.HornetQServerImpl : trying to deploy queue jms.queue.DLQ

2012-10-05 14:24:39,077 INFO  [Thread-10] org.hornetq.core.server.impl.HornetQServerImpl : trying to deploy queue jms.queue.ExpiryQueue

2012-10-05 14:24:39,098 INFO  [Thread-10] org.hornetq.core.server.impl.HornetQServerImpl : trying to deploy queue jms.queue.RBS_Dokument_Error

2012-10-05 14:24:39,109 INFO  [Thread-10] org.hornetq.core.server.impl.HornetQServerImpl : trying to deploy queue jms.queue.gfTestQueueXa

2012-10-05 14:24:39,119 INFO  [Thread-10] org.hornetq.core.server.impl.HornetQServerImpl : trying to deploy queue jms.queue.testQueue

2012-10-05 14:24:39,130 INFO  [Thread-10] org.hornetq.core.server.impl.HornetQServerImpl : trying to deploy queue jms.queue.RBS_Dokument

2012-10-05 14:24:39,139 INFO  [Thread-10] org.hornetq.core.server.impl.HornetQServerImpl : trying to deploy queue jms.queue.CFTProcessingMDB

2012-10-05 14:24:39,150 INFO  [Thread-10] org.hornetq.core.server.impl.HornetQServerImpl : trying to deploy queue jms.queue.Rules_Test

2012-10-05 14:24:39,160 INFO  [Thread-10] org.hornetq.core.server.impl.HornetQServerImpl : trying to deploy queue jms.queue.Rules_Test2

2012-10-05 14:24:39,169 INFO  [Thread-10] org.hornetq.core.server.impl.HornetQServerImpl : trying to deploy queue jms.queue.gfTestQueue

2012-10-05 14:24:39,179 INFO  [Thread-10] org.hornetq.core.server.impl.HornetQServerImpl : trying to deploy queue jms.topic.update

2012-10-05 14:24:39,208 INFO  [Thread-10] org.hornetq.core.server.impl.HornetQServerImpl : trying to deploy queue jms.queue.RBS_Test2_Error

2012-10-05 14:24:39,217 INFO  [Thread-10] org.hornetq.core.server.impl.HornetQServerImpl : trying to deploy queue jms.queue.RBS_Test

2012-10-05 14:24:39,228 INFO  [Thread-10] org.hornetq.core.server.impl.HornetQServerImpl : trying to deploy queue jms.topic.bms

2012-10-05 14:24:39,236 INFO  [Thread-10] org.hornetq.core.server.impl.HornetQServerImpl : trying to deploy queue jms.topic.impExpTopic

2012-10-05 14:24:39,244 INFO  [Thread-10] org.hornetq.core.server.impl.HornetQServerImpl : trying to deploy queue jms.queue.Rules_Test_Error

2012-10-05 14:24:39,253 INFO  [Thread-10] org.hornetq.core.server.impl.HornetQServerImpl : trying to deploy queue jms.queue.RBS_Test_Error

2012-10-05 14:24:39,262 INFO  [Thread-10] org.hornetq.core.server.impl.HornetQServerImpl : trying to deploy queue jms.topic.gfTestTopicXa

2012-10-05 14:24:39,270 INFO  [Thread-10] org.hornetq.core.server.impl.HornetQServerImpl : trying to deploy queue jms.queue.RBS_Test2

2012-10-05 14:24:39,279 INFO  [Thread-10] org.hornetq.core.server.impl.HornetQServerImpl : trying to deploy queue jms.topic.pool

2012-10-05 14:24:39,287 INFO  [Thread-10] org.hornetq.core.server.impl.HornetQServerImpl : trying to deploy queue jms.topic.gfTestTopic