fails to start: IllegalStateException: Router already has lo

[bartvh]

Setup:A two node cluster JBoss 4.2 with JBM 1.3.0 serverMySQL persistenceA distributed topic on both nodesAn MDB with a durable subscription and client ID, installed on both nodes
During tests, I have been starting and stopping the nodes one after the other and also together, but now I have arrived at a situation where starting either node fails:

java.lang.IllegalStateException: Router[4da9ec16] already has local queue
 at org.jboss.messaging.core.plugin.postoffice.cluster.RoundRobinRouter.add(RoundRobinRouter.java:171)
 at org.jboss.messaging.core.plugin.postoffice.cluster.RoundRobinRouter.add(RoundRobinRouter.java:114)
 at org.jboss.messaging.core.plugin.postoffice.cluster.DefaultClusteredPostOffice.addToConditionMap(DefaultClusteredPostOffice.java:1536)
 at org.jboss.messaging.core.plugin.postoffice.DefaultPostOffice.addBinding(DefaultPostOffice.java:753)
 at org.jboss.messaging.core.plugin.postoffice.DefaultPostOffice.loadBindings(DefaultPostOffice.java:619)
 at org.jboss.messaging.core.plugin.postoffice.cluster.DefaultClusteredPostOffice.loadBindings(DefaultClusteredPostOffice.java:1594)
 at org.jboss.messaging.core.plugin.postoffice.DefaultPostOffice.start(DefaultPostOffice.java:141)
 at org.jboss.messaging.core.plugin.postoffice.cluster.DefaultClusteredPostOffice.start(DefaultClusteredPostOffice.java:327)
 at org.jboss.messaging.core.plugin.ClusteredPostOfficeService.startService(ClusteredPostOfficeService.java:449)
 at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
 at org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:196)

A trace log of the server shows that the server is trying two times to add a binding for the durable subscription.

[timfox]

Have you configured each node to have a unique server id?

http://labs.jboss.com/file-access/default/members/jbossmessaging/freezone/docs/userguide-1.3.0.GA/html/installation.html#install.automatedclustered

[bartvh]

"timfox" wrote:
Have you configured each node to have a unique server id?

yes, I have server peer id 0 and 2.

[timfox]

Can you post the contents of your JBM_POSTOFFICE table?

[bartvh]

It is rather big. Here's a subselect:

mysql> select * from JBM_POSTOFFICE where JBM_POSTOFFICE.QUEUE_NAME = 'TraceListener.TraceListener';
+-----------------+---------+-----------------------------+----------------+----------+------------+-----------+
| POSTOFFICE_NAME | NODE_ID | QUEUE_NAME | COND | SELECTOR | CHANNEL_ID | CLUSTERED |
+-----------------+---------+-----------------------------+----------------+----------+------------+-----------+
| Clustered JMS | 2 | TraceListener.TraceListener | topic.Tracking | NULL | 1018 | Y |
| Clustered JMS | 0 | TraceListener.TraceListener | topic.Tracking | NULL | 1053 | Y |
| Clustered JMS | 0 | TraceListener.TraceListener | topic.Tracking | NULL | 1083 | Y |
+-----------------+---------+-----------------------------+----------------+----------+------------+-----------+

In this case, I first started node 0, and then node 2. I even deleted all of JBM_POSTOFFICE before restarting the two servers.

Additional notes:
- I am using RoundRobin post office.
- I am currently using svn trunk version 2779, because of a bug in the .GA version.

[timfox]

Can you replicate this (from a cleared DB) ?

If so, can you give instructions?

Thx

[bartvh]

I did a delete of JBM_POSTOFFICE, JBM_MSG and JBM_MSGREF. And then:
Start up the two nodes, wait until both are started.kill -9 the first node and start it up againwait until the first node is started again
Now I have the the situation like in the select statement above. If I now stop the two servers and try to start one again, it will always fail.

I hope this is enough to reproduce? I can try to give more information after this weekend.

[bartvh]

This problem appeared to have something to do with how the run.sh script works: my kill was killing the run.sh script, but the jboss server kept running.
(I was using kill instead of plainly hitting Ctrl-C because the latter shuts down jboss cleanly, so there is no cluster instance failure detected.)

[timfox]

If you're running Linux, pstree is a nice utility which displays your processes in a tree. It should be pretty clear to see from that the processes corresponding to the script and java itself.