HornetQ 2.3.0 RA + JGroups

[jmesnil]

For AS7-6124, I want to use JGroups with HornetQ RA.

I updated AS7 integration code so that the RA would use the same JChannel that is created when a discovery-group is configured with JGroups.

It does *not* create a new JChannel object for the RA (otherwise JGroups complains that I can't connect twice to a singleton transport).

I have a test that does not pass with that modification and I can't find the reason yet.

Could one of you give an eye on it and check that I'm not doing anything stupid?

The AS7 branch is at  https://github.com/jmesnil/jboss-as/tree/AS7-6124_messaging_jgroups_in_RA and uses hornetq master branch (not 2.3.0.BETA3 tag).

1. build AS7 with "mvn clean install"

2. copy https://gist.github.com/4494044 standalone-full.xml file to AS7 build/target/jboss-as-7.2.0.Alpha1-SNAPSHOT/standalone/configuration/standalone-full.xml

This conf use HornetQ's own UDP stack by default ("dg-group1")

3. run the test with ./integration-tests.sh -Dts.smoke -Dtest=org.jboss.as.test.smoke.jms.SendToQueueFromWithinTransactionTest

=> the test will fail with an error: "No, the test did pass but I wanted to see the test output". This means the test passed but I fail it at the end to keep the logs

4. modify AS7 build/target/jboss-as-7.2.0.Alpha1-SNAPSHOT/standalone/configuration/standalone-full.xml to use the JGroups stack for discovery by using "dg-group2" instead of "dg-group1" for the "hornetq-ra" pooled-connection-factory (https://gist.github.com/4494044#file-standalone-full-xml-L384)

=> the test now really fails because the discovery group never receive the broadcast group data.

dg-group2  is using the default UDP stack from JGroups which should work without any modification out of the box.

Could one of you give the test a try to see if my issue is environmental (unlikely since HornetQ's own UDP stack is working) or if there is something else going on?

The relevant code in AS7 is in HornetQService when the JChannel object is created[1] and JGroupsChannelLocator[2] that is called back by the HornetQ RA to get a reference on this channel. The RA has specific properties for that[3].

[1] https://github.com/jmesnil/jboss-as/blob/AS7-6124_messaging_jgroups_in_RA/messaging/src/main/java/org/jboss/as/messaging/HornetQService.java#L227

[2] https://github.com/jmesnil/jboss-as/blob/AS7-6124_messaging_jgroups_in_RA/messaging/src/main/java/org/jboss/as/messaging/jms/JGroupsChannelLocator.java

[3] https://github.com/jmesnil/jboss-as/blob/AS7-6124_messaging_jgroups_in_RA/messaging/src/main/java/org/jboss/as/messaging/jms/PooledConnectionFactoryService.java#L267

[gaohoward]

Hi Jeff I'll take a look.

[gaohoward]

I reproduced the same failure. I'm going to investigate a bit. But for the moment I search the log and found that I can see log

~~~~~~~~ created broadcast group channel whith key broadcastbg-group2 org.jboss.as.clustering.jgroups.MuxChannel@3b3219ed null howard-thinkpad-w510/hq-cluster

but I can't see something like:

~~~~~~~~ created discovery group .....

in the logs. Does that suggest the discovery group hasn't been created at all?

Howard

[gaohoward]

So far I've located where the real exception is:

----------------Exception java.lang.IllegalStateException: channel is closed------------------

org.jgroups.JChannel.checkClosed(JChannel.java:861)

org.jgroups.JChannel._preConnect(JChannel.java:516)

org.jgroups.JChannel.connect(JChannel.java:277)

org.jgroups.JChannel.connect(JChannel.java:268)

org.hornetq.api.core.JGroupsBroadcastGroupConfiguration$JGroupsBroadcastEndpoint.internalOpen(JGroupsBroadcastGroupConfiguration.java:175)

org.hornetq.api.core.JGroupsBroadcastGroupConfiguration$JGroupsBroadcastEndpoint.openClient(JGroupsBroadcastGroupConfiguration.java:159)

org.hornetq.core.cluster.DiscoveryGroup.start(DiscoveryGroup.java:108)

org.hornetq.core.client.impl.ServerLocatorImpl.initialise(ServerLocatorImpl.java:389)

org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:777)

org.hornetq.ra.inflow.HornetQActivation.setup(HornetQActivation.java:314)

org.hornetq.ra.inflow.HornetQActivation$SetupActivation.run(HornetQActivation.java:704)

org.jboss.jca.core.workmanager.WorkWrapper.run(WorkWrapper.java:218)

org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33)

org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:801)

org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45)

org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:821)

java.lang.Thread.run(Thread.java:662)

org.jboss.threads.JBossThread.run(JBossThread.java:122)

Turns out the JChannel passed to RA is closed already. However I don't know how is so yet.

[gaohoward]

Hi Jeff

The reason why the test fails is that the jgroups channel is shared in the VM. If a connection is created and then closed (for example the TX recovery manager may periodically create a connection to recover the transactions) it will cause the channel closed. To fix this, I added a 'wrapper' to each channel. The wrapper maintains a reference count to the channel. The channel will be closed only when the reference count is zero.

In closing the channel, we actually 'disconnect' it so that when a new connection is created, the channel can be re-connected.

I'm not sure this is the best way to do it. Besides I'm having a NoSuchMethod issue to be investigated. I'm uploading the patch. could you have a look? The test should pass after applying the patch.

Thanks,

Howard

[gaohoward]

The patch.