Problem configuring JMSBridgeImpl in Jboss 7

[dieguinho]

Hi all, I'm pretty new to JMS so please bare with me, I'm trying to learn it as fast as I can. Any help is greatly appreciated.

At my work I need to implement a broadcast system for some entities that we have. We were working with Jboss6 but due to some issues we were forced to migrate to Jboss 7 (precisely 7.1.1), but bridges are not supported in this version, so looking around with google found that they recommend to get the bridge up and running manualy using JMSBridgeImpl.

I made a simple app to learn how this works, its designed in a star configuration where:

the central server has an inputQueue where it receibes messages. Through divert it replicates the messages in N outputQueues (one for each client application). I've created a bean that starts at startup and creates the N JMSBridges (it takes the configuration from an XML file). I managed to make the connections with the bridges and messages are being sent, but, there are a couple of issues:

1) The messages are only sent to the clients IF the clients were up and running (in remote servers) before the central application (the one that receives and diverts the messages to the outputQueues) gets started.

2) If we first get running the central application, the bridges start trying to connect to the clients with no success (due to them being down), we then start the clients and the bridges manage to connect, but the messages are not being delivered to the clients.

3) If all is working OK and we stop one of the clients, the corresponding bridge loses the connection and starts to retry connecting, if then we get the client running, the bridge keeps tryng to connect to no avail, even though it had made a successful connection a couple of minutes earlier. The Exception being thrown again and again is:

17:47:48,564 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-9-thread-2) Failed to set up JMS bridge connections. Most probably the source or target servers are unavailable. Will retry after a pause of 3000 ms

17:47:51,566 WARN  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-9-thread-2) Failed to connect bridge: javax.naming.NamingException: JBAS011843: Failed instantiate InitialContextFactory org.jboss.naming.remote.client.InitialContextFactory from classloader ModuleClassLoader for Module "org.hornetq:main" from local module loader @2acdb06e (roots: /opt/jboss-as-7.1.1.Final-SINCRO/modules)

        at org.jboss.as.naming.InitialContextFactoryBuilder.createInitialContextFactory(InitialContextFactoryBuilder.java:64)

        at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:664) [rt.jar:1.6.0_30]

        at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:288) [rt.jar:1.6.0_30]

        at javax.naming.InitialContext.init(InitialContext.java:223) [rt.jar:1.6.0_30]

        at javax.naming.InitialContext.<init>(InitialContext.java:197) [rt.jar:1.6.0_30]

        at org.hornetq.jms.bridge.impl.JNDIFactorySupport.createObject(JNDIFactorySupport.java:55) [hornetq-jms-2.2.13.Final.jar:]

        at org.hornetq.jms.bridge.impl.JNDIDestinationFactory.createDestination(JNDIDestinationFactory.java:40) [hornetq-jms-2.2.13.Final.jar:]

        at org.hornetq.jms.bridge.impl.JMSBridgeImpl.setupJMSObjects(JMSBridgeImpl.java:1083) [hornetq-jms-2.2.13.Final.jar:]

        at org.hornetq.jms.bridge.impl.JMSBridgeImpl.setupJMSObjectsWithRetry(JMSBridgeImpl.java:1341) [hornetq-jms-2.2.13.Final.jar:]

        at org.hornetq.jms.bridge.impl.JMSBridgeImpl.access$2100(JMSBridgeImpl.java:71) [hornetq-jms-2.2.13.Final.jar:]

        at org.hornetq.jms.bridge.impl.JMSBridgeImpl$FailureHandler.run(JMSBridgeImpl.java:1864) [hornetq-jms-2.2.13.Final.jar:]

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]

        at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]

can someone give me a hand with this?

greatly appreciated for any help

cheers,

Diego

[clebert.suconic]

It seems you passed a wrong JNDI Context to the Bridge. To test if you passed the right JNDIContext, see if you can do a simple lookup at the instance you passed on.

The JMSBridge is nothing more than a consumer/producer pair.. nothing else....  it's as simple as a consumer with a producer.

[dieguinho]

Thanks for the reply Clebert!

The thing is that it actually connects and the messages get delivered in case 1 (when I first start the clients, and once the clients are up and running I then start the server that makes the broadcast of the messages). So I asume the context is right. This is how I'm creating the JMSBridgeImpl:

Properties jndiSourceProperties = new Properties();

jndiSourceProperties.put(Context.INITIAL_CONTEXT_FACTORY,"org.jboss.naming.remote.client.InitialContextFactory");

jndiSourceProperties.put(Context.PROVIDER_URL, "remote://" + ipCentral + ":4447");

jndiSourceProperties.put(Context.SECURITY_PRINCIPAL, "guest");

jndiSourceProperties.put(Context.SECURITY_CREDENTIALS, "pass");

Properties jndiTargetProperties = new Properties();

jndiTargetProperties.put(Context.INITIAL_CONTEXT_FACTORY,"org.jboss.naming.remote.client.InitialContextFactory");

jndiTargetProperties.put(Context.PROVIDER_URL, "remote://" + ipSucursal + ":4447");

jndiTargetProperties.put(Context.SECURITY_PRINCIPAL, "guest");

jndiTargetProperties.put(Context.SECURITY_CREDENTIALS, "pass");

sourceConnectionFactoryFactory = new JNDIConnectionFactoryFactory(jndiSourceProperties, "jms/MyConnectionFactory");

targetConnectionFactoryFactory = new JNDIConnectionFactoryFactory(jndiTargetProperties, "jms/MyConnectionFactory");

sourceDestinationFactory = new JNDIDestinationFactory(jndiSourceProperties, colaSalida);

targetDestinationFactory = new JNDIDestinationFactory(jndiTargetProperties, colaEntrada);

JMSBridge bridge = new JMSBridgeImpl(sourceConnectionFactoryFactory, targetConnectionFactoryFactory, sourceDestinationFactory, targetDestinationFactory, "guest", "pass", "guest", "pass", null, delay, veces, QualityOfServiceMode.AT_MOST_ONCE, 1, -1, null, null, true);

Where ipCentral is the server IP, and ipSucursal is the clients IP.

Any ideas?

[clebert.suconic]

You need to figure out why this Lookup is not working:

instantiate InitialContextFactory org.jboss.naming.remote.client.InitialContextFactory from classloader ModuleClassLoader for Module "org.hornetq:main" from local module loader @2acdb06e (roots: /opt/jboss-as-7.1.1.Final-SINCRO/modules)

        at org.jboss.as.naming.InitialContextFactoryBuilder.createInitialContextFactory(InitialContextFactoryBuilder.java:64)

        at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:664) [rt.jar:1.6.0_30]

        at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:288) [rt.jar:1.6.0_30]

        at javax.naming.InitialContext.init(InitialContext.java:223) [rt.jar:1.6.0_30]

        at javax.naming.InitialContext.<init>(InitialContext.java:197) [rt.jar:1.6.0_30]

        at org.hornetq.jms.bridge.impl.JNDIFactorySupport.createObject(JNDIFactorySupport.java:55) [hornetq-jms-2.2.13.Final.jar:]

        at org.hornetq.jms.bridge.impl.JNDIDestinationFactory.createDestination(JNDIDestinationFactory.java:40) [hornetq-jms-2.2.13.Final.jar:]

        at org.hornetq.jms.bridge.impl.JMSBridgeImpl.setupJMSObjects(JMSBridgeImpl.java:1083) [hornetq-jms-2.2.13.Final.jar:]

        at org.hornetq.jms.bridge.impl.JMSBridgeImpl.setupJMSObjectsWithRetry(JMSBridgeImpl.java:1341) [hornetq-jms-2.2.13.Final.jar:]

        at org.hornetq.jms.bridge.impl.JMSBridgeImpl.access$2100(JMSBridgeImpl.java:71) [hornetq-jms-2.2.13.Final.jar:]

        at org.hornetq.jms.bridge.impl.JMSBridgeImpl$FailureHandler.run(JMSBridgeImpl.java:1864) [hornetq-jms-2.2.13.Final.jar:]

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]

        at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]

[clebert.suconic]

It looks a classLoading isolation thing to me. What would be a configuration issue on JBoss 7 on the modules dependency.

[jbertram]

If you want to use a JMS bridge in AS7 I recommend you use a 7.2.0 nightly.  Documentation for configuring the bridge is here.

[jmesnil]

Is your Java client a simple Java app or is it based on JBoss modules?

When we added JMS bridges support to AS7, we had to update its messaging module definition[1].

If you app is based on JBoss modules (given that you have a ModuleClassLoader), you will have to add a least the org.jboss.remote-naming dependency to org.hornetq module.

[1] https://github.com/jbossas/jboss-as/commit/286244a08bbc55848a8a1a536f99514600bee9f5#diff-4

[dieguinho]

Great guys! thats right, the clients are jboss apps too, not standalone java apps. Added the remote-naming dependency to org.hornetq module and got it working for cases 1 and 3, but for case 2 (when the server app gets up and running before the clients are running) still has problems. In this case the bridges keep trying to connect until I start the clients, then the bridges manage to connect to them, but at the time of delivering messages to the clients, they just don't seem to get to the other side.

@Justin: is there a release date for 7.2? currently using 7.2.0 is not an option for us cause we will need this for production and we are already taking a leap of faith using 7.1.1 in production.

[jbertram]

I'm not aware of any release date for JBoss AS 7.2.

If you want to make your choice of AS 7.1.1 less of a leap of faith then I recommend a support contract with Red Hat for EAP 6.

[dieguinho]

I wanted to clarify that by "leap of faith" I'm not saying it in a bad way, I love Jboss, but I agree completely with you in taking a Support Contract. Thats a thing to talk with the bosses, but being that we are in Argentina I dubt we can afford (or even can) pay for a support contract, due to all the restrictions that we have with transactions in dollars (thanks to our great all mighty politicians )

[clebert.suconic]

As a Brazilian Hermano and neighbour I know what you mean.

But I think there's a RedHat subsidiary in Argentina. I'm not telling you to buy  a contract by this statement. I'm just mentioning that there's a RedHat Office in Argentina with Local Policies and Local prices.

Anyways.. I think you would be able to just get 7.2 nightly. It would probably be a good choice for now?

[dieguinho]

Oi, como vc ta irmao?

Yes, I've been looking at that RedHat subsidiary, but to tell the truth we have a couple guys that are partners of RedHat giving us a hand....but I don't wanna burn them in public hehe.

We could get 7.2 nightly but we can't use it in production, and this is something we gotta get to production as soon as we can.

Any ideas on the other issue?

If I get the clients running first and then I get the server running, the bridges connect and deliver the messages successfully. In this case I can stop the clients and start them again and no problem, the server will reconnect and send any pending messages with no problem to the clients. But in the other case where I get the server running first, the bridges connect to the clients but no messages get delivered, even if i stop the clients and start them again, it will reconnect but no messages delivered...

[dmoreira]

Hi Folks:

We (Diego and Me) were debugging the JMSBridgeImpl (Version 2.2.14.Final) and i can guess, that the reconnection problem is a bug in that version. The bug situation is that:

When you start the bridge and the target server is down not start the BatchTimeChecker or the SourceReceiver

the OnStartFailureHandler class restart the connection when target servers but nothing send the messages from the message's source because not start those helpers.

That bug is solved with that lite hack.

at lines 1901, of  JMSBridgeImpl source code, put the follow code:

if (maxBatchTime != -1)

         {

            if (JMSBridgeImpl.trace)

            {

               JMSBridgeImpl.log.trace("Starting time checker thread");

            }

            timeChecker = new BatchTimeChecker();

            executor.execute(timeChecker);

            batchExpiryTime = System.currentTimeMillis() + maxBatchTime;

            if (JMSBridgeImpl.trace)

            {

               JMSBridgeImpl.log.trace("Started time checker thread");

            }

         }

         executor.execute(new SourceReceiver());

         if (JMSBridgeImpl.trace)

         {

            JMSBridgeImpl.log.trace("Started " + this);

         }

Best

Diogenes

[dieguinho]

Thats correct, Diogenes came through and found the problem. Jboss 7.1.1 comes with HornetQ-2.2.13, we couldn't find the source for that version so we downloaded source for version 2.2.14. We can say that at least in that version of HornetQ if the bridge gets started and the connection to the target can't be stablished, the above setup code snipet is not executed. Therefore when you start the client, the bridge connects but theres no timeChecker to check for messages and deliver them to the target destination.

[dmoreira]

Hi folks:

Continuing with the implementation of a bridge we found a new "bug", i'm guessing,

When you instance a bridge in non-transaction mode and QualityOfServiceMode.AT_MOST_ONCE mode the message not leave de source queue. because no one send de ack message..

That problem is solved replacing  sendBatchNonTransacted() with de following Hack

private void sendBatchNonTransacted()

   {

      try

      {

        

         sendMessages();

         if (maxBatchSize > 1)

         {

            // The sending session is transacted - we need to commit it

            if (JMSBridgeImpl.trace)

            {

               JMSBridgeImpl.log.trace("Committing target session");

            }

            targetSession.commit();

            if (JMSBridgeImpl.trace)

            {

               JMSBridgeImpl.log.trace("Committed source session");

            }

         }

         for(Message msg :messages){

       JMSBridgeImpl.log.trace("Do ACK to message");

       msg.acknowledge();

         }

        

         // Clear the messages

         messages.clear();

      }

      catch (Exception e)

      {

         JMSBridgeImpl.log.warn("Failed to send + acknowledge batch, closing JMS objects", e);

         handleFailureOnSend();

      }

   }

Best D.

PD: the previous "if" over QualityOfServiceMode has not sense en this method.. I can Guess.