14 Replies Latest reply on Sep 21, 2012 3:49 PM by ndipiazza

    HornetQ 2.2.13 - HA not working on MDB client topic - IllegalStateException: Cannot create session factory, server locator is closed (maybe it has been garbage collected)

    ndipiazza

      I have a situation where HA failover fails to take place. Related to this issue: https://community.jboss.org/thread/204023

       

      We have a JBoss 7.1.1.Final JMS Server cluster with two nodes (port 5445, and port 5545).

       

      We have a third JBoss 7.1.1.Final JMS Client that connects to the HornetQ server cluster.

       

      We are connecting with an MDB that has the two node's hard coded in the MDB annotations. It is a durable topic. Discovery is not enabled.

       

      Here is the MDB:

       

      package test.mdb;
      import javax.ejb.EJBException;
      import javax.ejb.MessageDrivenBean;
      import javax.ejb.MessageDrivenContext;
      import javax.jms.Message;
      import javax.jms.MessageListener;
      import javax.naming.Context;
      import javax.naming.InitialContext;
      import javax.naming.NamingException;
      import javax.ejb.ActivationConfigProperty;
      import javax.ejb.MessageDriven;
      @MessageDriven(name = "MDB_CMT_TxRequiredExample", activationConfig = { @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Topic"),
              @ActivationConfigProperty(propertyName = "destination", propertyValue = "ndipiazza.pl.testing"),
              @ActivationConfigProperty(propertyName = "setupAttempts", propertyValue = "-1"),
              @ActivationConfigProperty(propertyName = "hA", propertyValue = "true"),
              @ActivationConfigProperty(propertyName = "clientID", propertyValue = "ndipiazza.pl.testing"),
              @ActivationConfigProperty(propertyName = "subscriptionName", propertyValue = "ndipiazza.pl.testing"),        
              @ActivationConfigProperty(propertyName = "connectorClassName", propertyValue = "org.hornetq.core.remoting.impl.netty.NettyConnectorFactory,org.hornetq.core.remoting.impl.netty.NettyConnectorFactory"),        
              @ActivationConfigProperty(propertyName = "connectionParameters", propertyValue = "host=127.0.0.1;port=5445,host=127.0.0.1;port=5545"),
              @ActivationConfigProperty(propertyName = "subscriptionDurability", propertyValue ="Durable")})
      public class TestMDB implements MessageDrivenBean, MessageListener
      {
          MessageDrivenContext mContext = null;
          Context jndiContext = null;
          int count = 0;
          private static final long serialVersionUID = -8023536477112837766L;
          public void onMessage(Message message)
          {
              System.out.println("\n ************************* In onMessage of TestMDB : " + (++count));
          }
          public void ejbRemove() throws EJBException
          {
              mContext = null;
          }
          public void setMessageDrivenContext(MessageDrivenContext messageDrivenContext) throws EJBException
          {
              mContext = messageDrivenContext;
              try
              {
                  jndiContext = new InitialContext();
              }
              catch (NamingException ne)
              {
                  throw new EJBException(ne);
              }
          }
      }
      

       

      So the MDB listens on both 127.0.0.1:5445 and 127.0.0.1:5545

       

      We start up the cluster and the client then start sending the cluster some messages. You at that point correctly see the messages being consumed from the client from the first JMS server listed (127.0.0.1:5445).

       

      Now we stop the node 127.0.0.1:5445, and instead of failing over to 127.0.0.1:5545, you get this:

       

      Thread-29 (HornetQ-client-global-threads-7213879)) Notified of connection failure in xa recovery connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=127-0-0-1, org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5545&host=127-0-0-1], discoveryGroupConfiguration=null], connectorConfig=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5545&host=localhost, backupConfig=null] will attempt reconnect on next pass: HornetQException[errorCode=4 message=The connection was disconnected because of server shutdown]
       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1467) [hornetq-core-2.2.13.Final.jar:]
              at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [hornetq-core-2.2.13.Final.jar:]
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) [rt.jar:1.6.0_24]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [rt.jar:1.6.0_24]
              at java.lang.Thread.run(Unknown Source) [rt.jar:1.6.0_24]
      

       

       

      This error message says: "will attempt reconnect on next pass" but the failover never occurs and the message processing will not resume until the client server redeploys the WAR file with the MDB.

       

      Later, another error appears:

       

      17:31:14,354 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016027: Local XARecoveryModule.xaRecovery got XA exception XAException.XAER_RMERR: javax.transaction.xa.XAException: Error trying to connect to any providers for xa recovery
       at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.getDelegate(HornetQXAResourceWrapper.java:275) [hornetq-jms-2.2.13.Final.jar:]
       at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.recover(HornetQXAResourceWrapper.java:77) [hornetq-jms-2.2.13.Final.jar:]
       at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecovery(XARecoveryModule.java:503) [jbossjts-4.16.2.Final.jar:]
       at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecoveryForRecoveryHelpers(XARecoveryModule.java:471) [jbossjts-4.16.2.Final.jar:]
       at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:385) [jbossjts-4.16.2.Final.jar:]
       at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:166) [jbossjts-4.16.2.Final.jar:]
       at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) [jbossjts-4.16.2.Final.jar:]
       at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [jbossjts-4.16.2.Final.jar:]
      Caused by: java.lang.IllegalStateException: Cannot create session factory, server locator is closed (maybe it has been garbage collected)
       at org.hornetq.core.client.impl.ServerLocatorImpl.assertOpen(ServerLocatorImpl.java:1823) [hornetq-core-2.2.13.Final.jar:]
       at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:699) [hornetq-core-2.2.13.Final.jar:]
       at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connect(HornetQXAResourceWrapper.java:321) [hornetq-jms-2.2.13.Final.jar:]
       at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.getDelegate(HornetQXAResourceWrapper.java:251) [hornetq-jms-2.2.13.Final.jar:]
       ... 7 more
      
      
      

       

       

      Even when i turn back on the node i shut down... it still won't start processing again. So not only does it not failover, it won't even reconnect to the existing node after it returns.

       

      This error message eventually repeats.

       

      So it looks like this is the next pass it was talking about. The error complains about this: Cannot create session factory, server locator is closed (maybe it has been garbage collected)

       

      When I debug that line of code, The state is

      ServerLocatorImpl$STATE  (id=327) --> name "CLOSED" (id=331) , ordinal 1 
      

       

      Any ideas why this might happen?

        • 1. Re: HornetQ 2.2.13 - HA not working on MDB client topic - IllegalStateException: Cannot create session factory, server locator is closed (maybe it has been garbage collected)
          clebert.suconic

          Would be too bad to ask you to try a checkout from git, Branch_2_2_AS7 and replace the JARs?

           

          I remember this message somewhere not long ago. If you're still seeing this we can take a look...

           

          Maybe there's some different also on the Transaction Manager on AS7.. and if that's the case we will have to take a look.

          • 2. Re: HornetQ 2.2.13 - HA not working on MDB client topic - IllegalStateException: Cannot create session factory, server locator is closed (maybe it has been garbage collected)
            ndipiazza

            Hey I build the Branch_2_2_AS7 but I'm not quite sure how to update the JARs in JBoss 7.1.1.Final now.

             

            There seem to just be 2 modules that JBoss AS7 uses...

             

            modules\org\hornetq\main and modules\org\hornetq\ra\main

             

            So I updated the modules to the versions that I built, now I get this error:

             

            class org.jboss.as.messaging.jms.AS7RecoveryRegistry in Module "org.jboss.as.me
            ssaging:main" from local module loader @9444d1 (roots: C:\JBoss\jboss-as-7.1.1.F
            inal-c\modules): java.lang.LinkageError: Failed to link org/jboss/as/messaging/j
            ms/AS7RecoveryRegistry (Module "org.jboss.as.messaging:main" from local module l
            oader @9444d1 (roots: C:\JBoss\jboss-as-7.1.1.Final-c\modules))
                    at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.jav
            a:396)
                    at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.
            java:243)
                    at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoade
            r.java:73)
                    at org.jboss.modules.Module.loadModuleClass(Module.java:517)
                    at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:
            182)
                    at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(Con
            currentClassLoader.java:468)
                    at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(Concu
            rrentClassLoader.java:456)
                    at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(Concu
            rrentClassLoader.java:423)
                    at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentCl
            assLoader.java:398)
                    at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoad
            er.java:120)
                    at org.jboss.as.messaging.jms.PooledConnectionFactoryService.createServi
            ce(PooledConnectionFactoryService.java:245) [jboss-as-messaging-7.1.1.Final.jar:
            7.1.1.Final]
                    at org.jboss.as.messaging.jms.PooledConnectionFactoryService.start(Poole
            dConnectionFactoryService.java:180) [jboss-as-messaging-7.1.1.Final.jar:7.1.1.Fi
            nal]
                    at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(Se
            rviceControllerImpl.java:1811) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
                    at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceCont
            rollerImpl.java:1746) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source
            ) [rt.jar:1.6.0_24]
                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [r
            t.jar:1.6.0_24]
                    at java.lang.Thread.run(Unknown Source) [rt.jar:1.6.0_24]
            Caused by: java.lang.NoClassDefFoundError: org/hornetq/jms/server/recovery/Recov
            eryRegistry
                    at java.lang.ClassLoader.defineClass1(Native Method) [rt.jar:1.6.0_24]
                    at java.lang.ClassLoader.defineClassCond(Unknown Source) [rt.jar:1.6.0_2
            4]
                    at java.lang.ClassLoader.defineClass(Unknown Source) [rt.jar:1.6.0_24]
                    at java.security.SecureClassLoader.defineClass(Unknown Source) [rt.jar:1
            .6.0_24]
                    at org.jboss.modules.ModuleClassLoader.doDefineOrLoadClass(ModuleClassLo
            ader.java:327)
                    at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.jav
            a:391)
                    ... 16 more
            Caused by: java.lang.ClassNotFoundException: org.hornetq.jms.server.recovery.Rec
            overyRegistry from [Module "org.jboss.as.messaging:main" from local module loade
            r @9444d1 (roots: C:\JBoss\jboss-as-7.1.1.Final-c\modules)]
                    at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:
            190)
                    at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(Con
            currentClassLoader.java:468)
                    at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(Concu
            rrentClassLoader.java:456)
                    at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(Concu
            rrentClassLoader.java:423)
                    at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(Concu
            rrentClassLoader.java:423)
                    at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentCl
            assLoader.java:398)
                    at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoad
            er.java:120)
                    ... 22 more

            • 3. Re: HornetQ 2.2.13 - HA not working on MDB client topic - IllegalStateException: Cannot create session factory, server locator is closed (maybe it has been garbage collected)
              jbertram

              I believe there are now incompatibilities between HornetQ and the messaging subsystem in 7.1.1.  You'd need to use 7.2 Alpha.

              1 of 1 people found this helpful
              • 4. Re: HornetQ 2.2.13 - HA not working on MDB client topic - IllegalStateException: Cannot create session factory, server locator is closed (maybe it has been garbage collected)
                ndipiazza

                No change when I moved to jboss-as-7.2.0.Alpha1-SNAPSHOT. I am able to reproduce the same issue.

                 

                The MDB listens to the first node in the list.

                 

                I force a shutdown to the node, and the server does not fail over to the next server in the list.

                 

                I'm going to keep digging, please let me know anything i can do to help.

                • 5. Re: HornetQ 2.2.13 - HA not working on MDB client topic - IllegalStateException: Cannot create session factory, server locator is closed (maybe it has been garbage collected)
                  ndipiazza

                  No I was mistaken. The problem is different now.

                   

                  The org.hornetq.core.client.impl.ServerLocatorImpl.assertOpen() method now has a change to the if statement:

                   

                   

                   

                   

                  if (state != null && state != STATE.INITIALIZED

                   

                  )

                  {

                   

                      throw new IllegalStateException("Cannot create session factory, server locator is closed (maybe it has been garbage collected)");

                  }

                   

                  So no more of those messages are appearing in error. Still no failover. But now I can get deeper. Still digging.

                  • 6. Re: HornetQ 2.2.13 - HA not working on MDB client topic - IllegalStateException: Cannot create session factory, server locator is closed (maybe it has been garbage collected)
                    ndipiazza

                    Hi Justin and Clebert,

                     

                    I have made some more progress on the research of the issue.

                     

                    I noticed that the recovery notification code is working:

                     

                    --> When the node goes down, it calls  org.hornetq.core.client.impl.ServerLocatorImpl.notifyNodeDown(long, String)

                     

                    --> When the node comes back up, it calls org.hornetq.core.client.impl.ServerLocatorImpl.notifyNodeUp(long, String, Pair<TransportConfiguration, TransportConfiguration>, boolean)

                     

                    After notifyNodeDown has been called, the failed endpoint is removed from the org.hornetq.core.client.impl.ServerLocatorImpl.topology object.

                     

                    After notifyNodeUp has been called, the endpoint is added once again to the org.hornetq.core.client.impl.ServerLocatorImpl.topology object.

                     

                    So this seems to be working exactly as expect. I'm thinking what's happening is that the MDB itself lacks the ability to re-activate after the node it was using went down then back up again.

                     

                    Continuing to research....

                     

                    • 7. Re: HornetQ 2.2.13 - HA not working on MDB client topic - IllegalStateException: Cannot create session factory, server locator is closed (maybe it has been garbage collected)
                      jbertram

                      Can you attach the configuration for all the nodes in the cluster to which you are connecting? 

                       

                      I ask because even though you have set ha = true on your MDB's activation configuration properties this just results in this method call:

                       

                      HornetQJMSClient.createConnectionFactoryWithHA(JMSFactoryType.XA_CF, transportConfigurations);
                      

                       

                      And this does not guarantee that fail-over will actually occur.  As the JavaDoc states, that method will:

                      Create a HornetQConnectionFactory which will receive cluster topology updates from the cluster as servers leave or join and new backups are appointed or removed.  The initial list of servers supplied in this method is simply to make an initial connection to the cluster, once that connection is made, up to date cluster topology information is downloaded and automatically updated whenever the cluster topology changes. If the topology includes backup servers that information is also propagated to the client so that it can know which server to failover onto in case of live server failure.

                       

                      Therefore, if your servers aren't configured with a back-up then fail-over won't occur.  That is what looks like is happening in your case.

                      • 8. Re: HornetQ 2.2.13 - HA not working on MDB client topic - IllegalStateException: Cannot create session factory, server locator is closed (maybe it has been garbage collected)
                        ndipiazza

                        That sounds like the answer. So I just set one as Backup?

                         

                        Attached is the ZIP file for this issue with the configurations you requested.

                         

                        I tried making the Servers both on the JBoss 7.2.0.Alpha1 but I'm getting an error --> the securityStore is not null and is trying to authenticate the client to the server even though in my standalone.xml's messaging subsystem I have:

                         

                         

                        <security-enabled>false</security-enabled>

                         

                        Thanks,

                        -Nicholas

                        • 9. Re: HornetQ 2.2.13 - HA not working on MDB client topic - IllegalStateException: Cannot create session factory, server locator is closed (maybe it has been garbage collected)
                          jbertram

                          For live/backup configuration see http://docs.jboss.org/hornetq/2.2.14.Final/user-manual/en/html/ha.html.

                           

                          For the other issue with 7.2 alpha open a new thread on the AS7 forum and fully explain what you're seeing.

                          • 10. Re: HornetQ 2.2.13 - HA not working on MDB client topic - IllegalStateException: Cannot create session factory, server locator is closed (maybe it has been garbage collected)
                            ndipiazza

                            Justin,

                             

                            OK so I went back to JBoss 7.1.1.Final (HornetQ 2.2.13) and then here is what I added...

                             

                            ServerA: (Master)

                             

                            <clustered>true</clustered>
                            <shared-store>true</shared-store>
                            <security-enabled>false</security-enabled>
                            <persistence-enabled>true</persistence-enabled>
                            <journal-file-size>102400</journal-file-size>
                            <journal-min-files>2</journal-min-files>
                            <journal-directory path="c:\JBoss\journal-dir" />
                            

                             

                            ServerB: (Backup)

                             

                            <clustered>true</clustered>
                            <backup>true</backup>
                            <shared-store>true</shared-store>
                            <security-enabled>false</security-enabled>
                            <persistence-enabled>true</persistence-enabled>
                            <journal-file-size>102400</journal-file-size>
                            <journal-directory path="c:\JBoss\journal-dir" />
                            <journal-min-files>2</journal-min-files>
                            

                            ServerC: Stayed the same.

                             

                            Upon starting server B, it first shows that the Backup was registered...

                             

                            14:14:12,607 INFO  [org.hornetq.core.server.cluster.impl.ClusterConnectionImpl]
                            (Thread-0 (HornetQ-server-HornetQServerImpl::serverUUID=00fda6f7-01c3-11e2-b6e5-
                            2477036e7584-10532909)) backup announced
                            

                             

                            then it just loops and shows this error:

                             

                            14:14:19,966 WARN  [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (P
                            eriodic Recovery) Can't connect to any hornetq server on recovery [XARecoveryCon
                            fig [hornetQConnectionFactory=HornetQConnectionFactory [serverLocator=ServerLoca
                            torImpl [initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFac
                            tory?server-id=0], discoveryGroupConfiguration=null], clientID=null, dupsOKBatch
                            Size=1048576, transactionBatchSize=1048576,], username=null, pass
                            word=null]]
                            14:14:19,997 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016027: Local
                            XARecoveryModule.xaRecovery got XA exception XAException.XAER_RMERR: javax.trans
                            action.xa.XAException: Error trying to connect to any providers for xa recovery
                                    at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.getDelegate(
                            HornetQXAResourceWrapper.java:275) [hornetq-jms-2.2.13.Final.jar:]
                                    at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.recover(Horn
                            etQXAResourceWrapper.java:77) [hornetq-jms-2.2.13.Final.jar:]
                                    at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRe
                            covery(XARecoveryModule.java:503) [jbossjts-4.16.2.Final.jar:]
                                    at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.reso
                            urceInitiatedRecoveryForRecoveryHelpers(XARecoveryModule.java:471) [jbossjts-4.1
                            6.2.Final.jar:]
                                    at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bott
                            omUpRecovery(XARecoveryModule.java:385) [jbossjts-4.16.2.Final.jar:]
                                    at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.peri
                            odicWorkSecondPass(XARecoveryModule.java:166) [jbossjts-4.16.2.Final.jar:]
                                    at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkIntern
                            al(PeriodicRecovery.java:789) [jbossjts-4.16.2.Final.jar:]
                                    at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(Periodic
                            Recovery.java:371) [jbossjts-4.16.2.Final.jar:]
                            Caused by: HornetQException[errorCode=2 message=null]
                                    at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connect(Horn
                            etQXAResourceWrapper.java:351) [hornetq-jms-2.2.13.Final.jar:]
                                    at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.getDelegate(
                            HornetQXAResourceWrapper.java:251) [hornetq-jms-2.2.13.Final.jar:]
                                    ... 7 more
                            
                            

                             

                            If I try then turning on the client... I am no longer able to recieve messages.

                             

                            I get this error from the client:

                             

                             

                            javax.jms.JMSSecurityException: Unable to validate user: null
                             at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:312)
                             at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:780)
                             at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:279)
                             at org.hornetq.jms.client.HornetQConnection.authorize(HornetQConnection.java:601)
                             at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:684)
                             at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:119)
                             at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:114)
                             at test.mdb.PublishMessage.main(PublishMessage.java:52)
                            Caused by: HornetQException[errorCode=105 message=Unable to validate user: null]
                             ... 8 more
                            javax.jms.JMSSecurityException: Unable to validate user: null
                             at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:312)
                             at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:780)
                             at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:279)
                             at org.hornetq.jms.client.HornetQConnection.authorize(HornetQConnection.java:601)
                             at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:684)
                             at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:119)
                             at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:114)
                             at test.mdb.PublishMessage.main(PublishMessage.java:52)
                            Caused by: HornetQException[errorCode=105 message=Unable to validate user: null]
                             ... 8 more
                            

                             

                            Removing the <backup> attribute from ServerB and restarting each server allows me to recieve messages from the MDB again.

                             

                            What am I doing wrong here?

                             

                            • 11. Re: HornetQ 2.2.13 - HA not working on MDB client topic - IllegalStateException: Cannot create session factory, server locator is closed (maybe it has been garbage collected)
                              jbertram

                              You can ignore the WARN messages about recovery.  That is an integration detail that hasn't yet been ironed out.  Basically what happens is that the Hornet JCA RA registers itself with JBossTS transaction recovery manager but since the server is in back-up it doesn't fully start.  Then when the JBossTS transaction recovery manager runs (every 120 seconds by default) it emits this WARN because it can't find the server.

                               

                              Regarding the security problem, do you see any errors or stack traces on the server?

                              • 12. Re: HornetQ 2.2.13 - HA not working on MDB client topic - IllegalStateException: Cannot create session factory, server locator is closed (maybe it has been garbage collected)
                                ndipiazza

                                OK thanks for all this info. Equipped with this information I was able to get a bit farther.

                                 

                                1) Start Server Master

                                2) Start Server Backup

                                3) Verified Server Backup is ready to go because log says 15:25:09,815 INFO  [org.hornetq.core.server.cluster.impl.ClusterConnectionImpl] (Thread-0 (HornetQ-server-HornetQServerImpl::serverUUID=00fda6f7-01c3-11e2-b6e5-2477036e7584-12198625)) backup announced

                                4) Started the Client Server.

                                5) Sent some messages to Master and

                                5b) Verified MDB is successfully collecting subscribed messages for the Topic from Server Master.

                                6) I Open Process Manager -> Kill Server Master process

                                7) Verified the Backup jumps into action: 15:38:49,033 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (Activation for server HornetQServerImpl::serverUUID=b8fe11f6-01d0-11e2-a37f-2477036e7584) Backup Server is now live

                                8) Sent some messages to the Backup server and verified messages were sent successfully.

                                9) Unfortunatley, client server is no longer listening to the messages... even though it is supposed to be configured with HA with the first server being the Master server, and the secondary server was listed as the Backup.

                                 

                                Here is what the logs show from the Client Server.

                                 

                                15:42:23,552 WARN  [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Thread-13 (HornetQ-client-global-threads-2932814)) Notified of connection failure in xa recovery connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=127-0-0-1, org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5545&host=127-0-0-1], discoveryGroupConfiguration=null], connectorConfig=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=localhost, backupConfig=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5545&host=localhost] will attempt reconnect on next pass: HornetQException[errorCode=0 message=Netty exception]
                                at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.exceptionCaught(HornetQChannelHandler.java:108) [hornetq-core-2.2.13.Final.jar:]
                                at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:142) [netty-3.2.6.Final.jar:]
                                at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:372) [netty-3.2.6.Final.jar:]
                                at org.jboss.netty.channel.StaticChannelPipeline$StaticChannelHandlerContext.sendUpstream(StaticChannelPipeline.java:534) [netty-3.2.6.Final.jar:]
                                at org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148) [netty-3.2.6.Final.jar:]
                                at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) [netty-3.2.6.Final.jar:]
                                at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:372) [netty-3.2.6.Final.jar:]
                                at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:367) [netty-3.2.6.Final.jar:]
                                at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432) [netty-3.2.6.Final.jar:]
                                at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:95) [netty-3.2.6.Final.jar:]
                                at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.2.6.Final.jar:]
                                at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) [netty-3.2.6.Final.jar:]
                                at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:181) [netty-3.2.6.Final.jar:]
                                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_24]
                                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_24]
                                at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_24]
                                Caused by: java.net.SocketException: Connection reset
                                at java.net.SocketInputStream.read(SocketInputStream.java:168) [rt.jar:1.6.0_24]
                                at java.net.SocketInputStream.read(SocketInputStream.java:182) [rt.jar:1.6.0_24]
                                at java.io.FilterInputStream.read(FilterInputStream.java:66) [rt.jar:1.6.0_24]
                                at java.io.PushbackInputStream.read(PushbackInputStream.java:122) [rt.jar:1.6.0_24]
                                at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:86) [netty-3.2.6.Final.jar:]
                                ... 4 more
                                

                                 

                                10) If you restart the client server... it will start collecting the messages it had been missing this whole time.

                                • 13. Re: HornetQ 2.2.13 - HA not working on MDB client topic - IllegalStateException: Cannot create session factory, server locator is closed (maybe it has been garbage collected)
                                  ndipiazza

                                  Created https://issues.jboss.org/browse/HORNETQ-1034 to track this issue.

                                   

                                  Now it contains an Ant project that fully reproduces the problem with one step.

                                  • 14. Re: HornetQ 2.2.13 - HA not working on MDB client topic - IllegalStateException: Cannot create session factory, server locator is closed (maybe it has been garbage collected)
                                    ndipiazza

                                    There is a bug that does not automatically overwrite the default value for reconnectAttempts to -1 (for unlimited) when hA is being used. It was instead a default of 0...

                                     

                                    This made it so org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(Object, HornetQException) will simply ignore any attempt to reconnect to the jms servers. And all is lost at that point.

                                     

                                    Simply specifying an MDB activation configuration property:

                                     
                                    @ActivationConfigProperty
                                    (propertyName = "reconnectAttempts", propertyValue = "-1")
                                    
                                    
                                    
                                     
                                    

                                    This makes it so the failover starts working as it is supposed to.