7 Replies Latest reply on Mar 28, 2011 4:08 PM by rhinmass

    Journal replication in jboss as6

    rhinmass

      Hello,

       

      I'm trying to set up replication with JBoss AS6 with embedded HornetQ.  Should the master and slave jboss instances be clustered or non-clustered? I've actually tried both ways and neither works.

       

      Here is what I did:

       

      1) On master hornetq-configuration.xml, added:

       

      <backup-connector-ref connector-name="backup-connector" />

       

      and ...

      <connectors>

      ...

        <connector name="backup-connector">

             <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

             <param key="host" value="169.254.25.129"/>

             <param key="port" value="5445"/>

        </connector>

      </connectors>   <!-- 169.254.25.129 is local loopback address that the other node is bound to. -->

       

      2) On slave hornetq-configuration.xml, added:

       

      <backup>true</backup>

      <shared-store>false</shared-store>

      <!-- left acceptors as configured out-of-the-box -->

       

      3) Copied masters data\hornetq directory onto the slave

       

      4) Started slave jboss.  No problem.

       

      5) Started master jboss.

       

      So I am wondering if there is a way to have journal replication in JBoss AS6, and what I might be doing wrong here.  Thanks for any help!!

        • 1. Journal replication in jboss as6
          rhinmass

          Adding more info...

           

          In non-clustered setup, starting slave yields:

           

          DEPLOYMENTS IN ERROR:

            Deployment "org.hornetq:module=JMS,type=Queue,name="testqueue"" is in error due to the following reason(s): java.lang.IllegalStateException: Cannot access JMS Server, core server is not yet active

            Deployment "org.hornetq:module=JMS,type=Queue,name="ExpiryQueue"" is in error due to the following reason(s): java.lang.IllegalStateException: Cannot access JMS Server, core server is not yet active

            Deployment "org.hornetq:module=JMS,name="NettyThroughputConnectionFactory",type=ConnectionFactory" is in error due to the following reason(s): java.lang.IllegalStateException: Cannot access JMS Server, core server is not yet active

            Deployment "org.hornetq:module=JMS,name="NettyConnectionFactory",type=ConnectionFactory" is in error due to the following reason(s): java.lang.IllegalStateException: Cannot access JMS Server, core server is not yet active

            Deployment "org.hornetq:module=JMS,name="InVMConnectionFactory",type=ConnectionFactory" is in error due to the following reason(s): java.lang.IllegalStateException: Cannot access JMS Server, core server is not yet active

            Deployment "org.hornetq:module=JMS,type=Queue,name="DLQ"" is in error due to the following reason(s): java.lang.IllegalStateException: Cannot access JMS Server, core server is not yet active

           

           

          Starting master, yields:

           

          DEPLOYMENTS MISSING DEPENDENCIES:

            Deployment "JMSConnectionFactoryManageMO" is missing the following dependencies:

              Dependency "<UNKNOWN JMSConnectionFactoryManageMO>" (should be in state "Installed", but is actually in state "** UNRESOLVED Demands 'JMSServerManager' **")

            Deployment "JMSQueueManageMO" is missing the following dependencies:

              Dependency "<UNKNOWN JMSQueueManageMO>" (should be in state "Installed", but is actually in state "** UNRESOLVED Demands 'JMSServerManager' **")

            Deployment "JMSServerMO" is missing the following dependencies:

              Dependency "<UNKNOWN JMSServerMO>" (should be in state "Installed", but is actually in state "** UNRESOLVED Demands 'JMSServerManager' **")

            Deployment "JMSTopicManageMO" is missing the following dependencies:

              Dependency "<UNKNOWN JMSTopicManageMO>" (should be in state "Installed", but is actually in state "** UNRESOLVED Demands 'JMSServerManager' **")

            Deployment "org.hornetq:module=JMS,name="InVMConnectionFactory",type=ConnectionFactory" is missing the following dependencies:

              Dependency "JMSServerManager" (should be in state "Installed", but is actually in state "**ERROR**")

            Deployment "org.hornetq:module=JMS,name="NettyConnectionFactory",type=ConnectionFactory" is missing the following dependencies:

              Dependency "JMSServerManager" (should be in state "Installed", but is actually in state "**ERROR**")

            Deployment "org.hornetq:module=JMS,name="NettyThroughputConnectionFactory",type=ConnectionFactory" is missing the following dependencies:

              Dependency "JMSServerManager" (should be in state "Installed", but is actually in state "**ERROR**")

            Deployment "org.hornetq:module=JMS,type=Queue,name="DLQ"" is missing the following dependencies:

              Dependency "JMSServerManager" (should be in state "Installed", but is actually in state "**ERROR**")

            Deployment "org.hornetq:module=JMS,type=Queue,name="testqueue"" is missing the following dependencies:

              Dependency "JMSServerManager" (should be in state "Installed", but is actually in state "**ERROR**")

             Deployment "org.hornetq:module=JMS,type=Queue,name="ExpiryQueue"" is missing the following dependencies:

              Dependency "JMSServerManager" (should be in state "Installed", but is actually in state "**ERROR**")

           

          DEPLOYMENTS IN ERROR:

            Deployment "JMSServerManager" is in error due to the following reason(s): HornetQException[errorCode=104 message=Backup server MUST be started before live server. Initialisation will not proceed.], **ERROR**

            Deployment "<UNKNOWN JMSServerMO>" is in error due to the following reason(s): ** UNRESOLVED Demands 'JMSServerManager' **

            Deployment "<UNKNOWN JMSConnectionFactoryManageMO>" is in error due to the following reason(s): ** UNRESOLVED Demands 'JMSServerManager' **

            Deployment "<UNKNOWN JMSTopicManageMO>" is in error due to the following reason(s): ** UNRESOLVED Demands 'JMSServerManager' **

            Deployment "<UNKNOWN JMSQueueManageMO>" is in error due to the following reason(s): ** UNRESOLVED Demands 'JMSServerManager' **

           

              at org.jboss.deployers.plugins.deployers.DeployersImpl.checkComplete(DeployersImpl.java:1228) [:2.2.0.GA]

              at org.jboss.deployers.plugins.main.MainDeployerImpl.checkComplete(MainDeployerImpl.java:905) [:2.2.0.GA]

              at org.jboss.system.server.profileservice.deployers.MainDeployerPlugin.checkComplete(MainDeployerPlugin.java:87) [:6.0.0.Final]

              at org.jboss.profileservice.deployment.ProfileDeployerPluginRegistry.checkAllComplete(ProfileDeployerPluginRegistry.java:107) [:0.2.2]

              at org.jboss.system.server.profileservice.bootstrap.BasicProfileServiceBootstrap.start(BasicProfileServiceBootstrap.java:135) [:6.0.0.Final]

              at org.jboss.system.server.profileservice.bootstrap.BasicProfileServiceBootstrap.start(BasicProfileServiceBootstrap.java:56) [:6.0.0.Final]

              at org.jboss.bootstrap.impl.base.server.AbstractServer.startBootstraps(AbstractServer.java:827) [jboss-bootstrap-impl-base.jar:2.1.0-alpha-5]

              at org.jboss.bootstrap.impl.base.server.AbstractServer$StartServerTask.run(AbstractServer.java:417) [jboss-bootstrap-impl-base.jar:2.1.0-alpha-5]

              at java.lang.Thread.run(Thread.java:619) [:1.6.0_21]

           

           

          In clustered setup - here is what happens:

           

          Slave starts normall - no errors

           

          Master yields:

           

          DEPLOYMENTS MISSING DEPENDENCIES:

            Deployment "JMSConnectionFactoryManageMO" is missing the following dependencies:

              Dependency "<UNKNOWN JMSConnectionFactoryManageMO>" (should be in state "Installed", but is actually in state "** UNRESOLVED Demands 'JMSServerManager' **")

            Deployment "JMSQueueManageMO" is missing the following dependencies:

              Dependency "<UNKNOWN JMSQueueManageMO>" (should be in state "Installed", but is actually in state "** UNRESOLVED Demands 'JMSServerManager' **")

            Deployment "JMSServerMO" is missing the following dependencies:

              Dependency "<UNKNOWN JMSServerMO>" (should be in state "Installed", but is actually in state "** UNRESOLVED Demands 'JMSServerManager' **")

            Deployment "JMSTopicManageMO" is missing the following dependencies:

              Dependency "<UNKNOWN JMSTopicManageMO>" (should be in state "Installed", but is actually in state "** UNRESOLVED Demands 'JMSServerManager' **")

            Deployment "org.hornetq:module=JMS,name="InVMConnectionFactory",type=ConnectionFactory" is missing the following dependencies:

              Dependency "JMSServerManager" (should be in state "Installed", but is actually in state "**ERROR**")

            Deployment "org.hornetq:module=JMS,name="NettyConnectionFactory",type=ConnectionFactory" is missing the following dependencies:

              Dependency "JMSServerManager" (should be in state "Installed", but is actually in state "**ERROR**")

            Deployment "org.hornetq:module=JMS,name="NettyThroughputConnectionFactory",type=ConnectionFactory" is missing the following dependencies:

              Dependency "JMSServerManager" (should be in state "Installed", but is actually in state "**ERROR**")

            Deployment "org.hornetq:module=JMS,type=Queue,name="DLQ"" is missing the following dependencies:

              Dependency "JMSServerManager" (should be in state "Installed", but is actually in state "**ERROR**")

            Deployment "org.hornetq:module=JMS,type=Queue,name="ExpiryQueue"" is missing the following dependencies:

              Dependency "JMSServerManager" (should be in state "Installed", but is actually in state "**ERROR**")

            Deployment "org.hornetq:module=JMS,type=Queue,name="testqueue"" is missing the following dependencies:

              Dependency "JMSServerManager" (should be in state "Installed", but is actually in state "**ERROR**")

           

          DEPLOYMENTS IN ERROR:

            Deployment "JMSServerManager" is in error due to the following reason(s): HornetQException[errorCode=104 message=Connected server is not a backup server], **ERROR**

            Deployment "<UNKNOWN JMSServerMO>" is in error due to the following reason(s): ** UNRESOLVED Demands 'JMSServerManager' **

            Deployment "<UNKNOWN JMSConnectionFactoryManageMO>" is in error due to the following reason(s): ** UNRESOLVED Demands 'JMSServerManager' **

            Deployment "<UNKNOWN JMSTopicManageMO>" is in error due to the following reason(s): ** UNRESOLVED Demands 'JMSServerManager' **

            Deployment "<UNKNOWN JMSQueueManageMO>" is in error due to the following reason(s): ** UNRESOLVED Demands 'JMSServerManager' **

           

              at org.jboss.deployers.plugins.deployers.DeployersImpl.checkComplete(DeployersImpl.java:1228) [:2.2.0.GA]

              at org.jboss.deployers.plugins.main.MainDeployerImpl.checkComplete(MainDeployerImpl.java:905) [:2.2.0.GA]

              at org.jboss.system.server.profileservice.deployers.MainDeployerPlugin.checkComplete(MainDeployerPlugin.java:87) [:6.0.0.Final]

              at org.jboss.profileservice.deployment.ProfileDeployerPluginRegistry.checkAllComplete(ProfileDeployerPluginRegistry.java:107) [:0.2.2]

              at org.jboss.system.server.profileservice.bootstrap.BasicProfileServiceBootstrap.start(BasicProfileServiceBootstrap.java:135) [:6.0.0.Final]

              at org.jboss.system.server.profileservice.bootstrap.BasicProfileServiceBootstrap.start(BasicProfileServiceBootstrap.java:56) [:6.0.0.Final]

              at org.jboss.bootstrap.impl.base.server.AbstractServer.startBootstraps(AbstractServer.java:827) [jboss-bootstrap-impl-base.jar:2.1.0-alpha-5]

              at org.jboss.bootstrap.impl.base.server.AbstractServer$StartServerTask.run(AbstractServer.java:417) [jboss-bootstrap-impl-base.jar:2.1.0-alpha-5]

              at java.lang.Thread.run(Thread.java:619) [:1.6.0_21]

          • 2. Journal replication in jboss as6
            clebert.suconic

            We are releasing 2.2. in the next few days.. However data-replication will not be supported in a first moment.

             

            We have done a brand new failover using shared store with distributed file locks, and we have done a lot of testing on it.

             

            We will start working on the next release, which will be 2.3, which will basically be the addition of replication and other tweaks as usual.

             

             

            If you're still interested in pursue replication on 2.1, I can help you figuring out. The main issue now is the lack of the possibility of adding a node and copy the data as the backup is becoming live, which will be done as part of the design on 2.3.

            • 3. Journal replication in jboss as6
              rhinmass

              Does data-replication work in standalone HornetQ 2.1.2, just not when running in JBoss?

               

              Thank you for offering to help figure it out.  I think, for now, I could live with procedural work-arounds, such as making sure the backup is fully live before starting the master.  And manually copying the data before bringing them up.

               

              Do my errors above mean anything to you?  And do you know if the 2 nodes should be clustered or not?

               

              Thanks!!

               

              Robin

              • 4. Journal replication in jboss as6
                clebert.suconic

                You probably changed some dependencies on the AS.  You have to keep deployers and other things the way they are.

                • 5. Journal replication in jboss as6
                  rhinmass

                  I really haven't changed anything with dependencies. I copied the clustered and default configurations and made the changes described in the original post, as well as adding the queue "testqueue" to hornet-jms.xml.   If I remove the changes to hornet-configuration.xml, everything works fine.  (Or course no replication happens.) 

                   

                  Would you say that journal replication is *supposed* to work when running in jbossas 6?  I'm just trying to figure out whether to keep working at it, or just accept that it isn't there yet in the current version.

                   

                  And should the master and slave nodes be configured with the clustered ("all") configuration or the normal ("default") configuration?

                   

                  Thanks again for keeping with me.  I'm wondering if anyone has gotten this to work in jboss6 or am I in uncharted territory.

                  • 6. Journal replication in jboss as6
                    clebert.suconic

                    The problem is that if you use the AS a backup, you will miss several dependencies on startup. I would suggest you have the backup a standalong, at least for now.

                     

                    We have fixed that as part of the EAP integration, but we don't have replication yet. (that will enter into development very soon).

                     

                     

                    BTW: You need to keep clustered=true.

                    • 7. Journal replication in jboss as6
                      rhinmass

                      To make sure I am clear on your suggestion, for each jbossas6 node, I should have a standalone hornetq node.  The jboss nodes would be running in a cluster, and both would be configured as masters for replications purposes.  The slaves would be the corresponding standalone hornetq instances.

                       

                      I'll give it a try.  Thanks!