13 Replies Latest reply on Jul 18, 2011 6:03 AM by Andy Taylor

    server side load balancing issue

    nareshpa Newbie

      HI,

       

      I would like to have setup with two HornetQ Servers(server 0, server 1) in a cluster, without backup servers or replication - just simple configuration: when one server is dead client should not use it and simply use other still running servers from cluster.


      My application contains only address of the Server 0: java.naming.provider.url=jnp://192.168.32.70:1099


      When server 0 is dead, After some time my application gets following exception:


      • org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Timed out waiting for response when sending packet 71

            at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)

            at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:168)

            at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:469)

            at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:543)

            at com.advertise.benchmark.protocol.JMSSender.executeRequest(JMSSender.java:46)

            at com.advertise.benchmark.components.searchtracking.AdsSearchMessageExecutor.run(AdsSearchMessageExecutor.java:69)

            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

            at java.lang.Thread.run(Unknown Source)

      Caused by: javax.jms.JMSException: Timed out waiting for response when sending packet 71

            at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:276)

            at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:287)

            at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:142)

            at org.hornetq.jms.client.HornetQMessageProducer.doSend(HornetQMessageProducer.java:451)

            at org.hornetq.jms.client.HornetQMessageProducer.send(HornetQMessageProducer.java:210)

            at org.springframework.jms.connection.CachedMessageProducer.send(CachedMessageProducer.java:117)

            at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:592)

            at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:569)

            at org.springframework.jms.core.JmsTemplate$4.doInJms(JmsTemplate.java:546)

            at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:466)

            ... 6 more

      Caused by: HornetQException[errorCode=3 message=Timed out waiting for response when sending packet 71]

            ... 16 more

        • 1. Re: server side load balancing issue
          Andy Taylor Master

          I would expect that since you have no backup servers for the client to failover too

          • 2. Re: server side load balancing issue
            nareshpa Newbie

            in cluster I configured two servers (0, 1), if  server 0 is down, but server 1 is working properly in the same cluster why my application not able to connect to server1 and server side clustering only for messages distributing.

            • 3. Re: server side load balancing issue
              Andy Taylor Master

              for your client to failover, you need a  backup server

              • 4. Re: server side load balancing issue
                nareshpa Newbie

                can I use server 1 part of the cluster and as well as backup server

                 

                hornetq-2.2.2.Final is support to backup configuration.

                • 5. Re: server side load balancing issue
                  Andy Taylor Master

                  like i saidf before you need  a backup server for each live server you have on your cluster

                  • 6. Re: server side load balancing issue
                    nareshpa Newbie

                    Couple of day’s back, while configuring backup server configuration I got  exception from hornetq-2.2.2.Final

                     

                    following SAXParseException while configuring the "backup-connector-name" attribute in "connector-ref" tag.

                     

                    Please help me how to resolve this issue.

                     

                    [main] 15:32:35,946 INFO [org.hornetq.integration.bootstrap.HornetQBootstrapServer]  Starting HornetQ Server

                    [main] 15:32:38,197 SEVERE [org.hornetq.utils.XMLUtil]  Invalid configuration

                    org.xml.sax.SAXParseException: cvc-complex-type.2.4.a: Invalid content was found starting with element 'backup-connector-re

                    ed, "urn:hornetq":security-invalidation-interval, "urn:hornetq":wild-card-routing-enabled, "urn:hornetq":management-address

                    n, "urn:hornetq":message-counter-enabled, "urn:hornetq":message-counter-sample-period, "urn:hornetq":message-counter-max-da

                    eriod, "urn:hornetq":message-expiry-thread-priority, "urn:hornetq":id-cache-size, "urn:hornetq":persist-id-cache, "urn:horn

                    -delivery, "urn:hornetq":live-connector-ref, "urn:hornetq":connectors, "urn:hornetq":acceptors, "urn:hornetq":broadcast-gro

                    :bindings-directory, "urn:hornetq":create-bindings-dir, "urn:hornetq":journal-directory, "urn:hornetq":create-journal-dir,

                    ite-rate, "urn:hornetq":journal-file-size, "urn:hornetq":journal-min-files, "urn:hornetq":journal-compact-percentage, "urn:

                    netq":memory-measure-interval, "urn:hornetq":large-messages-directory, "urn:hornetq":security-settings, "urn:hornetq":addre

                            at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)

                            at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.error(Unknown Source)

                            at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(Unknown Source)

                            at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(Unknown Source)

                            at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator$XSIErrorReporter.reportError(Unknown Source)

                            at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.reportSchemaError(Unknown Source)

                            at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.handleStartElement(Unknown Source)

                            at com.sun.org.apache.xerces.internal.impl.xs.XMLSchemaValidator.startElement(Unknown Source)

                            at com.sun.org.apache.xerces.internal.jaxp.validation.DOMValidatorHelper.beginNode(Unknown Source)

                            at com.sun.org.apache.xerces.internal.jaxp.validation.DOMValidatorHelper.validate(Unknown Source)

                            at com.sun.org.apache.xerces.internal.jaxp.validation.DOMValidatorHelper.validate(Unknown Source)

                            at com.sun.org.apache.xerces.internal.jaxp.validation.ValidatorImpl.validate(Unknown Source)

                            at javax.xml.validation.Validator.validate(Unknown Source)

                            at org.hornetq.utils.XMLUtil.validate(XMLUtil.java:490)

                            at org.hornetq.core.deployers.impl.FileConfigurationParser.parseMainConfig(FileConfigurationParser.java:164)

                            at org.hornetq.core.config.impl.FileConfiguration.start(FileConfiguration.java:86)

                            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                            at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

                            at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

                            at java.lang.reflect.Method.invoke(Unknown Source)

                            at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:59)

                            at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:150)

                            at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)

                            at org.jboss.kernel.plugins.dependency.KernelControllerContextAction$JoinpointDispatchWrapper.execute(KernelControl

                            at org.jboss.kernel.plugins.dependency.ExecutionWrapper.execute(ExecutionWrapper.java:47)

                            at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchExecutionWrapper(KernelControllerConte

                            at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchJoinPoint(KernelControllerContextActio

                            at org.jboss.kernel.plugins.dependency.LifecycleAction.installActionInternal(LifecycleAction.java:221)

                            at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:54)

                            at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:42)

                            at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAct

                            at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)

                            at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)

                            at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

                            at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1631)

                            at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934)

                            at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

                            at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984)

                            at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:774)

                            at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:540)

                            at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.deployBean(AbstractKernelDeployer.java:319)

                            at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.deployBeans(AbstractKernelDeployer.java:297)

                            at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.deploy(AbstractKernelDeployer.java:130)

                            at org.jboss.kernel.plugins.deployment.xml.BeanXMLDeployer.deploy(BeanXMLDeployer.java:96)

                            at org.hornetq.integration.bootstrap.HornetQBootstrapServer.deploy(HornetQBootstrapServer.java:236)

                            at org.hornetq.integration.bootstrap.HornetQBootstrapServer.deploy(HornetQBootstrapServer.java:206)

                            at org.hornetq.integration.bootstrap.HornetQBootstrapServer.bootstrap(HornetQBootstrapServer.java:155)

                            at org.jboss.kernel.plugins.bootstrap.AbstractBootstrap.run(AbstractBootstrap.java:83)

                            at org.hornetq.integration.bootstrap.HornetQBootstrapServer.run(HornetQBootstrapServer.java:116)

                            at org.hornetq.integration.bootstrap.HornetQBootstrapServer.main(HornetQBootstrapServer.java:73)

                    [main] 15:32:38,369 SEVERE [org.hornetq.integration.bootstrap.HornetQBootstrapServer]  Failed to start server

                    java.lang.IllegalStateException: Incompletely deployed:

                     

                    DEPLOYMENTS IN ERROR:

                      Deployment "Configuration" is in error due to: java.lang.IllegalStateException: Invalid configuration

                     

                    DEPLOYMENTS MISSING DEPENDENCIES:

                      Deployment "JMSServerManager" is missing the following dependencies:

                        Dependency "HornetQServer" (should be in state "Installed", but is actually in state "Described")

                      Deployment "HornetQServer" is missing the following dependencies:

                        Dependency "Configuration" (should be in state "Installed", but is actually in state "**ERROR**")

                     

                            at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.internalValidate(AbstractKernelDeployer.java:278)

                            at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.validate(AbstractKernelDeployer.java:174)

                            at org.hornetq.integration.bootstrap.HornetQBootstrapServer.bootstrap(HornetQBootstrapServer.java:158)

                            at org.jboss.kernel.plugins.bootstrap.AbstractBootstrap.run(AbstractBootstrap.java:83)

                            at org.hornetq.integration.bootstrap.HornetQBootstrapServer.run(HornetQBootstrapServer.java:116)

                            at org.hornetq.integration.bootstrap.HornetQBootstrapServer.main(HornetQBootstrapServer.java:73)

                    Exception in thread "main" java.lang.IllegalStateException: Incompletely deployed:

                     

                    DEPLOYMENTS IN ERROR:

                      Deployment "Configuration" is in error due to: java.lang.IllegalStateException: Invalid configuration

                     

                    DEPLOYMENTS MISSING DEPENDENCIES:

                      Deployment "JMSServerManager" is missing the following dependencies:

                        Dependency "HornetQServer" (should be in state "Installed", but is actually in state "Described")

                      Deployment "HornetQServer" is missing the following dependencies:

                        Dependency "Configuration" (should be in state "Installed", but is actually in state "**ERROR**")

                     

                            at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.internalValidate(AbstractKernelDeployer.java:278)

                            at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.validate(AbstractKernelDeployer.java:174)

                            at org.hornetq.integration.bootstrap.HornetQBootstrapServer.bootstrap(HornetQBootstrapServer.java:158)

                            at org.jboss.kernel.plugins.bootstrap.AbstractBootstrap.run(AbstractBootstrap.java:83)

                            at org.hornetq.integration.bootstrap.HornetQBootstrapServer.run(HornetQBootstrapServer.java:116)

                            at org.hornetq.integration.bootstrap.HornetQBootstrapServer.main(HornetQBootstrapServer.java:73)

                    • 7. Re: server side load balancing issue
                      Andy Taylor Master

                      start by looking at the clustered examples and go from there, its obvious your config is wrong

                      • 8. Re: server side load balancing issue
                        nareshpa Newbie

                        the following file is my hornetq-configuration.xml

                         

                         

                        <configuration xmlns="urn:hornetq"

                                       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                                       xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">

                         

                           <clustered>true</clustered>

                           <shared-store>false</shared-store>

                         

                           <paging-directory>${data.dir:../data}/paging</paging-directory>

                         

                           <bindings-directory>${data.dir:../data}/bindings</bindings-directory>

                         

                           <journal-directory>${data.dir:../data}/journal</journal-directory>

                         

                           <journal-min-files>10</journal-min-files>

                         

                           <large-messages-directory>${data.dir:../data}/large-messages</large-messages-directory>

                         

                           <connectors>     

                              <connector name="netty">

                                 <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

                                 <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>

                                 <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>

                              </connector>

                         

                              <connector name="netty-throughput">

                                 <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

                                 <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>

                                 <param key="port"  value="${hornetq.remoting.netty.batch.port:5455}"/>

                                 <param key="batch-delay" value="50"/>

                              </connector>

                         

                           <connector name="backup-connector">

                                 <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

                                  <param key="host"  value="${hornetq.remoting.netty.host:localhost1"/>

                                 <param key="port" value="5446"/>

                               </connector>

                           </connectors>

                         

                           <acceptors>

                              <acceptor name="netty">

                                 <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>

                                 <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>

                                 <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>

                              </acceptor>

                         

                              <acceptor name="netty-throughput">

                                 <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>

                                 <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>

                                 <param key="port"  value="${hornetq.remoting.netty.batch.port:5455}"/>

                                 <param key="batch-delay" value="50"/>

                                 <param key="direct-deliver" value="false"/>

                              </acceptor>

                           </acceptors>

                         

                           <broadcast-groups>

                              <broadcast-group name="bg-group1">

                                 <group-address>231.7.7.7</group-address>

                                 <group-port>9876</group-port>

                                 <broadcast-period>5000</broadcast-period>

                               <connector-ref connector-name="netty" backup-connector-name="backup-connector"/>

                               </broadcast-group>

                           </broadcast-groups>

                         

                           <discovery-groups>

                              <discovery-group name="dg-group1">

                                 <group-address>231.7.7.7</group-address>

                                 <group-port>9876</group-port>

                                 <refresh-timeout>10000</refresh-timeout>

                              </discovery-group>

                           </discovery-groups>

                         

                           <cluster-connections>

                              <cluster-connection name="my-cluster">

                                 <address>jms</address>    

                                 <connector-ref connector-name="netty" backup-connector-name="backup-connector"/>

                                  <discovery-group-ref discovery-group-name="dg-group1"/>

                              </cluster-connection>

                           </cluster-connections>

                         

                           <security-settings>

                              <security-setting match="#">

                                 <permission type="createNonDurableQueue" roles="guest"/>

                                 <permission type="deleteNonDurableQueue" roles="guest"/>

                                 <permission type="consume" roles="guest"/>

                                 <permission type="send" roles="guest"/>

                              </security-setting>

                           </security-settings>

                         

                           <address-settings>

                              <!--default for catch all-->

                              <address-setting match="#">

                                 <dead-letter-address>jms.queue.DLQ</dead-letter-address>

                                 <expiry-address>jms.queue.ExpiryQueue</expiry-address>

                                 <redelivery-delay>0</redelivery-delay>

                                 <max-size-bytes>10485760</max-size-bytes>      

                                 <message-counter-history-day-limit>10</message-counter-history-day-limit>

                                 <address-full-policy>BLOCK</address-full-policy>

                              </address-setting>

                           </address-settings>

                         

                         

                         

                        </configuration>

                        • 9. Re: server side load balancing issue
                          Andy Taylor Master

                          like i said, your config is wrong, take a look at the examples shipped

                          • 10. Re: server side load balancing issue
                            nareshpa Newbie

                            can I get the connection using jndi properties from discovery-group IP and Port

                             

                             

                            I am using following discovery-group

                             

                               <discovery-groups>

                                  <discovery-group name="dg-group1">

                                     <group-address>231.7.7.7</group-address>

                                     <group-port>9876</group-port>

                                     <refresh-timeout>10000</refresh-timeout>

                                  </discovery-group>

                               </discovery-groups>

                            • 11. Re: server side load balancing issue
                              Andy Taylor Master

                              yes, via a connection factory, all this is explained the user manual

                              • 12. Re: server side load balancing issue
                                nareshpa Newbie

                                I am using following provider to specifying server ip & port, same like that any provider is available to specify group-address & port

                                 

                                java.naming.provider.url=jnp://localhost:3099

                                • 13. Re: server side load balancing issue
                                  Andy Taylor Master

                                  That is jndi and has nothing at all to do with HornetQ, we just use the naming service to rtegister JMS resources.