1 2 Previous Next 23 Replies Latest reply on Apr 28, 2014 8:23 AM by mlange

    Separate netty connections for producer and consumer?

    mlange

      I am trying to use different acceptors for producer and consumer clients. Defined three acceptors in domain.xml (one for producers, one for consumers one for clustering communication):

       

      <acceptors>

        <!-- producer -->

        <netty-acceptor name="netty-producer-acceptor" socket-binding="messaging-producer-acceptor">

        <param key="use-nio" value="true"/>

        <param key="batch-delay" value="50"/>

        <param key="direct-deliver" value="false"/>

      </netty-acceptor>

      <netty-acceptor name="netty-consumer-acceptor" socket-binding="messaging-consumer-acceptor">

         <!-- consumer -->

         <param key="use-nio" value="true"/>

         <param key="batch-delay" value="50"/>

         <param key="direct-deliver" value="false"/>

      </netty-acceptor>

      <netty-acceptor name="netty-cluster-acceptor" socket-binding="messaging-internal-acceptor">

        <!-- for clustering -->

        <param key="use-nio" value="true"/>

        <param key="batch-delay" value="50"/>

        <param key="direct-deliver" value="false"/>

        </netty-acceptor>

       

      <!-- referenced in cluster connection-->

      <connectors>

        <netty-connector name="netty-internal-connector" socket-binding="messaging-internal-acceptor"/>

      </connectors>

       

      All of them have a different socket binding and are thus listening on another port.

      <socket-binding name="messaging-producer-acceptor" port="11830"/>

      <socket-binding name="messaging-consumer-acceptor" port="11831"/>

      <socket-binding name="messaging-internal-acceptor" port="11832"/>

       

      On the client side different connection factories are used for the producer and for the consumer:

      <tx-connection-factory>

          <jndi-name>HornetqProducerPooledConnectionFactory</jndi-name>

          <config-property name="ConnectionParameters" type="java.lang.String">host=host1;port=11830</config-property>

       

      <tx-connection-factory>

          <jndi-name>HornetqProducerPooledConnectionFactory</jndi-name>

          <config-property name="ConnectionParameters" type="java.lang.String">host=host1;port=11831</config-property>

       

      What I am seeing is that the client has one initial connection to the first acceptor (11830) and the other connections lead to the third acceptor (port 11832), although it has port 11830 defined in its ConnectionParameters.

       

      Do you have any idea why this happens and how is it possible to achieve a clear separattion of producer and consumer traffic?

       

      Thanks!

        • 1. Re: Separate netty connections for producer and consumer?
          ataylor

          a couple of things:

           

          1.

          Do you have any idea why this happens and how is it possible to achieve a clear separattion of producer and consumer traffic?

          what is it you want separating, just connections, you dont need different acceptors for this.

          2. i notice you are using a pooled connection factory, how are you configuring your clients (and what are they)

          3. what is tx-connection-factory?

          • 2. Re: Separate netty connections for producer and consumer?
            mlange

             

             



            2. i notice you are using a pooled connection factory, how are you configuring your clients (and what are they)

            3. what is tx-connection-factory?

            Hi Andy!

             

            >1. what is it you want separating, just connections, you dont need different acceptors for this.

             

            I want to achieve that the producer traffic can be stopped independently from the consumer traffic (using the org.hornetq:module=Core,type=Acceptor,name="netty-producer-acceptor" mbean). Do you have another option how this could be done?

             

            >2. i notice you are using a pooled connection factory, how are you configuring your clients (and what are they)

             

            These are pure JMS clients looking up the pooled factory using JNDI. They are configured in *-ds.xml files (client is JBoss4), this is for the producer:

             

            <connection-factories>

              <tx-connection-factory>

                <jndi-name>HornetqProducerPooledConnectionFactory</jndi-name>

                <xa-transaction/>

                <rar-name>hornetq-ra.rar</rar-name>

                <connection-definition>org.hornetq.ra.HornetQRAConnectionFactory</connection-definition>

                <config-property name="ConnectionParameters" type="java.lang.String">host=host1;port=11830,</config-property>

                <config-property name="ConnectorClassName" type="java.lang.String">org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</config-property>

                <min-pool-size>0</min-pool-size>

                <max-pool-size>10000</max-pool-size>

                <blocking-timeout-millis>30000</blocking-timeout-millis>

                <idle-timeout-minutes>99999999</idle-timeout-minutes>

                <track-connection-by-tx>true</track-connection-by-tx>

              </tx-connection-factory>

            </connection-factories>

             

            > 3. what is tx-connection-factory?

             

            See the snippet above - it is from the datasource configuration for JCA.

             

            Thanks!

            • 3. Re: Separate netty connections for producer and consumer?
              ataylor

              I want to achieve that the producer traffic can be stopped independently from the consumer traffic (using the org.hornetq:module=Core,type=Acceptor,name="netty-producer-acceptor" mbean). Do you have another option how this could be done?

              seems a bit heavy handed? closeConnectionsForAddress on HornetQServerControl might help

              These are pure JMS clients looking up the pooled factory using JNDI. They are configured in *-ds.xml files (client is JBoss4), this is for the producer:

              pooled connectionfactories are local only, i.e. MDB's, servlets etc and managed by the AS

              • 4. Re: Separate netty connections for producer and consumer?
                ataylor

                actually ignore my comment on pooled connection factories, i mis read your config. why it is connecting to a different port i have no idea, how is your Resource adapter configured?

                • 5. Re: Separate netty connections for producer and consumer?
                  mlange
                  seems a bit heavy handed? closeConnectionsForAddress on HornetQServerControl might help

                  Yes, but the problem is that the consumers should still stay active. Closing the connections for the address would disable both producers and consumers. What did you mean by "you don't need different acceptors"? How can producers and consumers use different ports then? Or is there another way to realize this?

                  • 6. Re: Separate netty connections for producer and consumer?
                    ataylor

                    Yes, but the problem is that the consumers should still stay active. Closing the connections for the address would disable both producers and consumers. What did you mean by "you don't need different acceptors"? How can producers and consumers use different ports then? Or is there another way to realize this?

                    The consumers will react the same whether you stop the acceptor or close the connection, the fact that they are connected to different port makes no difference.

                    • 7. Re: Separate netty connections for producer and consumer?
                      mlange

                      actually ignore my comment on pooled connection factories, i mis read your config. why it is connecting to a different port i have no idea, how is your Resource adapter configured?

                      ra.xml:

                       

                      <resourceadapter>

                            <resourceadapter-class>org.hornetq.ra.HornetQResourceAdapter</resourceadapter-class>

                            <config-property>

                               <description>

                                  The transport type. Multiple connectors can be configured by using a comma separated list,

                                  i.e. org.hornetq.core.remoting.impl.invm.InVMConnectorFactory,org.hornetq.core.remoting.impl.invm.InVMConnectorFactory.

                               </description>

                               <config-property-name>ConnectorClassName</config-property-name>

                               <config-property-type>java.lang.String</config-property-type>

                               <config-property-value>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</config-property-value>

                            </config-property>

                            <config-property>

                               <description>The transport configuration. These values must be in the form of key=val;key=val;,

                                  if multiple connectors are used then each set must be separated by a comma i.e. host=host1;port=5445,host=host2;port=5446.

                                  Each set of params maps to the connector classname specified.

                               </description>

                               <config-property-name>ConnectionParameters</config-property-name>

                               <config-property-type>java.lang.String</config-property-type>

                               <config-property-value>host=host1;port=11830</config-property-value>

                            </config-property>

                            [...]

                            <outbound-resourceadapter>

                               <connection-definition>

                                  <managedconnectionfactory-class>org.hornetq.ra.HornetQRAManagedConnectionFactory</managedconnectionfactory-class>

                       

                                  <config-property>

                                     <description>The default session type</description>

                                     <config-property-name>SessionDefaultType</config-property-name>

                                     <config-property-type>java.lang.String</config-property-type>

                                     <config-property-value>javax.jms.Queue</config-property-value>

                                  </config-property>

                                  <config-property>

                                     <description>Try to obtain a lock within specified number of seconds; less than or equal to 0 disable this functionality</description>

                                     <config-property-name>UseTryLock</config-property-name>

                                     <config-property-type>java.lang.Integer</config-property-type>

                                     <config-property-value>0</config-property-value>

                                  </config-property>

                       

                                  <connectionfactory-interface>org.hornetq.ra.HornetQRAConnectionFactory</connectionfactory-interface>

                                  <connectionfactory-impl-class>org.hornetq.ra.HornetQRAConnectionFactoryImpl</connectionfactory-impl-class>

                                  <connection-interface>javax.jms.Session</connection-interface>

                                  <connection-impl-class>org.hornetq.ra.HornetQRASession</connection-impl-class>

                               </connection-definition>

                               <transaction-support>XATransaction</transaction-support>

                               <authentication-mechanism>

                                  <authentication-mechanism-type>BasicPassword</authentication-mechanism-type>

                                  <credential-interface>javax.resource.spi.security.PasswordCredential</credential-interface>

                               </authentication-mechanism>

                               <reauthentication-support>false</reauthentication-support>

                            </outbound-resourceadapter>

                       

                            <inbound-resourceadapter>

                               <messageadapter>

                                  <messagelistener>

                                     <messagelistener-type>javax.jms.MessageListener</messagelistener-type>

                                     <activationspec>

                                        <activationspec-class>org.hornetq.ra.inflow.HornetQActivationSpec</activationspec-class>

                                        <required-config-property>

                                            <config-property-name>destination</config-property-name>

                                        </required-config-property>

                                     </activationspec>

                                  </messagelistener>

                               </messageadapter>

                            </inbound-resourceadapter>

                       

                         </resourceadapter>

                      • 8. Re: Separate netty connections for producer and consumer?
                        mlange
                        The consumers will react the same whether you stop the acceptor or close the connection, the fact that they are connected to different port makes no difference.

                        Not sure if I get your point. When the acceptors are using different ports and if the clients make sure that the producer and consumer is using different ports then it should be possible to stop one acceptor while the other continues to run. Or what am I missing?

                        • 9. Re: Separate netty connections for producer and consumer?
                          ataylor

                          Not sure if I get your point. When the acceptors are using different ports and if the clients make sure that the producer and consumer is using different ports then it should be possible to stop one acceptor while the other continues to run. Or what am I missing?

                          Yes, but the client connected to the acceptor will react accordingly, i.e. disconnect depending on your reconnect config, and you said you wanted the client to stay connected. my point is you dont have to define an acceptor to listen on a port for every connection you have. you have 1 acceptor and simply close the connection that you want via the method I explained.

                           

                          to be honest, if you need some sort of flow control i would either use our inbuilt flow control or build it into your app, saying that i dont really know what your trying to achieve.

                          • 10. Re: Separate netty connections for producer and consumer?
                            mlange

                            Yes, but the client connected to the acceptor will react accordingly, i.e. disconnect depending on your reconnect config, and you said you wanted the client to stay connected. my point is you dont have to define an acceptor to listen on a port for every connection you have. you have 1 acceptor and simply close the connection that you want via the method I explained.

                             

                            to be honest, if you need some sort of flow control i would either use our inbuilt flow control or build it into your app, saying that i dont really know what your trying to achieve.

                             

                            We are looking for a way to temporarily stop the incoming messages on a node while the consumers are still receiving from this node (for avoiding an OOM due to heavy paging). My assumption was that by separating the connections on different acceptors this could be achieved by simply stopping the acceptor which takes the connections from the clients sending the messages. Afterwards the consumers can empty the paging store, the instance goes out of paging mode and the acceptor can be started again to receive messages. I have no other idea how to realize this and the possibility to define different acceptors looked like a suitable option.

                            • 11. Re: Separate netty connections for producer and consumer?
                              ataylor

                              We are looking for a way to temporarily stop the incoming messages on a node while the consumers are still receiving from this node (for avoiding an OOM due to heavy paging). My assumption was that by separating the connections on different acceptors this could be achieved by simply stopping the acceptor which takes the connections from the clients sending the messages. Afterwards the consumers can empty the paging store, the instance goes out of paging mode and the acceptor can be started again to receive messages. I have no other idea how to realize this and the possibility to define different acceptors looked like a suitable option.

                              having one acceptor per connection and stopping it is probably the wrong approach, an acceptor has resources that it creates such as thread pools and other netty stuff that is really meant to be used by many clients.

                               

                              saying that, avoiding OOM on the server is exactly what flow control is for, you can either configure an address to PAGE rather than use memory or you can configure the producer to BLOCK. you do this via the address settings

                              • 12. Re: Separate netty connections for producer and consumer?
                                mlange
                                saying that, avoiding OOM on the server is exactly what flow control is for, you can either configure an address to PAGE rather than use memory or you can configure the producer to BLOCK. you do this via the address settings

                                I am aware of these concepts. But they are not suitable to be used dynamically. So it is not possible to introduce a producer throttling on demand (e.g. when a coming OOM is detected in operations). Blocking the producers is no option because there is always the question how these clients could handle this situation. What is the use case to define different netty acceptors then?

                                • 13. Re: Separate netty connections for producer and consumer?
                                  ataylor

                                  I am aware of these concepts. But they are not suitable to be used dynamically. So it is not possible to introduce a producer throttling on demand (e.g. when a coming OOM is detected in operations)

                                  I dont see why you need it on demand, you simply tune your server so an OOM never occurs. Im curious to know how you detect an OOM before it happens?

                                   

                                  What is the use case to define different netty acceptors then?

                                  before 2.4 when protocols werent detected automatically it was so you could define different ports for different protocols. The other use is if you are using different ip addresses/ports internally to externally.

                                  • 14. Re: Separate netty connections for producer and consumer?
                                    mlange

                                    > I dont see why you need it on demand, you simply tune your server so an OOM never occurs. Im curious to know how you detect an OOM before it happens?

                                     

                                    A long story :-) see Memory problems in paging mode makes server unresponsive

                                     

                                    We still have no option to come out of paging mode unless to stop the producers temporarily (what is no option in a production system).

                                    1 2 Previous Next