8 Replies Latest reply on Apr 18, 2017 9:38 AM by jbertram

    Wildfly 10 aretmis activemq seems to be throttling the producer

    dkjeldgaard

      I am running into an issue where activemq in wildfly 10 seems to run well for some time. After several hours our producers seem to be slowed down artificially like they are being throttled by artemis activemq. We have several different java clients that run in sequence and produce messages that are sent to various queues. The first time we run them they seem to run quickly and load up the queues quickly, which is the desired behavior. If we run the same programs several hours later it takes way too long for the programs (producers) to send the messages to the queues. In this scenario the consumers are processing the messages way faster than the consumer is producing them so we have consumers that are starving. Our goal is for our producer to send an unlimited number of messages to the queue (limited only by physical disk space for paging). This way our consumers (MDBs) can consume them as quickly as they possibly can.

       

      It seems as though artemis activemq is throttling the message producer somehow. Any help with this would be greatly appreciated. Here is our messaging configuration in wildfly 10.

       

                  <subsystem xmlns="urn:jboss:domain:messaging-activemq:1.0">
                      <server name="default">
                          <security enabled="false"/>
                          <cluster password="${jboss.messaging.cluster.password:defaultPWD}"/>
                          <security-setting name="#">
                              <role name="guest" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>
                              <role name="bpc" send="true" consume="true" create-non-durable-queue="true" delete-non-durable-queue="true"/>
                          </security-setting>
                          <address-setting name="#" dead-letter-address="jms.queue.DLQ" expiry-address="jms.queue.ExpiryQueue" redelivery-delay="5000" max-delivery-attempts="5"/>
                          <address-setting name="jms.queue.BPCLoggingQueue" dead-letter-address="jms.queue.DLQ" expiry-address="jms.queue.ExpiryQueue" max-delivery-attempts="1"/>
                          <address-setting name="jms.queue.PASNettingProcessorQueue" max-delivery-attempts="1"/>
                          <http-connector name="http-connector" socket-binding="http" endpoint="http-acceptor"/>
                          <http-connector name="http-connector-throughput" socket-binding="http" endpoint="http-acceptor-throughput">
                              <param name="batch-delay" value="50"/>
                          </http-connector>
                          <in-vm-connector name="in-vm" server-id="0"/>
                          <http-acceptor name="http-acceptor" http-listener="default"/>
                          <http-acceptor name="http-acceptor-throughput" http-listener="default">
                              <param name="batch-delay" value="50"/>
                              <param name="direct-deliver" value="false"/>
                          </http-acceptor>
                          <in-vm-acceptor name="in-vm" server-id="0"/>
                          <broadcast-group name="bg-group1" jgroups-channel="activemq-cluster" connectors="http-connector"/>
                          <discovery-group name="dg-group1" jgroups-channel="activemq-cluster"/>
                          <cluster-connection name="my-cluster" address="jms" connector-name="http-connector" discovery-group="dg-group1"/>
                          <jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>
                          <jms-queue name="AMSBatchPosterQueue" entries="java:jboss/exported/jms/queue/AMSBatchPosterQueue" durable="false"/>
                          <jms-queue name="AMSIntradayPosterQueue" entries="java:jboss/exported/jms/queue/AMSIntradayPosterQueue" durable="false"/>
                          <jms-queue name="PASBatchActivityPosterQueue" entries="java:jboss/exported/jms/queue/PASBatchActivityPosterQueue" durable="false"/>
                          <jms-queue name="PASIntradayActivityPosterQueue" entries="java:jboss/exported/jms/queue/PASIntradayActivityPosterQueue" durable="false"/>
                          <jms-queue name="PASPositionReconProcessorQueue" entries="java:jboss/exported/jms/queue/PASPositionReconProcessorQueue" durable="false"/>
                          <jms-queue name="DLQ" entries="java:/jboss/exported/jms/queue/DLQ" durable="false"/>
                          <jms-queue name="AMSAccountDataPosterQueue" entries="java:jboss/exported/jms/queue/AMSAccountDataPosterQueue" durable="false"/>
                          <jms-queue name="PASAccountDataPosterQueue" entries="java:jboss/exported/jms/queue/PASAccountDataPosterQueue" durable="false"/>
                          <jms-queue name="PASSATBProcessorQueue" entries="java:jboss/exported/jms/queue/PASSATBProcessorQueue" durable="false"/>
                          <jms-queue name="PASSecurityDataPosterQueue" entries="java:jboss/exported/jms/queue/PASSecurityDataPosterQueue" durable="false"/>
                          <jms-queue name="BPCLoggingQueue" entries="java:jboss/exported/jms/queue/BPCLoggingQueue" durable="false"/>
                          <jms-queue name="PASExternalSublotImportQueue" entries="java:jboss/exported/jms/queue/PASExternalSublotImportQueue" durable="false"/>
                          <jms-queue name="PASLotDeltaProcessorQueue" entries="java:jboss/exported/jms/queue/PASLotDeltaProcessorQueue" durable="false"/>
                          <jms-queue name="PASLotReconProcessorQueue" entries="java:jboss/exported/jms/queue/PASLotReconProcessorQueue" durable="false"/>
                          <jms-queue name="PortfolioDataPosterQueue" entries="java:jboss/exported/jms/queue/PortfolioDataPosterQueue" durable="false"/>
                          <jms-queue name="PASBaselineProcessorQueue" entries="java:jboss/exported/jms/queue/PASBaselineProcessorQueue" durable="false"/>
                          <jms-queue name="PASSerialNettingAccountProcessorQueue" entries="java:jboss/exported/jms/queue/PASSerialNettingAccountProcessorQueue" durable="false"/>
                          <jms-queue name="PASNettingProcessorQueue" entries="java:jboss/exported/jms/queue/PASNettingProcessorQueue" durable="false"/>
                          <connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>
                          <connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector" ha="true" min-large-message-size="1024" block-on-acknowledge="true" reconnect-attempts="-1"/>
                          <pooled-connection-factory name="activemq-ra" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" connectors="in-vm" transaction="xa"/>
                      </server>
                  </subsystem>
      
        • 1. Re: Wildfly 10 aretmis activemq seems to be throttling the producer
          jbertram

          Have a test-case which reproduces the behavior you're seeing?

          • 2. Re: Wildfly 10 aretmis activemq seems to be throttling the producer
            dkjeldgaard

            I'm not sure I could create a test case to produce the same behavior without providing all of our proprietary code or spending an inordinate amount of time writing dummy MDBs. Basically we run a java program (producer) that connects via the RemoteConnectionFactory and sends about 100,000 messages to the PASLotDeltaProcessorQueue queue. Then we wait for all the messages to be processed. Then another java program(producer) sends a similar number of messages to the PASNettingProcessorQueue and then we wait for all those to process before running the next java program(producer). Eventually we run the same programs over again and at some point the producer slows down tremendously. It slows it down way too slow. The consumers are starving. It is as if artemis mq is throttling the producer.

             

            Also, I have seen some documentation stating that activemq can throttle producers in order to prevent the server from being overloaded with messages or when there is a slow consumer. We don't want any producer throttling in our application.

             

            I was hoping someone would be able to see my configuration and be able to tell me what to change to stop artemis activemq from ever throttling the producer (if in fact that is what it is doing), or that someone would be able to tell me some other configuration change that would allow artemis to accept messages from producers as fast as possible without regard for overloading the server.

             

            Any help or suggestions to try would be greatly appreciated.

            • 3. Re: Wildfly 10 aretmis activemq seems to be throttling the producer
              jbertram

              Basically we run a java program (producer) that connects via the RemoteConnectionFactory and sends about 100,000 messages to the PASLotDeltaProcessorQueue queue. Then we wait for all the messages to be processed. Then another java program(producer) sends a similar number of messages to the PASNettingProcessorQueue and then we wait for all those to process before running the next java program(producer). Eventually we run the same programs over again...

              It sounds to me like that' would be pretty easy to replicate (at least in broad terms) with a simple producer and consumer.  I don't think there would be any need to reproduce all the MDBs and their logic, etc.  You're really just concerned with the sending portion.  You might not even need a consumer at all.  You could just trying deleting all the messages from the queue(s) to simulate the consumer fetching all the messages.  Of course, that assumes your consumers are, in fact, consuming all of the messages.

               

              Also, I have seen some documentation stating that activemq can throttle producers in order to prevent the server from being overloaded with messages or when there is a slow consumer. We don't want any producer throttling in our application.

              If you've read the documentation then I assume you've seen the configuration options necessary to disable producer flow-control.  Is that correct?  If so, have you disabled producer flow-control and retested?  If not, why not?

               

              I was hoping someone would be able to see my configuration and be able to tell me what to change to stop artemis activemq from ever throttling the producer (if in fact that is what it is doing), or that someone would be able to tell me some other configuration change that would allow artemis to accept messages from producers as fast as possible without regard for overloading the server.

              As mentioned previously, if you've read the documentation then you should already know how to disable producer flow-control.  That said, you've offered no evidence that flow-control is actually the culprit here.  Do you have thread-dumps from the client during the slow-down which might help identify what the code is actually doing at the time?  Do you have any statistics from the server at the time of the slow-down (e.g. message-count, delivering-count, consumer-count, etc. for the queue(s) in question)?  Is the broker paging?  For what it's worth, if the broker enters into paging mode producers sending durable messages will see a fairly significant slow-down.

               

              It's not really possible to provide conclusive help given the information you've provided so far.  You've simply described the problem in qualitative terms and provided your server configuration.

              • 4. Re: Wildfly 10 aretmis activemq seems to be throttling the producer
                dkjeldgaard

                Thanks Justin.

                 

                I think disabling flow-control is what we need to do but I don't know how to do it. I can't find any setting in the wildfly model reference that tells how to disable flow-control. Any instructions on how to do this would be helpful.

                 

                Thanks again.

                • 5. Re: Wildfly 10 aretmis activemq seems to be throttling the producer
                  dkjeldgaard

                  To answer your question about the queue's state when the slowdown happens and the queue state. It happens right away when the producer java program starts up. There are now messages in the queue and the queues never have more than a few messages in them at any given time. The producer sends a few messages and the consumers consumer them and then starve until the producer sends a few more. The consumers are starving most of the time. The producers should be able to flood the queues but are being artificialy slowed down by artemis for some reason (and they do flood the queues during the first few hours while wildfly is running). The consumers are starving most of the time because the producer is being artificially slowed down. This sounds like flow control problem to me. We just want to turn it off but don't know how.

                  • 6. Re: Wildfly 10 aretmis activemq seems to be throttling the producer
                    jbertram

                    You can change the producer-windows-size on the RemoteConnectionFactory a couple of different ways.  You can modify the XML like so:

                     

                      <connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector" producer-window-size="-1"/>

                     

                    Or you can execute this command from the JBoss CLI:

                     

                      /subsystem=messaging-activemq/server=default/connection-factory=RemoteConnectionFactory:write-attribute(name=producer-window-size, value=-1)

                     

                    For future reference, if you are curious about settings you can either look at the XSD for the messaging subsystem (i.e. <WILDFLY_HOME>/docs/schema/wildfly-messaging-activemq_1_0.xsd) or you can just browse around the CLI using tab-completion.

                     

                    I'm curious to know if this works for you because based on your description I wouldn't expect flow-control to be the issue.

                    • 7. Re: Wildfly 10 aretmis activemq seems to be throttling the producer
                      dkjeldgaard

                      Thanks for your suggestion. We didn't find anywhere in the documentation that said we could set producer-window-size to -1. We did see this mentioned for the consumer-window-size however. We wouldn't have known to try this without your input. I will let you know how it goes.

                      • 8. Re: Wildfly 10 aretmis activemq seems to be throttling the producer
                        jbertram

                        Did you get this sorted?