7 Replies Latest reply on Feb 10, 2015 9:55 AM by jbertram

    Blocked JMS Topic

    roman.pawliczek

      Hello,

       

      we are using WildFly 8.2 an have a problem with a JMS Topic. Couple of hours after deployment of the application the message count of one topic is getting very hight and the consumers are waiting to long for messages. Is there a configuration parameter which increases the throughput of messages. Strange to say there is another topic with similar amount of produced messages where the count is always 0 and subscribers are receiving new messages immediately.

       

      jms-topic.jpeg

        • 1. Re: Blocked JMS Topic
          jbertram

          There no single parameter that will increase message throughput regardless of use-case.  In order to know how to increase throughput you'll first need to know why throughput is less than expected.  The first place to look to investigate why throughput is less than expected is the consumers themselves. 

           

          You said, "the consumers are waiting to long for messages."  Can you clarify this a bit?  Are the consumers idle waiting for messages to arrive or are they busy consuming messages albeit more slowly than desired?  The screenshot you posted indicates that there are 257 messages currently being delivered (i.e. they have been sent to a client but have not yet been acknowledged) which leads me to believe your consumers may be consuming messages too slowly to keep up with the rate of messages arriving on the server.

           

          A few other questions:

          • Are the subscriptions durable or non-durable?
          • If the subscriptions are durable are the clients connected most of the time so that they actively consuming messages rather than letting them pile up in the server-side subscription?
          • Are the messages durable or non-durable?
          • How large are the messages?
          • What's the network like between these clients and the server?
          • 2. Re: Re: Blocked JMS Topic
            roman.pawliczek

            Subscriptions and messages are non-durable. We migrated our application from JBoss 4.2.3 to WildFly 8.2, since then we have the problems. Message amount, message size and network is still the same. Maybe there is another hint to the problem. In the log there are plenty of hornetq warnings:

             

            WARN  [org.hornetq.core.client] (Thread-2102 (HornetQ-client-global-threads-1899849122)) HQ212054: Destination address=jms.topic.acd-applet-info is blocked. If the system is configured to block make sure you consume messages on this configuration.

             

            How can the topic be blocked having address-full-policy set to PAGE in standalone-full.xml?

                            <address-settings>
                                <address-setting match="#">
                                    <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                                    <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                                    <redelivery-delay>0</redelivery-delay>
                                    <max-size-bytes>10485760</max-size-bytes>
                                    <page-size-bytes>2097152</page-size-bytes>
                                    <address-full-policy>PAGE</address-full-policy>
                                    <message-counter-history-day-limit>10</message-counter-history-day-limit>
                                </address-setting>
                            </address-settings>
            
            • 3. Re: Re: Blocked JMS Topic
              jbertram

              How can the topic be blocked having address-full-policy set to PAGE in standalone-full.xml?

              I'm not sure.  Can you provide a test-case which causes a queue to block even though the address-full-policy is set to PAGE?

              • 4. Re: Blocked JMS Topic
                roman.pawliczek

                That's the problem. We are trying to generate a testcase which causes the topic to block. But with no success. The problem occurs only in production.

                 

                It is a topic which gets eventually blocked, not a queue. Applies the address-full-policy also to topics?

                • 5. Re: Blocked JMS Topic
                  jbertram

                  The address-full-policy does apply to JMS topics.  The server aggregates the sizes of all the subscriptions on the topic and that is used to determine the total size of the topic.

                   

                  Have you tried getting a thread dump from the blocked client when it's blocked?  If so, could you attach it?

                  • 6. Re: Blocked JMS Topic
                    roman.pawliczek

                    Unfortunaly we weren't able to create a thread dump.

                    Maybe if I describe the problem in more detail it will help you to give us a hint to what is going wrong.

                     

                    We have a topic with some consumers (around 70 - ignore the 14 in the screenshot) and around 200 messages/sec. When the problem occurs we can see that the "Message Count" for the topic rises. Sometimes the problem resolves

                    itself if we just wait long enough (maybe around 5-15 minutes). It seems to us, that a single consumer blocks all other consumers. Is it possible, that if a single consumer isn't able to consume messages (e.g. some transaction problem occurred or something else we don't know yet), the internal structure of the topic gets filled up and blocks the other consumers? If so, how can we investigate or even better fix the problem? Is there some kind of property that has to be configured in the standalone-full.xml?

                     

                    Thanks in advance for your help.

                    • 7. Re: Blocked JMS Topic
                      jbertram

                      Actually it sounds like you need to get a thread dump from the server rather than the client when this occurs.  I've seen in the past where the delivery of a message from a topic to a subscriber might take longer than normal if the subscriber's network connection was congested and that would delay the delivery of messages to other subscribers.  I would recommend you switch to NIO.