4 Replies Latest reply on Jul 23, 2014 3:36 AM by k koushik

    OutOfMemory Error with hornetq-failure-check-thread

    k koushik Newbie

      Hi All,

      I have an application which runs on JBOSS AS 7.2.0Final. we are using Hornetq as messaging subsystem.

      while performing the Longevity tests we observed a Java Heapspace issue OOM occurred.

      on Analyzing the Heap Dump and thread dump i could see that hornetq-failure-check-thread is consuming lot of memory up to 1.2 GB.

      Could anyone please let me know if this is a known issue. or please guide me how to resolve this.

      here is the stack trace of Hornetq Failure thread

       

      "hornetq-failure-check-thread" prio=5 tid=116 RUNNABLE

          at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:48)

          at java.util.Arrays.copyOf(Arrays.java:2219)

          at java.util.ArrayList.grow(ArrayList.java:242)

          at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:216)

          at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:208)

          at java.util.ArrayList.add(ArrayList.java:440)

             Local Variable: java.util.ArrayList#4750

          at org.hornetq.core.server.impl.QueueImpl$RefsOperation.addAck(QueueImpl.java:2404)

             Local Variable: org.hornetq.core.server.impl.QueueImpl$RefsOperation#1

          at org.hornetq.core.server.impl.QueueImpl.acknowledge(QueueImpl.java:961)

             Local Variable: org.hornetq.core.server.impl.ServerMessageImpl#61690

          at org.hornetq.core.server.impl.QueueImpl$5.actMessage(QueueImpl.java:1105)

          at org.hornetq.core.server.impl.QueueImpl.iterQueue(QueueImpl.java:1147)

             Local Variable: org.hornetq.core.server.impl.MessageReferenceImpl#61779

             Local Variable: org.hornetq.core.server.impl.QueueImpl$5#1

             Local Variable: org.hornetq.core.transaction.impl.TransactionImpl#3

             Local Variable: org.hornetq.core.server.impl.QueueImpl$SynchronizedIterator#1

          at org.hornetq.core.server.impl.QueueImpl.deleteMatchingReferences(QueueImpl.java:1099)

          at org.hornetq.core.server.impl.QueueImpl.deleteAllReferences(QueueImpl.java:1094)

          at org.hornetq.core.server.impl.HornetQServerImpl.destroyQueue(HornetQServerImpl.java:1112)

             Local Variable: org.hornetq.core.postoffice.impl.LocalQueueBinding#3

          at org.hornetq.core.server.impl.HornetQServerImpl.destroyQueue(HornetQServerImpl.java:1070)

             Local Variable: org.hornetq.core.server.impl.HornetQServerImpl#1

             Local Variable: org.hornetq.api.core.SimpleString#147994

          at org.hornetq.core.server.impl.ServerSessionImpl$TempQueueCleanerUpper.run(ServerSessionImpl.java:472)

          at org.hornetq.core.server.impl.ServerSessionImpl$TempQueueCleanerUpper.connectionFailed(ServerSessionImpl.java:488)

          at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:570)

             Local Variable: org.hornetq.core.server.impl.ServerSessionImpl$TempQueueCleanerUpper#1

             Local Variable: java.util.ArrayList$Itr#111

             Local Variable: java.util.ArrayList#4694

          at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:341)

             Local Variable: java.lang.Object#332551

             Local Variable: org.hornetq.api.core.HornetQConnectionTimedOutException#1

          at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread.run(RemotingServiceImpl.java:652)

             Local Variable: java.util.HashSet#143121

             Local Variable: java.util.HashMap$KeyIterator#3

             Local Variable: java.lang.Integer#77518

             Local Variable: org.hornetq.core.protocol.core.impl.RemotingConnectionImpl#8

        • 1. Re: OutOfMemory Error with hornetq-failure-check-thread
          Clebert Suconic Master

          your thread dump tells me you are using a temporary queue, with lots of messages on it... and you are deleting the temporary queue with lots of messages...

           

           

          I'm not sure that's a valid scenario.. Usually temporary queues are small and for short period of usage.... and you are actually consuming messages.

          • 2. Re: Re: OutOfMemory Error with hornetq-failure-check-thread
            k koushik Newbie

            HI Thanks for the response.

            Could you please let me know what exactly meant by temporary queue. we have a topic hosted on the AS and the clients are subscribing to this topic. and whenever the topic connection breaks a reconnection would be initiated

            and here is my messaging configuration.

             

            <subsystem xmlns="urn:jboss:domain:messaging:1.3">

                        <hornetq-server>

                            <paging-directory path="lan/paging"/>

                            <bindings-directory path="lan/bindings"/>

                            <journal-directory path="lan/journal"/>

                            <large-messages-directory path="lan/largeMessage"/>

                            <security-domain>cnmJaas</security-domain>

                            <persistence-enabled>false</persistence-enabled>

                            <journal-type>NIO</journal-type>

                            <journal-min-files>2</journal-min-files>

                            <connectors>

                                <netty-connector name="netty" socket-binding="messaging"/>

                                <in-vm-connector name="in-vm" server-id="0"/>

                            </connectors>

                            <acceptors>

                                <netty-acceptor name="netty" socket-binding="messaging"/>

                                <in-vm-acceptor name="in-vm" server-id="0"/>

                            </acceptors>

                            <security-settings>

                                <security-setting match="#">

                                    <permission type="send" roles="user"/>

                                    <permission type="consume" roles="user"/>

                                    <permission type="createNonDurableQueue" roles="user"/>

                                    <permission type="deleteNonDurableQueue" roles="user"/>

                                </security-setting>

                            </security-settings>

                            <address-settings>

                                <address-setting match="#">

                                    <dead-letter-address>jms.queue.DLQ</dead-letter-address>

                                    <expiry-address>jms.queue.ExpiryQueue</expiry-address>

                                    <redelivery-delay>0</redelivery-delay>

                                    <!--  <max-size-bytes>10485760</max-size-bytes>  -->

                                    <max-size-bytes>52428800</max-size-bytes>

                                    <address-full-policy>PAGE</address-full-policy>

                                    <message-counter-history-day-limit>10</message-counter-history-day-limit>

                                </address-setting>

                            </address-settings>

                            <jms-connection-factories>

                                <connection-factory name="InVmConnectionFactory">

                                    <connectors>

                                        <connector-ref connector-name="in-vm"/>

                                    </connectors>

                                    <entries>

                                        <entry name="java:/ConnectionFactory"/>

                                    </entries>

                                </connection-factory>

                                <connection-factory name="RemoteConnectionFactory">

                                    <connectors>

                                        <connector-ref connector-name="netty"/>

                                    </connectors>

                                    <entries>

                                        <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>

                                    </entries>

                                </connection-factory>

                                <connection-factory name="cnmConnectionFactory">

                                    <connectors>

                                        <connector-ref connector-name="netty"/>

                                    </connectors>

                                    <entries>

                                        <entry name="java:jboss/exported/jms/cnmConnectionFactory"/>

                                    </entries>

                                </connection-factory>

                                <pooled-connection-factory name="hornetq-ra">

                                    <transaction mode="xa"/>

                                    <connectors>

                                        <connector-ref connector-name="in-vm"/>

                                    </connectors>

                                    <entries>

                                        <entry name="java:/JmsXA"/>

                                    </entries>

                                </pooled-connection-factory>

                            </jms-connection-factories>

                            <jms-destinations>

                                <jms-queue name="testQueue">

                                    <entry name="queue/test"/>

                                    <entry name="java:jboss/exported/jms/queue/test"/>

                                </jms-queue>

                                <jms-topic name="eventTopic">

                                    <entry name="topic/eventTopic"/>

                                    <entry name="java:jboss/exported/jms/topic/eventTopic"/>

                                </jms-topic>

                            </jms-destinations>

                        </hornetq-server>

            • 3. Re: OutOfMemory Error with hornetq-failure-check-thread
              Clebert Suconic Master

              a non durable topic subscription is a temporary queue.

               

               

              When you cancel the subscription, pending messages will need to be removed.. as they will have their references in the memory.

               

               

              Doing a massive subscription, and removing it is somewhat of an anti-pattern. if you need that kind of massive subscription use it as a durable subscription. Otherwise just consume your messages.

              • 4. Re: OutOfMemory Error with hornetq-failure-check-thread
                k koushik Newbie

                Hi Sorry for the delayed response..

                I was on vacation so couldn't respond back.

                So i understand the the subscription that we make for the topic should be durable?.

                If this is the case what impact it has when there is a disconnection and the application try a reconnect by establishing new subscription?