OutOfMemory Error with hornetq-failure-check-thread

[sketcha]

Hi All,

I have an application which runs on JBOSS AS 7.2.0Final. we are using Hornetq as messaging subsystem.

while performing the Longevity tests we observed a Java Heapspace issue OOM occurred.

on Analyzing the Heap Dump and thread dump i could see that hornetq-failure-check-thread is consuming lot of memory up to 1.2 GB.

Could anyone please let me know if this is a known issue. or please guide me how to resolve this.

here is the stack trace of Hornetq Failure thread

"hornetq-failure-check-thread" prio=5 tid=116 RUNNABLE

    at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:48)

    at java.util.Arrays.copyOf(Arrays.java:2219)

    at java.util.ArrayList.grow(ArrayList.java:242)

    at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:216)

    at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:208)

    at java.util.ArrayList.add(ArrayList.java:440)

       Local Variable: java.util.ArrayList#4750

    at org.hornetq.core.server.impl.QueueImpl$RefsOperation.addAck(QueueImpl.java:2404)

       Local Variable: org.hornetq.core.server.impl.QueueImpl$RefsOperation#1

    at org.hornetq.core.server.impl.QueueImpl.acknowledge(QueueImpl.java:961)

       Local Variable: org.hornetq.core.server.impl.ServerMessageImpl#61690

    at org.hornetq.core.server.impl.QueueImpl$5.actMessage(QueueImpl.java:1105)

    at org.hornetq.core.server.impl.QueueImpl.iterQueue(QueueImpl.java:1147)

       Local Variable: org.hornetq.core.server.impl.MessageReferenceImpl#61779

       Local Variable: org.hornetq.core.server.impl.QueueImpl$5#1

       Local Variable: org.hornetq.core.transaction.impl.TransactionImpl#3

       Local Variable: org.hornetq.core.server.impl.QueueImpl$SynchronizedIterator#1

    at org.hornetq.core.server.impl.QueueImpl.deleteMatchingReferences(QueueImpl.java:1099)

    at org.hornetq.core.server.impl.QueueImpl.deleteAllReferences(QueueImpl.java:1094)

    at org.hornetq.core.server.impl.HornetQServerImpl.destroyQueue(HornetQServerImpl.java:1112)

       Local Variable: org.hornetq.core.postoffice.impl.LocalQueueBinding#3

    at org.hornetq.core.server.impl.HornetQServerImpl.destroyQueue(HornetQServerImpl.java:1070)

       Local Variable: org.hornetq.core.server.impl.HornetQServerImpl#1

       Local Variable: org.hornetq.api.core.SimpleString#147994

    at org.hornetq.core.server.impl.ServerSessionImpl$TempQueueCleanerUpper.run(ServerSessionImpl.java:472)

    at org.hornetq.core.server.impl.ServerSessionImpl$TempQueueCleanerUpper.connectionFailed(ServerSessionImpl.java:488)

    at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:570)

       Local Variable: org.hornetq.core.server.impl.ServerSessionImpl$TempQueueCleanerUpper#1

       Local Variable: java.util.ArrayList$Itr#111

       Local Variable: java.util.ArrayList#4694

    at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:341)

       Local Variable: java.lang.Object#332551

       Local Variable: org.hornetq.api.core.HornetQConnectionTimedOutException#1

    at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread.run(RemotingServiceImpl.java:652)

       Local Variable: java.util.HashSet#143121

       Local Variable: java.util.HashMap$KeyIterator#3

       Local Variable: java.lang.Integer#77518

       Local Variable: org.hornetq.core.protocol.core.impl.RemotingConnectionImpl#8

[clebert.suconic]

your thread dump tells me you are using a temporary queue, with lots of messages on it... and you are deleting the temporary queue with lots of messages...

I'm not sure that's a valid scenario.. Usually temporary queues are small and for short period of usage.... and you are actually consuming messages.

[sketcha]

HI Thanks for the response.

Could you please let me know what exactly meant by temporary queue. we have a topic hosted on the AS and the clients are subscribing to this topic. and whenever the topic connection breaks a reconnection would be initiated

and here is my messaging configuration.

<subsystem xmlns="urn:jboss:domain:messaging:1.3">

            <hornetq-server>

                <paging-directory path="lan/paging"/>

                <bindings-directory path="lan/bindings"/>

                <journal-directory path="lan/journal"/>

                <large-messages-directory path="lan/largeMessage"/>

                <security-domain>cnmJaas</security-domain>

                <persistence-enabled>false</persistence-enabled>

                <journal-type>NIO</journal-type>

                <journal-min-files>2</journal-min-files>

                <connectors>

                    <netty-connector name="netty" socket-binding="messaging"/>

                    <in-vm-connector name="in-vm" server-id="0"/>

                </connectors>

                <acceptors>

                    <netty-acceptor name="netty" socket-binding="messaging"/>

                    <in-vm-acceptor name="in-vm" server-id="0"/>

                </acceptors>

                <security-settings>

                    <security-setting match="#">

                        <permission type="send" roles="user"/>

                        <permission type="consume" roles="user"/>

                        <permission type="createNonDurableQueue" roles="user"/>

                        <permission type="deleteNonDurableQueue" roles="user"/>

                    </security-setting>

                </security-settings>

                <address-settings>

                    <address-setting match="#">

                        <dead-letter-address>jms.queue.DLQ</dead-letter-address>

                        <expiry-address>jms.queue.ExpiryQueue</expiry-address>

                        <redelivery-delay>0</redelivery-delay>

                        <!--  <max-size-bytes>10485760</max-size-bytes>  -->

                        <max-size-bytes>52428800</max-size-bytes>

                        <address-full-policy>PAGE</address-full-policy>

                        <message-counter-history-day-limit>10</message-counter-history-day-limit>

                    </address-setting>

                </address-settings>

                <jms-connection-factories>

                    <connection-factory name="InVmConnectionFactory">

                        <connectors>

                            <connector-ref connector-name="in-vm"/>

                        </connectors>

                        <entries>

                            <entry name="java:/ConnectionFactory"/>

                        </entries>

                    </connection-factory>

                    <connection-factory name="RemoteConnectionFactory">

                        <connectors>

                            <connector-ref connector-name="netty"/>

                        </connectors>

                        <entries>

                            <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>

                        </entries>

                    </connection-factory>

                    <connection-factory name="cnmConnectionFactory">

                        <connectors>

                            <connector-ref connector-name="netty"/>

                        </connectors>

                        <entries>

                            <entry name="java:jboss/exported/jms/cnmConnectionFactory"/>

                        </entries>

                    </connection-factory>

                    <pooled-connection-factory name="hornetq-ra">

                        <transaction mode="xa"/>

                        <connectors>

                            <connector-ref connector-name="in-vm"/>

                        </connectors>

                        <entries>

                            <entry name="java:/JmsXA"/>

                        </entries>

                    </pooled-connection-factory>

                </jms-connection-factories>

                <jms-destinations>

                    <jms-queue name="testQueue">

                        <entry name="queue/test"/>

                        <entry name="java:jboss/exported/jms/queue/test"/>

                    </jms-queue>

                    <jms-topic name="eventTopic">

                        <entry name="topic/eventTopic"/>

                        <entry name="java:jboss/exported/jms/topic/eventTopic"/>

                    </jms-topic>

                </jms-destinations>

            </hornetq-server>

[clebert.suconic]

a non durable topic subscription is a temporary queue.

When you cancel the subscription, pending messages will need to be removed.. as they will have their references in the memory.

Doing a massive subscription, and removing it is somewhat of an anti-pattern. if you need that kind of massive subscription use it as a durable subscription. Otherwise just consume your messages.

[sketcha]

Hi Sorry for the delayed response..

I was on vacation so couldn't respond back.

So i understand the the subscription that we make for the topic should be durable?.

If this is the case what impact it has when there is a disconnection and the application try a reconnect by establishing new subscription?