Can MDB deployment fail on missing Queue - memory problems on missing queue and MDB deployment

[cornhoolio22]

Hello,

right now, I'm facing a heap space problem caused by a deployment of an MDB where the queue is missing. I'm running on JBoss 7.2.0 Alpha release with HornetQ 2.3.0.CR1 included.

The details:

I'm deploying and MBD where the queue is missing and it succeeds.

The serverlog only logs:

2014-01-22 10:48:17,748 INFO  [vm-mfalk-5660] [org.hornetq.ra] (default-threads - 2) HQ151001: awaiting topic/queue creation queue/GibtsNedQueue

And one time:

2014-01-22 10:48:19,765 INFO  [vm-mfalk-5660] [org.hornetq.ra] (default-threads - 2) HQ151002: Attempting to reconnect org.hornetq.ra.inflow.HornetQActivationSpec(ra=org.hornetq.ra.HornetQResourceAdapter@508cb3a8 destination=queue/GibtsNedQueue destinationType=javax.jms.Queue ack=Auto-acknowledge durable=false clientID=null user=null maxSession=15)

The implementation of the HornetQActivation in the runs in its exception handling part (handleFailure(Throwable failure)), which tries until infinity (reconnectAttempts = -1 by default) to reconnect and logs the INFO mentioned above.

In this exception handling there is again an exception handling included but no longer logged anymore because of the check '!(t instanceof HornetQNonExistentQueueException)'.

There are some Infos logged out in DEBUG level and an exception logged out in TRACE level. I would like to post them too here, but I don't know how to format it correctly in some kind of scroll block and it would be too long :-(

So now to my question:

Is it possible to configure HornetQ to fail during deployment, when the queue doesn't exist?

I looked into the properties of the connectors, connection-factories, pooled-connection-factories and HornetQ itself but couldn't find a property which fulfils my needs.

Thx for you answer,

Markus

PS: Some more infos to the memory leak:

In a server having this problem, there are 278 Threads of HornetQ-remoting-threads-HornetQServerImp and 113 Threads of HornetQ-client-global-threads, just to give some numbers. Seems that the cleanup doesn't work correctly (at least in version 2.3.0.CR1). HornetQ components use approx. 70 % of the heap space. This will cause the CPU and LOAD of the whole machine to rise. The LOAD on some point quite heavily.

[ataylor]

setup-attempts (setupAttempts) should do the trick, set it to something low.

could you test 2.4.0.Final to see if you get the leak there?

[cornhoolio22]

Ok setupAttempts and reconnectAttempts will stop the server from leaking. I saw this too in the code.

But I'm still not aware that my application is not working, because of the missing queue.

Is there a way to get the application not deployed when the queue is missing?

When using the setupAttempts I run in this (known and already fixed) issue:

[JBPAPP6-1388] <inbound-config> vanishes from standalone.xml after MDB deployment - JBoss Issue Tracker

And I just had a look how to integrate the 2.4.0 in JBoss 7 (http://downloads.jboss.org/hornetq/hornetq-2.4.0.Final-bin.zip). But in the download there are more libs than in my hornetq module folder of my JBoss 7 installation. Is there a prebuilt JBoss7 package or a howto how to integrate it with JBoss 7? The configfiles included are only for JBoss 4, 5, and 6 :-(

EDIT 1:

Oh and thanks for your answer! :-)

EDIT 2:

Ok I used the WildFly CR1 and deployed my application. Can give you feedback tomorrow about the memory and the threads.

But the deployment is still successful. Also on WildFly.

It seems not possible to me to answer my own post our your post a second time. I can only edit this one...

[cornhoolio22]

Wildfly test

2 GB Heap

512 GB Perm

2:30 pm startup

------------------------

heap 100 MB

perm 70 MB

hq-client-global-threads 3

hq-remoting-threads 2

4:30 pm between

-----------------------------

Heap 400

Perm 70

hq-client-global-threads 13  

hq-remoting-threads 15

during the night

-----------------

Many times (starting at this time):

2014-01-23 02:45:07,020 WARN  [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

2014-01-23 04:58:03,225 ERROR [org.hornetq.core.client] (Thread-2205 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=d50df06f-8366-11e3-a685-4b432785f3fa-1252340401-5

97895360)) HQ214017: Caught unexpected Throwable: java.lang.OutOfMemoryError: GC overhead limit exceeded

0900 am (not sure if this is correct due to OOM)

----------------------------------------------------------------------

Heap -

Perm -

hq-client-global-threads 10

hq-remoting-threads 16

Memory Analyze - Leak suspects

-------------------------------------------------

One instance of "org.hornetq.core.remoting.server.impl.RemotingServiceImpl" loaded by "org.jboss.modules.ModuleClassLoader @ 0x780387d68" occupies 558.046.672 (26,86%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "<system class loader>".

Keywords

java.util.concurrent.ConcurrentHashMap$Segment[]

org.jboss.modules.ModuleClassLoader @ 0x780387d68

org.hornetq.core.remoting.server.impl.RemotingServiceImpl

344.311 instances of "org.hornetq.core.client.impl.ClientSessionFactoryImpl", loaded by "org.jboss.modules.ModuleClassLoader @ 0x780387d68" occupy 983.453.152 (47,33%) bytes. These instances are referenced from one instance of "java.util.HashMap$Entry[]", loaded by "<system class loader>"

Keywords

org.jboss.modules.ModuleClassLoader @ 0x780387d68

java.util.HashMap$Entry[]

org.hornetq.core.client.impl.ClientSessionFactoryImpl

---------------------------------------------------------------------------------------------------------------------------------------------------------------

Edit 1:

So I ran the test yesterday to today. It was only a simple MDB without the queue. The threads look better now, but the memory problem still exists.

Again my question :-) Is there a way to configure that the deployment fails during deployment?

Thx,

Markus

[cornhoolio22]

Hi Andy,

I answered to my post not to yours. So this time I answer to your post. Don't know how it works here with notifications. But I've posted the results of the HornetQ 2.4.0 tests above.

Cheers,

Markus

[jbertram]

Again my question :-) Is there a way to configure that the deployment fails during deployment?

No, I don't believe there is a way to configure this behavior.  We'd have to change the code on the HornetQ JCA RA to provide that functionality. 

Did you happens to save the log from the test you ran on Wildfly?  If so, could you attach it here?

[jbertram]

Nevermind about the log.  I believe I've found the leak.  Working on a fix now...

[jbertram]

The leak has been fixed via https://issues.jboss.org/browse/HORNETQ-1313.

[cornhoolio22]

Ok, good to know. Thank you for your input here.

Cheers,

Markus

[jbertram]

Thanks for reporting the issue.  We appreciate it.

[kskandala]

Hi,

          I am facing exact same issue with JBoss EAP 6.2.0.GA & HornetQ 2.3.12.Final combination. I would like to upgrade my HornetQ above fixed version (2.4.1). But replacing jars at jboss-eap-6.2/modules/system/layers/base/org/hornetq/main/ and jboss-eap-6.2/modules/system/layers/base/org/hornetq/ra/main/ not helping.

Getting following exception:

Failed to define class org.hornetq.api.config.HornetQDefaultConfiguration in Module "org.hornetq:main" from local module loader @1f297e7 (finder: local module finder @1d2fc36 (roots: /usr/local/jboss-eap-6.2/modules,/usr/local/jboss-eap-6.2/modules/system/layers/base)): java.lang.UnsupportedClassVersionError: org/hornetq/api/config/HornetQDefaultConfiguration : Unsupported major.minor version 51.0

any help would be greatly appreciated!

Thanks!.

[jbertram]

Unsupported major.minor version 51.0

Look here.

As I told you on this thread, the messaging subsystem shipped with EAP 6.x is likely not even compatible with HornetQ 2.4.x.  I recommend you stick with HornetQ 2.3.x.  The fix for the issue you're seeing was committed to that branch over a year ago (see 4c0040e6f95193407757de88e0ee42d6ea9a8388).  Any release beyond 2.3.20.Final will contain the fix.  FYI - JBoss EAP 6.3.0.Final ships with HornetQ 2.3.20.Final.

[kskandala]

Justin,

  Thanks a lot. After moving to 2.3.25, issue got resolved.