Queue Consumers stopped working

[newway]

Hello all,

this is kind of a vague situation and i was hoping someone could help me in ways to investigate the problem.

I am also not sure if i should post the question here or on the jboss development community, since I am working on JBOSS 7.2.0.Alpha - so all your directions will be appreciated.

basically I have a system that is running over 5 hours and it had been functioning OK so far.

suddenly I noticed from the log files that all the components which are reading information from queues - both MDBs and remote clients that connect using netty stopped working.

if i login in the admin console I see that the queues have 1000s of messages inside them - but none of the consumers is invoked.

i looked around in all the log locations i could think of and found nothing, looked for java crush reports and nothing.

i restarted one of the remote consumers and i see that it manages to connect - but it is not getting any messages.

my machine has 8 GB of RAM and 4 CPUs - i monitored it and it didn't look like resources are out

and idea what and where i should look for?

i assume that if i will restart JBOSS the system will start working properly - but i want to check if there are any ideas that should be tried before loosing the current state

thanks,

Noa

[jbertram]

Get a thread dump from the stalled instance of AS 7.2 and attach it here.

[newway]

Hi,

I am trying to figure out how to obtain a thread dump from a jboss that is already running without loosing it's state.

i run jboss as a service on win 2008R2 using the tanuki service wrapper -but i got nowhere so far.

tried jconsole or visual VM - but they are not finding jboss

any ideas on what to do?

- I decided to start the server again in console mode where i can get the thread dump.

you can look at this specific question as an academic exercise if you have time for it

thanks,

Noa

[sergiu_pienar]

Try the MemoryAnalyzerTool, maybe it helps : http://www.eclipse.org/mat/

[jbertram]

How about jstack (http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstack.html)?  It should be included with your JDK.

[newway]

Looks like jstack should have worked if I still had the process running.

the thing is that when I used the jps command it looked like it doesn't recognize any java process except for itself, same for the visualVM

I am attaching the thread dump, although it happened now a lot faster than before - so I am not sure that it's the same problem

* forgot to say one thing - I am now runnig on jboss 7.1.1 and not 7.2 - the only diff from 7.1.1 is that i merged the fix that enables to change the thread pool min and max size

[jbertram]

Your thread dump is clean as far as I can tell.  Can you attach your standalone*.xml for the instance in question?

Also, do you happen to have a set of steps I could run to reproduce the issue?  It sounds like you see it pretty consistently.

[newway]

Looks like while I was trying to figure out how to produce the thread dump I stumbled over the real problem.

I thought that I was configuring Jboss to use maximum heap size of 1.5G - turns out that the Tanuki service wrapper was overriding my definitions using :

# Initial Java Heap Size (in MB)

wrapper.java.initmemory=64

# Maximum Java Heap Size (in MB)

wrapper.java.maxmemory=512

after commenting out those definitions I rerun my scenario and now the system is running over half a day without errors.

so I believe the problem was that it reached the max memory, the thing that puzzles me is why I didn't see errors that are related to this problem, but I might have been too aggressive in the logging configurations I had