JBoss 4.2.2 - Unable to read messages from queue or topic after some days server is started

[klmr57monisha]

Hi,

Im using JBoss 4.2.2. Im facing the problem where my application unable to read messages from queue or topic after some days server is started.

My listener onMessage() method is not being invoked when message is available on topic destination.

Previously we were using 3.2.7 version and never had this issue. after upgrading to 4.2.2, we are facing this issue.

[adrian.brock]

Impossible to say with so little information.

Most likely you haven't done Connection.setExceptionListener() and some error occurred you don't know about.

[klmr57monisha]

Adrian,

We have exception listener mechanism and we reconnect when exception occurs.

We have the current system running with 3.2.7 version since 7 years. we have upgraded to 4.2.2 a month back and facing this issue. I couldnt see any exception or connection errors in log files.

It happens only for particular topic or queue on that connection. The rest of topics or queue on same connection is working fine. messages are not persistent and we set expiry time as 3 minutes.

could you see did we miss any configuration with new Jboss?

[adrian.brock]

I've converted the old READ THIS FIRST topic to an article.

Read that on how to debug and what information you need to provide.

e.g. thread dump, trace logs, etc.

For config, check the tuning link on the FAQ linked in that article, out-of-the-box

the server is not tuned very well, but it should still work.

[klmr57monisha]

can you point me to link for the article?

I have attached thread dumps here.

maki_jboss.td - jboss server thread dumps.

rest of the attached logs are our application logs whic connect to jboss server.

[adrian.brock]

To find the article go back to the main forum by clicking "Up to Discussions in MQ" then click on the articles tab.

It's also listed on the first page of the forum at the moment since I just published it. It used to be a sticky discussion

at the top of the forum so you could also search for "READ THIS FIRST".

The thread dumps aren't much use without understanding what is happening, for that I need the logs at TRACE level

and also some idea of which destination is broken so I can look at what happened during the last send/receive()

requests. Having client TRACE logs for the same time will also be useful.

The client trace logs will let me see what your code is doing, especially since some of it appears to be

Spring code, so you're probably not sure yourself. ;-)

You can get a vague idea of whether it is in the correct state by looking at the destination under JMX

to see if it has messages (or InProcess messages - if a client is processing them) and what receivers

are currently waiting for messages (if there's none, then they are currently busy doing something).

It is possible to have both receivers and messages if the messages don't match any messageSelector on the receivers.

I did a quick check to see if there is anything stuck running a query in the database or similar, but couldn't see anything obvious.

The only running threads appear to be doing something with crypto in threads unrelated to JMS.

I also checked you had the same number of Read/Write tasks, so it isn't a problem with a client connection

not getting cleaned up properly.

$ grep ReadTask token.td | wc -l
200
$ grep WriteTask token.td | wc -l
200

[klmr57monisha]

Adrian,

Thanks for your reply.

Is there any chance why JBoss cant able to delete messages after expiry time is over?

Can you also point me to the JMX console link to set default expiry time when its not set explicitly during message send?

[klmr57monisha]

Adrian,

Im also observing some of client connections are not getting closed and we also observe client does reconnect many time when invoked by exception listener.

each time we create connection, we set identifier. for ex: Q_1_BINGO_10.1.66.128_2474

Q_1_BINGO_10.1.66.128 - is fixed part.

2474 is counter which we will increment every time we create connection from the respective JVM.

I could see from JMX Console destination manager

Q_1_BINGO_10.1.66.128_2474

Q_1_BINGO_10.1.66.128_2100

Q_1_BINGO_10.1.66.128_3145

Q_1_BINGO_10.1.66.128_2227

if you take max of last part 3145 is max. it means this process created 3145 connections to JBoss out of which most of the connections are closed and 3 other are open except 3145.

our configuration is as below.

<attribute name="ReadTimeout">120000</attribute>
    <!-- Used to disconnect the client on the clientside if there is no activity -->
    <!-- Ensure this is greater than the ping period -->
    <attribute name="ClientReadTimeout">120000</attribute>

can you help us on why the rest of connections are not getting closed? we see each connection creates some threads at server side which causes server to perform very slowly.

[adrian.brock]

Look at the bottom of this FAQ for how to determine what connections are leaking:

http://community.jboss.org/message/531063#531063

[klmr57monisha]

The link again points to me to my post only. can you recheck once?

But even if leak is there, will it not close connections as per configuration data?

[adrian.brock]

The correct link is this one:

http://community.jboss.org/wiki/IGetIOExceptionPingTimeoutHowDoIFixIt

I don't understand your second comment. Why would any configuration close connections it thinks you are still using?

[klmr57monisha]

Adrian,

Again we have faced same issue in production after restart which is 15 days back. I have attached jboss server thread dump. I observe that all Read Tasks are in runnable state and WriteTasks are in wait state. do you have any clue on why all write tasks are in wait state all of sudden?

[klmr57monisha]

one more doubt..

if you see thread dump,

"UIL2(SocketManager.MsgPool@10b6862 client=10.70.50.115:53699)#23432

the last part #23432 seems very high.. is it expected?

[adrian.brock]

leela mahesh wrote:

Adrian,

Again we have faced same issue in production after restart which is 15 days back. I have attached jboss server thread dump. I observe that all Read Tasks are in runnable state and WriteTasks are in wait state. do you have any clue on why all write tasks are in wait state all of sudden?

The read tasks are reading a socket so they are runnable.

The write tasks are waiting for something to write to the cleint.

Like I said before, the thread dumps don't show anything,

except that the server is idle as far as JMS is concerned.

[adrian.brock]

leela mahesh wrote:

one more doubt..

if you see thread dump,

"UIL2(SocketManager.MsgPool@10b6862 client=10.70.50.115:53699)#23432

the last part #23432 seems very high.. is it expected?

The number is how many threads have been constructed for read/write tasks.

If you divide it by 2 you get the number of remote connections ever established since reboot.

Most likely you are doing lots of opening and closing of remote connections (usually an anti-pattern)

instead of pooling or caching connections?