Reset button

[jaap]

Hi,

At this moment i have configured JBossMQ to use about 20 concurrent 'Workers' to eat up a queue. Unfortunately these Workers tend to 'hang' and after about 3, 4000 messages the whole thing has stopped.

So, i am considering the flee forward solution: Jboss Messaging. But i was wondering. Does it provide in some better tools to manage the queue? For starters, can i restart JBoss Messaging wihout having to restart JBossAS? And are there means to see which workers are stalled (and maybe why :), or give it a certain maximum life span (a sort of TTL, but then different).

What can i expect in work load to change a system with about four different queues from JBossMQ to Messaging? I assume/hope the jboss.xml adn ejb-jar.xml generated by XDoclet can still be used?

Jaap

[timfox]

Not sure what you mean by workers. Is this code you have written yourself?

[timfox]

Please can you restate your questions

[jaap]

Hello Timfox

With 'workers' i guess i mean consumers. In the jbos.xml i have a config like this:
<invoker-proxy-binding>
--bla bla
20
20
/-bla bla

meaning i want 20 concurrent consumers (the logging calls them workers) to work parallel to pick up the messages that are sent to the queue. A nice feature, but during that process consumers get lazy and do not respond anymore :(. So, with JBossMQ the only solution is to restart JBoss after about 24 hours.

I presume that any other solution, ie JBoss Messaging, could have the same problems so a nice feature would be if it would have the ability to recover from dying consumers. If not, a reset of the messaging system would hlp a lot because other services would be uneffected by a restart.

Another problem with JBossMQ (most probably) is that JBoss JCA connection pool is very itchy about not closing database connections. So, when a consumer stalls, it might keep the connections open indefinitly and this will lead to terrible other problems. Just do a search on the forums and you will see that a lot of people who switch to JBoss experience significant problems with JCA.
And in my configuration you see a lot of dead database connections combined with a decrease in MQ activity.

So, to solve my problem i see a few possible solutions:
1. improve the code that is executed in the consumer (because the consumer hangs because my code within it hangs)
2. get a MQ system that have some managing capability (to cover up for 1.)
3. drop JBoss JCA, it is simplly too itchy about connection closing (but this is a different story)

So, basically i my situation i would like to configure the following:
"Dear messaging system. You have to work through thousands of messages per day. I would like you to process them swiftly, and with accuracy i could expect from a computer program. But, every now and then you will get an evil message that might take too much of your attention. Please discard these messages if you spend more then 10 minutes of your time (or put them in a evil messages group where a few dedicated consumers handle these nasty cases)." And if a tsunami of evil messages arrive at your get i understand, so no heart feelings when i want to restart you without bothering other processes that are running in the application server!

FYI, a single message in our application will lead to several hundreds http connections with other servers, serveral thousand database requests, a couple of external webservices (AXIS, JBoss WS... cool features!) etc. So, a lot of situations where things can stall or go wrong. To get a feeling, if the concurrent cosumers are set to 20++, JBoss will die within the hour (only option "kill -9 PID").
And one other thing. We have a couple of message queues. Some can be considered 'bulk queues', but others are processing financial transactions, so i would wish to discriminate them (the pigs queue is more equal then others).

In a nutshell:
- messaging managability
- restart, preferably on queue level
- discriminating queues
- recovering of dead consumers, or...
- regrouping slow/fast messages, so the bulk will be processed in time

[weston.price]

Another problem with JBossMQ (most probably) is that JBoss JCA connection pool is very itchy about not closing database connections. So, when a consumer stalls, it might keep the connections open indefinitly and this will lead to terrible other problems. Just do a search on the forums and you will see that a lot of people who switch to JBoss experience significant problems with JCA.
And in my configuration you see a lot of dead database connections combined with a decrease in MQ activity.

This is absurd. Instead of making blanket generalizations like this, point to specific examples or at the very least post your issue to the JCA forum. You are always free to include your *-ds.xml file and any *specfics* you think makes JBoss JCA 'itchy'.

3. drop JBoss JCA, it is simplly too itchy about connection closing (but this is a different story)

More drivel. Again, provide specfic examples and some sort of working example that you believe exhibits this problem.

[jaap]

At your service :)

JCA posts:
http://www.jboss.com/index.html?module=bb&op=viewtopic&t=91882
http://www.jboss.com/index.html?module=bb&op=viewtopic&t=91064
http://www.jboss.com/index.html?module=bb&op=viewtopic&t=87175
http://www.jboss.com/index.html?module=bb&op=viewtopic&t=87807
http://www.jboss.com/index.html?module=bb&op=viewtopic&t=61812

hmmmmm....

We solved a lot of our problems by closing the connections. You can turn logging on for this with the Debug set to true. And no warnings we receive for that anymore. In our code we open and close connections like this:

 Session s = null;
 try {
 s = SessionLocator.newSession();
 //bla bla
 } catch (Exception e) {
 l.fatal("bla more bla", e);
 } finally {
 try {
 s.close();
 } catch (Exception e) {}
 }

We also tried to check closing Resultsets (<track-statements>true</track-statements>), but that produces a lot of warnings, the reason for that is we use Hibernate (cannot find forum entry, but it is a known (non) issue).

We have these problems more or less since JBoss 3, now using 4.0.4. Hibernate vs 2 and 3.1.

mysql-ds:

 <local-tx-datasource>
 <jndi-name>Database</jndi-name>

<connection-url>jdbc:mysql://localhost:3306/jst_objects?characterEncoding=ISO8859-1&amp;cachePrepStmts=true&amp;jdbcCompliantTruncation=false</connection-url>
 <driver-class>com.mysql.jdbc.Driver</driver-class>
 <user-name>web</user-name>
 <password>1234</password>
 <min-pool-size>5</min-pool-size>
 <max-pool-size>1050</max-pool-size>
 <idle-timeout-minutes>5</idle-timeout-minutes> <exception-sorter-class-name>com.mysql.jdbc.integration.jboss.ExtendedMysqlExceptionSorter</exception-sorter-class-name> <valid-connection-checker-class-name>com.mysql.jdbc.integration.jboss.TimeoutMySQLValidConnectionChecker</valid-connection-checker-class-name>
 <exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.MySQLExceptionSorter</exception-sorter-class-name>
 <metadata>
 <type-mapping>mySQL</type-mapping>
 </metadata>
 <attribute name="Debug">true</attribute>
 <track-statements>true</track-statements>
 </local-tx-datasource>

The database reports:

show processlist;

| 5864 | web | localhost:57348 | jst_objects | Sleep | 16207 | | 5865 | web | localhost:57104 | jst_objects | Sleep | 16279 |
| 5866 | web | localhost:42541 | jst_objects | Sleep | 15912 |
| 5875 | web | localhost:49491 | jst_objects | Sleep | 15806 |
| 5877 | web | localhost:37892 | jst_objects | Sleep | 15755 | | ......
157 rows in set (0.00 sec)

status;
.....
Threads: 155 Questions: 33777278 Slow queries: 0 Opens: 41 Flush tables: 1 Open tables: 64 Queries per second avg: 244.696

You can see that MySQL has a lot of unused, idle connections (16207 for example means a hell of a lot more then <idle-timeout-minutes>5</idle-timeout-minutes>).

With a system load of 250 queries per second, 50 concurrent connections should be enouh.

So, IMHO, there is a lot of drivel on the forums about runaway connections in JCA. And we have problems with one heavy loaded JBossMQ queue that is stalling. The other queues are running flawless. My hunch is that the stalling MQ is causing the connections to become idle. So, i want to remove that and put in another messaging system.

Looking for replacements ActiveMQ for example seems to be a mature replacement for JBossMQ, but i believe (searching the posts) it cannot be embedded in JBoss when using MDB:
http://www.jboss.org/index.html?module=bb&op=viewtopic&t=85381

Would be JBoss Messaging be a good replacement candidate? Therefor my post.

Another solution i drivelled would indeed be to drop JCA. It is, looking at the posts and my experience, proven itchy. Itchy is no problem when the software running on it is closing connections like they should, but that is with more complex system not always the case and will always be a threat to the robustness of the system.

or at the very least post your issue to the JCA forum

Especially the JCA forum is a place where i would hesitate to post an issue. Posters often get cyberkilled there, exemplified by their supposed stupidity:
http://www.jboss.org/index.html?module=bb&op=viewtopic&t=68044
and yet another 'too many connections' would not contribute to the many similar unanswered posts.

[timfox]

jaap-

If you think you have a genuine issue with JCA then the burden of proof is on you to submit a testcase/example to Weston so he can investigate.

Either way it should be done on the JCA forum so I suggest you move it there. Otherwise I'll move it there myself.

[jaap]

JCA does not really interest me. I was looking for some more managable/control in a new messaging solution.

I already found the answers to my questions myself. I have installed JBoss Messaging. There is as far as i can see no 'reset button' where you can repair a stalled queue on the fly. And the management interface (JMX) is as good as JBossMQ. No new features concerning robustness. I did see some extra configuration to tackle performance issues.

[ovidiu.feodorov]

jaap wrote:
There is as far as i can see no 'reset button' where you can repair a stalled queue on the fly.

A queue is not supposed to "stall". What do you mean a "stalled" queue?

jaap wrote:

And the management interface (JMX) is as good as JBossMQ. No new features concerning robustness.

Meaning?

If you have suggestions regarding new functionality, you can use JIRA (http://jira.jboss.org/jira/browse/JBMESSAGING?report=com.atlassian.jira.plugin.system.project:roadmap-panel) to raise feature requests.

[jaap]

Hi Ovidio,

On the JIRA link i can only seem to browse issues, the submit issue link below on the page is as far as i can see only for JIRA itself.

A feature that would help me a lot is an addition to the JMX function of JBoss Messaging. When you open a queue in JMX you can see the number of messages that are still in the queue. There is also a purge queue function. Nothing new sofar.

What i would like is to get a list of active consumers in the queue. Each consumer would have a JMX interface where you can see some kind of reference set by the MDB and an indication how long this consumer is running. So, then i can see for what messages are taking a (too) long time for the consumer to finish.
If i would be possible, this JMX interface would have a method to stop the consumer from processing and to pick up a new message from the queue.

In short i would get two functions to manage the queue. One is to reset a consumer through JMX. The othere is to set a reference for the consumer programmatically within the MDB that will be visible in JMX at consumer level.

----------------

Antoher approach, that might slightly too advannced, is to make a dynamic optimisation of the maximum number of consumers.
You might compare it with old school RBDMS that needed skilled DBA's to configure specific parameters for each table. 'Modern' systems can configure themselves on the fly.

In a real life situation where you have say 20% slow messages ad 80% fast messages where for example the slow messages only use few system resources. You need an old school DBA to configure the optimal consumer settings. The consideraion is: too many consumers will kill the application server and too little consumers will lead to traffic jams.

A solution would be to have a "maximum resource setting", instead of setting a fixed number of maximum consumers. JBoss Messages could add as many consumers as it would wish as long as the whole queue would not exceed XX% of the available resources.
So, in terms of speed, JBoss Messaging would then become superior because it will always utilise all available computing power and it will not be restricted to some arbritrary, manual setting.

If you add the above mentioned JMX with reference, then that would be similar to a 'log slow query' function that will give the system manager a clue how to improve on the slow or troublesome messages.

Jaap

[jaap]

Excuse me: I meant to say 'Hi Ovidiu'!

Jaap