JMS audit/logging/resubmission web tool

[exto]

Hey everyone, I am hoping this email might stir some good conversation, I would love to hear what everyone else is doing in this space.

I'm searching around to see if a particular tool exists, I can't seem to find one so I thought I would see what everyone else is doing.  I plan on posting this in a few of the open source projects around to get the best feedback.

I am looking for a software tool which I will primarily use to audit, monitor and resubmit messages sent to a JMS message broker.  Ideally this tool has the following features:

- Open Source with a good community

- Web based console to view and search messages

- Messages are persisted in a database (SQL/NoSQL)

- Ability to add custom "Action" buttons so that you could easily write a plugin that "resubmits" a message back to a JMS queue/topic as well as write whatever else you need.   Would also need the ability to "edit" the message before resubmitting.

- Ability to acknowledge/comment on failed messages

- Support for displaying/viewing Java objects, (probably limits me to Java based tools...)

- Have the ability to send email alerts based on user defined conditions

- Graphing, statistics and reporting regarding how many messages we're sent to each queue/topic in 5 minute intervals grouped by topic/message type.

- Does not need to have a JMS adaptor, as long as their is an API to get the data into the tool, i'm sure this would be easy to write/contribute.

- Supports millions of messages/day

- BAM style dashboard

I would really like to have a tool like this to monitor/audit/log/replay messages in my JMS environment.  I see a lot of similarities with what I want in the tools like Zenoss/OpenNMS/syslog-ng/splunk etc.. as they all have "Eventing/Syslog" capabilities. 

I would prefer to add/contribute to an existing project as opposed to building something custom myself...

- What is everyone else out there doing for auditing and tracking of failed messages in your JMS environment?

- Which open source tool do you think is the closest to what i'm looking for?

[clebert.suconic]

That's a freaking awesome idea!

I have actually thought about doing something like this...   but I reckoned it would require a whole project effort.

In terms of plugging something like that on HornetQ you could either have a subscription for each  JMS queue and JMS topics, but that would be a bit invasive, or you could plug the StorageManager (which I think it would be a better approach).

I liked this post so much.. and I'm moving it to the dev's forum since this is a dev's thread IMO.

[clebert.suconic]

HornetQ has a OperationContext class. We never block on any persistence at all.

I'm wondering if you would require persistence on the DB before before completing the context? The best would be to not require it. You could implement something on the JournalStorageManager to check if the message is available upon restart. So, you would keep it totally asynchronous and not affecting (or minimally affecting performance).

[exto]

Thanks for the reply and thoughs Clebert.

Unfortunately I think getting the data from the broker is the less difficult part.   I like your ideas on where to put the plugin, the only caveat I would add is that it would be ideal if the data was persisted and available on JMS so that common adaptors can be written.  Perhaps subscribing to '#' and get all messages to all topics seems like a simple approach.   Maybe add a '#!' which means all messages to all topics & queues.

The part I find hard is deciding if there is a project out there that has the Web Frontend that can display this data with a pluggable framework for custom actions.    Since I deal with a lot of OpenSource and not one particular JMS broker, I want to ensure that it would also work with any JMS vendor.

On other messaging boards, people have recommended HermesJMS, however this is more of a developer tool and would need to have a full web gui written.

It seems like no such tool exists...  so I am thinking either:

a) Ignore the auditing/logging requirement and send failed/dmq messages to a BPEL process which could give users the ability to view/resubmit, etc..

b) Start a new Open source project, given our teams skillset we would probably write the frontend in grails or gwt.  My gui skills aren't great, so was hoping to have a front-end to start with and add to it.. 

Thanks again for taking the time to respond!

[clebert.suconic]

I like the idea of gwt... Maybe Errai?

I don't think you can subscribe to the data with pure-jms. You can't for instance have multiple consumers to a single jms queue.

(You can with hornetq-core).

I would suggest you having plugin forms to providers. With hornetq the easiest would probably be through a special StorageManager.