using hornetq as a log server

[augustsimonelli]

Hi All,

Disclaimer: I'm a sysadmin and not a developer. So i'm kinda slow with this stuff :-)

I have tomcat logging nicely with log4j. Our developers also use log4j. Right now, the logging goes to a file for quick review and syslog for long term storage. This is nice, but i thought it'd be neat to try and log to hornetq. Well, actually, this (http://www.ibm.com/developerworks/websphere/library/techarticles/0207_barcia/barcia.html) got me thinking that way.

So, i'm embarking on this little journey and thought i'd see if anyone else has forged ahead and had some pointers etc? Or if anyone else was interested in sharing recipes/ideas/failures/success/etc!

August

[timfox]

I don't see why you can't use HornetQ in this way, although I've never seen it before.

Tell us how it goes

[leosbitto]

I think that studying Log4J's JDBCAppender should give you the idea how to implement JMSAppender which would send the data to be logged to any JMS destination. Be prepared that it is going to be much slower than logging to the local file, though. Anyway, you could configre Log4J to send only some selected messages to your new JMSAppender, so the lower speed might be acceptable...

[augustsimonelli]

Leos -

Good idea ... i'll check out http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/net/JMSAppender.html and see what i learn.

As far as speed, i was hoping this could be faster ... because (naiive sysadmin hat goes on) if i use the async appender to handle the JMSappender output i could batch it there and then send on to hornetq. That way we are quickly done as the async appender has kicked in. Then i thought JMS would be faster (ie geting the message and sending to DB) than syslog (in my case rsyslog or syslog-ng) writing to a file. But maybe not. And perhaps the java devs would rather a good grep and awk fest than having their logs dumped in a DB.

Still, it seems a good challenge; i think it may not be as "revolutionary" as i may have been thinking :-)

I'll report how i go!

August

[timfox]

HornetQ internally can write to it's files much faster than you can write to a local log file, since we do all sorts of tricks to optimise this, use AIO etc.

If you're consuming from a queue and then logging to a file, or inserting in a DB, that's going to be the limiting factor, not HornetQ.

Make sure you send and consume messages async for best perf.

[augustsimonelli]

thanks everyone! i can see that this is more an interesting project for me to learn about hornetq, jms and log4j - and may not be a huge "improvement" over my current set up. but as a nice bonding exercise between dev and ops i'm gonna go for it.

thanks for the details and i'll report back any interesting results!

love the way i get such quick and useful AND friendly responses form y'all ... this is what it's all about!

- August