Processing apache log files with a UNIX tail type route

[unwired]

I am trying to write a route that would consume rotating log files from apache.

such as

server.log

server.log.1

server.log.2

etc.

I tried a route similar to

from("stream:file?fileName=/server/logs/server.log?scanStream=true&scanStreamDelay=1000").to("bean:logService?method=parseLogLine");

However this does not work if the log file has rotated once the route has started i.e the route continues to process the older file and never comes to the new file.  (the stream component hangs on to the inputstream it had opened when route started)

Is there a simple way to do this where the stream is reopened if the file has been rotated.

Thanks

[davsclaus]

Hmm

I wonder how we can improve that so it detects the file has been rotated?

If possible can you create a small unit tests or project that demonstrates this? Its kinda hard to simulate in the same JVM as the JDK uses JVM wide locks on files. So basically you need two separate JVM to test this.

We could maybe in case of some errors re attach the input stream to the file and then it may help when the other process rotates the log file.

Do you see any kind of error / warn etc in the logs when this happens?

[unwired]

There are no errors in the log files.  The only behavior is that the new file is not processed and after the original file is processed the tail stops.

If it is possible to add an additional flag that re-opens the file if the file has been rotated that would be the best solution.

If you want an example I can create one but it is essentially the route described above at its core that is reading a directory in which apache writes it log files.

Another issue that I noticed was that the scanStreamDelay was after every line that was processed rather that processing the entire file and then having the delay waiting for further data in the file.  Is that the expected behavior? 

It seems to be a bit different from tail where the file is dumped to stdout and then as more data is available it is streamed to stdout.

Edited by: unwired on Oct 8, 2009 10:59 PM

[davsclaus]

Hi

Good observations.

I have created a ticket to track this issue: MR-269