Dealing real life issues : improve statistics, robots crawl,

[cnovara]

I'm working on the following issues, any comments, suggestions, laughs are welcome :

Allowing robots to index nukes sites. Usual URLs rewriting methods seem difficult to set up (IMHO, JBoss gurus can you hear me ?)

There is another, simpler way to achieve the same result :
create some "dummy pages" presenting a flat view of database content, organised with "clean" (friendly) URL links.
Robots can easily follow those links to index the full content.

Queryng a "dummy page" should not result in a "FIle ID not found" but redirected on home page, or better "de-frindlys'ed" to view the real page with normal URL.

A batch process (a module, sorry) can compute those pages each night for example ...

Another way : detect bots crawl and presents "custom pages to them. thi pages presents only friendly URLs.
At the same time, we could log the web crawl for futur analyses ...

secondly, I would like Nukes to log each visitor, each page (independently) and presents nice graphs and reportings. Ip address will be "dns looked up" in background (ideally during low cpu activity ...)

And last, bots activity will be detected by Ip criteria and reported.

I'm allready working on those futurs modules.

Thanks for your attention

[theute]

I don't think a statistical tool is needed, there are already plenty free or not that analyze logs (like those of Apache) and display results in fancy graphs.

I am not sure about all this google's friendly urls... so i'd rather shut up and think for now.

[cnovara]

Indeed, we could use the Apache log and analyse it. But here is a typical Apache log entry :
193.253.222.57 - - [27/Sep/2004:08:49:24 +0200] "GET /index_ado?topicid=3 HTTP/1.1" 200 2267 "http://mysite.com/index.html?module=news&op=story&storyid=5&mode=3" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312461)"

I would like to see the news title for exemple, instead of storyid=5.
I mean a kind of "logical level" reporting, but not necesserly a nice graph. An SQL query is enough.

keep on studying