Performance with many projects

[bjoy]

I have experimented by configuring Kosmos to monitor many small projects (27 projects each with 13 folders and 8 files) all in the same repository.

I have a Liferay portal page with only the Kosmos portlet on it. When I select it, nothing shows for 5 minutes and the Tomcat log grows by 180K. Once the page shows everything works as expected.

The Tomcat log shows the timings on executing many Subversion commands. The timestamps agree with the timings and there does not seem to be any gaps. So I think that 5 minutes is mostly spent gathering data.

I would like to see Kosmos appear more quickly and do the data gathering in the background. Is there anything I can configure which would speed this initial startup?

Also, is there a way to disable the informational logging?

[aron.gombas]

I would like to see Kosmos appear more quickly and do the data gathering in the background. Is there anything I can configure which would speed this initial startup?

Unfortunately, the SVN traversal can be very expensive. There is no way to save this time, otherwise the commit- or file information would not available.
After each traversal, the result is cached and redisplayed until the next traversal completes.

Also, is there a way to disable the informational logging?

Kosmos is using log4j for logging, so you can easily configure the logging output as you wish. (You can disable the whole Kosmos logging by disabling it for the hu.midori.kosmos package.)

[bjoy]

I have started/stopped Tomcat several times trying to disable Kosmos informational logging using a log4j.properties file but have given up. Can you provide details about how/where to do this?

In the process of doing the above, I noticed that Kosmos startup was getting slower. Currently the Tomcat log is 1.1GB and Kosmos now takes 10 minutes to startup. Looking back in the log, only the first startup was 5 minutes. Every startup following the first one took 10 minutes.

Any idea why that might be?

The log entries which match the following pattern are recording much longer times than they did on the first startup:

PROPFIND, 207 "Multi-Status", 1609 ms, /files/kosmos-cache

[bjoy]

I have tried modifying the ehcache.xml to point at a different directory.

The startup still took 10 minutes and the data/index files created for the cache were both 0 length.

I compared the log entries made on the first run against the last and noticed that there were 300 more entries logged in the last run.

The first run did not have entries like the following:

http-8080-Processor22, 07-Dec-2006 22:14:52, unauthenticated, DELETE, 204 "No Content", 250 ms, /files/kosmos-cache/1165528202000_http%3A%2F%2Fussvd-bjoy%3A8070%2Fsubversion%2FSampleProject%282%29%2Ftrunk_files_per_file_type_small.png

Given that all the extra log entries seem to be like this one, that only shows where approximately 75 seconds of the extra time is being spent.

And since I changed the cache location (and the cache staying zero length after VM shutdown), I don't understand why the last run was finding anything to delete. I was expecting startup behavior to match the timing of the first run.

[aron.gombas]

Some explanation.
Kosmos is caching two kind of data:

1. Java objects (the model itself) that represent SVN repositories in this case. These are stored in the ehcache storage.

2. Generated content files (mostly images) that are charts in this case. These files must be cleaned up, when a more uptodate version gets generated.
These are stored in the WebDAV storage (by default), and it might be expensive to delete the files from those. The first time you launch Kosmos, there's nothing to clean, so that's why it's faster. If you like, you can manually delete the stuffs from WebDAV before starting up the container.

Of course, changing the ehcache configuration will not have any impact on the second.
And, 2 is pluggable, so you can use other mechanisms to store your images. (AFAIK the JBoss Labs guys developed some custom solution based on their Shotoku for this purpose.)

I hope this helps to understand what's going on.