The first question is how do you the TreeCacheAop? Do you use putObject() inside your session for every request? If it is, this may be the problem since it is slow (and should be used only initially).
Also, if you use web clustering, we will support fine-grained replication in 4.0.4 (using TreecacheAop). It is not yet fully released. But you can give it a try soon.
I'm using a homegrown session manager because I can't set the subdomain cookie scope on Tomcat's sessions (long story). I create a session once and put it into the cache with putObject(). The key I use is a unique string generated from the request (url + subdomain + some other stuff). So on each request I retrieve the session using that key and the getObject() method. The putObject() is not used on every request.
One thing I've considered is that at least one property is updated on every request. I have a property called UserSession.mostrecentactivity which I update every request so that I can expire sessions properly (sysadmins for our app can set the timeout via the web). My hope was that because of AOP this property update wouldn't cause that much traffic across the wire.
Of course, the problem I'm seeing isn't network related. It's the performance of the app itself. I've done some testing and the performance degradation seems to relate to how many times I read a property from the object that was cached... even after I've retrieved the object from the cache into local memory.
For example, if I pull a UserSession from the cache and then use the getFoo() method to access the foo String 1000 times the app is a lot slower than if i had only accessed the foo String 10 times. In other words, because of AOP the performance of simple property gets is degraded.
Watching the profiler it seems that there is some jbosscache code that is called by AOP each time that I do a property get call. This doesn't make sense to me. I understand that jbosscache needs to know about property puts/updates because it needs to replicate those changes across the cache. But I don't understand why org.jboss.aop.joinpoint.FieldReadInvocation.invokeNext is called on gets.
Question: is there a way to tell AOP to not worry about property gets and only notify jbosscache of property puts/updates?
The good news is that TreeCacheAOP is working as advertised. Now it's just a matter of tweaking the performance for my particular app. I appreciate your help and look forward to running our cluster on TreeCacheAOP! (Looking forward to your training session on webapp clustering next week.)
Ok, it is diffcult for me to see now because I have run some profiling myself last week and I didn't see it. Which version of JBossCache you are using?