when eviction is configured in the clustered environment, th

[murthytvsn]

when eviction is configured in the clustered environment, the tomcat server is not shutting down properly. And I have to kill the process manually. My configuration is as below

<?xml version="1.0"?>


READ_COMMITTED
REPL_ASYNC
JForumCacheStage



<!-- UDP: if you have a multihomed machine,
set the bind_addr attribute to the appropriate NIC IP address -->

<!-- UDP: On Windows machines, because of the media sense feature
being broken with multicast (even after disabling media sense)
set the loopback attribute to true -->
<UDP mcast_addr="228.1.2.3" mcast_port="48866"
ip_ttl="64" ip_mcast="true" bind_addr="xxx.xx.xx.xx"
mcast_send_buf_size="300000" mcast_recv_buf_size="640000"
ucast_send_buf_size="300000" ucast_recv_buf_size="640000"
loopback="false"/>

<PING timeout="2000" num_initial_members="3" up_thread="false" down_thread="false"/>
<MERGE2 min_interval="60000" max_interval="90000"/>
<FD shun="true" up_thread="true" down_thread="true" timeout="2500" max_tries="5"/>
<VERIFY_SUSPECT timeout="1500" up_thread="false" down_thread="false"/>

<pbcast.NAKACK gc_lag="50" retransmit_timeout="100,200,300,600,1200,2400,4800"
max_xmit_size="32768" up_thread="false" down_thread="false"/>

<UNICAST timeout="300,600,1200,2400,3600" window_size="100" min_threshold="10" down_thread="false"/>
<pbcast.STABLE desired_avg_gossip="40000" up_thread="false" down_thread="false" max_bytes="300000"/>

<FRAG frag_size="16384" down_thread="false" up_thread="false"/>
<pbcast.GMS join_timeout="5000" join_retry_timeout="2000" shun="true" print_local_addr="true"/>
<pbcast.STATE_TRANSFER up_thread="false" down_thread="false"/>



org.jboss.cache.eviction.LRUPolicy
<!-- Specific eviction policy configurations. This is LRU -->


<!-- Cache wide default -->
3600

<!-- Cache wide default -->

2000
3600


200
0
0


300
0
0


200
0
0


200
0
0


<!--attribute name="maxNodes">300</attribute-->
10
3600
0

[brian.stansberry]

This is a bug -- http://jira.jboss.com/jira/browse/JBCACHE-963 . Thanks for reporting it.

If Tomcat undeploys webapps as part of normal shutdown (not sure if it does -- JBoss AS definitely does), the workaround is to ensure that your webapp calls cache.stopService() as part of undeploy. You should do that anyway.

[murthytvsn]

Tkanks bstansberry. I will add cache.stopService while undeploying the webapp

[murthytvsn]

After adding cache.stopService() the tomcat shuts down proeperly if the cache is not triggered(not added any data). If cache is triggered by adding nodes the tomcat doesn't shutdown properly.

[brian.stansberry]

Can you post your config again?

Use the 'code' button to surround it with 'code' tags; this will allow it to display properly.

I'm guessing there's another thread not getting cleaned up properly; your config will likely indicate where. Although it's odd that we don't see this with the AS.

[murthytvsn]

Here is the configuration

<?xml version="1.0"?>
<server>
 <mbean code="org.jboss.cache.TreeCache" name="jboss.cache:service=TreeCache">
 <attribute name="IsolationLevel">READ_COMMITTED</attribute>
 <attribute name="CacheMode">REPL_ASYNC</attribute>
 <attribute name="ClusterName">CacheStage</attribute>

 <attribute name="ClusterConfig">
 <config>
 <!-- UDP: if you have a multihomed machine,
 set the bind_addr attribute to the appropriate NIC IP address -->

 <!-- UDP: On Windows machines, because of the media sense feature
 being broken with multicast (even after disabling media sense)
 set the loopback attribute to true -->
 <UDP mcast_addr="228.1.2.3" mcast_port="48866"
 ip_ttl="64" ip_mcast="true" bind_addr="xxx.xx.xx.xx"
 mcast_send_buf_size="300000" mcast_recv_buf_size="640000"
 ucast_send_buf_size="300000" ucast_recv_buf_size="640000"
 loopback="false"/>

 <PING timeout="2000" num_initial_members="3" up_thread="false" down_thread="false"/>
 <MERGE2 min_interval="60000" max_interval="90000"/>
 <FD shun="true" up_thread="true" down_thread="true" timeout="2500" max_tries="5"/>
 <VERIFY_SUSPECT timeout="1500" up_thread="false" down_thread="false"/>

 <pbcast.NAKACK gc_lag="50" retransmit_timeout="100,200,300,600,1200,2400,4800"
 max_xmit_size="32768" up_thread="false" down_thread="false"/>

 <UNICAST timeout="300,600,1200,2400,3600" window_size="100" min_threshold="10" down_thread="false"/>
 <pbcast.STABLE desired_avg_gossip="40000" up_thread="false" down_thread="false" max_bytes="300000"/>

 <FRAG frag_size="16384" down_thread="false" up_thread="false"/>
 <pbcast.GMS join_timeout="5000" join_retry_timeout="2000" shun="true" print_local_addr="true"/>
 <pbcast.STATE_TRANSFER up_thread="false" down_thread="false"/>
 </config>
 </attribute>

 <attribute name="EvictionPolicyClass">org.jboss.cache.eviction.LRUPolicy</attribute>
 <!-- Specific eviction policy configurations. This is LRU -->
 <attribute name="EvictionPolicyConfig">
 <config>
 <!-- Cache wide default -->
 <attribute name="wakeUpIntervalSeconds">3600</attribute>

 <!-- Cache wide default -->
 <region name="/_default_">
 <attribute name="maxNodes">2000</attribute>
 <attribute name="timeToLiveSeconds">3600</attribute>
 </region>
 <region name="/modules/">
 <attribute name="maxNodes">200</attribute>
 <attribute name="timeToLiveSeconds">0</attribute>
 <attribute name="maxAgeSeconds">0</attribute>
 </region>
 <region name="/temps/">
 <attribute name="maxNodes">300</attribute>
 <attribute name="timeToLiveSeconds">0</attribute>
 <attribute name="maxAgeSeconds">0</attribute>
 </region>
 <region name="/images/">
 <attribute name="maxNodes">200</attribute>
 <attribute name="timeToLiveSeconds">0</attribute>
 <attribute name="maxAgeSeconds">0</attribute>
 </region>
 <region name="/codes/">
 <attribute name="maxNodes">200</attribute>
 <attribute name="timeToLiveSeconds">0</attribute>
 <attribute name="maxAgeSeconds">0</attribute>
 </region>
 <region name="/cats">
 <!--attribute name="maxNodes">300</attribute-->
 <attribute name="maxNodes">10</attribute>
 <attribute name="timeToLiveSeconds">3600</attribute>
 <attribute name="maxAgeSeconds">0</attribute>
 </region>
 </config>
 </attribute>

 </mbean>
</server>

From the log:
INFO: Illegal access: this web application instance has been stopped already. Could not load java.io.PrintWriter. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1238)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1198)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at org.jgroups.util.Util.printStackTrace(Util.java:162)
at org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:1305)
at java.lang.Thread.run(Thread.java:595)
ERROR: [153.18.96.62:35325] exception=java.lang.NullPointerException, stack trace=java.lang.NullPointerException
at org.jgroups.Version.compareTo(Version.java:54)
at org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:1280)
at java.lang.Thread.run(Thread.java:595)
(2007-01-31 16:01:01,959 UDP.UcastReceiverThread_org.jgroups.protocols.UDP)
Exception in thread "TimeScheduler.Thread" java.lang.NullPointerException
at org.jgroups.util.TimeScheduler$Loop.run(TimeScheduler.java:138)
at java.lang.Thread.run(Thread.java:595)
ERROR: exception=java.lang.NullPointerException, stack trace=java.lang.NullPointerException
at org.jgroups.Version.compareTo(Version.java:54)
at org.jgroups.protocols.UDP.run(UDP.java:227)
at java.lang.Thread.run(Thread.java:595)
(2007-01-31 16:01:02,882 UDP mcast receiver_org.jgroups.protocols.UDP)
ERROR: [153.18.96.62:35325] exception=java.lang.NullPointerException, stack trace=java.lang.NullPointerException
at org.jgroups.Version.compareTo(Version.java:54)
at org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:1280)
at java.lang.Thread.run(Thread.java:595)
(2007-01-31 16:01:04,460 UDP.UcastReceiverThread_org.jgroups.protocols.UDP)
ERROR: [153.18.96.62:35325] exception=java.lang.NullPointerException, stack trace=java.lang.NullPointerException
at org.jgroups.Version.compareTo(Version.java:54)
at org.jgroups.protocols.UDP$UcastReceiver.run(UDP.java:1280)
at java.lang.Thread.run(Thread.java:595)
(2007-01-31 16:01:06,962 UDP.UcastReceiverThread_org.jgroups.protocols.UDP)
ERROR: exception=java.lang.NullPointerException, stack trace=java.lang.NullPointerException
at org.jgroups.Version.compareTo(Version.java:54)
at org.jgroups.protocols.UDP.run(UDP.java:227)
at java.lang.Thread.run(Thread.java:595)
(2007-01-31 16:01:07,297 UDP mcast receiver_org.jgroups.protocols.UDP)

[brian.stansberry]

How are you stopping the cache; i.e. what code invokes stopService() and where is it invoked?

Your log shows continuing operations of the JGroups channel, which should have stopped when stopService() returns.

[murthytvsn]

I used cache.stopService() in servlet destory and I can see in the log that code is executed

[brian.stansberry]

What JBoss Cache release? If you're not using the jgroups.jar that comes with that release, please advise the JGroups version as well.

[murthytvsn]

we are using jboss-cache-1.3.0.SP2.jar and jgroups-all-2.2.7.jar

[brian.stansberry]

It's not clear to me what's going on in the messages you log. Apparently JGroups is reading a packet(s) with null data; that's what's causing the NPEs mentioned in the error messages. Perhaps this is preventing the threads associated with the UDP protocol stopping properly. But what the root issue is I don't know.

Can you try a more recent JGroups release? Basically any subsequent release is compatible with JBC 1.3.0.SP2, so I'd say try 2.4.1. (See http://labs.jboss.com/portal/jbosscache/compatibility/index.html for compatibility listings.)

[murthytvsn]

I will try that.

[murthytvsn]

I tried with 2.4.1, 2.2.8 and 2.2.9.1 But I was not succesfull. For time being I commented the eviction attribute.

[brian.stansberry]

Commenting the eviction attribute fixed it?

[murthytvsn]

Here is the summary from my findings.
If eviction attribute is commented the tomcat is shutting down properly. But when eviction is uncommented the tomcat is not shutting down properly no matter cache is loaded or not loaded. After adding cache.stopservice code the server was shutdown properly with eviction attribute uncommented but the cache is not loaded with data. When eviction attribute is uncommented and cache is loaded the server is not shutting down properly.