Hi,

I have faced the following exception, which has probably led to 100% CPU Utilization of our JBoss nodes

01:06:11,947 ERROR [JBossCacheService] externalizeSession(): exception  occurred externalizing session SessionBasedClusteredSession[id:  5fv4Kd9YXgvt6S0VcwnEPA**.node2 lastAccessedTime: 1284570371909 version:  2736 lastOutdated: 0]
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:841)
at java.util.HashMap$EntryIterator.next(HashMap.java:883)
at java.util.HashMap$EntryIterator.next(HashMap.java:881)
at java.util.HashMap.writeObject(HashMap.java:1037)
at sun.reflect.GeneratedMethodAccessor380.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
at java.util.concurrent.ConcurrentHashMap.writeObject(ConcurrentHashMap.java:1380)
at sun.reflect.GeneratedMethodAccessor560.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
at  org.jboss.web.tomcat.service.session.SessionBasedClusteredSession.writeExternal(SessionBasedClusteredSession.java:175)
at org.jboss.web.tomcat.service.session.JBossCacheService.externalizeSession(JBossCacheService.java:1033)
at org.jboss.web.tomcat.service.session.JBossCacheService.putSession(JBossCacheService.java:322)
at  org.jboss.web.tomcat.service.session.JBossCacheClusteredSession.processSessionRepl(JBossCacheClusteredSession.java:125)
at org.jboss.web.tomcat.service.session.JBossCacheManager.processSessionRepl(JBossCacheManager.java:1153)
at org.jboss.web.tomcat.service.session.JBossCacheManager.storeSession(JBossCacheManager.java:702)
at org.jboss.web.tomcat.service.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:49)
at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:108)
at org.jboss.web.tomcat.service.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
at org.jboss.web.tomcat.service.sso.ClusteredSingleSignOn.invoke(ClusteredSingleSignOn.java:638)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:437)
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:381)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:595)

We are using 2 clustered JBoss node. But due to many non-serialized classes, we are not able to achieve complete session replication across the nodes. In few minutes of getting this exception, the CPU utilization on both the JBoss nodes went up to 100% and didn't come down till we rebooted the JBoss nodes.

What can be the possible cause of this exception?

Thanks in advance,

Sanket