8 Replies Latest reply on Aug 19, 2008 5:07 AM by funnyone

RMI invoke fail, ConcurrentModificationException was thrown.

funnyone Aug 5, 2008 12:22 AM

Hi, everybody.
In our project, we hava a Jboss cluster on Solaris10 which has three nodes. Our client application have a thread which will invoke MBean's getStateList() method through RMI ervery 5 seconds. Generally, there is nothing wrong with it. But when the number of client is enough, A ConcurrentModificationException will be thrown. The full stack trace is:

java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:841)
at java.util.HashMap$KeyIterator.next(HashMap.java:877)
at java.util.HashSet.writeObject(HashSet.java:254)
at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
at org.jboss.invocation.MarshalledInvocation.writeExternal(MarshalledInvocation.java:620)
at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1307)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1288)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:258)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:117)
at org.jboss.invocation.jrmp.server.JRMPInvoker_Stub.invoke(Unknown Source)
at org.jboss.invocation.jrmp.interfaces.JRMPInvokerProxy.invoke(JRMPInvokerProxy.java:133)
at org.jboss.invocation.InvokerInterceptor.invokeInvoker(InvokerInterceptor.java:365)
at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:197)
at org.jboss.jmx.connector.invoker.client.InvokerAdaptorClientInterceptor.invoke(InvokerAdaptorClientInterceptor.java:66)
at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:70)
at org.jboss.proxy.ClientMethodInterceptor.invoke(ClientMethodInterceptor.java:74)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:100)
at $Proxy7.invoke(Unknown Source)
at com.datangmobile.oss.platform.servermonitor.client.rmi.RMIAdaptorHelper.invoke(RMIAdaptorHelper.java:35)
at com.datangmobile.oss.platform.servermonitor.client.ServerBeanCheckImp_J2EE.check(ServerBeanCheckImp_J2EE.java:39)
at com.datangmobile.oss.platform.servermonitor.client.ServerBeanListener.onTimer(ServerBeanListener.java:66)
at com.datangmobile.oss.platform.util.HeartTimer$1.run(HeartTimer.java:47)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
2008-08-04 10:05:58,703 DEBUG - [ServerBean Monitor]rmiConnect starting.................
2008-08-04 10:05:58,703 DEBUG - [ServerBean Monitor]rmiConnect={rmiIp=192.166.32.8,rmiPort=1100,rmiTimeout=5000
2008-08-04 10:05:58,750 DEBUG - [ServerBean Monitor]rmiConnect started..................
2008-08-04 10:05:58,750 DEBUG - [ServerBean Monitor]checking...................try[2]
2008-08-04 10:05:59,578 ERROR - [ServerBean Monitor]check error.
javax.management.MBeanException
at org.jboss.mx.interceptor.ReflectedDispatcher.handleInvocationExceptions(ReflectedDispatcher.java:180)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:163)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at org.jboss.jmx.connector.invoker.InvokerAdaptorService.invoke(InvokerAdaptorService.java:214)
at sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
at org.jboss.jmx.connector.invoker.SerializableInterceptor.invoke(SerializableInterceptor.java:74)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at org.jboss.invocation.jrmp.server.JRMPProxyFactory.invoke(JRMPProxyFactory.java:179)
at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at org.jboss.invocation.jrmp.server.JRMPInvoker$MBeanServerAction.invoke(JRMPInvoker.java:818)
at org.jboss.invocation.jrmp.server.JRMPInvoker.invoke(JRMPInvoker.java:419)
at sun.reflect.GeneratedMethodAccessor100.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
at sun.rmi.transport.Transport$1.run(Transport.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701)
at java.lang.Thread.run(Thread.java:595)
Caused by: javax.management.InstanceNotFoundException: datang:service=SvrMon is not registered.
at org.jboss.mx.server.registry.BasicMBeanRegistry.get(BasicMBeanRegistry.java:523)
at org.jboss.mx.server.registry.BasicMBeanRegistry.getValue(BasicMBeanRegistry.java:561)
at sun.reflect.GeneratedMethodAccessor103.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
... 41 more

We have only one clustered JBossAS(4.2.2.GA) and multiple clients(at least 20 in the real environment). One of the three nodes is master node whick deploys a MBean in scheduler-service.xml under server/default/deploy. the configeration is:

true
datang:name=SvrMon
check
NOW
3000
-1

the MBean which was deployed in the master cluster node is:
public class SvrMonService extends ServiceMBeanSupport implements SvrMonServiceMBean {
public void check()
public ArrayList getStatList(){ ...... }
.......
}

and the SvrMonServiceMBean is:
public interface SvrMonServiceMBean extends ServiceMBean{
/**
* return all MBean's state in the JbossAS
* @return SvrBeanList
*/
ArrayList getStateList();
void check();

}

ServiceMBean and ServiceMBeanSupport are the classes in the jboss-system.jar or jbossall-client.jar.

the client invoke code are here:
public class RMIAdaptorHelper {

private String _rmiHost = "127.0.0.1";
private int _rmiPort = 1099;
private int _timeout = 2000; // ms
private RMIAdaptor _server;

public RMIAdaptorHelper(String rmiHost, int rmiPort, int timeout)
throws CommunicationException, NamingException
{
this._rmiHost = rmiHost;
this._rmiPort = rmiPort;
this._timeout = timeout;
}

/**
* è¿œç¨‹è°ƒç”¨ MBEAN
*/
public Object invoke(RmiInvokeObject rmido) throws Exception
{
ObjectName name = new ObjectName(rmido.getJndiname());
return _server.invoke(name, rmido.getMethodname(), rmido
.getMethodArgs(), rmido.getMethodArgsType());
}

/**
* è¿žæŽ¥ Jboss rmi æœ�åŠ¡å™¨
*/
public void connect() throws NamingException, CommunicationException
{
Properties prop = System.getProperties();
prop.put(Context.INITIAL_CONTEXT_FACTORY,
"org.jnp.interfaces.NamingContextFactory");
prop.put(Context.URL_PKG_PREFIXES,
"org.jboss.naming:org.jnp.interfaces");
prop.put(Context.PROVIDER_URL, "jnp://" + _rmiHost + ":" + _rmiPort);

prop.put("jnp.discoveryTimeout", String.valueOf(_timeout));

InitialContext ic = new InitialContext(prop);
if (_server != null)
{
Logger.getLogger(getClass()).debug("RMIAdaptorHelper not null");
_server = null;
}
Object o = ic.lookup("jmx/rmi/RMIAdaptor");
_server = (RMIAdaptor) o;

}
}

1. Re: RMI invoke fail, ConcurrentModificationException was thr

funnyone Aug 5, 2008 12:25 AM (in response to funnyone)

Sorry . the configeration in scheduler-service.xml is:

true
datang:name=SvrMon
check
NOW
5000
-1
Actions

2. Re: RMI invoke fail, ConcurrentModificationException was thr

funnyone Aug 5, 2008 12:29 AM (in response to funnyone)

Sorry again. the configeration in scheduler-service.xml is:

<mbean code="com.datangmobile.oss.platform.servermonitor.mbean.SvrMonService" name="datang:name=SvrMon">
 </mbean>
 <mbean code="org.jboss.varia.scheduler.Scheduler"
 name="datang:service=Scheduler,name=SvrMon">
 <attribute name="StartAtStartup">true</attribute>
 <attribute name="SchedulableMBean">datang:name=SvrMon</attribute>
 <attribute name="SchedulableMBeanMethod">check</attribute>
 <attribute name="InitialStartDate">NOW</attribute>
 <attribute name="SchedulePeriod">3000</attribute>
 <attribute name="InitialRepetitions">-1</attribute>
 </mbean>

3. Re: RMI invoke fail, ConcurrentModificationException was thr

funnyone Aug 5, 2008 12:36 AM (in response to funnyone)

Something must be reminded of is: it's just happened incidentally, not always. I have already seen JBAS-1823 in the following url:
https://jira.jboss.org/jira/browse/JBAS-1823

But the stack trace is not the same with mine. So i don't know is there any relations between them.

Please help, Thanks.
Actions
4. Re: RMI invoke fail, ConcurrentModificationException was thr

funnyone Aug 6, 2008 3:22 AM (in response to funnyone)

Anyone help me?
Actions
5. Re: RMI invoke fail, ConcurrentModificationException was thr

funnyone Aug 15, 2008 4:03 AM (in response to funnyone)

Can anyone help me?
Actions
6. Re: RMI invoke fail, ConcurrentModificationException was thr

skajotde Aug 17, 2008 7:31 PM (in response to funnyone)
this Exception:

Caused by: javax.management.InstanceNotFoundException: datang:service=SvrMon is not registered.

is not by twice name SvrMon scheduler-service.xml ?
Actions
7. Re: RMI invoke fail, ConcurrentModificationException was thr

skajotde Aug 17, 2008 7:32 PM (in response to funnyone)

And in main question:

SvrMonService mayby should be synchronized and is written stateless ?
Actions
8. Re: RMI invoke fail, ConcurrentModificationException was thr

funnyone Aug 19, 2008 5:07 AM (in response to funnyone)

"skajotde" wrote:
And in main question:

SvrMonService mayby should be synchronized and is written stateless ?

you mean the check() method or the getStateList() method should be synchronized?
Actions

Go to original post