8 Replies Latest reply on Aug 19, 2008 5:07 AM by feng xueyong

    RMI invoke fail, ConcurrentModificationException was thrown.

    feng xueyong Newbie

      Hi, everybody.
      In our project, we hava a Jboss cluster on Solaris10 which has three nodes. Our client application have a thread which will invoke MBean's getStateList() method through RMI ervery 5 seconds. Generally, there is nothing wrong with it. But when the number of client is enough, A ConcurrentModificationException will be thrown. The full stack trace is:

      java.util.ConcurrentModificationException
      at java.util.HashMap$HashIterator.nextEntry(HashMap.java:841)
      at java.util.HashMap$KeyIterator.next(HashMap.java:877)
      at java.util.HashSet.writeObject(HashSet.java:254)
      at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
      at org.jboss.invocation.MarshalledInvocation.writeExternal(MarshalledInvocation.java:620)
      at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1307)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1288)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
      at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
      at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:258)
      at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:117)
      at org.jboss.invocation.jrmp.server.JRMPInvoker_Stub.invoke(Unknown Source)
      at org.jboss.invocation.jrmp.interfaces.JRMPInvokerProxy.invoke(JRMPInvokerProxy.java:133)
      at org.jboss.invocation.InvokerInterceptor.invokeInvoker(InvokerInterceptor.java:365)
      at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:197)
      at org.jboss.jmx.connector.invoker.client.InvokerAdaptorClientInterceptor.invoke(InvokerAdaptorClientInterceptor.java:66)
      at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:70)
      at org.jboss.proxy.ClientMethodInterceptor.invoke(ClientMethodInterceptor.java:74)
      at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:100)
      at $Proxy7.invoke(Unknown Source)
      at com.datangmobile.oss.platform.servermonitor.client.rmi.RMIAdaptorHelper.invoke(RMIAdaptorHelper.java:35)
      at com.datangmobile.oss.platform.servermonitor.client.ServerBeanCheckImp_J2EE.check(ServerBeanCheckImp_J2EE.java:39)
      at com.datangmobile.oss.platform.servermonitor.client.ServerBeanListener.onTimer(ServerBeanListener.java:66)
      at com.datangmobile.oss.platform.util.HeartTimer$1.run(HeartTimer.java:47)
      at java.util.TimerThread.mainLoop(Timer.java:512)
      at java.util.TimerThread.run(Timer.java:462)
      2008-08-04 10:05:58,703 DEBUG - [ServerBean Monitor]rmiConnect starting.................
      2008-08-04 10:05:58,703 DEBUG - [ServerBean Monitor]rmiConnect={rmiIp=192.166.32.8,rmiPort=1100,rmiTimeout=5000
      2008-08-04 10:05:58,750 DEBUG - [ServerBean Monitor]rmiConnect started..................
      2008-08-04 10:05:58,750 DEBUG - [ServerBean Monitor]checking...................try[2]
      2008-08-04 10:05:59,578 ERROR - [ServerBean Monitor]check error.
      javax.management.MBeanException
      at org.jboss.mx.interceptor.ReflectedDispatcher.handleInvocationExceptions(ReflectedDispatcher.java:180)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:163)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
      at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      at org.jboss.jmx.connector.invoker.InvokerAdaptorService.invoke(InvokerAdaptorService.java:214)
      at sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
      at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
      at org.jboss.jmx.connector.invoker.SerializableInterceptor.invoke(SerializableInterceptor.java:74)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      at org.jboss.invocation.jrmp.server.JRMPProxyFactory.invoke(JRMPProxyFactory.java:179)
      at sun.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      at org.jboss.invocation.jrmp.server.JRMPInvoker$MBeanServerAction.invoke(JRMPInvoker.java:818)
      at org.jboss.invocation.jrmp.server.JRMPInvoker.invoke(JRMPInvoker.java:419)
      at sun.reflect.GeneratedMethodAccessor100.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
      at sun.rmi.transport.Transport$1.run(Transport.java:153)
      at java.security.AccessController.doPrivileged(Native Method)
      at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
      at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460)
      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701)
      at java.lang.Thread.run(Thread.java:595)
      Caused by: javax.management.InstanceNotFoundException: datang:service=SvrMon is not registered.
      at org.jboss.mx.server.registry.BasicMBeanRegistry.get(BasicMBeanRegistry.java:523)
      at org.jboss.mx.server.registry.BasicMBeanRegistry.getValue(BasicMBeanRegistry.java:561)
      at sun.reflect.GeneratedMethodAccessor103.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
      ... 41 more


      We have only one clustered JBossAS(4.2.2.GA) and multiple clients(at least 20 in the real environment). One of the three nodes is master node whick deploys a MBean in scheduler-service.xml under server/default/deploy. the configeration is:



      true
      datang:name=SvrMon
      check
      NOW
      3000
      -1


      the MBean which was deployed in the master cluster node is:
      public class SvrMonService extends ServiceMBeanSupport implements SvrMonServiceMBean {
      public void check()
      public ArrayList getStatList(){ ...... }
      .......
      }

      and the SvrMonServiceMBean is:
      public interface SvrMonServiceMBean extends ServiceMBean{
      /**
      * return all MBean's state in the JbossAS
      * @return SvrBeanList
      */
      ArrayList getStateList();
      void check();

      }

      ServiceMBean and ServiceMBeanSupport are the classes in the jboss-system.jar or jbossall-client.jar.

      the client invoke code are here:
      public class RMIAdaptorHelper {

      private String _rmiHost = "127.0.0.1";
      private int _rmiPort = 1099;
      private int _timeout = 2000; // ms
      private RMIAdaptor _server;

      public RMIAdaptorHelper(String rmiHost, int rmiPort, int timeout)
      throws CommunicationException, NamingException
      {
      this._rmiHost = rmiHost;
      this._rmiPort = rmiPort;
      this._timeout = timeout;
      }

      /**
      * 远程调用 MBEAN
      */
      public Object invoke(RmiInvokeObject rmido) throws Exception
      {
      ObjectName name = new ObjectName(rmido.getJndiname());
      return _server.invoke(name, rmido.getMethodname(), rmido
      .getMethodArgs(), rmido.getMethodArgsType());
      }

      /**
      * 连接 Jboss rmi �务器
      */
      public void connect() throws NamingException, CommunicationException
      {
      Properties prop = System.getProperties();
      prop.put(Context.INITIAL_CONTEXT_FACTORY,
      "org.jnp.interfaces.NamingContextFactory");
      prop.put(Context.URL_PKG_PREFIXES,
      "org.jboss.naming:org.jnp.interfaces");
      prop.put(Context.PROVIDER_URL, "jnp://" + _rmiHost + ":" + _rmiPort);

      prop.put("jnp.discoveryTimeout", String.valueOf(_timeout));

      InitialContext ic = new InitialContext(prop);
      if (_server != null)
      {
      Logger.getLogger(getClass()).debug("RMIAdaptorHelper not null");
      _server = null;
      }
      Object o = ic.lookup("jmx/rmi/RMIAdaptor");
      _server = (RMIAdaptor) o;

      }
      }