1 Reply Latest reply on Dec 1, 2011 8:34 PM by amathewjboss1

    Web to App tier Remote call is taking more time in Jboss 5.1

    amathewjboss1
    amathewjboss1 Nov 29, 2011 1:50 AM

      Hi,

       

         We have an application running on Jboss 5.1 + JDK 1.6. In our application we use SLSB and we do remote calls as the enviroment is clustered with seperate web & app tiers. We are facing a big issue during high load where Jboss is not responding to the web tier - remote EJB "create" call.  Below is the thread dump which we took. In the thread dump we could see 100's of threads like below where web tier is waiting on the app tier to respond.

       

      "TP-Processor681" daemon prio=10 tid=0x61a37c00 nid=0x5a79 waiting for monitor entry [0x66764000..0x66765db0]

         java.lang.Thread.State: BLOCKED (on object monitor)

              at java.lang.Class.forName0(Native Method)

              at java.lang.Class.forName(Class.java:247)

              at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:154)

              at java.rmi.server.RMIClassLoader$2.loadClass(RMIClassLoader.java:620)

              at java.rmi.server.RMIClassLoader.loadClass(RMIClassLoader.java:247)

              at sun.rmi.server.MarshalInputStream.resolveClass(MarshalInputStream.java:197)

              at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)

              at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)

              at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1583)

              at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)

              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)

              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)

              at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)

              at org.jboss.invocation.jrmp.interfaces.JRMPInvokerProxyHA.readExternal(JRMPInvokerProxyHA.java:384)

              at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1792)

              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1751)

              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)

              at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)

              at java.util.HashMap.readObject(HashMap.java:1030)

              at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)

              at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)

              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)

              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)

              at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)

              at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)

              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)

              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)

              at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)

              at org.jboss.proxy.ClientContainer.readExternal(ClientContainer.java:158)

              at java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:1792)

              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1751)

              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)

              at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)

              at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)

              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)

              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)

              at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)

              at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)

              at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)

              at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)

              at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)

              at java.rmi.MarshalledObject.get(MarshalledObject.java:142)

              at org.jboss.invocation.jrmp.interfaces.JRMPInvokerProxyHA.invoke(JRMPInvokerProxyHA.java:228)

              at org.jboss.invocation.InvokerInterceptor.invokeInvoker(InvokerInterceptor.java:365)

              at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:197)

              at org.jboss.proxy.ejb.RetryInterceptor.invoke(RetryInterceptor.java:177)

              at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)

              at org.jboss.proxy.ejb.SecurityContextInterceptor.invoke(SecurityContextInterceptor.java:64)

              at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:68)

              at org.jboss.proxy.ejb.HomeInterceptor.invoke(HomeInterceptor.java:184)

              at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:101)

              at $Proxy14.create(Unknown Source)

              at com.myapp.testworker.MyAppWorkerWrapper.getRemoteReference(MyAppWorkerWrapper.java:467)

       

      Any thoughts on this please?

       

      Thanks

      Anil Mathew

      • 965 Views
      • Categories: JBoss AS 5.1
      • Tags:
        • 1. Re: Web to App tier Remote call is taking more time in Jboss 5.1
          amathewjboss1
          amathewjboss1 Dec 1, 2011 8:34 PM (in response to amathewjboss1)

          Hi,

           

             After couple of days of brainstroming, we found the issue and the culprit was usage of jrmpha invoker in the clustered stateless environment. We changed this to poolha and now we don't the above slowness we are seeing. It seems like JBoss 5.1 has some major changes in terms of using jrmpha compared to JBoss 4.x.

           

             One thing which we are noticing right now is when you use the poolha, what ever values we set in deploy/cluster/ha-legacy-jboss-beans.xml file for maxPoolSize/clientMaxPoolSize, it is reaching to that point as we get more loads.

           

                <property name="numAcceptThreads">1</property>
                <property name="maxPoolSize">400</property>
                <property name="clientMaxPoolSize">400</property>
                <property name="socketTimeout">60000</property>

           

             i.e if I set it to 400, it reaches 400 after one hour of high load. Any idea why it is doing this? Please help.

           

          Thanks

          Anil Mathew

           

           

           

            • Actions