6 Replies Latest reply on May 3, 2011 12:52 AM by dinukadev

    java.lang.RuntimeException: Unreachable?: Service unavailable

    dinukadev
    dinukadev Apr 29, 2011 2:12 AM

      Hi guys,

       

      This is my first ever topic im posting here. We are having a major issue with our production servers. Let me explain our architecture so you can get an idea of what could be wrong. We have three web servers and two app servers. All servers are running jboss 4.2.3 GA. Communication between the web and app is through ejb3 remoting.

       

      We have an issue with production servers when the number of remote connections reach above 100 then we get a  java.lang.RuntimeException: Unreachable?: Service unavailable error and the web server is unable to find the app server and application hands.

       

      Any help anyone can give regarding this is highly apprecaited.

       

       

       

      Cheers,

       

      Dinuka

        • 1. java.lang.RuntimeException: Unreachable?: Service unavailable
          wdfink
          wdfink Apr 29, 2011 5:53 AM (in response to dinukadev)

          Could you attach the error?

          Where you see the error? At the Web-server (as client) or the EJB server?

           

          What you mean by 'reach 100 conections'? Is it 100conn. from one web-server to one EJB?

           

          BTW, is there a special reason to separate web and ejb? I think it might better if the WEB instance have local access to the EJB's for a better performance.

            • Actions
          • 2. Re: java.lang.RuntimeException: Unreachable?: Service unavailable
            dinukadev
            dinukadev Apr 29, 2011 6:39 AM (in response to wdfink)

            Hi Fink,

             

            Thank you for your reply. This error actually comes in the WEB layer saying it cant find the app servers. Our app servers are clustered. This is our architecture and its a bit hard to change it right now. I totally agree with you on performance. What we want is to load balance and fail over our app layer by separating it. We have three web servers and two app servers. The error is coming from within the web layer. I have attached the log file here with. Pls any help is much appreciated;

             

            2011-04-28 11:22:40,560 ERROR [com.AvailabilityRH] java.lang.RuntimeException: Unreachable?: Service unavailable.

            2011-04-28 11:22:40,562 ERROR [STDERR] java.lang.RuntimeException: Unreachable?: Service unavailable.

            2011-04-28 11:22:40,562 ERROR [STDERR]  at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:176)

            2011-04-28 11:22:40,562 ERROR [STDERR]  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)

            2011-04-28 11:22:40,562 ERROR [STDERR]  at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:67)

            2011-04-28 11:22:40,562 ERROR [STDERR]  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)

            2011-04-28 11:22:40,562 ERROR [STDERR]  at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:53)

            2011-04-28 11:22:40,562 ERROR [STDERR]  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)

            2011-04-28 11:22:40,562 ERROR [STDERR]  at org.jboss.ejb3.remoting.ClusteredIsLocalInterceptor.invoke(ClusteredIsLocalInterceptor.java:55)

            2011-04-28 11:22:40,562 ERROR [STDERR]  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)

            2011-04-28 11:22:40,562 ERROR [STDERR]  at org.jboss.ejb3.stateless.StatelessClusteredProxy.invoke(StatelessClusteredProxy.java:112)

            2011-04-28 11:22:40,562 ERROR [STDERR]  at $Proxy72.checkFlightAvailability(Unknown Source)

            2011-04-28 11:22:40,562 ERROR [STDERR]  at AvailabilityRH.getAvailabilityOnLoad(AvailabilityRH.java:914)

            2011-04-28 11:22:40,561 ERROR [com.AvailabilityRH] java.lang.RuntimeException: Unreachable?: Service unavailable.

            2011-04-28 11:22:40,562 ERROR [STDERR]  at AvailabilityRH.setDisplayData(AvailabilityRH.java:303)

            2011-04-28 11:22:40,562 ERROR [STDERR]  at AvailabilityRH.execute(AvailabilityRH.java:127)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.ShowAvailabilityAction.execute(ShowAvailabilityAction.java:41)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at sun.reflect.GeneratedMethodAccessor518.invoke(Unknown Source)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at java.lang.reflect.Method.invoke(Method.java:597)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation.invokeAction(DefaultActionInvocation.java:404)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation.invokeActionOnly(DefaultActionInvocation.java:267)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:229)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:221)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:150)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.doIntercept(AnnotationValidationInterceptor.java:48)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.intercept(ConversionErrorInterceptor.java:123)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:167)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.interceptor.StaticParametersInterceptor.intercept(StaticParametersInterceptor.java:105)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:83)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

            2011-04-28 11:22:40,563 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at org.apache.struts2.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:207)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.interceptor.ModelDrivenInterceptor.intercept(ModelDrivenInterceptor.java:74)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.interceptor.ScopedModelDrivenInterceptor.intercept(ScopedModelDrivenInterceptor.java:127)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at org.apache.struts2.interceptor.ProfilingActivationInterceptor.intercept(ProfilingActivationInterceptor.java:107)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at org.apache.struts2.interceptor.debugging.DebuggingInterceptor.intercept(DebuggingInterceptor.java:206)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.interceptor.ChainingInterceptor.intercept(ChainingInterceptor.java:115)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:143)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:121)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)

            2011-04-28 11:22:40,564 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:170)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at com.opensymphony.xwork2.interceptor.AliasInterceptor.intercept(AliasInterceptor.java:123)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:176)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at org.apache.struts2.impl.StrutsActionProxy.execute(StrutsActionProxy.java:50)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:504)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at org.apache.struts2.dispatcher.FilterDispatcher.doFilter(FilterDispatcher.java:419)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)

            2011-04-28 11:22:40,565 ERROR [STDERR]  at java.lang.Thread.run(Thread.java:619)

            2011-04-28 11:22:40,566 ERROR [com.AvailabilityRH] Saving Performance Staistics Failed

            2011-04-28 11:22:40,567 ERROR [STDERR] java.lang.RuntimeException: Unreachable?: Service unavailable.

            2011-04-28 11:22:40,568 ERROR [STDERR]  at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:176)

            2011-04-28 11:22:40,568 ERROR [STDERR]  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)

            2011-04-28 11:22:40,568 ERROR [STDERR]  at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:67)

            2011-04-28 11:22:40,568 ERROR [STDERR]  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)

            2011-04-28 11:22:40,568 ERROR [STDERR]  at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:53)

            2011-04-28 11:22:40,568 ERROR [STDERR]  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)

            2011-04-28 11:22:40,568 ERROR [STDERR]  at org.jboss.ejb3.remoting.ClusteredIsLocalInterceptor.invoke(ClusteredIsLocalInterceptor.java:55)

            2011-04-28 11:22:40,568 ERROR [STDERR]  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)

            2011-04-28 11:22:40,568 ERROR [STDERR]  at org.jboss.ejb3.stateless.StatelessClusteredProxy.invoke(StatelessClusteredProxy.java:112)

            2011-04-28 11:22:40,568 ERROR [STDERR]  at $Proxy72.checkFlightAvailability(Unknown Source)

              • Actions
            • 3. java.lang.RuntimeException: Unreachable?: Service unavailable
              dinukadev
              dinukadev Apr 29, 2011 6:27 AM (in response to wdfink)

              100 connections meaning 100 concurrent users accessing the web which is load balanced. From within the web layer we access remote EJBs. This is the error we get when 100 users access concurrently. It works fine for less than that.

                • Actions
              • 4. Re: java.lang.RuntimeException: Unreachable?: Service unavailable
                wdfink
                wdfink Apr 29, 2011 7:59 AM (in response to dinukadev)

                I suppose it depends to the ejb3 remoting configuration.

                See server/.../deploy/ejb3.deployer/META-INF/jboss-service.xml

                 

                You will find a commented section for the remoting. I suppose it depends to numAcceptThreads or clientMaxPoolSize. AFAIK clientMaxPoolSize default is 50 which will match your 100 concurent accesses spit for two server

                 

                Remember you have to change it within the EJB-server configuration

                  • Actions
                • 5. Re: java.lang.RuntimeException: Unreachable?: Service unavailable
                  dinukadev
                  dinukadev Apr 29, 2011 10:35 AM (in response to wdfink)

                  Hi Wolf,

                   

                  Thank you for your prompt reply. Unfortunately this did not solve the issue as well. We still get the same error. Stumped for an aswer .. Thx for your co-operation in this matter. If you do have any more ideas which you think might work pls do leave a reply whenever your free.

                   

                   

                  Cheers

                   

                  Dinuka

                    • Actions
                  • 6. Re: java.lang.RuntimeException: Unreachable?: Service unavailable
                    dinukadev
                    dinukadev May 3, 2011 12:52 AM (in response to dinukadev)

                    Hi Guys,

                     

                    Just in case anyone else will face this problem in the future. One of our architects in the company found the answer. The issue was we were getting the EJB references through Spring JNDI Object Factory Bean. As these are singletone references if by any chance the remote stubs get invalidated at any point it cannot reconnect. We tried making the scop to prototype but that did not work as well. So going back to basics we initialized through IniticalContext without going through Spring. And now everything works fine no matter what the load is. Note that this issue only arise if you are doing remote calls to beans.

                     

                    Hope this helps anyone else who might face this problem.

                     

                    Cheers

                      • Actions