3 Replies Latest reply on Jan 13, 2014 11:37 AM by Christian Sadilek

    What might cause this error? Running Errai on scalable openshift

    Anton Hughes Master

      Hi

       

      We are spending a lot of time trying to get Errai to run on a scalable Jboss EAP on OpenShift.

       

      We had our app running for about 2min, then we get a 503 error.

       

      Below is what is in the logs. What might cause this?

       

      2014/01/07 05:39:04,336 INFO  [org.apache.tomcat.util] (http-127.9.155.1/127.9.155.1:8080-Acceptor-0) JBWEB003008: Maximum number of threads (64) created for connector with address 127.9.155.1/127.9.155.1 and port 8080

      2014/01/07 06:05:24,876 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/].[ErraiServlet]] (http-127.9.155.1/127.9.155.1:8080-6) JBWEB000236: Servlet.service() for servlet ErraiServlet threw exception: java.net.SocketException: Connection reset

        at java.net.SocketInputStream.read(SocketInputStream.java:196) [rt.jar:1.7.0_45]

        at java.net.SocketInputStream.read(SocketInputStream.java:122) [rt.jar:1.7.0_45]

        at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:737) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:767) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:116) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at org.apache.coyote.http11.InternalInputBuffer.doRead(InternalInputBuffer.java:697) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at org.apache.coyote.Request.doRead(Request.java:438) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:341) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:422) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:373) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:193) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283) [rt.jar:1.7.0_45]

        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325) [rt.jar:1.7.0_45]

        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177) [rt.jar:1.7.0_45]

        at java.io.InputStreamReader.read(InputStreamReader.java:184) [rt.jar:1.7.0_45]

        at java.io.BufferedReader.fill(BufferedReader.java:154) [rt.jar:1.7.0_45]

        at java.io.BufferedReader.read1(BufferedReader.java:205) [rt.jar:1.7.0_45]

        at java.io.BufferedReader.read(BufferedReader.java:279) [rt.jar:1.7.0_45]

        at java.io.Reader.read(Reader.java:100) [rt.jar:1.7.0_45]

        at org.jboss.errai.marshalling.server.JSONStreamDecoder.read(JSONStreamDecoder.java:86) [errai-marshalling-2.4.3.Final.jar:2.4.3.Final]

        at org.jboss.errai.marshalling.server.JSONStreamDecoder._parse(JSONStreamDecoder.java:107) [errai-marshalling-2.4.3.Final.jar:2.4.3.Final]

        at org.jboss.errai.marshalling.server.JSONStreamDecoder._parse(JSONStreamDecoder.java:114) [errai-marshalling-2.4.3.Final.jar:2.4.3.Final]

        at org.jboss.errai.marshalling.server.JSONStreamDecoder._parse(JSONStreamDecoder.java:114) [errai-marshalling-2.4.3.Final.jar:2.4.3.Final]

        at org.jboss.errai.marshalling.server.JSONStreamDecoder._parse(JSONStreamDecoder.java:114) [errai-marshalling-2.4.3.Final.jar:2.4.3.Final]

        at org.jboss.errai.marshalling.server.JSONStreamDecoder._parse(JSONStreamDecoder.java:114) [errai-marshalling-2.4.3.Final.jar:2.4.3.Final]

        at org.jboss.errai.marshalling.server.JSONStreamDecoder._parse(JSONStreamDecoder.java:110) [errai-marshalling-2.4.3.Final.jar:2.4.3.Final]

        at org.jboss.errai.marshalling.server.JSONStreamDecoder.parse(JSONStreamDecoder.java:97) [errai-marshalling-2.4.3.Final.jar:2.4.3.Final]

        at org.jboss.errai.marshalling.server.JSONStreamDecoder.decode(JSONStreamDecoder.java:74) [errai-marshalling-2.4.3.Final.jar:2.4.3.Final]

        at org.jboss.errai.bus.server.io.MessageFactory.createCommandMessage(MessageFactory.java:92) [errai-bus-2.4.3.Final.jar:2.4.3.Final]

        at org.jboss.errai.bus.server.servlet.DefaultBlockingServlet.doPost(DefaultBlockingServlet.java:140) [errai-bus-2.4.3.Final.jar:2.4.3.Final]

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-1.jar:1.0.2.Final-redhat-1]

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-1.jar:1.0.2.Final-redhat-1]

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) [shiro-web-1.2.2.jar:1.2.2]

        at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) [shiro-web-1.2.2.jar:1.2.2]

        at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) [shiro-core-1.2.2.jar:1.2.2]

        at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) [shiro-core-1.2.2.jar:1.2.2]

        at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) [shiro-core-1.2.2.jar:1.2.2]

        at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) [shiro-web-1.2.2.jar:1.2.2]

        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) [shiro-web-1.2.2.jar:1.2.2]

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]

        at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]

        at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

        at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]

        • 1. Re: What might cause this error? Running Errai on scalable openshift
          Anton Hughes Master

          I tried to switch on clustering - as we are attempting to run the app on a jboss HA, to see if that would help, and now I get the following error.

           

          How do we resolve this error?

           

           

          2014/01/07 17:40:12,940 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/].[ErraiServlet]] (http-127.9.155.1/127.9.155.1:8080-48) JBWEB000235: Allocate exception for servlet ErraiServlet: java.net.BindException: Permission denied

              at java.net.PlainDatagramSocketImpl.bind0(Native Method) [rt.jar:1.7.0_45]

              at java.net.AbstractPlainDatagramSocketImpl.bind(AbstractPlainDatagramSocketImpl.java:95) [rt.jar:1.7.0_45]

              at java.net.DatagramSocket.bind(DatagramSocket.java:376) [rt.jar:1.7.0_45]

              at java.net.MulticastSocket.<init>(MulticastSocket.java:172) [rt.jar:1.7.0_45]

              at java.net.MulticastSocket.<init>(MulticastSocket.java:137) [rt.jar:1.7.0_45]

              at org.jgroups.util.DefaultSocketFactory.createMulticastSocket(DefaultSocketFactory.java:70) [jgroups-3.2.0.Final.jar:3.2.0.Final]

              at org.jgroups.util.Util.createMulticastSocket(Util.java:3443) [jgroups-3.2.0.Final.jar:3.2.0.Final]

              at org.jgroups.protocols.UDP.createSockets(UDP.java:350) [jgroups-3.2.0.Final.jar:3.2.0.Final]

              at org.jgroups.protocols.UDP.start(UDP.java:225) [jgroups-3.2.0.Final.jar:3.2.0.Final]

              at org.jgroups.stack.ProtocolStack.startStack(ProtocolStack.java:962) [jgroups-3.2.0.Final.jar:3.2.0.Final]

              at org.jgroups.JChannel.startStack(JChannel.java:811) [jgroups-3.2.0.Final.jar:3.2.0.Final]

              at org.jgroups.JChannel._preConnect(JChannel.java:510) [jgroups-3.2.0.Final.jar:3.2.0.Final]

              at org.jgroups.JChannel.connect(JChannel.java:280) [jgroups-3.2.0.Final.jar:3.2.0.Final]

              at org.jgroups.JChannel.connect(JChannel.java:271) [jgroups-3.2.0.Final.jar:3.2.0.Final]

              at org.jboss.errai.bus.server.cluster.jgroups.JGroupsClusteringProvider.<init>(JGroupsClusteringProvider.java:82) [errai-bus-2.4.3.Final.jar:2.4.3.Final]

              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [rt.jar:1.7.0_45]

              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) [rt.jar:1.7.0_45]

              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [rt.jar:1.7.0_45]

              at java.lang.reflect.Constructor.newInstance(Constructor.java:526) [rt.jar:1.7.0_45]

              at com.google.inject.internal.DefaultConstructionProxyFactory$2.newInstance(DefaultConstructionProxyFactory.java:85) [guice-3.0.jar:]

              at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) [guice-3.0.jar:]

              at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) [guice-3.0.jar:]

              at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978) [guice-3.0.jar:]

              at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) [guice-3.0.jar:]

              at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974) [guice-3.0.jar:]

              at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1013) [guice-3.0.jar:]

              at org.jboss.errai.bus.server.ServerMessageBusImpl.<init>(ServerMessageBusImpl.java:219) [errai-bus-2.4.3.Final.jar:2.4.3.Final]

              at org.jboss.errai.bus.server.service.ErraiServiceImpl.<init>(ErraiServiceImpl.java:61) [errai-bus-2.4.3.Final.jar:2.4.3.Final]

              at org.jboss.errai.bus.server.service.ErraiServiceImpl$$FastClassByGuice$$7879947c.newInstance(<generated>) [guice-3.0.jar:2.4.3.Final]

              at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) [guice-3.0.jar:]

              at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:60) [guice-3.0.jar:]

              at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) [guice-3.0.jar:]

              at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) [guice-3.0.jar:]

              at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) [guice-3.0.jar:]

              at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) [guice-3.0.jar:]

              at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [guice-3.0.jar:]

              at com.google.inject.Scopes$1$1.get(Scopes.java:65) [guice-3.0.jar:]

              at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) [guice-3.0.jar:]

              at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:54) [guice-3.0.jar:]

              at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978) [guice-3.0.jar:]

              at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1024) [guice-3.0.jar:]

              at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:974) [guice-3.0.jar:]

              at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1013) [guice-3.0.jar:]

              at org.jboss.errai.bus.server.service.ErraiServiceFactory.create(ErraiServiceFactory.java:30) [errai-bus-2.4.3.Final.jar:2.4.3.Final]

              at org.jboss.errai.bus.server.service.ErraiServiceSingleton.initSingleton(ErraiServiceSingleton.java:39) [errai-bus-2.4.3.Final.jar:2.4.3.Final]

              at org.jboss.errai.bus.server.servlet.ServletBootstrapUtil.initService(ServletBootstrapUtil.java:110) [errai-bus-2.4.3.Final.jar:2.4.3.Final]

              at org.jboss.errai.bus.server.servlet.ServletBootstrapUtil.getService(ServletBootstrapUtil.java:61) [errai-bus-2.4.3.Final.jar:2.4.3.Final]

              at org.jboss.errai.bus.server.servlet.AbstractErraiServlet.init(AbstractErraiServlet.java:86) [errai-bus-2.4.3.Final.jar:2.4.3.Final]

              at org.jboss.errai.bus.server.servlet.DefaultBlockingServlet.init(DefaultBlockingServlet.java:88) [errai-bus-2.4.3.Final.jar:2.4.3.Final]

              at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1194) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:950) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:151) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]

              at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]

              at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.2.1.Final-redhat-10.jar:7.2.1.Final-redhat-10]

              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]

              at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]

          • 2. Re: What might cause this error? Running Errai on scalable openshift
            Anton Hughes Master

            csa, Max suggested I talk to you about this. Do you have any insight?

             

            Thanks

            • 3. Re: What might cause this error? Running Errai on scalable openshift
              Christian Sadilek Master

              Hi,

               

              The first problem you described means that you hit the configured thread limit in your app server. See here for how to configure this in AS7:

              https://community.jboss.org/thread/199845 (please note that there also resource limitations in case you're using a small gear on Openshift: https://www.openshift.com/faq)

               

              The second problem is described here: https://www.openshift.com/kb/kb-e1038-i-cant-bind-to-a-port

               

              Basically, Openshift doesn't allow for binding to arbitrary ports and the default JGroups configuration doesn't work (https://www.openshift.com/forums/openshift/jgroup-port-in-openshift). You should be able to configure an alternative JGroups protocol stack using environment variables that are available on your Openshift instance.

               

              For Errai's JGroupClusteringProvider, you should be able to configure an alternative JGroups protocol stack using the property errai.jgroups.protocol_stack in your ErraiApp.properties.

               

              Cheers,

              Christian