5 Replies Latest reply on Dec 5, 2013 6:01 AM by pathduck

    Server 4.9; waiting for ever for initalization of Remoting

    pathduck

      Hi,

      after installing 4.9 from scratch (I had problems with the upgrade), the server won't start properly and no login window is shown. The RHQ logo flashes for an instant when going to coregui, then just a blank page (which actually is just JS output). Even after several days, it never finishes this.

       

      The following is logged constantly in server.log, I assume this is every time a running Agent tries to connect to the server:

       

      09:41:48,845 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/jboss-remoting-servlet-invoker].[ServerInvokerServlet]] (http-/0.0.0.0:7080-6) JBWEB000235: Allocate exception for servlet ServerInvokerServlet: javax.servlet.ServletException: Could not find the remoting servlet invoker: jboss.remoting:service=invoker,rhq.communications.connector.rhqtype=server,* - need to wait for remoting to be initialized later
              at org.rhq.enterprise.communications.servlet.ServerInvokerServlet.getInvokerFromInvokerName(ServerInvokerServlet.java:78) [classes:]
              at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.init(ServerInvokerServlet.java:78) [jboss-remoting-2.5.4.SP4.jar:]
              at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) [:1.7.0_40]
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_40]
              at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_40]
              at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:263) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
              at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:261) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
              at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.7.0_40]
              at javax.security.auth.Subject.doAsPrivileged(Subject.java:536) [rt.jar:1.7.0_40]
              at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:295) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
              at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:155) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
              at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:108) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
              at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1188) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
              at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:950) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:151) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
              at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
              at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
              at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.2.0.Alpha1-redhat-4.jar:7.2.0.Alpha1-redhat-4]
              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
              at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
              at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
              at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.0.Final.jar:7.2.0.Final]
              at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_40]
      
      
      
      

       

      The DB upgrade process during install went fine, can't find any errors in the other related logs.

       

      I've seen this before in earlier installs, but it always goes away after a few minutes when remoting is done with init.

       

      Our platform is:

      Red Hat Enterprise Linux Server release 6.4 (Santiago)

      Linux d26apvl007.test.local 2.6.32-358.14.1.el6.x86_64 #1 SMP Mon Jun 17 15:54:20 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux

      Running on Vmware x86_64, 2 cpu cores, 8GB ram, RHQ server set to 2GB max heap.

       

      java version "1.7.0_40"

      Java(TM) SE Runtime Environment (build 1.7.0_40-b43)

      Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, mixed mode)

       

      Attaching logs as well as server.log with DEBUG (about 90 megs).

       

      Would really appreciate if someone could have a look; I've been thinking of just waiting for 4.10 but it might take a while, and I've been struggling with this install for a long time now.

       

      cheers,

      Stian

        • 1. Re: Server 4.9; waiting for ever for initalization of Remoting
          pathduck

          Just FYI, in case someone else has this same problem:

           

          It was caused by a misconfiguration in the table rhq_storage_node, apparently values of CTIME and MTIME was set to zero. It was fixed by inserting a row with ID=1 and CTIME/MTIME set to '121212', and deleting the faulty row with ID=10001

           

          The error was actually in an earliger EJBException that stated the error could be solved by reinstalling the Storage Node, that I did not notice the first time around.

          • 2. Re: Server 4.9; waiting for ever for initalization of Remoting
            jeryma

            hi,can you show me the postgreps command for this ? thanks very much.

             

            my  ctime | mtime isn't zero,but I still have this show page problem.Question about RHQ visiting to http://localhost:7080

            rhq=# select * from rhq_storage_node;

              id  |  address  | cql_port | operation_mode |     ctime     |     mtime     | error_msg | maintenance_pending | resource_id | resource_op_hist_id

            ------+-----------+----------+----------------+---------------+---------------+-----------+---------------------+-------------+---------------------

            1001 | 127.0.0.1 |     9142 | NORMAL         | 1386065213789 | 1386065213789 |           | f                   |       10002 |                   

            (1 row)

            • 3. Re: Server 4.9; waiting for ever for initalization of Remoting
              pathduck

              Hi - for me it looks like this:

               

              ID ADDRESSCQL_PORT OPERATION_MODE CTIME  MTIME  RESOURCE_ID MAINTENANCE_PENDING ERROR_MSG RESOURCE_OP_HIST_ID

              -- ---------- -------- -------------- ------ ------ ----------- ------------------- --------- -------------------

              1  10.51.9.38 9142 NORMAL     121212 121212 21334   0               (null)(null)        

               

              Maybe you could try setting the value of 'id' to '1' - since if you only have one storage node (I assume?)

               

              How you do it, it's up to you, we run Oracle but I'm sure it's a simple 'update x where y' sql.

               

              hth,

              Stian

              • 4. Re: Server 4.9; waiting for ever for initalization of Remoting
                jeryma

                thanks,I will try to use oracle database

                • 5. Re: Server 4.9; waiting for ever for initalization of Remoting
                  pathduck

                  xiaojie Ma skrev:

                   

                  thanks,I will try to use oracle database

                   

                  You misunderstand - it is not because you use Oracle or Postgresql, it seems to be caused by error in the data that is in the table rhq_storage_node.

                   

                  Stian