5 Replies Latest reply on Nov 12, 2012 4:59 PM by efrinmn

    Host Controller should be less forgiving on management port

    b.eckenfels
    b.eckenfels Jun 27, 2012 11:39 AM

      On Windows in a domain setup it sometimes happens that the host controller process or a server is hanging around. Especially if there is some external method who does restart the process the components of the AS7 should be a bit smarter about duplicate starts. For the host controller for example a very reliable way to detect if it is started as the correct instance is the availability of the management port.

       

      Currently the host controller will start a server even when it was not able to open the management port. I think it should retr creating that port for some time, and if this does not succeed it should not proceed with starting the server(s) but signal the process controller to actually terminate. (Because this is the most reliable way to signal a service manager or cluster framework that something is wrong). Starting a server when you are not the host controller in charge is IMHO a totally wrong decision. Here is a sample log of such a incident:

       

      02:49:25,946 INFO  [org.jboss.as.process.Host Controller.status] (main) JBAS012017: Starting process 'Host Controller'
      [Host Controller] 02:49:27,358 Information [org.jboss.modules] (main) JBoss Modules version 1.1.1.GA
      [Host Controller] 02:49:27,970 INFO  [org.jboss.as] (MSC service thread 1-3) JBAS015899: JBoss AS 7.1.1.Final "Brontes" starting
      [Host Controller] 02:49:32,243 INFO  [org.jboss.as] (Controller Boot Thread) JBAS010902: Creating http management service using network interface (management) port (9990) securePort (-1)
      [Host Controller] 02:49:32,283 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC00001: Failed to start service jboss.remoting.server.management: org.jboss.msc.service.StartException in service jboss.remoting.server.management:

       

      JBAS017112: Address already in use: bind /127.0.0.1:9999
      [Host Controller]       at org.jboss.as.remoting.AbstractStreamServerService.start(AbstractStreamServerService.java:109) [jboss-as-remoting-7.1.1.Final.jar:7.1.1.Final]
      [Host Controller]       at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]

      [Host Controller]       at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
      [Host Controller]       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_05]
      [Host Controller]       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_05]
      [Host Controller]       at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_05]
      [Host Controller] Caused by: java.net.BindException: Address already in use: bind

      ...

      [Host Controller] 02:49:32,339 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-14) MSC00001: Failed to start service jboss.serverManagement.controller.management.http: org.jboss.msc.service.StartException in service jboss.serverManagement.controller.management.http: Address already in use: bind /127.0.0.1:9990
      ...

      [Host Controller] 02:49:32,410 INFO  [org.jboss.as.controller] (MSC service thread 1-14) JBAS014774: Service status report
      [Host Controller] JBAS014777:   Services which failed to start:

           service jboss.serverManagement.controller.management.http: org.jboss.msc.service.StartException in service jboss.serverManagement.controller.management.http: Address already in use: bind /127.0.0.1:9990
           service jboss.remoting.server.management: org.jboss.msc.service.StartException in service jboss.remoting.server.management: JBAS017112: Address already in use: bind /127.0.0.1:9999
      [Host Controller] 02:49:32,774 INFO  [org.jboss.as.host.controller] (Controller Boot Thread) JBAS010922: Starting server server-one
      02:49:32,814 INFO  [org.jboss.as.process.Server:server-one.status] (ProcessController-threads - 3) JBAS012017: Starting process 'Server:server-one'
      [Host Controller] 02:49:32,829 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:9990
      [Host Controller] 02:49:32,831 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS 7.1.1.Final "Brontes" (Host Controller) started in 6580ms - Started 11 of 11 services (0 services are passive or on-demand)

      [Server:server-one] 02:49:47,306 INFO  [org.jboss.as] (MSC service thread 1-3) JBAS015899: JBoss AS 7.1.1.Final "Brontes" starting
      [Server:server-one] 02:49:47,637 INFO  [org.jboss.remoting] (MSC service thread 1-10) JBoss Remoting version 3.2.3.GA
      [Server:server-one] 02:49:49,955 ERROR [org.jboss.remoting.remote.connection] (Remoting "master:server-one:MA) JBREM000200: Remote connection failed: javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed


      Note also (red) that the host controller is reporting wrongly that the admin console is started and it also reports that 11 services are running. There is also a missleading JAAS Exception in JBoss Remoting in case of IOExceptions, I will discuss that in a seperate report.

       

      Do you agree that the host controller should treat errors on the management port as fatal (after configurable retries). I think it could ignore a startup failure on the http port, but then again it might be better to take this as a warning sign, too.