12 Replies Latest reply on Sep 30, 2013 7:49 AM by codergeek

    JBoss Cluster: Service Temporarily Unavailable

    codergeek

      I am using JBoss 6.1.0 Alpha in domain mode (cluster with 2 servers on 2 physical machines) and the mod_cluster binary bundle 1.2.0 Final. The setup is as follows:

      Physical machine 1: JBoss server 1 which also acts as the domain controller and the Apache webserver.

      Physical machine 2: JBoss server 2.

       

      When I start up everything and deploy a distributable webapp, everything is fine. Communication between the 2 JBoss servers works and in the mod_cluster console I can see that the Apache server also sees both JBoss servers. The webapp starts on JBoss server 2 (which is not the domain controller) and when I shutdown server 2 the session is replicated and executed on server 1 as expected. Problems arise, when I start server 2 again and shutdown server 1 (the domain controller). I can see in the console that the Apache webserver still sees server 2 but the call to the webapp is not routed to this server. Instead the following message is shown:

       

      Service Temporarily Unavailable. The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

       

       

      The same thing happens in the following scenario with only one server  (the domain controller): Apache is running, the domain controller is running => everything is fine. Domain controller is shut down and restarted => same message as above.

       

      It looks like there is a problem when the domain controller goes down. Is there a way to solve this issue?

        • 1. Re: JBoss Cluster: Service Temporarily Unavailable
          jfclere

          I don't think that is a domain problem, could you test in reverse order? Do you get the same?

          • 2. Re: Re: JBoss Cluster: Service Temporarily Unavailable
            codergeek

            Hello,

            I have tried it the other way around: When the application starts on the domain controller (server 1) and the domain controller is shut down, then the session is replicated to server 2. When I restart the domain controller and shut down server 2, I get the same message again.

            I have also tried this in standalone mode and it is working perfectly - no matter which and how often a standalone server is shut down and restarted, the session is always transferred to the other server.

             

            http.conf:

            <IfModule manager_module>

              Listen vm-server-1:6666

              ManagerBalancerName cluster-server-group

              <VirtualHost vm-server-1:6666>

                <Location />

                 Order deny,allow

                 Deny from all

                 Allow from all

                </Location>

             

             

                KeepAliveTimeout 60

                MaxKeepAliveRequests 0

                #ServerAdvertise on http://127.0.0.1:6666

                AdvertiseFrequency 5

                #AdvertiseSecurityKey secret

                #AdvertiseGroup 224.0.1.105:23364

                EnableMCPMReceive

             

             

                <Location /mod_cluster_manager>

                   SetHandler mod_cluster-manager

                   Order deny,allow

                   Deny from all

                   Allow from all

                </Location>

             

             

              </VirtualHost>

            </IfModule>

             

             

            Server-side mod_cluster configuration in the domain.xml (ha-profile) of the domain controller:

            <subsystem xmlns="urn:jboss:domain:modcluster:1.1">
              <mod-cluster-config advertise-socket="modcluster" connector="ajp">
              <dynamic-load-provider>
              <load-metric type="busyness"/>
              </dynamic-load-provider>
              </mod-cluster-config>
            </subsystem>
            

             

             

            Command to start the domain controller:

            domain -b vm-server-1 -Djboss.bind.address.management=vm-server-1

             

            Command to start the host controller:

            domain -b vm-server-2 -Djboss.domain.master.address=vm-server-1 --backup

            • 3. Re: Re: JBoss Cluster: Service Temporarily Unavailable
              jfclere

              What do you see in /mod_cluster_manager when you have the problem?

              Does the error lasts for ever once you run the test or does it disappear after ~ 5 minutes?

              • 4. Re: Re: Re: JBoss Cluster: Service Temporarily Unavailable
                codergeek

                The output is for the scenario in my first post: Application starts on server 2 - server 2 shut down => session is replicated to server 1 (domain controller) - server 2 is restarted and server 1 is shut down => "Service Temporarily Unavailabe" message.

                I have waited around 10 min and also tried it with new sessions from a different browser, but the problem persists.

                 

                The mod_cluster manager output:

                mod_cluster/1.2.0.Final

                Auto Refresh show DUMP output show INFO output

                Node 9eb1c69b-1a04-316f-967a-17c5a6debc08 (ajp://vm-server-2:8009):

                Enable Contexts Disable Contexts

                Balancer: cluster-server-group,LBGroup: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 65,Ttl: 60000000,Status: OK,Elected: 0,Read: 0,Transferred: 0,Connected: 0,Load: 100

                Virtual Host 1:

                Contexts:

                 

                 

                /ClusterTest, Status: ENABLED Request: 0 Disable

                 

                 

                Aliases:

                 

                 

                example.com

                localhost

                default-host

                 

                When I restart server 1 again, I get:

                 

                mod_cluster/1.2.0.Final

                Auto Refresh show DUMP output show INFO output

                Node be1d0b16-48a1-3e07-b57e-4a592910b722 (ajp://vm-server-1:8009):

                Enable Contexts Disable Contexts

                Balancer: cluster-server-group,LBGroup: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 65,Ttl: 60000000,Status: OK,Elected: 0,Read: 0,Transferred: 0,Connected: 0,Load: 100

                Virtual Host 1:

                Contexts:

                 

                 

                /ClusterTest, Status: ENABLED Request: 0 Disable

                 

                 

                Aliases:

                 

                 

                example.com

                localhost

                default-host

                 

                 

                 

                Node 9eb1c69b-1a04-316f-967a-17c5a6debc08 (ajp://vm-server-2:8009):

                Enable Contexts Disable Contexts

                Balancer: cluster-server-group,LBGroup: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 65,Ttl: 60000000,Status: OK,Elected: 0,Read: 0,Transferred: 0,Connected: 0,Load: 100

                Virtual Host 1:

                Contexts:

                 

                 

                /ClusterTest, Status: ENABLED Request: 0 Disable

                 

                 

                Aliases:

                 

                 

                example.com

                localhost

                default-host

                Nevertheless, the problem still persists, even if I wait 10 min again and trying with new sessions from differnt browsers. I can only solve this by restarting httpd. Then the application is executed again.

                 

                In the httpd error log, I get:

                [Thu Sep 26 10:01:35 2013] [notice] Advertise initialized for process 2228

                [Thu Sep 26 10:01:35 2013] [notice] Apache/2.2.21 (Win64) mod_cluster/1.2.0.Final configured -- resuming normal operations

                [Thu Sep 26 10:01:35 2013] [notice] Server built: Feb  9 2012 22:51:47

                [Thu Sep 26 10:01:35 2013] [notice] Parent: Created child process 2244

                [Thu Sep 26 10:01:35 2013] [notice] Child 2244: Child process is running

                [Thu Sep 26 10:01:35 2013] [notice] Child 2244: Acquired the start mutex.

                [Thu Sep 26 10:01:35 2013] [notice] Child 2244: Starting 64 worker threads.

                [Thu Sep 26 10:01:35 2013] [notice] Child 2244: Starting thread to listen on port 6666.

                [Thu Sep 26 10:01:35 2013] [notice] Child 2244: Starting thread to listen on port 8000.

                [Thu Sep 26 10:06:06 2013] [error] [client 10.0.0.6] File does not exist: C:/JBossCluster/httpd-2.2/htdocs/favicon.ico

                [Thu Sep 26 10:09:24 2013] [error] proxy: CLUSTER: (balancer://cluster-server-group). All workers are in error state

                [Thu Sep 26 10:12:24 2013] [error] proxy: CLUSTER: (balancer://cluster-server-group). All workers are in error state

                [Thu Sep 26 10:12:35 2013] [error] proxy: CLUSTER: (balancer://cluster-server-group). All workers are in error state

                [Thu Sep 26 10:12:47 2013] [error] proxy: CLUSTER: (balancer://cluster-server-group). All workers are in error state

                [Thu Sep 26 10:17:49 2013] [error] proxy: CLUSTER: (balancer://cluster-server-group). All workers are in error state

                [Thu Sep 26 10:17:56 2013] [error] proxy: CLUSTER: (balancer://cluster-server-group). All workers are in error state

                [Thu Sep 26 10:18:07 2013] [error] proxy: CLUSTER: (balancer://cluster-server-group). All workers are in error state

                • 5. Re: JBoss Cluster: Service Temporarily Unavailable
                  erasmomarciano

                  Are you sure that this command it's work?

                  domain -b vm-server-

                   

                  Why you have disabled

                  #AdvertiseGroup 224.0.1.105:23364

                  • 6. Re: Re: JBoss Cluster: Service Temporarily Unavailable
                    codergeek

                    Hello,

                     

                    yes, "domain -b vm-server-1" works, since I can start the JBoss Admin Console via "http://vm-server-1:9990/console". If I do not use the -b option, the server is bound to localhost and I can only start the console with "http://localhost:9990/console".

                     

                    I use the mod_cluster distribution version 1.2.0 which contains the Apache webserver and the corresponding mod_cluster modules. The httpd.conf file is also preconfigured and "#AdvertiseGroup 224.0.1.105:23364" was already commented by default. Besides it was also commented or omitted in every tutorial that I have seen. Nevertheless, I removed the comment and tried again, but the result was still the same.

                    • 7. Re: Re: JBoss Cluster: Service Temporarily Unavailable
                      erasmomarciano

                      Hi

                      you check if this 224.0.1.105 23364 is up on the node Jboss

                      • 8. Re: Re: Re: JBoss Cluster: Service Temporarily Unavailable
                        codergeek

                        Yes, the configuration on JBoss side is

                        <socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>

                        • 9. Re: Re: Re: JBoss Cluster: Service Temporarily Unavailable
                          jfclere

                          Elected: 0,Read: 0,Transferred: 0,Connected: 0

                          that looks fuzzy... If you do a fail over test you should have something in one of nodes, what are you testing?

                          • 10. Re: Re: Re: Re: JBoss Cluster: Service Temporarily Unavailable
                            codergeek

                            Hello,

                             

                            I tried this with the evaluation version of Red Hat Webserver which uses mod_cluster 1.2.4 and it works perfectly. So it seems that this is a bug of mod_cluster 1.2.0 that has been resolved in later versions.

                             

                            I have some related questions: Why is mod_cluster 1.2.0 the last version that has a binary distribution? Do I Have to build mod_cluster from source, if I want a higher version? And if so, is there a tutorial how to do this for Windows binaries?

                             

                            Thank's for your help!

                            • 11. Re: JBoss Cluster: Service Temporarily Unavailable
                              jfclere

                              The binary that can be downloaded in the .org are produced by the developers we can't do that for all version we are releasing.

                              If you don't have a subscription you need to download the sources and compile them your self. That is easy on Linux but not on Windows sorry.

                              1 of 1 people found this helpful
                              • 12. Re: JBoss Cluster: Service Temporarily Unavailable
                                codergeek

                                I understand. Thank's for your help!