JBoss Cluster: Service Temporarily Unavailable

[codergeek]

I am using JBoss 6.1.0 Alpha in domain mode (cluster with 2 servers on 2 physical machines) and the mod_cluster binary bundle 1.2.0 Final. The setup is as follows:

Physical machine 1: JBoss server 1 which also acts as the domain controller and the Apache webserver.

Physical machine 2: JBoss server 2.

When I start up everything and deploy a distributable webapp, everything is fine. Communication between the 2 JBoss servers works and in the mod_cluster console I can see that the Apache server also sees both JBoss servers. The webapp starts on JBoss server 2 (which is not the domain controller) and when I shutdown server 2 the session is replicated and executed on server 1 as expected. Problems arise, when I start server 2 again and shutdown server 1 (the domain controller). I can see in the console that the Apache webserver still sees server 2 but the call to the webapp is not routed to this server. Instead the following message is shown:

Service Temporarily Unavailable. The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

The same thing happens in the following scenario with only one server  (the domain controller): Apache is running, the domain controller is running => everything is fine. Domain controller is shut down and restarted => same message as above.

It looks like there is a problem when the domain controller goes down. Is there a way to solve this issue?

[jfclere]

I don't think that is a domain problem, could you test in reverse order? Do you get the same?

[codergeek]

Hello,

I have tried it the other way around: When the application starts on the domain controller (server 1) and the domain controller is shut down, then the session is replicated to server 2. When I restart the domain controller and shut down server 2, I get the same message again.

I have also tried this in standalone mode and it is working perfectly - no matter which and how often a standalone server is shut down and restarted, the session is always transferred to the other server.

http.conf:

<IfModule manager_module>

  Listen vm-server-1:6666

  ManagerBalancerName cluster-server-group

  <VirtualHost vm-server-1:6666>

    <Location />

     Order deny,allow

     Deny from all

     Allow from all

    </Location>

    KeepAliveTimeout 60

    MaxKeepAliveRequests 0

    #ServerAdvertise on http://127.0.0.1:6666

    AdvertiseFrequency 5

    #AdvertiseSecurityKey secret

    #AdvertiseGroup 224.0.1.105:23364

    EnableMCPMReceive

    <Location /mod_cluster_manager>

       SetHandler mod_cluster-manager

       Order deny,allow

       Deny from all

       Allow from all

    </Location>

  </VirtualHost>

</IfModule>

Server-side mod_cluster configuration in the domain.xml (ha-profile) of the domain controller:

<subsystem xmlns="urn:jboss:domain:modcluster:1.1">
  <mod-cluster-config advertise-socket="modcluster" connector="ajp">
  <dynamic-load-provider>
  <load-metric type="busyness"/>
  </dynamic-load-provider>
  </mod-cluster-config>
</subsystem>

Command to start the domain controller:

domain -b vm-server-1 -Djboss.bind.address.management=vm-server-1

Command to start the host controller:

domain -b vm-server-2 -Djboss.domain.master.address=vm-server-1 --backup

[jfclere]

What do you see in /mod_cluster_manager when you have the problem?

Does the error lasts for ever once you run the test or does it disappear after ~ 5 minutes?

[codergeek]

The output is for the scenario in my first post: Application starts on server 2 - server 2 shut down => session is replicated to server 1 (domain controller) - server 2 is restarted and server 1 is shut down => "Service Temporarily Unavailabe" message.

I have waited around 10 min and also tried it with new sessions from a different browser, but the problem persists.

The mod_cluster manager output:

mod_cluster/1.2.0.Final

Auto Refresh show DUMP output show INFO output

Node 9eb1c69b-1a04-316f-967a-17c5a6debc08 (ajp://vm-server-2:8009):

Enable Contexts Disable Contexts

Balancer: cluster-server-group,LBGroup: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 65,Ttl: 60000000,Status: OK,Elected: 0,Read: 0,Transferred: 0,Connected: 0,Load: 100

Virtual Host 1:

Contexts:

/ClusterTest, Status: ENABLED Request: 0 Disable

Aliases:

example.com

localhost

default-host

When I restart server 1 again, I get:

mod_cluster/1.2.0.Final

Auto Refresh show DUMP output show INFO output

Node be1d0b16-48a1-3e07-b57e-4a592910b722 (ajp://vm-server-1:8009):

Enable Contexts Disable Contexts

Balancer: cluster-server-group,LBGroup: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 65,Ttl: 60000000,Status: OK,Elected: 0,Read: 0,Transferred: 0,Connected: 0,Load: 100

Virtual Host 1:

Contexts:

/ClusterTest, Status: ENABLED Request: 0 Disable

Aliases:

example.com

localhost

default-host

Node 9eb1c69b-1a04-316f-967a-17c5a6debc08 (ajp://vm-server-2:8009):

Enable Contexts Disable Contexts

Balancer: cluster-server-group,LBGroup: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 65,Ttl: 60000000,Status: OK,Elected: 0,Read: 0,Transferred: 0,Connected: 0,Load: 100

Virtual Host 1:

Contexts:

/ClusterTest, Status: ENABLED Request: 0 Disable

Aliases:

example.com

localhost

default-host

Nevertheless, the problem still persists, even if I wait 10 min again and trying with new sessions from differnt browsers. I can only solve this by restarting httpd. Then the application is executed again.

In the httpd error log, I get:

[Thu Sep 26 10:01:35 2013] [notice] Advertise initialized for process 2228

[Thu Sep 26 10:01:35 2013] [notice] Apache/2.2.21 (Win64) mod_cluster/1.2.0.Final configured -- resuming normal operations

[Thu Sep 26 10:01:35 2013] [notice] Server built: Feb  9 2012 22:51:47

[Thu Sep 26 10:01:35 2013] [notice] Parent: Created child process 2244

[Thu Sep 26 10:01:35 2013] [notice] Child 2244: Child process is running

[Thu Sep 26 10:01:35 2013] [notice] Child 2244: Acquired the start mutex.

[Thu Sep 26 10:01:35 2013] [notice] Child 2244: Starting 64 worker threads.

[Thu Sep 26 10:01:35 2013] [notice] Child 2244: Starting thread to listen on port 6666.

[Thu Sep 26 10:01:35 2013] [notice] Child 2244: Starting thread to listen on port 8000.

[Thu Sep 26 10:06:06 2013] [error] [client 10.0.0.6] File does not exist: C:/JBossCluster/httpd-2.2/htdocs/favicon.ico

[Thu Sep 26 10:09:24 2013] [error] proxy: CLUSTER: (balancer://cluster-server-group). All workers are in error state

[Thu Sep 26 10:12:24 2013] [error] proxy: CLUSTER: (balancer://cluster-server-group). All workers are in error state

[Thu Sep 26 10:12:35 2013] [error] proxy: CLUSTER: (balancer://cluster-server-group). All workers are in error state

[Thu Sep 26 10:12:47 2013] [error] proxy: CLUSTER: (balancer://cluster-server-group). All workers are in error state

[Thu Sep 26 10:17:49 2013] [error] proxy: CLUSTER: (balancer://cluster-server-group). All workers are in error state

[Thu Sep 26 10:17:56 2013] [error] proxy: CLUSTER: (balancer://cluster-server-group). All workers are in error state

[Thu Sep 26 10:18:07 2013] [error] proxy: CLUSTER: (balancer://cluster-server-group). All workers are in error state

[erasmomarciano]

Are you sure that this command it's work?

domain -b vm-server-

Why you have disabled

#AdvertiseGroup 224.0.1.105:23364

[codergeek]

Hello,

yes, "domain -b vm-server-1" works, since I can start the JBoss Admin Console via "http://vm-server-1:9990/console". If I do not use the -b option, the server is bound to localhost and I can only start the console with "http://localhost:9990/console".

I use the mod_cluster distribution version 1.2.0 which contains the Apache webserver and the corresponding mod_cluster modules. The httpd.conf file is also preconfigured and "#AdvertiseGroup 224.0.1.105:23364" was already commented by default. Besides it was also commented or omitted in every tutorial that I have seen. Nevertheless, I removed the comment and tried again, but the result was still the same.

[erasmomarciano]

Hi

you check if this 224.0.1.105 23364 is up on the node Jboss

[codergeek]

Yes, the configuration on JBoss side is

<socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>

[jfclere]

Elected: 0,Read: 0,Transferred: 0,Connected: 0

that looks fuzzy... If you do a fail over test you should have something in one of nodes, what are you testing?

[codergeek]

Hello,

I tried this with the evaluation version of Red Hat Webserver which uses mod_cluster 1.2.4 and it works perfectly. So it seems that this is a bug of mod_cluster 1.2.0 that has been resolved in later versions.

I have some related questions: Why is mod_cluster 1.2.0 the last version that has a binary distribution? Do I Have to build mod_cluster from source, if I want a higher version? And if so, is there a tutorial how to do this for Windows binaries?

Thank's for your help!

[jfclere]

The binary that can be downloaded in the .org are produced by the developers we can't do that for all version we are releasing.

If you don't have a subscription you need to download the sources and compile them your self. That is easy on Linux but not on Windows sorry.

[codergeek]

I understand. Thank's for your help!