JBOSS AS 7.1.1 Clustering with Appache HTTPD Server

[suwath123]

I observerd the following while doing test. please let me if you have solution to this. I setup domain in a single PC (no slave configuration) to test clustering with appache httpd.

A). to cluster to be correctly operate, socket group name of the

server group where the profile name=”full-ha”,must be “full-ha-sockets”.The bare minimum requirement is “ha-sockets”though it doesn’t correctly operate the “failover” concept. (if one server is down it will stop responding to user requests and the other servers in the cluster will not serve the requests). Additionally server start with errors (dependency issues) as follows.

JBAS014775: New missing/unsatisfied dependencies:

service jboss.binding.jacorb (missing) dependents: [service jboss.jacorb.orb-service]

service jboss.binding.jacorb-ssl (missing) dependents: [service jboss.jacorb.orb-service]

service jboss.binding.messaging (missing) dependents: [service jboss.messaging.default]

service jboss.binding.messaging-throughput (missing) dependents: [service jboss.messaging.default]

B). To operate on “full-ha-sockets”,you cannot “auto-start” more than one server in the server group. If you do, it will create a deadlock situation.

ERROR [org.hornetq.core.protocol.core.impl.HornetQPacketHandler] (Old I/O server worker (parentId: 18599604, [id: 0x011bceb4, localhost/127.0.0.1:5695])) Failed to create session : HornetQException[errorCode=105 message=Unable to validate user: HORNETQ.CLUSTER.ADMIN.USER]

at org.hornetq.core.security.impl.SecurityStoreImpl.authenticate(SecurityStoreImpl.java:126) [hornetq-core-2.2.13.Final.jar:]

at org.hornetq.core.server.impl.HornetQServerImpl.createSession(HornetQServerImpl.java:807) [hornetq-core-2.2.13.Final.jar:]

at org.hornetq.core.protocol.core.impl.HornetQPacketHandler.handleCreateSession(HornetQPacketHandler.java:187) [hornetq-core-2.2.13.Final.jar:]

at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) [netty-3.2.6.Final.jar:]

C) Even after you start the server and try to start other servers in the group through admin console, it will enter to deadlock state.

However difference between “full-ha-sockets”and “ha-sockets” configuration is “full-ha-sockets” configuration still operate and serve the user request in this situation. (Server which is auto started at boot, will continuously serve the request unless you stop this server, and keep other in the same server group).

[rhusar]

Hi

Are you trying to get HornetQ to cluster or general AS (web / ejb session) clustering? If you are not using messaging just remove that subsystem, otherwise configure it properly - Unable to validate user: HORNETQ.CLUSTER.ADMIN.USER -- seems like you havent. I think I saw this issue, everything deadlocked when the system was misconfigured but I didnt investigate further as its just a usability issue.

Rado

[suwath123]

No I am doing general web clusturing.  I commented the mdb-hornetq-ra refrerence and hornetQ subsystem. this solves the deadlock issue. i guess both mod-cluster and hornetQ clustering was competing before. if i have ejb project with messaging do we need to configure both? and how its done?  However one of my orginal issues stands still-  failover does not work. I think user session is stick to one server in the group , even if it is shutdown session will not pass over to the next in the cluster.

[suwath123]

I carried out further experiments to "fail-over" concept and found httpd server delivered from jboss, randomly supports it. I have first boot up three servers in the cluster , with "server-two" and "server-three" in other-server-group which is on "full-ha" profile, full-ha socket group. server-two responded initally to web request. then i shutdown server two using jboss console. now server console reported it has obtaned new cluster view  as follows;

[Server:server-three] 11:00:24,556 INFO  [org.infinispan.remoting.transport.jgro
ups.JGroupsTransport] (Incoming-8,null) ISPN000094: Received new cluster view: [master:server-three/web|2] [master:server-three/web]
[Server:server-two] 11:00:24,556 INFO  [org.jboss.as] (MSC service thread 1-3) JBAS015950: JBoss AS 7.1.1.Final "Brontes" stopped in 896ms11:00:24,806 INFO  [org.jboss.as.process.Server:server-two.status] (reaper for S
erver:server-two) JBAS012010: Process 'Server:server-two' finished with an exit
status of 0

Now server-three started responding to my web requests. (in this case fail-over works).

Again I brought back the server-two. still it respoded to my web requests by server-three.

Then I stoped server-three and for following web requests I got the error bellow.

But if I restart HTTPD it will pickup the server-two and respond to my web requests.

Because this is a random behaviour we may not observe it on Appache HTTPD orginal installation.

I am just wondering its something to do with "instance-id" configured in "web" subsystem (without it clustering will not work) . However I think there is an issue in mod_cluster_proxy module.

Please let me know whats the value for "nofailover" attribute, if I use following default(minimal) configuration suggested by JBOSS. I cannot see anywhere in this document describing how "fail-over" works.

(http://docs.jboss.org/mod_cluster/1.2.0/html/native.config.html)

LoadModule proxy_module modules/mod_proxy.so
 LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
 LoadModule slotmem_module modules/mod_slotmem.so
 LoadModule manager_module modules/mod_manager.so
 LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
 LoadModule advertise_module modules/mod_advertise.so
 
 Listen 10.33.144.3:6666
 <VirtualHost 10.33.144.3:6666>
 
   <Location />
      Order deny,allow
      Deny from all
      Allow from 10.33.144.
   </Location>
 
 KeepAliveTimeout 60
 MaxKeepAliveRequests 0
 
 ManagerBalancerName mycluster
 ServerAdvertise On
 EnableMCPMReceive
 
 </VirtualHost>

[suwath123]

I have now configured workers.properties for both JBOSS HTTPD server and Standard Appache HTTPD Server. I h

I have added following line to the file.

worker.balancer.sticky-session=false

"sticky-session" attribute is also sugested for proxy-configuration in

http://docs.jboss.org/mod_cluster/1.2.0/html/java.properties.html

to make the sessiion replication , it was required to use "tcp" instead of "udp" in jgroup subsystem. therefore i have tweak the configuration as follows;

<subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="tcp">

<stack name="tcp">

<transport type="TCP" socket-binding="jgroups-tcp" diagnostics-socket-binding="jgroups-diagnostics"/>

<protocol type="TCPPING">

<property name="timeout">

10000

</property>

<property name="initial_hosts">

127.0.0.1[8230],127.0.0.1[8330]

</property>

<property name="port_range">

1

</property>

<property name="num_initial_members">

2

</property>

</protocol>

</stack>

</subsystem>

this time I thought "fail-over" became certain thing for both servers(it was random on JBOSS HTTPD server). However it only works for first time. (ex: boot two servers for the clustre, stop one who is serving request, then other server in the cluster will be able to serve the requests).

after the stopped server is started again, it is attached to cluster but the session was not sucessfully transfered. (I suspected "session replication" is not taking place in jboss AS side, or load balancer is not handling "fail-over" and "load-balancing" consistently)

if we stop the server who is currently serving request, and thereafter for all web requests you will end up with "service temporary unavailable" error.

I think this proves additional configuration / fixes required to mod_cluster_proxy module or its not visible to users.

Digging further, to see consistency of "session replication" , "fail-over" and "load-balancing", I have done further observations as follows;

lets say, I boot up the cluster  c where c={server-two, server-three}

then i had web request http://localhost/cluster-demo-master

which is sucessfully served by server-two, as shown in the console.

then i shur down server-two , now new cluster c', is c'={server-three}.

then i had web request http://localhost/cluster-demo-master

which is sucessfully served by server-three, as shown in the console.

following is the logging extract from server-three. which proves infinispan and jgroup working properly, replicating the session, which obtained from web cache container and cache repl.

then i had web request http://localhost/cluster-demo-master

which is still sucessfully served by server-three, as shown in the console.

09:11:34,378 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-3) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
09:11:34,378 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
09:11:34,425 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-14-thread-1) ISPN000078: Starting JGroups Channel
09:11:34,440 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-14-thread-1) ISPN000094: Received new cluster view: [master:server-three/web|0] [master:server-three/web]
09:11:34,440 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-14-thread-1) ISPN000079: Cache local address is master:server-three/web, physical addresses are [127.0.0.1:7850]
09:11:34,440 INFO  [org.infinispan.factories.GlobalComponentRegistry] (pool-14-thread-1) ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.2.FINAL
09:11:34,440 INFO  [org.infinispan.config.ConfigurationValidatingVisitor] (pool-14-thread-1) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
09:11:34,550 INFO  [org.infinispan.jmx.CacheJmxRegistration] (pool-14-thread-1) ISPN000031: MBeans were successfully registered to the platform mbean server.
09:11:34,565 INFO  [org.jboss.as.clustering.infinispan] (pool-14-thread-1) JBAS010281: Started repl cache from web container
09:11:34,581 INFO  [org.jboss.as.clustering.impl.CoreGroupCommunicationService.web] (MSC service thread 1-3) JBAS010206: Number of cluster members: 1
09:11:34,612 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
09:11:34,612 INFO  [org.infinispan.config.ConfigurationValidatingVisitor] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
09:11:34,628 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-2) ISPN000031: MBeans were successfully registered to the platform mbean server.
09:11:34,628 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) JBAS010281: Started //default-host//cluster-demo-master cache from web container
09:11:34,644 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
09:11:34,644 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
09:11:34,737 INFO  [org.jboss.web] (MSC service thread 1-2) JBAS018210: Registering web context: /cluster-demo-master
09:11:36,628 INFO  [org.jboss.modcluster.ModClusterService] (MSC service thread 1-4) Initializing mod_cluster 1.2.0.Final
09:11:36,659 INFO  [org.jboss.modcluster.advertise.impl.AdvertiseListenerImpl] (MSC service thread 1-4) Listening to proxy advertisements on 224.0.1.105:23364
09:11:36,690 INFO  [org.jboss.as.server] (Controller Boot Thread) JBAS018559: Deployed "cluster-demo-master.war"
09:11:36,737 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS 7.1.1.Final "Brontes" started in 7890ms - Started 193 of 324 services (130 services are passive or on-demand)

I start the server-two again, now my new cluster c'', where c''={server-three,server-two}.

server-two log extract is as follows (similar to server-three above). session replication is happening

09:34:26,824 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-11-thread-1) ISPN000078: Starting JGroups Channel
09:34:26,839 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-11-thread-1) ISPN000094: Received new cluster view: [master:server-two/web|0] [master:server-two/web]
09:34:26,839 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-11-thread-1) ISPN000079: Cache local address is master:server-two/web, physical addresses are [127.0.0.1:7750]
09:34:26,839 INFO  [org.infinispan.factories.GlobalComponentRegistry] (pool-11-thread-1) ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.2.FINAL
09:34:26,855 INFO  [org.infinispan.config.ConfigurationValidatingVisitor] (pool-11-thread-1) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
09:34:26,964 INFO  [org.infinispan.jmx.CacheJmxRegistration] (pool-11-thread-1) ISPN000031: MBeans were successfully registered to the platform mbean server.
09:34:26,980 INFO  [org.jboss.as.clustering.infinispan] (pool-11-thread-1) JBAS010281: Started repl cache from web container
09:34:26,996 INFO  [org.jboss.as.clustering.impl.CoreGroupCommunicationService.web] (MSC service thread 1-2) JBAS010206: Number of cluster members: 1
09:34:27,043 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-1) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
09:34:27,043 INFO  [org.infinispan.config.ConfigurationValidatingVisitor] (MSC service thread 1-1) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
09:34:27,058 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-1) ISPN000031: MBeans were successfully registered to the platform mbean server.
09:34:27,058 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-1) JBAS010281: Started //default-host//cluster-demo-master cache from web container
09:34:27,074 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-1) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
09:34:27,089 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-1) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
09:34:27,199 INFO  [org.jboss.web] (MSC service thread 1-1) JBAS018210: Registering web context: /cluster-demo-master
09:34:29,011 INFO  [org.jboss.modcluster.ModClusterService] (MSC service thread 1-4) Initializing mod_cluster 1.2.0.Final
09:34:29,027 INFO  [org.jboss.modcluster.advertise.impl.AdvertiseListenerImpl] (MSC service thread 1-4) Listening to proxy advertisements on 224.0.1.105:23364
09:34:29,043 INFO  [org.jboss.as.server] (Controller Boot Thread) JBAS018559: Deployed "cluster-demo-master.war"
09:34:29,043 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS 7.1.1.Final "Brontes" started in 7828ms - Started 193 of 324 services (130 services are passive or on-demand)

then i had web request http://localhost/cluster-demo-master

which is still sucessfully served by server-three, as shown in the console.(here the order doent matter, as i observed doing this test number of times using two browsers. in some cases server-two takes the lead). which means "session-failover" and "load-balancing" working perfectly untill now.

to prove further, that the session replication is working, now i stoped server-three (current request-serving server). now my new cluster c''' is c'''={server-two}

then i had web request http://localhost/cluster-demo-master

"Service Temporary Unavailable" page appears as discribed in my previous post.

now I shutdown apache HTTPD server, then server-two try coonecting to apache proxy via socket channel fails. log extract is given bellow.

09:50:15,188 INFO  [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) IO error sending command STATUS to proxy localhost/127.0.0.1:10001: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.6.0_21]
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333) [rt.jar:1.6.0_21]
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195) [rt.jar:1.6.0_21]
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182) [rt.jar:1.6.0_21]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) [rt.jar:1.6.0_21]
at java.net.Socket.connect(Socket.java:529) [rt.jar:1.6.0_21]
at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler$Proxy.getConnection(DefaultMCMPHandler.java:812)
at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler$Proxy.getConnectionWriter(DefaultMCMPHandler.java:834)
at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:479)
at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:580)
at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:409)
at org.jboss.modcluster.ModClusterService.status(ModClusterService.java:468)
at org.jboss.modcluster.container.catalina.CatalinaEventHandlerAdapter.lifecycleEvent(CatalinaEventHandlerAdapter.java:239)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:115) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1337) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1601) [jbossweb-7.0.13.Final.jar:]
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1590) [jbossweb-7.0.13.Final.jar:]
at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_21]

this proves server-two become the active server when server-three is absent and it also has the current session. However mod_cluster configuration at apache side did not respond well in server-shutdown-startup, which may not recognise or were not communicated to apache by jboss AS properly or still session sticky ness causing issues. How can we over come this issue? any fixes in new version or workaround to this issue?

[suwath123]

Is there any workaround you suggest to this issue or do we need to wait for a fix? As we are planing to migrate from weblogic to Jboss AS 7.1.1 its a critical decision to make. I have configured clusters for weblogic on apache before and had no issues. If you follow my investigations , in previous posts, you wil realize its something to do with load-balancer which fail to consistently load-balance and fail-over. i have mod-cluster 1.0 subsystem on Jboss server side and mod-cluster 1.2.0.Final on apache side.

[mpgong]

Suwath,

Your issues sound very similar to the one I'm experiencing as well.  We are on 7.1 final and mod_cluster 1.1.3.  Still no resolution or work around for us but have you uncovered something since your last post that might have resolved this.

I was thinking of upgrading to newer versions of everything to see if things work but was hoping it could be a config issue.