[ranga033]

ok thank you for the prompt reply Tomaz !

[johanvermeij]

Maybe this is not the right thread but I try and try without result.

Using 7.1.0:

I can access the cluster using Apache httpd when I comment out ServerAdvertise (when I uncomment ServerAdvertise I don't find the cluster)

<IfModule manager_module>

  Listen 192.168.2.2:6666

  ManagerBalancerName other-server-group

  <VirtualHost 192.168.2.2:6666>

    <Location />

     Order deny,allow

     Deny from all

     Allow from all

    </Location>

    ErrorLog "logs/chatter_log"

    LogLevel debug

    KeepAliveTimeout 100

    MaxKeepAliveRequests 0

#   ServerAdvertise on http://127.0.0.1:6666

    AdvertiseFrequency 30

    #AdvertiseSecurityKey secret

    #AdvertiseGroup 224.0.1.105:23364

    EnableMCPMReceive

  </VirtualHost>

</IfModule>

</IfModule>

NameVirtualHost 192.168.2.3:8880

<VirtualHost 192.168.2.3:8880>

    ServerAdmin [e-mail]

    ServerName 192.168.2.3

    ErrorLog "logs/cluster_log"

    LogLevel debug

    ProxyPass '/beans/XMLTranServiceBean/XMLTranServiceBean' balancer://other-server-group stickysession=JSESSIONID|jsessionid nofailover=On

    ProxyPassReverse / balancer://other-server-group

    ProxyPreserveHost On

    <Location />

        Order deny,allow

        Allow from All

    </Location>

    <Location /mod_cluster_manager>

        SetHandler mod_cluster-manager

        Order deny,allow

        Deny from all

        Allow from all

    </Location>

</VirtualHost>

That works

I now would like to use 7.1.1 clustering.

When I use the same httpd.conf as above under 7.1.1 all I ever get is (I have tried everything I could think of for 2 days straight):

[Tue Aug 21 21:24:17 2012] [debug] mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED

[Tue Aug 21 21:24:17 2012] [error] proxy: CLUSTER: (balancer://other-server-group). All workers are in error state

[jfclere]

Does /mod_cluster_manager show the nodes?

[ranga033]

Let me know which version are you trying this ? Did you try on 7.1.1/7.1.2. ?

[johanvermeij]

I am now at the point where I can run master and slave instances but I cannot access the cluster (do transactions from a terminal) when I start up a slave.

As soon as I stop the slave I can do transactions again.

This is the output from mod_cluster_manager with the master running

mod_cluster/1.2.0.Final

Auto Refresh show DUMP output show INFO output

Node undefined (ajp://192.168.2.3:8259):

Enable Contexts Disable Contexts

Balancer: other-server-group,LBGroup: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 26,Ttl: 60000000,Status: OK,Elected: 1,Read: 9065,Transferred: 149,Connected: 0,Load: 100

Virtual Host 1:

Contexts:

/internetbanking, Status: ENABLED Request: 0 Disable /beans, Status: ENABLED Request: 0 Disable /wssimple, Status: ENABLED Request: 0 Disable 

Aliases:

localhost default-host example.com 

When I bring up a slave I get this intermittently (httpd is very busy stopping and starting workers)

mod_cluster/1.2.0.Final

Auto Refresh show DUMP output show INFO output

Node REMOVED (ajp://192.168.2.3:8259):

Enable Contexts Disable Contexts

Balancer: other-server-group,LBGroup: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 26,Ttl: 60000000,Status: NOTOK,Elected: 0,Read: 0,Transferred: 0,Connected: 0,Load: 100

modcluster settings in domain.xml are as follows (I have added the proxy-list and advertise-security-key this morning).

<subsystem xmlns="urn:jboss:domain:modcluster:1.0">

                <mod-cluster-config advertise-socket="modcluster" proxy-list="192.168.2.3:6666" advertise-security-key="secret">

                    <dynamic-load-provider>

                        <load-metric type="busyness"/>

                    </dynamic-load-provider>

                </mod-cluster-config>

            </subsystem>

As soon as I start a slave I get this error on both master and slave:

[Server:server-three-Linux] 12:50:48,703 ERROR [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Error [MEM: MEM: Can't read node: {4}] sending command STATUS to proxy 192.168.2.3/192.168.2.3:6666, configuration will be reset

I post my current httpd.conf settings as well (I copied again from 1.7.0 and this time things started working).

  Listen 192.168.2.3:6666

  ManagerBalancerName other-server-group

  <VirtualHost 192.168.2.3:6666>

    <Location />

     Order deny,allow

     Deny from all

     Allow from all

    </Location>

    ServerName vantagebanking.com

    ErrorLog "logs/chatter_log"

    LogLevel debug

    KeepAliveTimeout 100

    MaxKeepAliveRequests 0

    ServerAdvertise on http://127.0.0.1:6666

    AdvertiseFrequency 30

    AdvertiseSecurityKey secret

    AdvertiseGroup 224.0.1.105:23364

    EnableMCPMReceive

  </VirtualHost>

</IfModule>

NameVirtualHost 192.168.2.3:8000

<VirtualHost 192.168.2.3:8000>

    ServerAdmin [e-mail]

    ServerName vantagebanking.com

    ErrorLog "logs/cluster_log"

    LogLevel debug

    ProxyPass '/beans/XMLTranServiceBean/XMLTranServiceBean' balancer://other-server-group stickysession=JSESSIONID|jsessionid nofailover=On

    ProxyPassReverse / balancer://other-server-group

    ProxyPreserveHost On

    <Location />

        Order deny,allow

        Allow from All

    </Location>

    <Location /mod_cluster_manager>

        SetHandler mod_cluster-manager

        Order deny,allow

        Deny from all

        Allow from all

    </Location>

/VirtualHost>

Sorry, all a bit messy, for me there is a lot of trial and error in this.

I am amazed by JBoss clustering and I am determined to make it work

[johanvermeij]

I use 7.1.1

Is 7.1.2 out already ?

[ranga033]

Did you make the node names unique (specifying unique names for master and slave through host.xml) . If everything is fine, the mod_cluster_manager should show both the nodes in the paeg.

[ranga033]

7.1.2 is a tag for EAP release, it has to be built by downloading the JBoss source code. However for mod_cluster to work 7.1.1 should be fine.

[johanvermeij]

That is what I am looking for, I read that an instance-id can be added, also through the web console.

But I can't find where.

I suppose the instance-id can be set in this section of host.xml

Can you give me a hint ?

	<servers>
	<server name="server-one" group="main-server-group">
	<!-- Remote JPDA debugging for a specific server
	<jvm name="default">
	<jvm-options>
	<option value="-Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n"/>
	</jvm-options>
	</jvm>
	-->
	</server>
	<server name="server-two" group="main-server-group" auto-start="false">
	<!-- server-two avoids port conflicts by incrementing the ports in
	the default socket-group declared in the server-group -->
	<socket-bindings port-offset="150"/>
	</server>
	<server name="server-three-Linux" group="other-server-group" auto-start="true">
	<!-- server-three avoids port conflicts by incrementing the ports in
	the default socket-group declared in the server-group -->
	<socket-bindings port-offset="250"/>
	</server>
	</servers>

[jfclere]

the undefined is due to a bug... JBPAPP-8451?

[ranga033]

As mentioned in Jean-Frederic's post above try adding the workaround as you use 7.1.1

Try adding this in domain.xml 

   <subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="false">

Also, hope you have set the names for master and slave as per https://docs.jboss.org/author/display/AS71/AS7+Cluster+Howto

something like , <host name="slave" xmlns="urn:jboss:domain:1.2">

I hope this should solve your issue. Also go through the comments in  https://docs.jboss.org/author/display/AS71/AS7+Cluster+Howto . It has solution and workarounds for most of the issues for Cluster setup.

[johanvermeij]

Thank you very much, it all works fine

I am already running a jmeter test against a cluster with a HP Proliant, an iMac and a Linux PC

It is fast

[johanvermeij]

When I am running a stress test against the above cluster under 7.1.1 (domain mode) I get this error very often.

[Server:server-three-slave-HP] 18:46:25,182 ERROR [com.vantage.capital.service.xml.XMLTranServiceBean] (ajp--192.168.2.12-8259-22) DDF0001,CashTransaction:EXCP:org.hibernate.cache.CacheException: org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.hibernate.cache.CacheException: org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate

In fact, I don't need jmeter for this.

When I open 2 terminals and I do transactions quickly by hitting enter fast I get the same error.

Looking at the JBoss logs, netstat, Apache logs everything looks fine

[ranga033]

I am not too sure what kind of stress testing are you performing. If you can elaborate a bit, it would be better to understand the problem.

From your logs, I see that you are using Hibernate and perhaps the Infinispan cache for Hibernate. Are those caches up when you are performing a test ?  What kind of caches have you configured (any remote caches?)?

[johanvermeij]

I am testing a financial application.

I prepare jmeter jmx files using a sample soap message captured from a terminal.

I use the sample to propagate many other also simulating different users, bank branches, accounts etc.

I point the jmeter injector at Apache http on port 8000 (see httpd.conf below) using mod_cluster (latest)

I use one or more laptops as jmeter injectors

httpd then load balances this to a cluster which consist of a HP Proliant mini server (Centos 6 64 bit, 192.168.2.12), a PC (Centos 6 64 bit, 192.168.2.3) and an iMac (MacOs, 192.168.2.4)

I use an Oracle 11g database on the HP, httpd runs on the PC.

The stress test runs very fast when only the master is up (PC is the master)

As soon as I bring up a slave (and also when I bring up master and 2 slaves from the start) I get this error:

[Server:server-three-slave-HP] 19:28:03,650 ERROR [com.vantage.capital.service.xml.XMLTranServiceBean] (ajp--192.168.2.12-8259-146) DDF0002,CashWithdrawal:EXCP:org.hibernate.cache.CacheException: org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.hibernate.cache.CacheException: org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate

The above timeout happens on the PC and the HP.

It slows the test down to about 1TPS (by comparison I can do almost 100TPS against the HP in standalone mode)

The iMac has a different error:

[Server:server-three-slave-iMac] 19:30:22,593 WARNING [org.jgroups.protocols.UDP] (pool-23-thread-1) null: no physical address for 3cf4cecf-6754-af54-1c50-f2ba733aac93, dropping message

[Server:server-three-slave-iMac] 19:30:25,595 WARNING [org.jgroups.protocols.pbcast.GMS] (pool-23-thread-1) JOIN(slave1:server-three-slave-iMac/hibernate) sent to 3cf4cecf-6754-af54-1c50-f2ba733aac93 timed out (after 3000 ms), retrying

I have gone through the startup logs and everything looks fine there, there are no errors or warnings.

I don't expect you to fix this for me Rang, maybe you can give me a hint.

We have been working on this for almost a week now and we are almost there.

Problem is also that Googling these kinds of conditions is very "circular" and a lot of the discussions pertain to older JBoss releases.

I say it again, what you guys have done is incredible, no commercial software can come near you.

From httpd.conf

<IfModule manager_module>

  Listen 192.168.2.3:6666

  AllowDisplay On

  ManagerBalancerName other-server-group

  <VirtualHost 192.168.2.3:6666>

    <Location />

     Order deny,allow

     Deny from all

     Allow from all

    </Location>

    ServerName vantagebanking.com

    ErrorLog "logs/chatter_log"

    LogLevel debug

    KeepAliveTimeout 100

    MaxKeepAliveRequests 0

#   ServerAdvertise on http://192.168.2.3:6666

    AdvertiseFrequency 30

#   AdvertiseSecurityKey secret

#   AdvertiseGroup 224.0.1.105:23364

    EnableMCPMReceive

  </VirtualHost>

</IfModule>

NameVirtualHost 192.168.2.3:8000

<VirtualHost 192.168.2.3:8000>

    ServerAdmin [e-mail]

    ServerName vantagebanking.com

    ErrorLog "logs/cluster_log"

    LogLevel debug

    AllowDisplay On

    ProxyPass '/beans/XMLTranServiceBean/XMLTranServiceBean' balancer://other-server-group stickysession=JSESSIONID|jsessionid nofailover=On

    ProxyPassReverse / balancer://other-server-group

    ProxyPreserveHost On

    <Location />

        Order deny,allow

        Allow from All

    </Location>

    <Location /mod_cluster_manager>

        SetHandler mod_cluster-manager

        Order deny,allow

        Deny from all

        Allow from all

    </Location>

</VirtualHost>