1 2 3 Previous Next 35 Replies Latest reply on Aug 25, 2012 10:06 AM by ranga033 Go to original post
      • 15. Re: mod_cluster 1.2 + JBossAS 7.1 problem
        ranga033

        ok thank you for the prompt reply Tomaz !

        • 16. Re: mod_cluster 1.2 + JBossAS 7.1 problem
          johanvermeij

          Maybe this is not the right thread but I try and try without result.

           

          Using 7.1.0:

          I can access the cluster using Apache httpd when I comment out ServerAdvertise (when I uncomment ServerAdvertise I don't find the cluster)

          <IfModule manager_module>

            Listen 192.168.2.2:6666

            ManagerBalancerName other-server-group

            <VirtualHost 192.168.2.2:6666>

              <Location />

               Order deny,allow

               Deny from all

               Allow from all

              </Location>

           

              ErrorLog "logs/chatter_log"

              LogLevel debug

              KeepAliveTimeout 100

              MaxKeepAliveRequests 0

          #   ServerAdvertise on http://127.0.0.1:6666

              AdvertiseFrequency 30

              #AdvertiseSecurityKey secret

              #AdvertiseGroup 224.0.1.105:23364

              EnableMCPMReceive

           

            </VirtualHost>

          </IfModule>

          </IfModule>

          NameVirtualHost 192.168.2.3:8880

          <VirtualHost 192.168.2.3:8880>

              ServerAdmin johanvermeij@gmail.com

              ServerName 192.168.2.3

              ErrorLog "logs/cluster_log"

              LogLevel debug

           

              ProxyPass '/beans/XMLTranServiceBean/XMLTranServiceBean' balancer://other-server-group stickysession=JSESSIONID|jsessionid nofailover=On

              ProxyPassReverse / balancer://other-server-group

              ProxyPreserveHost On

           

              <Location />

                  Order deny,allow

                  Allow from All

              </Location>

           

              <Location /mod_cluster_manager>

                  SetHandler mod_cluster-manager

                  Order deny,allow

                  Deny from all

                  Allow from all

              </Location>

          </VirtualHost>

           

          That works

           

          I now would like to use 7.1.1 clustering.

           

          When I use the same httpd.conf as above under 7.1.1 all I ever get is (I have tried everything I could think of for 2 days straight):

          [Tue Aug 21 21:24:17 2012] [debug] mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED

          [Tue Aug 21 21:24:17 2012] [error] proxy: CLUSTER: (balancer://other-server-group). All workers are in error state

          • 17. Re: mod_cluster 1.2 + JBossAS 7.1 problem
            jfclere

            Does /mod_cluster_manager show the nodes?

            • 18. Re: mod_cluster 1.2 + JBossAS 7.1 problem
              ranga033

              Let me know which version are you trying this ? Did you try on 7.1.1/7.1.2. ?

              • 19. Re: mod_cluster 1.2 + JBossAS 7.1 problem
                johanvermeij

                I am now at the point where I can run master and slave instances but I cannot access the cluster (do transactions from a terminal) when I start up a slave.

                As soon as I stop the slave I can do transactions again.

                 

                This is the output from mod_cluster_manager with the master running

                mod_cluster/1.2.0.Final

                Auto Refresh show DUMP output show INFO output

                Node undefined (ajp://192.168.2.3:8259):

                Enable Contexts Disable Contexts

                Balancer: other-server-group,LBGroup: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 26,Ttl: 60000000,Status: OK,Elected: 1,Read: 9065,Transferred: 149,Connected: 0,Load: 100

                Virtual Host 1:

                Contexts:

                /internetbanking, Status: ENABLED Request: 0 Disable /beans, Status: ENABLED Request: 0 Disable /wssimple, Status: ENABLED Request: 0 Disable 

                Aliases:

                localhost default-host example.com 

                When I bring up a slave I get this intermittently (httpd is very busy stopping and starting workers)

                mod_cluster/1.2.0.Final

                Auto Refresh show DUMP output show INFO output

                Node REMOVED (ajp://192.168.2.3:8259):

                Enable Contexts Disable Contexts

                Balancer: other-server-group,LBGroup: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 26,Ttl: 60000000,Status: NOTOK,Elected: 0,Read: 0,Transferred: 0,Connected: 0,Load: 100

                 

                modcluster settings in domain.xml are as follows (I have added the proxy-list and advertise-security-key this morning).

                 

                <subsystem xmlns="urn:jboss:domain:modcluster:1.0">

                                <mod-cluster-config advertise-socket="modcluster" proxy-list="192.168.2.3:6666" advertise-security-key="secret">

                                    <dynamic-load-provider>

                                        <load-metric type="busyness"/>

                                    </dynamic-load-provider>

                                </mod-cluster-config>

                            </subsystem>

                 

                 

                As soon as I start a slave I get this error on both master and slave:

                [Server:server-three-Linux] 12:50:48,703 ERROR [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Error [MEM: MEM: Can't read node: {4}] sending command STATUS to proxy 192.168.2.3/192.168.2.3:6666, configuration will be reset

                 

                I post my current httpd.conf settings as well (I copied again from 1.7.0 and this time things started working).

                 

                  Listen 192.168.2.3:6666

                  ManagerBalancerName other-server-group

                  <VirtualHost 192.168.2.3:6666>

                    <Location />

                     Order deny,allow

                     Deny from all

                     Allow from all

                    </Location>

                 

                    ServerName vantagebanking.com

                    ErrorLog "logs/chatter_log"

                    LogLevel debug

                    KeepAliveTimeout 100

                    MaxKeepAliveRequests 0

                    ServerAdvertise on http://127.0.0.1:6666

                    AdvertiseFrequency 30

                    AdvertiseSecurityKey secret

                    AdvertiseGroup 224.0.1.105:23364

                    EnableMCPMReceive

                 

                  </VirtualHost>

                </IfModule>

                 

                NameVirtualHost 192.168.2.3:8000

                <VirtualHost 192.168.2.3:8000>

                    ServerAdmin johanvermeij@gmail.com

                    ServerName vantagebanking.com

                    ErrorLog "logs/cluster_log"

                    LogLevel debug

                 

                    ProxyPass '/beans/XMLTranServiceBean/XMLTranServiceBean' balancer://other-server-group stickysession=JSESSIONID|jsessionid nofailover=On

                    ProxyPassReverse / balancer://other-server-group

                    ProxyPreserveHost On

                 

                    <Location />

                        Order deny,allow

                        Allow from All

                    </Location>

                 

                    <Location /mod_cluster_manager>

                        SetHandler mod_cluster-manager

                        Order deny,allow

                        Deny from all

                        Allow from all

                    </Location>

                 

                /VirtualHost>

                 

                Sorry, all a bit messy, for me there is a lot of trial and error in this.

                 

                I am amazed by JBoss clustering and I am determined to make it work

                • 20. Re: mod_cluster 1.2 + JBossAS 7.1 problem
                  johanvermeij

                  I use 7.1.1

                   

                  Is 7.1.2 out already ?

                  • 21. Re: mod_cluster 1.2 + JBossAS 7.1 problem
                    ranga033

                    Did you make the node names unique (specifying unique names for master and slave through host.xml) . If everything is fine, the mod_cluster_manager should show both the nodes in the paeg.

                    • 22. Re: mod_cluster 1.2 + JBossAS 7.1 problem
                      ranga033

                      7.1.2 is a tag for EAP release, it has to be built by downloading the JBoss source code. However for mod_cluster to work 7.1.1 should be fine.

                      • 23. Re: mod_cluster 1.2 + JBossAS 7.1 problem
                        johanvermeij

                        That is what I am looking for, I read that an instance-id can be added, also through the web console.

                        But I can't find where.

                        I suppose the instance-id can be set in this section of host.xml

                        Can you give me a hint ?

                        <servers>
                            <server name="server-one" group="main-server-group">
                                <!-- Remote JPDA debugging for a specific server
                                <jvm name="default">
                                  <jvm-options>
                                    <option value="-Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n"/>
                                  </jvm-options>
                               </jvm>
                               -->
                            </server>
                            <server name="server-two" group="main-server-group" auto-start="false">
                                <!-- server-two avoids port conflicts by incrementing the ports in
                                     the default socket-group declared in the server-group -->
                                <socket-bindings port-offset="150"/>
                            </server>
                            <server name="server-three-Linux" group="other-server-group" auto-start="true">
                                <!-- server-three avoids port conflicts by incrementing the ports in
                                     the default socket-group declared in the server-group -->
                                <socket-bindings port-offset="250"/>
                            </server>
                        </servers>
                        • 24. Re: mod_cluster 1.2 + JBossAS 7.1 problem
                          jfclere

                          the undefined is due to a bug... JBPAPP-8451?

                          • 25. Re: mod_cluster 1.2 + JBossAS 7.1 problem
                            ranga033

                            As mentioned in Jean-Frederic's post above try adding the workaround as you use 7.1.1

                             

                            Try adding this in domain.xml 

                             

                               <subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="false">

                             

                            Also, hope you have set the names for master and slave as per https://docs.jboss.org/author/display/AS71/AS7+Cluster+Howto

                             

                            something like , <host name="slave" xmlns="urn:jboss:domain:1.2">

                             

                            I hope this should solve your issue. Also go through the comments in  https://docs.jboss.org/author/display/AS71/AS7+Cluster+Howto . It has solution and workarounds for most of the issues for Cluster setup.

                            • 26. Re: mod_cluster 1.2 + JBossAS 7.1 problem
                              johanvermeij

                              Thank you very much, it all works fine

                               

                              I am already running a jmeter test against a cluster with a HP Proliant, an iMac and a Linux PC

                               

                              It is fast

                              • 27. Re: mod_cluster 1.2 + JBossAS 7.1 problem
                                johanvermeij

                                When I am running a stress test against the above cluster under 7.1.1 (domain mode) I get this error very often.

                                 

                                [Server:server-three-slave-HP] 18:46:25,182 ERROR [com.vantage.capital.service.xml.XMLTranServiceBean] (ajp--192.168.2.12-8259-22) DDF0001,CashTransaction:EXCP:org.hibernate.cache.CacheException: org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.hibernate.cache.CacheException: org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate

                                 

                                In fact, I don't need jmeter for this.

                                When I open 2 terminals and I do transactions quickly by hitting enter fast I get the same error.

                                 

                                Looking at the JBoss logs, netstat, Apache logs everything looks fine

                                • 28. Re: mod_cluster 1.2 + JBossAS 7.1 problem
                                  ranga033

                                  I am not too sure what kind of stress testing are you performing. If you can elaborate a bit, it would be better to understand the problem.

                                   

                                  From your logs, I see that you are using Hibernate and perhaps the Infinispan cache for Hibernate. Are those caches up when you are performing a test ?  What kind of caches have you configured (any remote caches?)?

                                  • 29. Re: mod_cluster 1.2 + JBossAS 7.1 problem
                                    johanvermeij

                                    I am testing a financial application.

                                    I prepare jmeter jmx files using a sample soap message captured from a terminal.

                                    I use the sample to propagate many other also simulating different users, bank branches, accounts etc.

                                     

                                    I point the jmeter injector at Apache http on port 8000 (see httpd.conf below) using mod_cluster (latest)

                                    I use one or more laptops as jmeter injectors

                                     

                                    httpd then load balances this to a cluster which consist of a HP Proliant mini server (Centos 6 64 bit, 192.168.2.12), a PC (Centos 6 64 bit, 192.168.2.3) and an iMac (MacOs, 192.168.2.4)

                                    I use an Oracle 11g database on the HP, httpd runs on the PC.

                                     

                                    The stress test runs very fast when only the master is up (PC is the master)

                                     

                                    As soon as I bring up a slave (and also when I bring up master and 2 slaves from the start) I get this error:

                                    [Server:server-three-slave-HP] 19:28:03,650 ERROR [com.vantage.capital.service.xml.XMLTranServiceBean] (ajp--192.168.2.12-8259-146) DDF0002,CashWithdrawal:EXCP:org.hibernate.cache.CacheException: org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.hibernate.cache.CacheException: org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate

                                     

                                    The above timeout happens on the PC and the HP.

                                    It slows the test down to about 1TPS (by comparison I can do almost 100TPS against the HP in standalone mode)

                                     

                                    The iMac has a different error:

                                    [Server:server-three-slave-iMac] 19:30:22,593 WARNING [org.jgroups.protocols.UDP] (pool-23-thread-1) null: no physical address for 3cf4cecf-6754-af54-1c50-f2ba733aac93, dropping message

                                    [Server:server-three-slave-iMac] 19:30:25,595 WARNING [org.jgroups.protocols.pbcast.GMS] (pool-23-thread-1) JOIN(slave1:server-three-slave-iMac/hibernate) sent to 3cf4cecf-6754-af54-1c50-f2ba733aac93 timed out (after 3000 ms), retrying

                                     

                                    I have gone through the startup logs and everything looks fine there, there are no errors or warnings.

                                     

                                    I don't expect you to fix this for me Rang, maybe you can give me a hint.

                                    We have been working on this for almost a week now and we are almost there.

                                    Problem is also that Googling these kinds of conditions is very "circular" and a lot of the discussions pertain to older JBoss releases.

                                     

                                    I say it again, what you guys have done is incredible, no commercial software can come near you.

                                     

                                    From httpd.conf

                                    <IfModule manager_module>

                                      Listen 192.168.2.3:6666

                                      AllowDisplay On

                                      ManagerBalancerName other-server-group

                                      <VirtualHost 192.168.2.3:6666>

                                        <Location />

                                         Order deny,allow

                                         Deny from all

                                         Allow from all

                                        </Location>

                                     

                                        ServerName vantagebanking.com

                                        ErrorLog "logs/chatter_log"

                                        LogLevel debug

                                        KeepAliveTimeout 100

                                        MaxKeepAliveRequests 0

                                    #   ServerAdvertise on http://192.168.2.3:6666

                                        AdvertiseFrequency 30

                                    #   AdvertiseSecurityKey secret

                                    #   AdvertiseGroup 224.0.1.105:23364

                                        EnableMCPMReceive

                                     

                                      </VirtualHost>

                                    </IfModule>

                                     

                                    NameVirtualHost 192.168.2.3:8000

                                    <VirtualHost 192.168.2.3:8000>

                                        ServerAdmin johanvermeij@gmail.com

                                        ServerName vantagebanking.com

                                        ErrorLog "logs/cluster_log"

                                        LogLevel debug

                                        AllowDisplay On

                                        ProxyPass '/beans/XMLTranServiceBean/XMLTranServiceBean' balancer://other-server-group stickysession=JSESSIONID|jsessionid nofailover=On

                                        ProxyPassReverse / balancer://other-server-group

                                        ProxyPreserveHost On

                                     

                                        <Location />

                                            Order deny,allow

                                            Allow from All

                                        </Location>

                                     

                                        <Location /mod_cluster_manager>

                                            SetHandler mod_cluster-manager

                                            Order deny,allow

                                            Deny from all

                                            Allow from all

                                        </Location>

                                     

                                    </VirtualHost>