1 Reply Latest reply on Mar 27, 2014 5:21 PM by vikrammurugesan

    mod_cluster - Error [MEM: MEM: Can't read node: {4}]

    vikrammurugesan

      Hello,

       

      I have a 2 node standalone cluster that I am trying to configure with mod_cluster. I read this page before starting - https://docs.jboss.org/author/display/AS72/AS7+Cluster+Howto

       

      I grabbed the mod_cluster-1.2.0.Final-windows-x86-ssl.zip file from Mod_cluster 1.2.0.Final - JBoss Community  and grabbed the 4 files mod_advertise.so, mod_manager.so, mod_proxy_cluster.so, mod_slotmem.so and dropped it into my already existing Apache HTTP server's modules directory.

       

      I made sure I commented out the line that loads mod_proxy module. I appended the below lines to my httpd.conf file, (xxx.xxx.xxx.xx is the IP of my machine and -b property of both my server startups)

       

      ########### mod_cluster to JBossAS7 ###########
      LoadModule slotmem_module modules/mod_slotmem.so
      LoadModule manager_module modules/mod_manager.so
      LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
      LoadModule advertise_module modules/mod_advertise.so
      
      Listen xxx.xxx.xxx.xx:3001
      
      <VirtualHost xxx.xxx.xxx.xx:3001>
        <Directory />
             Order deny,allow
             Deny from all
             Allow from all
        </Directory>
      
        <Location /mod_cluster-manager>
             SetHandler mod_cluster-manager
             Order deny,allow
             Deny from all
             Allow from all
        </Location>
      
        KeepAliveTimeout 60
        MaxKeepAliveRequests 0
        AdvertiseFrequency 5
        AdvertiseSecurityKey secret
        EnableMCPMReceive
      </VirtualHost>
      ######### End mod_cluster to JBossAS7 #########
      
      
      
      
      

       

      I modified my standalone-full-ha.xml in three places,

       

      I added the node name to my server, (on the second node the name is 'nodeB')

      <server name="nodeA" xmlns="urn:jboss:domain:1.3">
      
      
      
      
      

       

      I added instance-id to the web subsystem, (I also pass jboss.node.name as system property with the same value 'nodeA' and 'nodeB')

      <subsystem xmlns="urn:jboss:domain:web:1.2" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="false">
      
      
      
      
      

       

      I added adverties=on, proxy-list and advertise-security-key to modcluster subsystem, (where xxx.xxx.xxx.xx is the same IP as in my httpd.conf file. It is also my -b bind address)

      <mod-cluster-config advertise-socket="modcluster" connector="ajp" advertise="true" proxy-list="xxx.xxx.xxx.xx:3001" advertise-security-key="secret">
      
      
      
      
      

       

      I bounced Apache server and it started up clean.

       

      After I start my first jboss node 'nodeA', I see this repeating in my logs every 10 seconds,

       

      15:03:19,204 ERROR [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] Error [MEM: MEM: Can't read node: {4}] sending command STATUS to proxy my.machine.com/xxx.xxx.xxx.xx:3001, configuration will be reset
      15:03:29,419 ERROR [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] Error [MEM: MEM: Can't read node: {4}] sending command STATUS to proxy my.machine.com/xxx.xxx.xxx.xx:3001, configuration will be reset
      15:03:39,632 ERROR [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] Error [MEM: MEM: Can't read node: {4}] sending command STATUS to proxy my.machine.com/xxx.xxx.xxx.xx:3001, configuration will be reset
      15:03:49,846 ERROR [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] Error [MEM: MEM: Can't read node: {4}] sending command STATUS to proxy my.machine.com/xxx.xxx.xxx.xx:3001, configuration will be reset
      15:04:00,060 ERROR [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] Error [MEM: MEM: Can't read node: {4}] sending command STATUS to proxy my.machine.com/xxx.xxx.xxx.xx:3001, configuration will be reset
      
      
      
      
      

       

      Also here is the apache error log, (LogLevel was debug) Sorry fot the big log file. I can attach it

       

      [Tue Oct 15 15:07:53 2013] [notice] Advertise initialized for process 14544
      [Tue Oct 15 15:07:53 2013] [notice] Apache/2.2.22 (Win32) mod_cluster/1.2.0.Final configured -- resuming normal operations
      [Tue Oct 15 15:07:53 2013] [notice] Server built: Jan 28 2012 11:16:39
      [Tue Oct 15 15:07:53 2013] [notice] Parent: Created child process 14184
      [Tue Oct 15 15:07:53 2013] [debug] mpm_winnt.c(477): Parent: Sent the scoreboard to the child
      [Tue Oct 15 15:07:53 2013] [debug] mod_advertise.c(577): [14184 - 14544] in child post config hook
      [Tue Oct 15 15:07:53 2013] [notice] Child 14184: Child process is running
      [Tue Oct 15 15:07:53 2013] [info] Parent: Duplicating socket 280 and sending it to child process 14184
      [Tue Oct 15 15:07:53 2013] [debug] mpm_winnt.c(398): Child 14184: Retrieved our scoreboard from the parent.
      [Tue Oct 15 15:07:53 2013] [info] Parent: Duplicating socket 272 and sending it to child process 14184
      [Tue Oct 15 15:07:53 2013] [debug] mpm_winnt.c(595): Parent: Sent 2 listeners to child 14184
      [Tue Oct 15 15:07:53 2013] [debug] mpm_winnt.c(554): Child 14184: retrieved 2 listeners from parent
      [Tue Oct 15 15:07:53 2013] [notice] Child 14184: Acquired the start mutex.
      [Tue Oct 15 15:07:53 2013] [notice] Child 14184: Starting 64 worker threads.
      [Tue Oct 15 15:07:53 2013] [notice] Child 14184: Starting thread to listen on port 3001.
      [Tue Oct 15 15:07:53 2013] [notice] Child 14184: Starting thread to listen on port 80.
      [Tue Oct 15 15:08:20 2013] [debug] mod_manager.c(1667): manager_trans INFO 
      [Tue Oct 15 15:08:20 2013] [debug] mod_manager.c(2323): manager_handler INFO (C:/) processing: ""
      [Tue Oct 15 15:08:20 2013] [debug] mod_manager.c(2366): manager_handler INFO  OK
      [Tue Oct 15 15:08:20 2013] [debug] mod_manager.c(1667): manager_trans CONFIG 
      [Tue Oct 15 15:08:20 2013] [debug] mod_manager.c(2323): manager_handler CONFIG (C:/) processing: "JVMRoute=nodeA&Host=localhost&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp&ping=10"
      [Tue Oct 15 15:08:20 2013] [debug] mod_manager.c(2366): manager_handler CONFIG  OK
      [Tue Oct 15 15:08:20 2013] [debug] mod_manager.c(1667): manager_trans STATUS 
      [Tue Oct 15 15:08:20 2013] [debug] mod_manager.c(2323): manager_handler STATUS (C:/) processing: "JVMRoute=nodeA&Load=100"
      [Tue Oct 15 15:08:20 2013] [debug] mod_manager.c(1404): Processing STATUS
      [Tue Oct 15 15:08:20 2013] [notice] Parent: child process exited with status 3221225477 -- Restarting.
      [Tue Oct 15 15:08:20 2013] [notice] Advertise initialized for process 14544
      [Tue Oct 15 15:08:20 2013] [notice] Apache/2.2.22 (Win32) mod_cluster/1.2.0.Final configured -- resuming normal operations
      [Tue Oct 15 15:08:20 2013] [notice] Server built: Jan 28 2012 11:16:39
      [Tue Oct 15 15:08:20 2013] [notice] Parent: Created child process 7836
      [Tue Oct 15 15:08:20 2013] [debug] mpm_winnt.c(477): Parent: Sent the scoreboard to the child
      [Tue Oct 15 15:08:20 2013] [debug] mod_advertise.c(577): [7836 - 14544] in child post config hook
      [Tue Oct 15 15:08:20 2013] [notice] Child 7836: Child process is running
      [Tue Oct 15 15:08:20 2013] [debug] mpm_winnt.c(398): Child 7836: Retrieved our scoreboard from the parent.
      [Tue Oct 15 15:08:20 2013] [info] Parent: Duplicating socket 280 and sending it to child process 7836
      [Tue Oct 15 15:08:20 2013] [info] Parent: Duplicating socket 272 and sending it to child process 7836
      [Tue Oct 15 15:08:20 2013] [debug] mpm_winnt.c(595): Parent: Sent 2 listeners to child 7836
      [Tue Oct 15 15:08:20 2013] [debug] mpm_winnt.c(554): Child 7836: retrieved 2 listeners from parent
      [Tue Oct 15 15:08:20 2013] [notice] Child 7836: Acquired the start mutex.
      [Tue Oct 15 15:08:20 2013] [notice] Child 7836: Starting 64 worker threads.
      [Tue Oct 15 15:08:20 2013] [notice] Child 7836: Starting thread to listen on port 3001.
      [Tue Oct 15 15:08:20 2013] [debug] mod_manager.c(1667): manager_trans STATUS 
      [Tue Oct 15 15:08:20 2013] [debug] mod_manager.c(2323): manager_handler STATUS (C:/) processing: "JVMRoute=nodeA&Load=100"
      [Tue Oct 15 15:08:20 2013] [debug] mod_manager.c(1404): Processing STATUS
      [Tue Oct 15 15:08:20 2013] [warn] manager_handler STATUS error: MEM: Can't read node
      [Tue Oct 15 15:08:20 2013] [notice] Child 7836: Starting thread to listen on port 80.
      [Tue Oct 15 15:08:30 2013] [debug] mod_manager.c(1667): manager_trans INFO 
      [Tue Oct 15 15:08:30 2013] [debug] mod_manager.c(2323): manager_handler INFO (C:/) processing: ""
      [Tue Oct 15 15:08:30 2013] [debug] mod_manager.c(2366): manager_handler INFO  OK
      [Tue Oct 15 15:08:30 2013] [debug] mod_manager.c(1667): manager_trans CONFIG 
      [Tue Oct 15 15:08:30 2013] [debug] mod_manager.c(2323): manager_handler CONFIG (C:/) processing: "JVMRoute=nodeA&Host=localhost&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp&ping=10"
      [Tue Oct 15 15:08:30 2013] [debug] mod_manager.c(2366): manager_handler CONFIG  OK
      [Tue Oct 15 15:08:30 2013] [debug] mod_manager.c(1667): manager_trans STATUS 
      [Tue Oct 15 15:08:30 2013] [debug] mod_manager.c(2323): manager_handler STATUS (C:/) processing: "JVMRoute=nodeA&Load=100"
      [Tue Oct 15 15:08:30 2013] [debug] mod_manager.c(1404): Processing STATUS
      [Tue Oct 15 15:08:30 2013] [notice] Parent: child process exited with status 3221225477 -- Restarting.
      [Tue Oct 15 15:08:30 2013] [notice] Advertise initialized for process 14544
      [Tue Oct 15 15:08:30 2013] [notice] Apache/2.2.22 (Win32) mod_cluster/1.2.0.Final configured -- resuming normal operations
      [Tue Oct 15 15:08:30 2013] [notice] Server built: Jan 28 2012 11:16:39
      [Tue Oct 15 15:08:30 2013] [notice] Parent: Created child process 1060
      [Tue Oct 15 15:08:30 2013] [debug] mpm_winnt.c(477): Parent: Sent the scoreboard to the child
      [Tue Oct 15 15:08:30 2013] [debug] mod_advertise.c(577): [1060 - 14544] in child post config hook
      [Tue Oct 15 15:08:30 2013] [notice] Child 1060: Child process is running
      [Tue Oct 15 15:08:30 2013] [info] Parent: Duplicating socket 280 and sending it to child process 1060
      [Tue Oct 15 15:08:30 2013] [debug] mpm_winnt.c(398): Child 1060: Retrieved our scoreboard from the parent.
      [Tue Oct 15 15:08:30 2013] [info] Parent: Duplicating socket 272 and sending it to child process 1060
      [Tue Oct 15 15:08:30 2013] [debug] mpm_winnt.c(595): Parent: Sent 2 listeners to child 1060
      [Tue Oct 15 15:08:30 2013] [debug] mpm_winnt.c(554): Child 1060: retrieved 2 listeners from parent
      [Tue Oct 15 15:08:30 2013] [notice] Child 1060: Acquired the start mutex.
      [Tue Oct 15 15:08:30 2013] [notice] Child 1060: Starting 64 worker threads.
      [Tue Oct 15 15:08:30 2013] [notice] Child 1060: Starting thread to listen on port 3001.
      [Tue Oct 15 15:08:30 2013] [notice] Child 1060: Starting thread to listen on port 80.
      [Tue Oct 15 15:08:30 2013] [debug] mod_manager.c(1667): manager_trans STATUS 
      [Tue Oct 15 15:08:30 2013] [debug] mod_manager.c(2323): manager_handler STATUS (C:/) processing: "JVMRoute=nodeA&Load=100"
      [Tue Oct 15 15:08:30 2013] [debug] mod_manager.c(1404): Processing STATUS
      [Tue Oct 15 15:08:30 2013] [warn] manager_handler STATUS error: MEM: Can't read node
      
      
      
      
      

       

      Not sure what I am doing wrong.

       

      Here is the version detail,

      OS - Windows 7 Professional 32 bit

      JBoss EAP 6.0.1

      mod_cluster 1.2.0.Final

      Apache HTTP Server 2.2.22 (4 modules were added to an existing installation)

      JDK 1.7.0_25

        • 1. Re: mod_cluster - Error [MEM: MEM: Can't read node: {4}]
          vikrammurugesan

          I was able to fix the problem. I just wanted to post it here so that people don't see an unanswered question.

           

          First thing - I did not use Apache 2.2.22. Instead I used mod_cluster 1.2.6.Final installation that I downloaded from here. I would suggest using apache httpd that you get from mod_cluster instead of trying to add mod_cluster module to an existing Apache. I might be wrong, but using httpd that comes with mod_cluster saves a lot of time.

           

          Here are my final settings.

           

          standalone-full-ha.xml

           

          <subsystem xmlns="urn:jboss:domain:modcluster:1.1">
                    <mod-cluster-config advertise-socket="modcluster" proxy-list="127.0.0.1:8796" connector="ajp">
                    <dynamic-load-provider>
                         <load-metric type="busyness"/>
                    </dynamic-load-provider>
                    </mod-cluster-config>
          </subsystem>
          

           

           

           

          httpd.conf

           

          # MOD_CLUSTER_ADDS
          # Adjust to you hostname and subnet.
          <IfModule manager_module>
            Listen 127.0.0.1:8796
          
            <VirtualHost 127.0.0.1:8796>
              <Directory />
               Order deny,allow
               Deny from all
               Allow from 127.0.0
              </Directory>
          
              KeepAliveTimeout 300
              MaxKeepAliveRequests 0
              #ServerAdvertise on http://127.0.0.1:8796
               AdvertiseFrequency 5
              #AdvertiseSecurityKey secret
              #AdvertiseGroup 224.0.1.105:23364
              EnableMCPMReceive
            ManagerBalancerName mycluster
          
            <Location /mod_cluster-manager>
                 SetHandler mod_cluster-manager
                 Order deny,allow
                 Deny from all
                 Allow from 127.0.0
                 AllowDisplay On
              </Location>
          
            </VirtualHost>
          </IfModule>
          

           

          Good luck and let me know if I have missed anything.