3 Replies Latest reply on Jan 12, 2007 9:45 AM by Brian Dill

    HOWTO Step by step clustering ( JBoss 4.0.1 - Linux - Oracle

    schaffal Newbie

      Hi,
      Making cluster is easy with JBoss, that's right... When it does work out-of-the box.

      Of course all the info needed is there, but scattered in many forums and Wiki pages. Moreover, infos are mostly for 'older' releases since 3.2.x are seem to be the most used. Unfortunately, that can lead to confusion sometimes.

      Well, anyway, here's a step by step 'how-to' I now use to make my simple hello examples work. Maybe that'll be helpfull to some, maybe not.

      Goal :
      Quick steps I followed to have a running cluster with
      JBoss. (Oracle DB and Linux multi-homed machines)
      
      JBoss release :
      4.0.1sp1
      
      JVM :
      Sun's 1.4.2_07-b05
      
      Machine1 :
      Linux (SUSE Pro8 - 2.4.18-64GB-SMP) ; multi-homed (IP1 used on eth0)
      
      Machine2 :
      Linux (SUSE SLES9 - 2.6.5-7.108-smp) ; multi-homed (IP2 used on eth2)
      
      Note :
      
      *******************************************
      * PART 1 : Make Oracle the Default database
      *******************************************
      
      Prerequisite :
      Oracle and listener installed on Machine1 :
      - sid is MYSID ;
      - uses IP1 and default port (1521);
      - valid user/password is LOGIN/PASSWORD
      - untarred brand new JBoss delivery 4.0.1sp1 (from sourceforge)
      
      Note : all path given relative to <JBOSS_HOME>
      
      ----------------------
      1.0 in conf/run.conf :
      ----------------------
      
      - set JAVA_HOME correctly to not trust env variables ( especially if
      you want to use a special jvm )
      
      ----------------------------------------------
      1.1 in file server/all/conf/login-config.xml :
      ----------------------------------------------
      
      From :
       <module-option name = "dsJndiName">java:/DefaultDS</module-option>
      To :
       <module-option name = "dsJndiName">java:/OracleDS</module-option>
      
      
      --------------------------------------------------
      1.2 in server/all/conf/standardjbosscmp-jdbc.xml :
      --------------------------------------------------
      
      From :
       <defaults>
       <datasource>java:/DefaultDS</datasource>
       <!-- optional since 4.0 <datasource-mapping>Hypersonic SQL</datasource-mapping> -->
      
      To :
       <defaults>
       <datasource>java:/OracleDS</datasource>
       <datasource-mapping>Oracle9i</datasource-mapping>
      
      ---
      
      From :
       <type-mapping>
       <name>Oracle9i</name>
      ...
       <mapping>
       <java-type>java.lang.Short</java-type>
       <jdbc-type>NUMERIC</jdbc-type>
       <!-- <jdbc-type>SMALLINT</jdbc-type> this resulted in a ClassCastException -->
       <sql-type>NUMBER(5)</sql-type>
      To :
       <type-mapping>
       <name>Oracle9i</name>
      ...
       <mapping>
       <java-type>java.lang.Short</java-type>
       <jdbc-type>SMALLINT</jdbc-type> <!-- NUMERIC does not Work for us -->
       <!-- <jdbc-type>SMALLINT</jdbc-type> this resulted in a ClassCastException -->
       <sql-type>NUMBER(5)</sql-type>
      
      
      ---------------------------------
      1.3 Change the service provider :
      ---------------------------------
      
      rm server/all/deploy-hasingleton/jms/hsqldb-jdbc2-service.xml
      
      cp docs/examples/jms/oracle-jdbc2-service.xml server/all/deploy-hasingleton/jms/
      
      cp docs/examples/jca/oracle-ds.xml server/all/deploy/
      
      mv server/all/deploy-hasingleton/jms/hsqldb-jdbc-state-service.xml server/all/deploy-hasingleton/jms/oracle-jdbc-state-service.xml
      
      
      ----------------------------------------
      1.4 in server/all/deploy/oracle-ds.xml :
      ----------------------------------------
      
      From :
       <connection-url>jdbc:oracle:thin:@youroraclehost:1521:yoursid</connection-url>
      
      To :
       <connection-url>jdbc:oracle:thin:@IP1:1521:MYSID</connection-url>
      
      ---
      
      From :
       <user-name>x</user-name>
       <password>y</password>
      
      To :
       <user-name>LOGIN</user-name>
       <password>PASSWORD</password>
      
      
      -----------------------------------------------------------------------
      1.5 in server/all/deploy-hasingleton/jms/oracle-jdbc-state-service.xml
      -----------------------------------------------------------------------
      
      From :
       <depends optional-attribute-name="ConnectionManager">jboss.jca:service=DataSourceBinding,name=DefaultDS</depends>
      
      To :
       <depends optional-attribute-name="ConnectionManager">jboss.jca:service=DataSourceBinding,name=OracleDS</depends>
      
      
      -----------------------------
      1.6 get the classes12 library
      -----------------------------
      
      From the oracle server, get classes12.zip ( file is in
      [ORACLE_HOME]/product/[ORACLE_VERSION]/jdbc/lib/classes12.zip )
      
      Copy file to server/all/lib/classes12.jar
      
      ( Note that if oci is desired instead of thin, it's a whole different
      story )
      
      
      *******************************************************
      * PART 2 : File Modifications and others for clustering
      *******************************************************
      
      -------------------------
      2.1 Avoid IPv6 problems :
      -------------------------
      
      See http://www.jboss.org/index.html?module=bb&op=viewtopic&t=49207
       and
       http://portal.suse.de/sdb/en/2003/10/90_mozilla_ipv6.html
      
      - in file /etc/modules.conf on Machine1 (SUSE 8 Distrib) :
      From :
      alias net-pf-10 ipv6
      #alias net-pf-10 off
      
      To :
      #alias net-pf-10 ipv6
      alias net-pf-10 off
      
      
      - in file /etc/modprobe.conf on Machine2 (SUSE 9.1 distrib) :
      From :
      alias net-pf-10 ipv6
      
      To :
      alias net-pf-10 ipv6
      install ipv6 /bin/true
      
      - For SUSE 9.0 distribs (c.f. SUSE web site), in file /etc/modprobe.conf :
      From :
      alias net-pf-10 ipv6
      #alias net-pf-10 off
      
      To :
      #alias net-pf-10 ipv6
      alias net-pf-10 off
      
      Note : /etc/sysconfig/scripts/SuSEfirewall2-custom can also be
      modified to allow incoming multicast packets.
      
      
      --------------------------
      2.2 add multicast routes :
      --------------------------
      
      On Machine1 :
      route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0
      
      On Machine2 :
      route add -net 224.0.0.0 netmask 240.0.0.0 dev eth2
      
      Note1 : do not mix-up network devices
      
      Note2 : The netmask is to allow other default multicast IPs
      (228.1.2.3, 230.0.0.4, 230.1.2.3 and 224.1.9.1)
      
      Note 3 : to check routes : netstat -grn
      On Machine1 the important line is :
      224.0.0.0 0.0.0.0 240.0.0.0 U 40 0 0 eth0
      
      
      ----------------------------------------
      2.3 in server/all/conf/standardjboss.xml
      ----------------------------------------
      
      From :
       <container-configuration>
       <container-name>Clustered Stateless SessionBean</container-name>
      ...
       <interceptor transaction="Container">org.jboss.ejb.plugins.ServiceEndpointInterceptor</interceptor>
      ...
       <interceptor transaction="Bean">org.jboss.ejb.plugins.ServiceEndpointInterceptor</interceptor>
      
      
      To :
       <container-configuration>
       <container-name>Clustered Stateless SessionBean</container-name>
      ...
       <interceptor transaction="Container">org.jboss.webservice.server.ServiceEndpointInterceptor</interceptor>
      ...
       <interceptor transaction="Bean">org.jboss.webservice.server.ServiceEndpointInterceptor</interceptor>
      
      
      ------------------------------------------------
      2.4 in server/all/deploy/jms/hajndi-jms-ds.xml :
      ------------------------------------------------
      
      From :
       <attribute name="Properties">
       java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
      
       java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces
      
       java.naming.provider.url=localhost:1100
       </attribute>
      
      To :
       <attribute name="Properties">
       java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
      
       java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces
      
       java.naming.provider.url=<IP1>:1100,<IP2>:1100
       </attribute>
      
      Note, you can also dos2unix the file since some eof are dos formated
      in this file.
      
      
      ----------------------------------------------
      2.5 in server/all/deploy/cluster-service.xml :
      ----------------------------------------------
      
      From :
       <UDP mcast_addr="228.1.2.3" mcast_port="45566"
       ip_ttl="32" ip_mcast="true"
       mcast_send_buf_size="800000" mcast_recv_buf_size="150000"
       ucast_send_buf_size="800000" ucast_recv_buf_size="150000"
       loopback="false" />
      
      To :
       <UDP mcast_addr="228.1.2.3" mcast_port="45566"
       ip_ttl="32" ip_mcast="true"
       mcast_send_buf_size="800000" mcast_recv_buf_size="150000"
       ucast_send_buf_size="800000" ucast_recv_buf_size="150000"
       loopback="false" bind_addr="<IP1 on Machine1 ; IP2 on Machine2>" />
      
      Note : that is the ONLY file to change from one machine to the other (
      use rsync to copy modifs for second install to avoid human errors.)
      
      
      ------------------------
      2.5 Start command used :
      ------------------------
      
      Machine1 :
      /opt/jboss/bin/run.sh -c all --host=IP1
      
      Machine2 :
      /opt/jboss/bin/run.sh -c all --host=IP2
      
      ******************************
      * PART 3 : Some Specific usage
      ******************************
      
      ---------------------------
      3.1 A singleton scheduler :
      ---------------------------
      
      Make use of the jboss:service=ScheduleManager defined in
      server/all/deploy/schedule-manager-service.xml
      
      Example : SchedulableMBeanExample
      
      
      -------------------------------------------------------------
      3.2 Access to a queue using HA-JNDI (from inside the cluster)
      -------------------------------------------------------------
      
      Solution1 :
      ~~~~~~~~~~~
      Set 'hardcoded' properties for the context :
      
      Properties p = new Properties();
      p.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory");
      p.put(Context.URL_PKG_PREFIXES, "jboss.naming:org.jnp.interfaces");
      p.put(Context.PROVIDER_URL, "IP1:1100,IP2:1100"); // HA-JNDI port.
      InitialContext context = new InitialContext(p);
      Queue queue = (Queue) context.lookup(queueName);
      
      pros/cons :
      - necessity to modify code if hardcoded or to reload properties in
       case of a change in IP used.
      - information is already in a property file (hajndi-jms-ds.xml)
      - complex for n-nodes cluster
      + easy to try/test
      
      => Solution1 should be avoided when lookup is done from inside the cluster
      
      
      Solution2 :
      ~~~~~~~~~~~
      Use the partition name with code :
      Properties p = new Properties();
      p.put("jnp.partitionName", "DefaultPartition");
      InitialContext context = new InitialContext(p);
      Queue queue = (Queue) context.lookup(queueName);
      
      Pro/cons :
      - may be non-applicable on some infrastructures (forbiden multicast msgs)
      
      
      Solution3
      ~~~~~~~~~
      Use only propertie files :
      This solution would be the preffered solution, just using :
      InitialContext context = new InitialContext();
      Queue queue = (Queue) context.lookup(queueName);
      
      <<Solution TBD>>
      
      Pros/cons :
      - may be non-applicable on some infrastructures (forbiden multicast msgs)
      + No need to change initial code : one person can be responsible of the
      code, while another person is in charge of the deployment ( no access
      to source code or impossibility to build ! )
      
      => preffered solution when lookup is done from inside the cluster.
      


      Btw, if someone knows how to deal with Solution3, that'll be great.
      I had this problem posted some time ago now : http://www.jboss.org/index.html?module=bb&op=viewtopic&t=60106 but I was not precise on the desired way to address the issue. Solution 1 above is often found ; Solution2 I did not found on the forum or Wiki ( I do not say it is not there, I just did not find it) ; Solution3 is the one desired : no change to the code, only use configuration files.

      Good day,
      Alex.