HOWTO Step by step clustering ( JBoss 4.0.1 - Linux - Oracle
Hi,
Making cluster is easy with JBoss, that's right... When it does work out-of-the box.
Of course all the info needed is there, but scattered in many forums and Wiki pages. Moreover, infos are mostly for 'older' releases since 3.2.x are seem to be the most used. Unfortunately, that can lead to confusion sometimes.
Well, anyway, here's a step by step 'how-to' I now use to make my simple hello examples work. Maybe that'll be helpfull to some, maybe not.
Goal :
Quick steps I followed to have a running cluster with
JBoss. (Oracle DB and Linux multi-homed machines)
JBoss release :
4.0.1sp1
JVM :
Sun's 1.4.2_07-b05
Machine1 :
Linux (SUSE Pro8 - 2.4.18-64GB-SMP) ; multi-homed (IP1 used on eth0)
Machine2 :
Linux (SUSE SLES9 - 2.6.5-7.108-smp) ; multi-homed (IP2 used on eth2)
Note :
*******************************************
* PART 1 : Make Oracle the Default database
*******************************************
Prerequisite :
Oracle and listener installed on Machine1 :
- sid is MYSID ;
- uses IP1 and default port (1521);
- valid user/password is LOGIN/PASSWORD
- untarred brand new JBoss delivery 4.0.1sp1 (from sourceforge)
Note : all path given relative to <JBOSS_HOME>
----------------------
1.0 in conf/run.conf :
----------------------
- set JAVA_HOME correctly to not trust env variables ( especially if
you want to use a special jvm )
----------------------------------------------
1.1 in file server/all/conf/login-config.xml :
----------------------------------------------
From :
<module-option name = "dsJndiName">java:/DefaultDS</module-option>
To :
<module-option name = "dsJndiName">java:/OracleDS</module-option>
--------------------------------------------------
1.2 in server/all/conf/standardjbosscmp-jdbc.xml :
--------------------------------------------------
From :
<defaults>
<datasource>java:/DefaultDS</datasource>
<!-- optional since 4.0 <datasource-mapping>Hypersonic SQL</datasource-mapping> -->
To :
<defaults>
<datasource>java:/OracleDS</datasource>
<datasource-mapping>Oracle9i</datasource-mapping>
---
From :
<type-mapping>
<name>Oracle9i</name>
...
<mapping>
<java-type>java.lang.Short</java-type>
<jdbc-type>NUMERIC</jdbc-type>
<!-- <jdbc-type>SMALLINT</jdbc-type> this resulted in a ClassCastException -->
<sql-type>NUMBER(5)</sql-type>
To :
<type-mapping>
<name>Oracle9i</name>
...
<mapping>
<java-type>java.lang.Short</java-type>
<jdbc-type>SMALLINT</jdbc-type> <!-- NUMERIC does not Work for us -->
<!-- <jdbc-type>SMALLINT</jdbc-type> this resulted in a ClassCastException -->
<sql-type>NUMBER(5)</sql-type>
---------------------------------
1.3 Change the service provider :
---------------------------------
rm server/all/deploy-hasingleton/jms/hsqldb-jdbc2-service.xml
cp docs/examples/jms/oracle-jdbc2-service.xml server/all/deploy-hasingleton/jms/
cp docs/examples/jca/oracle-ds.xml server/all/deploy/
mv server/all/deploy-hasingleton/jms/hsqldb-jdbc-state-service.xml server/all/deploy-hasingleton/jms/oracle-jdbc-state-service.xml
----------------------------------------
1.4 in server/all/deploy/oracle-ds.xml :
----------------------------------------
From :
<connection-url>jdbc:oracle:thin:@youroraclehost:1521:yoursid</connection-url>
To :
<connection-url>jdbc:oracle:thin:@IP1:1521:MYSID</connection-url>
---
From :
<user-name>x</user-name>
<password>y</password>
To :
<user-name>LOGIN</user-name>
<password>PASSWORD</password>
-----------------------------------------------------------------------
1.5 in server/all/deploy-hasingleton/jms/oracle-jdbc-state-service.xml
-----------------------------------------------------------------------
From :
<depends optional-attribute-name="ConnectionManager">jboss.jca:service=DataSourceBinding,name=DefaultDS</depends>
To :
<depends optional-attribute-name="ConnectionManager">jboss.jca:service=DataSourceBinding,name=OracleDS</depends>
-----------------------------
1.6 get the classes12 library
-----------------------------
From the oracle server, get classes12.zip ( file is in
[ORACLE_HOME]/product/[ORACLE_VERSION]/jdbc/lib/classes12.zip )
Copy file to server/all/lib/classes12.jar
( Note that if oci is desired instead of thin, it's a whole different
story )
*******************************************************
* PART 2 : File Modifications and others for clustering
*******************************************************
-------------------------
2.1 Avoid IPv6 problems :
-------------------------
See http://www.jboss.org/index.html?module=bb&op=viewtopic&t=49207
and
http://portal.suse.de/sdb/en/2003/10/90_mozilla_ipv6.html
- in file /etc/modules.conf on Machine1 (SUSE 8 Distrib) :
From :
alias net-pf-10 ipv6
#alias net-pf-10 off
To :
#alias net-pf-10 ipv6
alias net-pf-10 off
- in file /etc/modprobe.conf on Machine2 (SUSE 9.1 distrib) :
From :
alias net-pf-10 ipv6
To :
alias net-pf-10 ipv6
install ipv6 /bin/true
- For SUSE 9.0 distribs (c.f. SUSE web site), in file /etc/modprobe.conf :
From :
alias net-pf-10 ipv6
#alias net-pf-10 off
To :
#alias net-pf-10 ipv6
alias net-pf-10 off
Note : /etc/sysconfig/scripts/SuSEfirewall2-custom can also be
modified to allow incoming multicast packets.
--------------------------
2.2 add multicast routes :
--------------------------
On Machine1 :
route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0
On Machine2 :
route add -net 224.0.0.0 netmask 240.0.0.0 dev eth2
Note1 : do not mix-up network devices
Note2 : The netmask is to allow other default multicast IPs
(228.1.2.3, 230.0.0.4, 230.1.2.3 and 224.1.9.1)
Note 3 : to check routes : netstat -grn
On Machine1 the important line is :
224.0.0.0 0.0.0.0 240.0.0.0 U 40 0 0 eth0
----------------------------------------
2.3 in server/all/conf/standardjboss.xml
----------------------------------------
From :
<container-configuration>
<container-name>Clustered Stateless SessionBean</container-name>
...
<interceptor transaction="Container">org.jboss.ejb.plugins.ServiceEndpointInterceptor</interceptor>
...
<interceptor transaction="Bean">org.jboss.ejb.plugins.ServiceEndpointInterceptor</interceptor>
To :
<container-configuration>
<container-name>Clustered Stateless SessionBean</container-name>
...
<interceptor transaction="Container">org.jboss.webservice.server.ServiceEndpointInterceptor</interceptor>
...
<interceptor transaction="Bean">org.jboss.webservice.server.ServiceEndpointInterceptor</interceptor>
------------------------------------------------
2.4 in server/all/deploy/jms/hajndi-jms-ds.xml :
------------------------------------------------
From :
<attribute name="Properties">
java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces
java.naming.provider.url=localhost:1100
</attribute>
To :
<attribute name="Properties">
java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces
java.naming.provider.url=<IP1>:1100,<IP2>:1100
</attribute>
Note, you can also dos2unix the file since some eof are dos formated
in this file.
----------------------------------------------
2.5 in server/all/deploy/cluster-service.xml :
----------------------------------------------
From :
<UDP mcast_addr="228.1.2.3" mcast_port="45566"
ip_ttl="32" ip_mcast="true"
mcast_send_buf_size="800000" mcast_recv_buf_size="150000"
ucast_send_buf_size="800000" ucast_recv_buf_size="150000"
loopback="false" />
To :
<UDP mcast_addr="228.1.2.3" mcast_port="45566"
ip_ttl="32" ip_mcast="true"
mcast_send_buf_size="800000" mcast_recv_buf_size="150000"
ucast_send_buf_size="800000" ucast_recv_buf_size="150000"
loopback="false" bind_addr="<IP1 on Machine1 ; IP2 on Machine2>" />
Note : that is the ONLY file to change from one machine to the other (
use rsync to copy modifs for second install to avoid human errors.)
------------------------
2.5 Start command used :
------------------------
Machine1 :
/opt/jboss/bin/run.sh -c all --host=IP1
Machine2 :
/opt/jboss/bin/run.sh -c all --host=IP2
******************************
* PART 3 : Some Specific usage
******************************
---------------------------
3.1 A singleton scheduler :
---------------------------
Make use of the jboss:service=ScheduleManager defined in
server/all/deploy/schedule-manager-service.xml
Example : SchedulableMBeanExample
-------------------------------------------------------------
3.2 Access to a queue using HA-JNDI (from inside the cluster)
-------------------------------------------------------------
Solution1 :
~~~~~~~~~~~
Set 'hardcoded' properties for the context :
Properties p = new Properties();
p.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory");
p.put(Context.URL_PKG_PREFIXES, "jboss.naming:org.jnp.interfaces");
p.put(Context.PROVIDER_URL, "IP1:1100,IP2:1100"); // HA-JNDI port.
InitialContext context = new InitialContext(p);
Queue queue = (Queue) context.lookup(queueName);
pros/cons :
- necessity to modify code if hardcoded or to reload properties in
case of a change in IP used.
- information is already in a property file (hajndi-jms-ds.xml)
- complex for n-nodes cluster
+ easy to try/test
=> Solution1 should be avoided when lookup is done from inside the cluster
Solution2 :
~~~~~~~~~~~
Use the partition name with code :
Properties p = new Properties();
p.put("jnp.partitionName", "DefaultPartition");
InitialContext context = new InitialContext(p);
Queue queue = (Queue) context.lookup(queueName);
Pro/cons :
- may be non-applicable on some infrastructures (forbiden multicast msgs)
Solution3
~~~~~~~~~
Use only propertie files :
This solution would be the preffered solution, just using :
InitialContext context = new InitialContext();
Queue queue = (Queue) context.lookup(queueName);
<<Solution TBD>>
Pros/cons :
- may be non-applicable on some infrastructures (forbiden multicast msgs)
+ No need to change initial code : one person can be responsible of the
code, while another person is in charge of the deployment ( no access
to source code or impossibility to build ! )
=> preffered solution when lookup is done from inside the cluster.
Btw, if someone knows how to deal with Solution3, that'll be great.
I had this problem posted some time ago now : http://www.jboss.org/index.html?module=bb&op=viewtopic&t=60106 but I was not precise on the desired way to address the issue. Solution 1 above is often found ; Solution2 I did not found on the forum or Wiki ( I do not say it is not there, I just did not find it) ; Solution3 is the one desired : no change to the code, only use configuration files.
Good day,
Alex.
