JBossESB 4.8 clustering (with shared jUDDI registry)

[daniele.sacchetti]

I'm trying to implement the ESB clustering as described in the Architecture presented in Chaptyer 7 (Advanced Topics) of the JBossESB 4.8 Programmers Guide.

The first issue I'm facing is the setup of a jUDDI registry shared between two instances of JBoss ESB. I'm running two instances of jbossesb-server 4.8.

For the first I've just unzipped the distribution without any changes.

For the second instance I've modified the following configuration files:
-jbossesb-server-4.8/server/default/conf/jndi.properties
    added aline:     
           java.naming.provider.url=jnp://172.17.4.31:1099
- jbossesb-server-4.8/server/default/deploy/jbossesb.sar/esb.juddi.client.xml
    uncommented the RMI section

I start the first (IP address = 172.17.4.31) with command:
./run.sh  -c default --host=0.0.0.0

I start the second with command:
./run.sh  -Djboss.esb.bind.address=172.17.4.31 -c default --host=0.0.0.0

And on the console of the second instance I get the following exceptions:

16:27:56,335 WARN  [ServiceController] Problem starting service jboss.jmx:type=adaptor,name=Invoker,protocol=jrmp,service=proxyFactory
javax.naming.NameAlreadyBoundException
16:27:43,771 WARN  [ServiceController] Problem starting service jboss:service=proxyFactory,target=ClientUserTransactionFactory
javax.naming.NameAlreadyBoundException
16:28:04,021 ERROR [JNDIRegistration]
javax.naming.NameAlreadyBoundException; remaining name 'juddiv3'
16:28:05,095 ERROR [ExceptionUtil] Queue[/queue/DLQ] startService
javax.jms.InvalidDestinationException: Destination DLQ already exists
16:28:05,101 ERROR [ExceptionUtil] Queue[/queue/ExpiryQueue] startService
javax.jms.InvalidDestinationException: Destination ExpiryQueue already exists
16:28:05,121 ERROR [ExceptionUtil] Topic[/topic/testTopic] startService
javax.jms.InvalidDestinationException: Destination testTopic already exists
16:28:05,125 ERROR [ExceptionUtil] Topic[/topic/securedTopic] startService
javax.jms.InvalidDestinationException: Destination securedTopic already exists
16:28:05,145 ERROR [ExceptionUtil] Topic[/topic/testDurableTopic] startService
javax.jms.InvalidDestinationException: Destination testDurableTopic already exists
16:28:05,167 ERROR [ExceptionUtil] Queue[/queue/testQueue] startService
javax.jms.InvalidDestinationException: Destination testQueue already exists
.....
.....
16:28:05,258 ERROR [ExceptionUtil] Queue[/queue/PrivateDLQ] startService
javax.jms.InvalidDestinationException: Destination PrivateDLQ already exists
16:28:05,289 ERROR [ExceptionUtil] Queue[/queue/QueueWithOwnDLQAndExpiryQueue] startService
javax.jms.InvalidDestinationException: Destination QueueWithOwnDLQAndExpiryQueue already exists
16:28:12,149 ERROR [ExceptionUtil] Queue[/queue/DeadMessageQueue] startService
javax.jms.InvalidDestinationException: Destination DeadMessageQueue already exists

At the end of the startup phase of the second instance I get this message:

16:28:20,559 ERROR [URLDeploymentScanner] Incomplete Deployment listing:
16:28:21,168 ERROR [Server] Root deployment has missing dependencies; continuing Incomplete Deployment listing:

--- MBEANS THAT ARE THE ROOT CAUSE OF THE PROBLEM ---
ObjectName: jboss.jmx:type=adaptor,name=Invoker,protocol=jrmp,service=proxyFactory
  State: FAILED
  Reason: javax.naming.NameAlreadyBoundException
  I Depend On:
    jboss:service=invoker,type=jrmp
    jboss.jmx:type=adaptor,name=Invoker
    jboss:service=Naming
  Depends On Me:
    jboss.admin:service=PluginManager
    jboss.jmx:type=adaptor,name=MBeanProxyRemote,protocol=jrmp

ObjectName: jboss:service=proxyFactory,target=ClientUserTransactionFactory
  State: FAILED
  Reason: javax.naming.NameAlreadyBoundException
  I Depend On:
    jboss:service=invoker,type=unified
  Depends On Me:
    jboss:service=ClientUserTransaction

Are there any on line documents that could help me to properly implement my clustered architecture?
Am I missing any other configuration files to properly setup the connection between the two instances and the jUDDI registry?

Thank you,
Daniele

[massios]

Hello Danielle,

I am trying to cluster JBossESB 4.6 and I am getting same problems.

I have clustered JBoss 5.1GA with Apache and Mod-cluster but now I am trying to cluster the ESB and I am getting problems with the queues

It is trying to create the same queues again (even though they are already created by the first instance going up) and this is creating mayhem.

Have you managed to get a solution?

Nikos

[daniele.sacchetti]

Hello Nikos,

Still not fixed but I've solved some issues and I've moved forward:

Now I've two hosts with AS5.1.0+ESB4.6, a third host configured as jUDDI registry (AS5.1.0) and a forth host with PostgreSQL 8.4.4 (used for JBMessaging clustering).

For ESB clustering I'm not using Apache and Mod-cluster but JGroups+JBMessaging clustering.

I've found some documentation and links to help deploying clustering based on JGroups+JBMessaging clustering:

- Setup JBMessaging clustering:

http://community.jboss.org/message/552180#552180

- Replace hsqldb with other DB for message persistence:

http://community.jboss.org/wiki/JBossESBSwitchDatabase

http://community.jboss.org/wiki/configdatasources

Concerning the exception in my post, I think the problem is due to JNDI problem: I'm using the same JNDI registry located at 172.17.4.31:1099

(java.naming.provider.url=jnp://172.17.4.31:1099) for both AS+ESB hosts and I think (I have not solved this issue yet) this is not the correct solution to have a centralize jUDDI registry. I think the correct solution is as described in steps 2 and 4 of http://community.jboss.org/wiki/JBossESBSwitchDatabase,that is, just based on a shared DB.

Daniele

[massios]

Hello Daniele,

I got it working. I can call the ESB either by HTTP via the apache mod-cluster and by JMS gateway.

I did also some basic steps copied from Ryan Hochstetler (thanks Ryan)

http://community.jboss.org/message/552180#552180

I am using oracle for a db, clustered queues and a clustered post office as explained by Ryan

I am using the -g -u options at the start up. For example for node 2 I do

run.bat -c node2 -gLocalDevMassios -u 239.255.100.100 -b <YOUR IP HERE> -Djboss.messaging.ServerPeerID=2 -Djboss.service.binding.set=ports-01 -Djboss.mod_cluster.jvmRoute=node2

In the example above I am setting the serverpeerid for jboss messaging. I am also setting the jvm route for mod proxy. You do not need to set the jvm_route if you do not cluster http

After having done all those changes I was still getting the errors you mentioned. The extra changes that fixed the problem were on

C:\jboss-5.1.0.GA-ESB4.6\server\node2\deployers\esb.deployer\esb.juddi.xml

C:\jboss-5.1.0.GA-ESB4.6\server\node2\deployers\esb.deployer\jbossesb-properties.xml

These files are copied on startup to

C:\jboss-5.1.0.GA-ESB4.6\server\node2\data\

These files contained the ip's of node1. I changed those ip's to contain the local ones. I think this is something that could be improved in the esb because it took me quite some time to find out what was going on. These files should be fixed for each node of the ESB.

In terms of configuration I have 4 identical ESB nodes each with its own JUDDI, messaging, admin consoles etc.  (Although they all work with the same db)

Our ESB does service routing for a single request. Because of that and because of the clustering some of the processing happens on one node while some happens on another. So the clustering works even within the same request which I am not sure if it is a good idea or not. I would like some answer on whether this can be disabled somehow.

Nikos

[ingedeut]

Hi daniele,

I start the first (IP address = 172.17.4.31) with command:

./run.sh  -c default --host=0.0.0.0

with default you canno't start a JBossESB Cluster , you need all configuration because default configuration don't support clustering.

--Regards

younes