JBoss 3.2.1 cluster-wide hot-deployment

[sysuser1]

Hi All,

I managed to cluster two JBoss 3.2 nodes, which was pretty much out of the box requiring very little changes to the "all" configuration. I have outlined the scenario below :

1) I start node A
2) I deploy helloworld on A
3) I start node B
4) Node B gets helloworld from node A and deploys it
5) Clients access helloworld in a load-balanced fashion
6) I re-deploy a new helloworld on node A
7) But node B keeps responding with the OLD helloworld
8) I un-deploy helloworld on machine A
9) But node B keeps serving the OLD helloworld

It seems to me, the cluster-wide hot deployment only during the starting up of the nodes. Any subsequent changes (re-deployment, un-deployment) doesn't seem to propagate. Is this normal behaviour?

Also, the client that I ran, usually on the two machines, will get the EJB from the respective node. But if the JBoss server is down on that respective node, it will fail. It doesn't locate the other node. What code should be put in the client to indicate that the client can fail-over to some other JBoss server in case one doesn't work.

The client as it is now :

====

Properties prop = new Properties();
prop.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory");
prop.put(Context.URL_PKG_PREFIXES, "jboss.naming:org.jnp.interfaces");
prop.put(Context.PROVIDER_URL, "jnp://10.2.3.114:1100,jnp://10.2.3.78:1100");
InitialContext iContext = new InitialContext(prop);

HelloWorldHome home = (HelloWorldHome) iContext.lookup("greetings/HelloWorld");
HelloWorld object = home.create();
System.out.println("EJB says : " + object.sayHello());

====

I'm running JBoss 3.2.1 on Debian Linux 3.0.

Regards,

Anuradha.

[jackpot]

Hi,

There might be a problem with your cluster-service.xml. If you can forward it or list it out in the forum, I might be able to help.

Thanks,
JP

[slaboure]

- do you put your application in /deploy or in /farm?
- have you activated, for your EJB, the clustering (for each EJB you are using from your client side) i.e. set the True tags in your or tags of jboss.xml?

Cheers,


sacha

[sysuser1]

The cluster-service.xml file :

======

<?xml version="1.0" encoding="UTF-8"?>

<!-- ===================================================================== -->
<!-- -->
<!-- Sample Clustering Service Configuration -->
<!-- -->
<!-- ===================================================================== -->





<!-- ==================================================================== -->
<!-- Cluster Partition: defines cluster -->
<!-- ==================================================================== -->



<!-- Name of the partition being built -->
DefaultPartition
<!-- Determine if deadlock detection is enabled -->
False
<!-- The JavaGroups protocol configuration -->


<!-- UDP: if you have a multihomed machine,
set the bind_addr attribute to the appropriate NIC IP address -->
<!-- UDP: On Windows machines, because of the media sense feature
being broken with multicast (even after disabling media sense)
set the loopback attribute to true -->
<UDP mcast_addr="228.1.2.3" mcast_port="45566"
bind_addr="10.2.3.114"
ip_ttl="64" ip_mcast="true"
mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
loopback="false" />
<PING timeout="2000" num_initial_members="3"
up_thread="true" down_thread="true" />
<MERGE2 min_interval="5000" max_interval="10000" />
<FD shun="true" up_thread="true" down_thread="true" />
<VERIFY_SUSPECT timeout="1500"
up_thread="true" down_thread="true" />
<pbcast.STABLE desired_avg_gossip="20000"
up_thread="true" down_thread="true" />
<pbcast.NAKACK gc_lag="50" retransmit_timeout="300,600,1200,2400,4800"
up_thread="true" down_thread="true" />
<UNICAST timeout="5000" window_size="100" min_threshold="10"
down_thread="true" />
<FRAG frag_size="8192"
down_thread="true" up_thread="true" />
<pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
shun="true" print_local_addr="true" />
<pbcast.STATE_TRANSFER up_thread="true" down_thread="true" />




<!-- ==================================================================== -->
<!-- HA Session State Service for SFSB -->
<!-- ==================================================================== -->


jboss:service=DefaultPartition
<!-- Name of the partition to which the service is linked -->
DefaultPartition
<!-- JNDI name under which the service is bound -->
/HASessionState/Default
<!-- Max delay before cleaning unreclaimed state.
Defaults to 30*60*1000 => 30 minutes -->
0


<!-- ==================================================================== -->
<!-- HA JNDI -->
<!-- ==================================================================== -->


jboss:service=DefaultPartition
<!-- Added by Anuradha -->
10.2.3.114
<!-- -->
<!-- Name of the partition to which the service is linked -->
DefaultPartition
<!-- RmiPort to be used by the HA-JNDI service
once bound. 0 => auto. -->
0
<!-- Port on which the HA-JNDI stub is made available -->
1100
<!-- Backlog to be used for client-server RMI
invocations during JNDI queries -->
50

<!-- Multicast Address and Group used for auto-discovery -->
230.0.0.4
1102

<!-- Client socket factory to be used for client-server
RMI invocations during JNDI queries -->
<!--attribute name="ClientSocketFactory">custom</attribute-->
<!-- Server socket factory to be used for client-server
RMI invocations during JNDI queries -->
<!--attribute name="ServerSocketFactory">custom</attribute-->



<!--
0
custom
custom
custom
-->


<!-- ==================================================================== -->
<!-- Distributed cache invalidation -->
<!-- ==================================================================== -->


jboss:service=DefaultPartition
jboss.cache:service=InvalidationManager
jboss.cache:service=InvalidationManager
DefaultPartition
DefaultJGBridge




======

[sysuser1]

Also, I put the EJBs in /farm and below is my xdoclet generated jboss.xml file :

=====

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE jboss PUBLIC "-//JBoss//DTD JBOSS 3.0//EN" "http://www.jboss.org/j2ee/dtd/jboss_3_0.dtd">



<enterprise-beans>

<!--
To add beans that you have deployment descriptor info for, add
a file to your XDoclet merge directory called jboss-beans.xml that contains
the , and <message-driven></message-driven>
markup for those beans.
-->


<ejb-name>greetings/HelloWorld</ejb-name>
<jndi-name>greetings/HelloWorld</jndi-name>
true
<cluster-config>
<partition-name>DefaultPartition</partition-name>
<home-load-balance-policy>org.jboss.ha.framework.interfaces.RoundRobin</home-load-balance-policy>
<bean-load-balance-policy>org.jboss.ha.framework.interfaces.RoundRobin</bean-load-balance-policy>
</cluster-config>



</enterprise-beans>

<resource-managers>
</resource-managers>



=====

Anuradha.

[buckman1]

Actually, 3.2.2RC1 fixes the issue of un-deploying a farmed node. However, it does not yet re-deploy a farmed node. We see this occur... if I drop a file in /farm, AND I have farm-service.xml in my /deploy dir specifying ./farm as the farm dir, it will automatically pick it up and deploy it. If I remove the file from my /farm dir, I see the other nodes un-deploy and remove the file out of their /farm dir. If I drop in the file again, it gets deployed again. The problem seems to be in re-deployment not farming properly. I think Sacha is working on that for a 3.2.2 final (Sacha??). Once that is fixed, the redeployment for farming should be working.

Sacha, any idea when 3.2.2 will be final?

[slaboure]

3.2.2. is for very soon.

As for your problem, I don't get it: you say that removing it and putting it again works but that re-deployment doesn't. What is the difference?

Again, if you think it fails, create a small scenario, put jboss in debug mode, play the scenario, fully describe it in an e-mail and send be both log files (one for each node). Otherwise, it is not very helpful.

Cheers,



sacha

[buckman1]

Sacha,

I agree with you it doesn't make sense. But others report the same thing.

Try this. Maybe you have. If I deploy a jar in the farm dir, it properly propogates to each node, deploys, etc. That works every time as we have seen. Where it fails, is if we bring a new node online that doesn't have the file yet, it just sits there. It never gets propogated to that new node. That is bad.

Then, try this, try removing the jar from one node, delete the .jar file out of the farm dir. This also seems to work for us, although we only tested it once. The other nodes also delete/undeploy the jar. That works fine. But rather than deleting, if you drop a new version of the .jar file in the farm dir, it properly redeploys on the server it is in, but the other nodes do not get updated. We have let it sit for a few minutes and never saw it get redeployed. Therefore, that also seems to fail, at least for us.

Please try those two tests. If you can get them both to work, please let us know how long if any wait was required, to see the redeploy and/or the new node to get their copy of the .jar and redeploy it. If you can't get those to work, then obviously you will be aware that it is a bug (or unimplemented feature at that point).

In a couple of days I will test all this again. We'll try the port 1100 (we are using 1099), hajndi-client.jar, and all that stuff and see if that helps also. I'll report my findings, but please give these a try and see if you have no problems with them or not.

Thanks.

[slaboure]

I don't have time to test it now, please add a bug report to sf.net/projects/jboss and assign it to slaboure.

thanks.

sacha

[buckman1]

done.

[pbrant]

Sacha,

i have also experienced the same (?) problem with farming, but my test scenario was slightly different...

i have one web server, with servlets and jsp, and two app servers (1 and 2), clustered, and deployed with stateless session and entity beans.

the web server is configured with HA-JNDI, and will call session beans only...


1. startup JBoss on both app servers (1 and 2), but do not deploy any application resources to JBoss...just startup JBoss as an empty shell, to keep it raw and simple.

both app servers synch with each other and this is indicated by a message to the console.


2. deploy the application ear file to the /farm directory on app server 1,

it deploys to app server 1 and then deploys to app server 2 across the LAN...

all is good.


3. startup the web server, configured with HA-JNDI,


4. Test the application using an Internet browser

login works, and all is fine.

5. take app server 1 down by pulling the plug...

6. keep testing the app by clicking on something...

failover to app server 2 works, and all is good.


7. now restart app server 1....

and nothing...

the application does not redeploy to app server 1, and there is no redeployment to app server 2...



perhaps this test scanario helps.


Peter

[buckman1]

Peter,

I am glad we are not the only ones seeing this. ;) I am sure Sacha will fix it soon.

A few thoughts...

You say you take app server 1 down, then bring it back up? Why would you need to redeploy to that server? As long as appserver 1 communicates with app server 2 again, and you have not deployed a NEW .ear to either one, they should both be the same deployed app. If anything, I think what you may be describing is that you are not seeing load balancing to the rebooted appserver 1?

In our situation, we did not try that scenario yet, or should I say, we did do that, but we weren't testing to see if it load balanced, although now that you bring that up, I recall we saw similar issues, in that disconnecting the network cable and reconnecting it would show messages on the nodes, but the client would never start load balancing back to the node that was reconnected. We also saw other unusual issues. We have 3 nodes, load balancing starts round-robin to all 3. When we brought down #2, the load balancing would only go to #1, and not #1 and #3. When we brougth #2 back up, it would stay on #1. A new client WOULD go to #1 and #2 though, so while our first client would only go to #1 and not round robin beyond that when we brought #2 down, starting a new client would never get to #3! It was definitely odd.

The way it should work is if #2 goes down, client 1 should see #1 and #3. Recall in the docu that each time a client makes a call, the updated "view" is sent back. So if #2 goes down, the updated view should be #1 and #3. For some reason, I don't know for sure, but it appears that didn't happen. And when client #2 starts, it only sees #1 and #2 (after #2 was restarted), but not #3. It's as if the "chain" was broken when #2 went down, and now #3 doesn't seem to work. When we brought #3 down then back up, a new client then saw #2 and #3, but missed #1.

I think its still a work in progress. Sacha and Bill have done a great job, and given its free nature and their full time careers, it may take time to get all this working perfectly, but so far its pretty impressive minus a few nagging issues.

Sacha, question for you. Given these few issues, is it coneivable that you'll fix this in 3.2.x and not move it off to 4.x? In our particular case, we wont be moving off of 3.2.x for some time. We just moved off of 2.4.x to 3.0.x. I just want to make sure that since we'll be on 3.2.x for some time, that these issues are going to be resolved in this release and not put off for 4.x.

Thanks.

[jbossnz]

Is this behaviour something to do with the default load-balancing options ?

Seems like stateless session beans have both a home-load-balance-policy and bean-load-balance-policy of RoundRobin (chap 5 clustering ejb of April 2003, the latest Clustering docs, page 35 and 35)
but
Stateful Session beans and Entity Beans have a home-load-balance-policy of RoundRobin BUT a bean-load-balance-policy of FirstAvailable (chap 5 clustering ejb of April 2003, the latest Clustering docs, page 37 and 38)

[buckman1]

I don't see how. The problem is not the behavior, its that the behavior seems to break while in use. If you have node 1,2 and 3 all working, and your first client connects and you see it working on 1, then 2, then 3, that is fine. Its if node 2 goes down, that the client should then go to 1, then 3, then 1, then 3 and so on. If node 2 comes back up, it should then go from 3, to 1, to 2, to 3 and so on. This is not occurring. We are not using stateful ejb either. I tend to believe that session should be round robin in stateless and first available in stateful, and entity should always be first available.

As a co worker points out, if the user connects to node 1 and that node loads a ton of entity data that is cached, then gets connected to node 2 on their next call, all that cached data is not yet available, so now it has to be loaded again, and so forth. Ideally, once a client connects to a server, they should remain with that server unless it fails, thus the first available algorithm.

[sysuser1]

After using 3.2.2RC1 (as buckman1 suggested) the hot-deployment problems went away. Thanks all! Now the farming works as expected. Even undeployment and redeployment works. Although I had a slight issue with the redeployment, undeploying and deploying again fixed that.

The issue I have now is with the client. I want the client to query the nodes in a roundrobin fashion. My client :

== snip ==

Properties prop = new Properties();
prop.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory");
prop.put(Context.URL_PKG_PREFIXES, "jboss.naming:org.jnp.interfaces");
prop.put(Context.PROVIDER_URL, "jnp://10.2.3.114:1100,jnp://10.2.3.78:1100");
InitialContext iContext = new InitialContext(prop);

HelloWorldHome home = (HelloWorldHome) iContext.lookup("greetings/HelloWorld");
HelloWorld object = home.create();
System.out.println("EJB says : " + object.sayHello());

== snip ==

The ejb that responds is always on the machine that the client runs from. The other node isn't queried even when the jboss on the client machine dies. Any input appreciated,

Btw, I am running the xdoclet generated run-helloworld-client.sh. The jndi.properties for it :

==
java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces
==

Anuradha.