Deploying EJB 3 ear via Farming

[marcreis]

Hi,
my config in short:
I have configured a 4.0.4rc2 to run clustered on one machine with farming for EJB 3 (followed the guides I found in the wiki and the admin guide. Merged the all and EJB 3 clustered installations, which works so far). I have a small test app which I wish to deploy via farming. It consists of an ear, with a HAR, JAR (for EJB 3) and a WAR.
Too sum up the package, it consist of one tiny EJB 3 (marked @Clustered with remote and local interface (bound through the @RemoteBinding and @LocalBinding annotations).
So far, everything works (the ear works for none farming deployment on one instance of the server).
When I deploy the EAR in the /farm Folder the 1st node does fine and deploys the EAR and pushes it to the 2nd node. This one receives the EAR and also starts deploying.
The Problem:
comes up at this point: as the 2nd node starts deploying, it try's to bind the remote and local interfaces to the JNDI tree (HAJndi Tree), which results in a conlict, since they are already bound, on the consol this looks something like this (shortend version):

...
2006-04-26 16:46:43,187 DEBUG [org.jboss.ejb3.ServiceDelegateWrapper] Starting failed jboss.j2ee:ear=testEJB.ear,jar=testEJB.jar,name=NatPerLstBean,service=EJB3
javax.naming.NamingException: Could not bind statless proxy with ejb name NatPerLstBean into JNDI under jndiName: /EJBTEST/NatPerLstBean/remote [Root exception is javax.naming.NameAlreadyBoundException]
 at org.jboss.ejb3.stateless.BaseStatelessProxyFactory.start(BaseStatelessProxyFactory.java:70)
 at org.jboss.ejb3.stateless.StatelessClusterProxyFactory.start(StatelessClusterProxyFactory.java:97)
 at org.jboss.ejb3.ProxyDeployer.start(ProxyDeployer.java:85)
 at org.jboss.ejb3.SessionContainer.start(SessionContainer.java:85)
 at org.jboss.ejb3.stateless.StatelessContainer.start(StatelessContainer.java:80)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.jboss.ejb3.ServiceDelegateWrapper.startService(ServiceDelegateWrapper.java:99)
 at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
 at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
...

ending with

16:46:43,187 INFO [EJB3Deployer] Deployed: file:/C:/work/_programme/jboss-4.0.4
.CR2/server/default2/tmp/deploy/tmp39879testEJB.ear-contents/testEJB.jar
16:46:43,203 INFO [TomcatDeployer] deploy, ctxPath=/testEJBWar, warUrl=.../tmp/
deploy/tmp39879testEJB.ear-contents/testEJBWar-exp.war/
16:46:43,421 INFO [EARDeployer] Started J2EE application: file:/C:/work/_progra
mme/jboss-4.0.4.CR2/server/default2/farm/testEJB.ear
16:46:43,421 ERROR [FarmMemberService] Incomplete Deployment listing:

--- MBeans waiting for other MBeans ---
ObjectName: jboss.j2ee:ear=testEJB.ear,jar=testEJB.jar,name=NatPerLstBean,servic
e=EJB3
 State: FAILED
 Reason: javax.naming.NamingException: Could not bind statless proxy with ejb n
ame NatPerLstBean into JNDI under jndiName: /EJBTEST/NatPerLstBean/remote [Root
exception is javax.naming.NameAlreadyBoundException]

--- MBEANS THAT ARE THE ROOT CAUSE OF THE PROBLEM ---
ObjectName: jboss.j2ee:ear=testEJB.ear,jar=testEJB.jar,name=NatPerLstBean,servic
e=EJB3
 State: FAILED
 Reason: javax.naming.NamingException: Could not bind statless proxy with ejb n
ame NatPerLstBean into JNDI under jndiName: /EJBTEST/NatPerLstBean/remote [Root
exception is javax.naming.NameAlreadyBoundException]

I thought I had configured all correct so far, but I guess I am missing something... Thanks for any help hint's and links ! :)

[marcreis]

Depending on my config I also get this problem with the "EJB3EntityTreeCache" on startup of the second server.
I really would appriciate a tip, since I still cant find any thing in the wiki, the there linked documents or the other guides and handbooks I have (google wont help either, and reading the stuff the fourth or fifth time also did not do the job).

[ben.wang]

Sorry but I don't get it. Why is the second node produced the "remote" HA-JNDI name already bound error? Did you put another same app under the regular deploy dir?

[marcreis]

Hi,
no I did not, that is what is confusing me also.
The deploy dir contains nothing but the elements that come with the default deploy dir (no personal stuff at all, I cleared all out and used a clean install made out of the all and ejb 3.0 cluster folders, to make sure there is nothing I have overseen).
I then start both servers, one after the other. And both find each other nicely. Then I drop my testEJB.ear into the /farm folder of node 1 (which is until now empty on both nodes). Node 1 then starts to deploy nicely and at the end does its push to the cluster, node 2 receives this push and starts deploying also and then the "already bound" exception is thrown for the interfaces, since they are already in the JNDI - Tree, which seems logically, since HAJindi is supposed to provide them to all in the cluster (or ?). I thought what would happen "automatically" is, that the second node would register that, and then just "link" to the same bindings.

[marcreis]

It seems to be a problem of my Test ear file. We deployed a larger application made out of jar, har, ejb3 and ear files and they were distributed fine (none are market @clustered though).

[smarlow]

As a test, could you take the application out of the farm folder. I would like to copy it into the deploy folder on each cluster node. I suspect that the same problem will reproduce (from what you have said).

This will help determine if the problem is caused by combining farm deployment with the deployment of the clustered application.

Should be interesting.

Scott

[marcreis]

You are right. It has the same effect if I manually do the deploy in each deploy folder. It seems to be my ear (and what is within) that causes the trouble. We have a real project we ported to 4.0.4rc2 and it worked, no matter if farmed or not (at least in the short time we tested it).

The test app is actually very simple, but I dont get why it produces such strange effects. If I leave the app in the farm folder and restart the server it will throw errors right away, that is something it doesnt do when I start the server and the ear is in the deploy folder.
After some more trying out I now also have the problem that the jmx consol's JNDI Tree view wont show me the bean and interfaces that I deploy in the ear.... very strange, the hibernate factory is visibile though.
The bean and interfaces are there though, when I run a client in debug mode I can see that it gets access via HAJNDI to the tree and can fetch the interface, but then when I try to call a mehtod through the interface i looked up, I get a cluster invocation error. I allready set back to server folder to what I started with (merged all and ejb3.0 clusterd) to make sure I didnt mess up the config too much, but the effect holds on.
I also get the effect if I just deploy/farm the testEJB.jar.

My ear and the containing jar, har and war look like this( I guess the configs would be more interresting, but I've got to start somewhere;):

testEJB.ear
 +--testEJB.jar
 +--testEJBHib.har
 +--testEJBWar.war
 +--META-INF/
 +--application.xml
 +--jboss-app.xml

testEJB.jar
 +--ejb-jar.xml
 +--hibernate.cfg.xml
 +--jndi.properties
 +--log4j.properties
 +--de/somewhere/test/
 +--ejbean/
 +--NatPersLstBean.class
 +--HelloWorld.class
 +--interfaces/
 +--IHelloWorld.class
 +--IHelloWorldRemote.class
 +--INatPersLst.class
 +--INatPersLstLocal.class
 +--INatPersLstRemote.class
 +--persi/
 +--NatPers.class
 +--util/
 +--HibTestUtil.class

testEJBHib.har
 +--NatPers.hbm.xml
 +--META-INF/
 +--hibernate-service.xml
+--de/somewhere/test/persi
 +--NatPers.class

testEJBWar.war
 +--hibernate.cfg.xml
 +--jndi.properties
 +--log4j.properties
 +--NatPersPage.html
 +--META-INF/
 +--WEB-INF/
 +--web.xml
 +--NatPers/
 +--Count2Ten.jsp
 +--LilstNatPers.jsp
 +--de/somewhere/test/
 +--ejbean/
 +--NatPersLstBean.class
 +--HelloWorld.class
 +--interfaces/
 +--IHelloWorld.class
 +--IHelloWorldRemote.class
 +--INatPersLst.class
 +--INatPersLstLocal.class
 +--INatPersLstRemote.class
 +--persi/
 +--NatPers.class
 +--util/
 +--HibTestUtil.class

Since I have the some problem with just the jar, I guess that should be the start point (the jar depends on the har, which I deploy before).
My ejb-jar.xml is mainly empty just contains the "main" xml node, but nothing defined or configured there.
I guess the jndi.properties could be interresting, so here it comes

#----- RMI
#java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
#java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces
#java.naming.provider.url=192.168.111.222:1099
#java.naming.provider.url=localhost:1099
#----- HTTP
#java.naming.factory.initial=org.jboss.naming.HttpNamingContextFactory
#java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces
#java.naming.provider.url=192.168.111.222:8080/invoker/JNDIFactory

java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces
jnp.partitionName=DefaultPartition
java.naming.provider.url=192.168.111.222:1100

For the cache the hibernate.cfg.xml might also be interessting:

<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE hibernate-configuration PUBLIC
 "-//Hibernate/Hibernate Configuration DTD 3.0//EN"
 "http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
 <session-factory>
 <!-- Database connection settings -->
 <property name="connection.driver_class">oracle.jdbc.OracleDriver</property>
 <property name="connection.url">jdbc:oracle:thin:@thedb.somewhere.de:1521:THEDBNAME</property>
 <property name="connection.username">myname</property>
 <property name="connection.password">mypass</property>

 <!-- JDBC connection pool (use the built-in) -->
 <!-- <property name="connection.pool_size">1</property> -->
 <!-- JDBC connection pool (use the c3p0 connection pool (a foreign lib included in hib's lib)) -->
 <property name="c3p0.min_size">1</property>
 <property name="c3p0.max_size">10</property>
 <property name="c3p0.timeout">1800</property>
 <property name="c3p0.max_statements">25</property>

 <!-- SQL dialect -->
 <property name="dialect">org.hibernate.dialect.Oracle9Dialect</property>

 <!-- Enable Hibernate's automatic session context management -->
 <property name="current_session_context_class">thread</property>

 <!-- Echo all executed SQL to stdout -->
 <property name="show_sql">false</property>

 <!-- Drop and re-create the database schema on startup -->
 <property name="hbm2ddl.auto">create</property>
 <mapping resource="NatPers.hbm.xml"/>

 <!-- Clustering -->
 <property name="hibernate.cache.provider_class"
 value="org.jboss.ejb3.entity.TreeCacheProviderHook"/>
 <property name="hibernate.treecache.mbean.object_name"
 vlaue="jboss.cache:service=EJB3EntityTreeCache"/>
 <property name="hibernate.cache.use_query_cache"
 value="true"/>


 <!-- Define cached entities and collections -->
 <class-cache class="de.somewhere.test.persi.NatPers" usage="transactional"/>



 <!-- Disable the second-level cache -->
<!-- <property name="cache.provider_class">org.hibernate.cache.NoCacheProvider</property>
-->

 </session-factory>
</hibernate-configuration>

The bean ist marked with:

@Remote({INatPersLstRemote.class})
@Local({INatPersLstLocal.class})
@RemoteBinding(jndiBinding="EJBTEST/NatPerLstBean/remote")
@LocalBinding(jndiBinding="comp/env/EJBTEST/NatPerLstBean/local")
@Stateless
@Clustered(partition= "DefaultPartition", loadBalancePolicy = org.jboss.ha.framework.interfaces.RoundRobin.class)
public class NatPerLstBean implements INatPersLstLocal,
 INatPersLstRemote {
...

The interfaces are marked also with @Local and @Remote.
I do the lookup in the client with

...
INatPersLst persLstI = INatPersLst)initCtx.lookup(INatPersLstRemote.JNDIName);
NamingEnumeration myNe = initCtx.list("/"); //just for testing

Object myObj = persLstI.getNatPersLst(lettersLastName); //Here it throws the clustered invocatio error since I used the last part in the jndi.properties for HAJNDI instead of the RMI marked part for JNDI
myNatPersLst = (List<NatPers>)myObj;
..

I hope the information is usefull, and doesnt just eat bandwith ;)

[marcreis]

... as always, just after posting, one problem or cause is solved. The culster invocation error was due to a typo (I read over it about 20times yesterday and did'nt see it, sorry) in the hibernate.cfg.xml at

<!-- Define cached entities and collections -->
 <class-cache class="de.somewhere.test.persi.NatPers" usage="transactional"/>

I had two letters mixed up. Now the invocation works.

Still the problem with the deployment causing the "allready bound error" ist there and the fact that I cant see my bean and interfaces in the JNDI tree via jmx consol (which is new).
As said, I run two instances of Jboss on one mashine, when I call the bean via the client multiple times (one non and one web app), it accesses the first mashine (do to the jndi.properties), should round robin still do its work and alternate the load ? (I thought so, but it doesnt, the way I'm doing it at the moment. Just the bean on one maschine is used).

Thx !

[marcreis]

There it is again...

java.lang.RuntimeException: cluster invocation failed, last exception was:
 at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:115)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:98)
 at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:46)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:98)
 at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:40)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:98)An Error has accured: java.lang.RuntimeException: cluster invocation failed, last exception was:
 at org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:41)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:98)
 at org.jboss.ejb3.stateless.StatelessClusteredProxy.invoke(StatelessClusteredProxy.java:85)
 at $Proxy1.getNatPersLst(Unknown Source)
 at de.somewhere.test.client.ClientNatPers.getNatPersLst(ClientNatPers.java:77)
 at de.somewhere.test.client.ClientNatPers.main(ClientNatPers.java:35)
Caused by: org.jboss.aop.NotFoundInDispatcherException: Object with oid: jboss.j2ee:ear=testEJB.ear,jar=testEJB.jar,name=NatPerLstBean,service=EJB3 was not found in the Dispatcher
 at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:83)
 at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:828)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:681)
 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:358)
 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:398)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:239)

I deployed (in the /deploy folder) the ear, first in the 2nd node, then the 1st node. So the "could not bind to jndi..." appeared on the first node. I then tried to access it via the non web-client, and the above error accured .
I guess thats the also the reason there is no load balancing happening.
If I deploy in ascending order (node1,node2) this doesnt happen (I guess since only node 1 is "used" ).

[marcreis]

I should have noticed it, the problem is that the jndi.properties got into the jar and ear package, so ofcourse causing trouble. Corrected the ant script and I could continue on to the next problems ;)