HASingletonDeployer ERROR upon secondary server startup?

[triumphthepup]

We're testing with JBoss 5.1.0, JDK 6. We've have two servers clustered, our app is deployed as an EAR (EJB & WAR). Things seem to run fine, including several aspects of clustering.

We have two EJB3s that depend on the , one using quartz and the other a simple MDB.

Example use of @Depends:

@MessageDriven(activationConfig = {
 @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),
 @ActivationConfigProperty(propertyName = "destination", propertyValue = "queue/BoardPoller") })
@Depends("jboss.ha:service=HASingletonDeployer,type=Barrier")
public class BoardPollerMDB implements MessageListener {...

However upon starting the second server, when the first is already running, we get the following ERROR on the second server:

---------------------------------------------------------
GMS: address is 192.168.71.60:46650 (cluster=beta-SessionCache)
---------------------------------------------------------
16:05:41,159 INFO [org.jboss.cache.RPCManagerImpl] Received new cluster view: [192.168.71.61:40473|25] [192.168.71.61:40473, 192.168.71.60:
46650]
16:05:41,651 INFO [org.jboss.cache.statetransfer.LegacyStateTransferIntegrator] Using version 4096
16:05:41,716 INFO [org.jboss.cache.RPCManagerImpl] Cache local address is 192.168.71.60:46650
16:05:41,716 INFO [org.jboss.cache.RPCManagerImpl] state was retrieved successfully (in 618 milliseconds)
16:05:41,725 INFO [org.jboss.cache.factories.ComponentRegistry] JBoss Cache version: JBossCache 'Cascabel' 3.1.0.GA
16:05:41,921 ERROR [org.jboss.system.server.profileservice.ProfileServiceBootstrap] Failed to load profile: Summary of incomplete deployment
s (SEE PREVIOUS ERRORS FOR DETAILS):

DEPLOYMENTS MISSING DEPENDENCIES:
 Deployment "jboss.j2ee:ear=app-ear.ear,jar=app-ejb.jar,name=BoardPollerMDB,service=EJB3" is missing the following dependencies:
 Dependency "<UNKNOWN jboss.j2ee:ear=app-ear.ear,jar=app-ejb.jar,name=BoardPollerMDB,service=EJB3>" (should be in state "Described", bu
 t is actually in state "** UNRESOLVED Demands 'jboss.ha:service=HASingletonDeployer,type=Barrier' **")

 DEPLOYMENTS IN ERROR:
 Deployment "<UNKNOWN jboss.j2ee:ear=app-ear.ear,jar=app-ejb.jar,name=BoardPollerMDB,service=EJB3>" is in error due to the following reas
 on(s): ** UNRESOLVED Demands 'jboss.ha:service=HASingletonDeployer,type=Barrier' **

16:05:42,035 INFO [org.apache.coyote.http11.Http11AprProtocol] Starting Coyote HTTP/1.1 on http-0.0.0.0-8080
16:05:42,077 INFO [org.apache.coyote.ajp.AjpAprProtocol] Starting Coyote AJP/1.3 on ajp-0.0.0.0-8009
16:05:42,085 INFO [org.jboss.bootstrap.microcontainer.ServerImpl] JBoss (Microcontainer) [5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=20090
5221634)] Started in 1m:12s:414ms

Using the JMX Console I can see that the state of the HASingletonDeployer is in "Created" state. It seems that JBoss is looking for a state of "Described". Am I missing something? Is there a way to avoid this error message? As far as I can tell there is no actual error occurring.

[brian.stansberry]

The "DEPLOYMENTS MISSING DEPENDENCIES:" error message is rather cryptic; I'm looking into it to make sure it means what I think it means. But my interpretation is not that the microcontainer expects the bean to be in "Described" but rather that it doesn't recognize the name at all.

Looking at the Barrier in the jmx-console and seeing "Create" state is what I would expect on the non-master node, so that aspect of it looks good.

Hmm, a possibility here is the BarrierController is directly registering the Barrier in JMX, and the microcontainer is bypassed. So the microcontainer is unaware of the barrier.

[brian.stansberry]

Hmm, a possibility here is the BarrierController is directly registering the Barrier in JMX, and the microcontainer is bypassed. So the microcontainer is unaware of the barrier.

OK, I checked that theory out and the microcontainer isn't bypassed; things work as they should.

[brian.stansberry]

I tried deploying a simple pojo with a dependency on the barrier:

<deployment xmlns="urn:jboss:bean-deployer:2.0">

 <bean name="Test"
 class="java.lang.Object">

 <depends>jboss.ha:service=HASingletonDeployer,type=Barrier</depends>

 </bean>

</deployment>

That gave me the result I expected on the 2nd node

DEPLOYMENTS MISSING DEPENDENCIES:
Deployment "Test" is missing the following dependencies:
Dependency "jboss.ha:service=HASingletonDeployer,type=Barrier" (should be in state "Start", but is actually in state "Create")

DEPLOYMENTS IN ERROR:
Deployment "jboss.ha:service=HASingletonDeployer,type=Barrier" is in error due to the following reason(s): Create

Using the barrier is going to result in log complaints about unresolved dependencies, until whatever trips the barrier occurs, e.g. the 2nd node becomes master. Getting rid of the log complaint would be nice, but I don't expect it to happen soon.

Your message is different, perhaps because of @Depends inside an EJB.

If you shut down the first node and the 2nd becomes master, does the EJB deploy properly?

[jaikiran]

Brian,

I was able to reproduce this with a simple SLSB:

import javax.ejb.Stateless;

import org.jboss.ejb3.annotation.Depends;

@Stateless
@Depends("jboss.ha:service=HASingletonDeployer,type=Barrier")
public class CalculatorBean implements CalculatorRemote, CalculatorLocal

Deploys fine on node1. When node1 is started and node2 is brought up with this deployment, it fails with (cryptic MC dependency error):

DEPLOYMENTS MISSING DEPENDENCIES:
 Deployment "jboss.j2ee:jar=jboss-ejb3-tutorial-stateless.jar,name=CalculatorBean,service=EJB3" is missing the following dependencies:
 Dependency "<UNKNOWN jboss.j2ee:jar=jboss-ejb3-tutorial-stateless.jar,name=CalculatorBean,service=EJB3>" (should be in state "Described", but is actually in state "** UNRESOLVED Demands 'jboss.ha:service=HASingletonDeployer,type=Barrier' **")
 Deployment "jboss.j2ee:jar=jboss-ejb3-tutorial-stateless.jar,name=CalculatorBean,service=EJB3_endpoint" is missing the following dependencies:
 Dependency "jboss.j2ee:jar=jboss-ejb3-tutorial-stateless.jar,name=CalculatorBean,service=EJB3" (should be in state "Configured", but is actually in state "PreInstall")

DEPLOYMENTS IN ERROR:
 Deployment "<UNKNOWN jboss.j2ee:jar=jboss-ejb3-tutorial-stateless.jar,name=CalculatorBean,service=EJB3>" is in error due to the following reason(s): ** UNRESOLVED Demands 'jboss.ha:service=HASingletonDeployer,type=Barrier' **

The real issue is that a @Depends add a depedency on a MC bean with state = INSTALLED/STARTED (don't exactly remember which of these 2).

However, looking at the JMX console, i can see that the

jboss.ha:service=HASingletonDeployer,type=Barrier

on Node1 is in STARTED state whereas on Node2 its in CREATED state. I don't know much of this type=Barrier MBean, but probably that's expected? This effectively results in MC dependency resolution error.

[jaikiran]

Ah, did not see your 15:12 post, while typing my reply.

[triumphthepup]

Thanks for looking into this.

If you shut down the first node and the 2nd becomes master, does the EJB deploy properly?

Yes, when I shut down node 1, the EJB does seem to properly deploy / start on node 2 when it becomes the master. So it appears to work. What alarmed me was the logging level of ERROR, but it sounds like that may simply be a side effect of how things are wired with the MC.

[triumphthepup]

Shall I write up a JIRA ticket for this issue so that it shows up on the appropriate "radars"?

We're getting ready to deploy to production with 5.1.0 so we're interested in tracking this topic.

[brian.stansberry]

Yes, please do. In the JBAS project. Actually, I see two issues in this thread:

1) Figuring out why the @Depends on your EJB results in a different error message than the expected one that I got when I deployed my example POJO.

2) Coming up with a mechanism to avoid the ERROR message altogether when the missing dependency is due to a known, acceptable condition. Perhaps some SPI, where processors can register themselves with the service that logs this and be given access to the objects that drive the logging. An impl of the SPI that works with/is part of a BarrierController would flag relevant missing dependencies as not being error conditions.

[farmerzen]

Hi to all,

Got the same error in the second node..... got very alarmed too, tought i got some problem in my deployment process on jboss 6... after google for a while came across this thread. Got a look on jira, and find out that this "problem" was not solved in jboss6.... probably it will be on jboss7. https://issues.jboss.org/browse/JBAS-7096

https://issues.jboss.org/browse/JBAS-7096It is not pretty to see error logs on startup, but what shall we do?!