Controlling deployments via a barrier bean

[brian.stansberry]

I've been looking at better ways to do HASingleton deployments vs the current (broken) scanner-based approach (http://jira.jboss.com/jira/browse/JBAS-5563).

I've long thought that a better way to do this would be by adding a deployer that adds a dependency on "barrier" bean to deployments that have appropriate metadata. Similar to the JMX-based jboss.ha:service=HASingletonDeployer,type=BarrierController concept, but with an MC bean.

An example set of deployers would be:

<!-- For deployments that have HASingletonMetaData attached,
 establishes a dependency on a barrier bean that is only
 satisfied when this node becomes the singleton master
 -->
 <bean name="HASingletonBarrierDependencyDeployer"
 class="org.jboss.ha.singleton.deployer.HASingletonBarrierDependencyDeployer">

 <property name="dependencyNamesByPartition">
 <map keyClass="java.lang.String" valueClass="java.lang.String">
 <entry>
 <key>${jboss.partition.name:DefaultPartition}</key>
 <value>HASingletonBarrier</value>
 </entry>
 <!-- Add more entries if you deploy more HAPartitions -->
 </map>
 </property>

 </bean>

 <!--
 Adds HASingletonMetaData to deployments that come from the specified
 URLs. Provides support for the legacy deploy-hasingleton directory
 concept.
 -->
 <bean name="HASingletonMetadataDeployer"
 class="org.jboss.ha.singleton.deployer.VFSHASingletonMetadataDeployer">

 <property name="partitionName">${jboss.partition.name:DefaultPartition}</property>

 <property name="URIList">
 <list elementClass="java.net.URI">
 <!-- Standard HASingleton deployment directory -->
 <value>${jboss.server.home.url}deploy-hasingleton/</value>
 </list>
 </property>
 </bean>

In deploy we'd have these beans:

 <!-- Bean on which HASingletons will depend. ControllerMode is MANUAL,
 so will not deploy until HASingletonBarrierInstaller installs it.
 -->
 <bean name="HASingletonBarrier" class="java.lang.Object" mode="MANUAL"/>

 <!-- Tracks cluster topology and tells the MainDeployer to bring the
 HASingletonBarrier to INSTALLED when it becomes singleton master, and
 to bring it to NOT_INSTALLED when it is no longer the master.
 -->
 <bean name="HASingletonBarrierInstaller"
 class="org.jboss.ha.singleton.deployer.HASingletonBarrierInstaller">

 <property name="barrierBeanName">HASingletonBarrier</property>
 <property name="mainDeployer"><inject bean="MainDeployer"/></property>
 <property name="HAPartition"><inject bean="HAPartition"/></property>

 </bean>

That's the concept at least. One issue I see is on the non-singleton-master nodes the MainDeployer.checkComplete() impl is going to report an unsatisfied dependency, which will get logged at ERROR. Not good.

Any comments on this basic approach to controlling deployments? If it seems reasonable, can we add a mechanism (perhaps a flag in DependencyItem) such that certain unresolved dependencies are not logged as an error?

(Note it's possible this will just work w/o the logging. I'm working through other issues and haven't gotten it to work right yet. But looking at the code it seems clear there will be ERROR logging.)

[starksm64]

I don't think a dependency is the way this should be handled as its not an optional dependency. Your using a dependency to try to identify the singleton bean in the cluster. Really we don't want the unelected singleton beans to be exposed as available for installation as runtime components.

To properly handle this we would need the ability to expand the states for the component in question to allow for alternate DESCRIBE, DESCRIBE_CLUSTERED, DESCRIBE_SINGLETON states depending on the cluster metadata associated with the bean. Currently the mc state machine does not allow for this choice type of state transition. I have not worked much with the lower level state manipulation so I could also be talking out of my ass, but I think this is the proper way to do it.

Given I don't believe there is proper mc support for this, how to do it? I'm not sure. Any idea I have would mix deployer and kernel bean state management behavior. Need some feedback from Adrian/Ales.

[adrian.brock]

I think as far as backwards compatibility goes Brian's approach is roughly correct,
but it should NOT be using the MainDeployer directly.

It should be based on a (sub-)profile.

If you look in broad brush terms, the profile service is made up of two components

1) A client side api that lets you define what a profile is
2) A server side that implements that profile at runtime

For the current default profile (hot deployment) that means three pojos that know how
to scan three sets of directories/urls (bootstrap/deployers/deploy)
which are the three sub-profiles that make up the profile.

There should be a way to add extra (sub-)profiles e.g. hasingleton, jbossmq, etc.
that could be based on normal, singleton or farm semantics.
These are just implementation details of how the server side profile is implementated.

In practice, these subprofiles are not implemented in such a flexible way
with the scanners being hardwired along with the subprofiles they represent
and a subprofile is always active.

In terms of where we want to get to in the profile service and metadata in
general, it should be possible using a management console to say
I want this sub-profile, deployment, service, etc. to be a singleton in a given
partition.

I'd say partition is too low level since what you really want is some election policy
which could be generalised beyond the hasingleton notion to some activation policy.
e.g. you activate a deployment based on a timer rather than clustered failover election

By taking a metadata approach, users can also programme this explicitly
without having to use the management console directly.
e.g. I add a META-INF/jboss-activation.xml to describe how/when I want
this deployment starting/stopping

At the service/pojo level this would probably be an annotation

<bean name="Whatever" ...>
 <annotation>@Singleton policy="DefaultPartitionSingleton"<annotation/>
...

Which could also be applied via the scoped metadata if you want to
externalise it from the main descriptors in the management console/profile service case.


Short term for now, I'd just go with extending the hot deployment profile service to allow
some kind of barrier service like Brian describes to add/remove a sub-profile dynamically.

I'm not sure it needs to be as complicated as Brian describes though?
i.e. messing around with dependencies
The hasingleton scanner should just respond to the elected/de-elected event
by adding/removing its profile to/from the profile service.

[adrian.brock]

There should be a way to add extra (sub-)profiles e.g. hasingleton, jbossmq, etc.
that could be based on normal, singleton or farm semantics.
These are just implementation details of how the server side profile is implementated.

As discussed elsewhere, singleton and farming are really orthogonal issues.
Even in the hot deployment profile, it should be possible to have the
hasingleton profile take part in farming even if the current member is
not actually deploying the deployments.

[adrian.brock]

OFF TOPIC, but on a related issue.

I said a while ago that a real profile service implementation
has a lot in common with JBoss Cache (at least in terms of aspects/technology if
not in implementation :-).

It is a (possibly transacted/distributed) tree like structure with an
abitrary backing store which should be able to scale from standalone to
clustered environments.

Similarly it has requirements to track fine grained changes inside the tree
(changes to the items of metadata - the fields on the managed properties)
and applying them atomically.

Some areas where it differs are that the (sub-)profile is versioned as a whole.
Related to this is that the distribution of the changes has a slightly different
semantic in that changes to the metadata would first be tried locally
or on some staging cluster before distributing the new version across the
production cluster.

There are other parts that are not related to the cache structure, e.g. retrieving statistics
or invoking managed operations like flushing a pool, but these have more
in common with jgroups clustered invocation sematics.

[adrian.brock]

If you look in broad brush terms, the profile service is made up of two components

1) A client side api that lets you define what a profile is
2) A server side that implements that profile at runtime

For the current default profile (hot deployment) that means three pojos that know how
to scan three sets of directories/urls (bootstrap/deployers/deploy)
which are the three sub-profiles that make up the profile.

The client side api for the default hot deployment profile is of course to
just modify the directory structure and the file contents. :-)

[alesj]

To properly handle this we would need the ability to expand the states for the component in question to allow for alternate DESCRIBE, DESCRIBE_CLUSTERED, DESCRIBE_SINGLETON states depending on the cluster metadata associated with the bean. Currently the mc state machine does not allow for this choice type of state transition. I have not worked much with the lower level state manipulation so I could also be talking out of my ass, but I think this is the proper way to do it.

This could be achieved.
OK, it would still depend on exactly what you wanted to do - no cyclic/fork states.

You can do this atm.
Add these two new states (DESCRIBE_CLUSTERED, DESCRIBE_SINGLETON) to Controller.
Then when you install a bean that would use any of those two states, you need to push him your own ControllerContextActions impl.
e.g. our alias handling

 Map<ControllerState, ControllerContextAction> map = Collections.<ControllerState, ControllerContextAction>singletonMap(ControllerState.INSTALLED, new AliasControllerContextAction());
 ControllerContextActions actions = new AbstractControllerContextActions(map);
 install(new AliasControllerContext(alias, original, actions));

[brian.stansberry]

Thanks guys, for the input. :) My apologies, I opened this thread a couple hours before leaving on a trip where it turned out I had no chance for e-mail. I'll respond bit by bit as I digest.

Short term for now, I'd just go with extending the hot deployment profile service to allow
some kind of barrier service like Brian describes to add/remove a sub-profile dynamically.

I'm not sure it needs to be as complicated as Brian describes though?
i.e. messing around with dependencies
The hasingleton scanner should just respond to the elected/de-elected event
by adding/removing its profile to/from the profile service.

This is essentially what's there now. Discovered last week that something is broken with it, but since it has a few flaws I wanted to focus on a more long-term replacement rather than just a quick fix. The flaws:

1) A scanner that populates the profile is a behavior of the basic profile service, but not of the full repository-based profile service. In the full version it's the DeploymentRepository impl itself that scans the filesystem for content.

2) The distinction betweens what's in a profile and how it's reflected in the runtime. If the profile service on a non-master node isn't aware at all of a set of deployments, those deployments can't be managed (e.g. some piece of metadata changed), at least not directly on that node.

3) Can't handle the fine-grained metadata driven approach you describe in your '@Singleton policy="DefaultPartitionSingleton"' example.

Properly dealing with #2 and #3 could possibly be deferred; i.e. use a short term approach for 5.0.0. But AIUI, ability to use the full profile service is a requirement for 5.0.0 though, so I'd need to solve #1. On the farming thread Scott and I agreed last week that a clustered DeploymentRepository would need to have a clustered comm resource (JGroups Channel) available to it, so it's possible its internal scan logic could have the ability to respond to an elected/unelected event and add/remove sub-profile content. But that could end up being a fair amount of work and if the work will just be thrown away when I solve #2 and #3, I'm reluctant to do it that way.

I need to better understand what Scott's driving at with the discussion of DESCRIBE, DESCRIBE_CLUSTERED, DESCRIBE_SINGLETON and how that solves the issue where we "(r)eally we don't want the unelected singleton beans to be exposed as available for installation as runtime components." Thanks, Ales for the links to the alias stuff; found where that's used.

[adrian.brock]

I need to better understand what Scott's driving at with the discussion of DESCRIBE, DESCRIBE_CLUSTERED, DESCRIBE_SINGLETON and how that solves the issue where we "(r)eally we don't want the unelected singleton beans to be exposed as available for installation as runtime components."

I'm saying that's the wrong approach. What's required (for the first iteration) is being
able to activate/deactivate a sub-profile.
That's what your singleton activation policy does (the contract).

The implementation of scanning to determine what's in that profile
is specific to the current profile implementation.
Another possible implementation is to pull that profile from some reference repository
(which may or may not be clustered).

The definition and implementation of the sub-profile is an orthogonal issue to how
and when it gets activated.

[brian.stansberry]

I'm saying that's the wrong approach.

OK, thanks. I knew you'd said that, but since it was under discussion I figured if I had time I'd figured out what was meant. :-)

The definition and implementation of the sub-profile is an orthogonal issue to how and when it gets activated.

Yes, it's the concept of content that's available in a profile that isn't active yet that needs to be fleshed out. That's fundamentally why I want to replace the current scanning based approach; it mixes the two.

There would probably need to be some API change related to this, in order to allow mgmt tools to activate/inactivate sub-profiles. If we didn't want to allow external clients to do that, it could just be done via an spi that an activation policy would use.

AIUI, though, the detection of an @Singleton annotation or some other form of deployment-specific metadata is always done by a deployer; i.e. is not part of the profile service itself. (Waiting to be told I'm wrong ;) ) So you end up with a chicken and egg problem; you need to start deploying to find out a deployment is in an inactive sub-profile. That's fundamentally what drove me to the idea of using dependencies. I'd much prefer an approach where the deployers aren't even aware of content that isn't "activated".

[adrian.brock]

AIUI, though, the detection of an @Singleton annotation or some other form of deployment-specific metadata is always done by a deployer; i.e. is not part of the profile service itself. (Waiting to be told I'm wrong ;) ) So you end up with a chicken and egg problem; you need to start deploying to find out a deployment is in an inactive sub-profile. That's fundamentally what drove me to the idea of using dependencies. I'd much prefer an approach where the deployers aren't even aware of content that isn't "activated".

That's more long term (i.e. a proper implementation ;-).
The more fine-grained stuff involves defining some callbacks that are triggered by some policy.

e.g. You might have (or it could be annotation based):

public interface Activation
{
 void activate();
 void deactivate();
}

But it will depend upon what you are doing on how the object gets registred with
the policy.

The simplest case might be something like;

@Activatable(policy="SomeNameInTheMC")
public MyPOJO
{
 @Activate
 public void method1();

 @Deactivate
 public void method2();
}

The registration can then be handled using something similar to the @JMX
handling. i.e. you write a lifecycle-install handler
that looks for the annotation/interface and does the relevant registrations.
(This requires another interface, i.e. ActivationPolicy or Activator)

More complicated would be an MDB that you want to start/stop delivery on some
kind of schedule.
That would involve registering the MDB container (or some activatable extension)
with the activation policy at deployment and wait for the callbacks.
Of course if the MDB container is just a POJO then it is the same as above.

The activation of the sub-profile also be the same thing.
The de/activation methods just to talk to the profile service.

@Activatable(policy="HASingletonDefaultDomain")
public class ProfileActivator
{
 private String subProfileName;
 @Inject
 private ProfileService profileService;

 @Activate
 public void activate()
 {
 profileService.deployProfile(subProfileName);
 }
// etc.

We could also get more complicated and with activation requiring multiple policies
and logic to decide whether something is active.
This probably ties in with what Ales was doing with the JBoss Rules integration in
the MC, but that's obviously in the 5% of use cases. ;-)

[starksm64]

There would probably need to be some API change related to this, in order to allow mgmt tools to activate/inactivate sub-profiles. If we didn't want to allow external clients to do that, it could just be done via an spi that an activation policy would use.

AIUI, though, the detection of an @Singleton annotation or some other form of deployment-specific metadata is always done by a deployer; i.e. is not part of the profile service itself. (Waiting to be told I'm wrong ;) ) So you end up with a chicken and egg problem; you need to start deploying to find out a deployment is in an inactive sub-profile. That's fundamentally what drove me to the idea of using dependencies. I'd much prefer an approach where the deployers aren't even aware of content that isn't "activated".

The profile service does not look at deployment metadata, and there is no profile metadata corresponding to a jboss-activation.xml descriptor currently. I don't see how we would go from a bean/component level annotation to a subprofile membership. It seems we are driving at having an additional ActivationPolicyMetaData component for each deployment (maybe even component), but this still leaves a gap between specifying an annotation on a bean and identifying that the bean's deployment has a singleton component.

The profile service does not currently deal with makeup of a profile down to the component level. So what do we need for 1)? Its not completely true that:

1) A scanner that populates the profile is a behavior of the basic profile service, but not of the full repository-based profile service. In the full version it's the DeploymentRepository impl itself that scans the filesystem for content.

in that you could still have an hasingleton scanner adding content to a profile regardless of what profile service implementation is in use. There should be a flag indicating that the deployment should not be a permanent part of the profile though. Is the APPLICATION_CLUSTERED phase going to be useful? Maybe this should be APPLICATION_TRANSIENT or some such to indicate a deployment that will not be remembered across restarts.

[starksm64]

That's more long term (i.e. a proper implementation ;-).
The more fine-grained stuff involves defining some callbacks that are triggered by some policy.

e.g. You might have (or it could be annotation based):

public interface Activation
{
 void activate();
 void deactivate();
}

Ok, I see, activation is a runtime behavior driven by metadata that could be talking to the profile service. That makes more sense.

[adrian.brock]

Its not completely true that:

1) A scanner that populates the profile is a behavior of the basic profile service, but not of the full repository-based profile service. In the full version it's the DeploymentRepository impl itself that scans the filesystem for content.

in that you could still have an hasingleton scanner adding content to a profile regardless of what profile service implementation is in use. There should be a flag indicating that the deployment should not be a permanent part of the profile though. Is the APPLICATION_CLUSTERED phase going to be useful? Maybe this should be APPLICATION_TRANSIENT or some such to indicate a deployment that will not be remembered across restarts.

If that is the case then you also could say (regardless of clustering) that a hot
deployment scanner could be added to any profile service implementation.

Again this orthogonal.

It's then a hybrid profile service implementation. Where some sub-profiles
come from a versioned repository and others are unversioned in a directory.

Whichever way you look at it, its still a sub-profile of the profile in operation
and needs some "metadata" to say how/if/when it gets activated.

Ideally this should be a part of the management console where somebody can
say this sub-profile/deployment gets activated according to this policy,
where the policy is a part of the profile itself.

As I was trying to explain above, (but got side-tracked in implementation details :-)
for fine-grained it gets harder because the link to the policy might be
annotation on a pojo or a piece of metadata in jboss.xml, etc.

That's why I suggested a jboss-activation.xml to try to make it easier to manage.
The downside is it pushes the responsibility of this new metadata to the subsystems
to read it, since only they know what it means to activate their fine-grained component.

[starksm64]

If that is the case then you also could say (regardless of clustering) that a hot deployment scanner could be added to any profile service implementation.

Again this orthogonal.

It's then a hybrid profile service implementation. Where some sub-profiles
come from a versioned repository and others are unversioned in a directory.

Right, but there is a notion of "modified deployments" in the profile spi, and the hot deployment scanner implementation uses this. If we add a APPLICATION_TRANSIENT notion, whether the profile should detect modifications needs to be defined. Currently the profile implementations only look at the APPLICATION phase deployments.

Whichever way you look at it, its still a sub-profile of the profile in operation
and needs some "metadata" to say how/if/when it gets activated.

Ideally this should be a part of the management console where somebody can say this sub-profile/deployment gets activated according to this policy,
where the policy is a part of the profile itself.

Ok, I'll have to look at how deployment level policies (sub-profiles) could be implemented consistently regardless of the profile service implementation.