MODE's component wiring

[ajs6f]

To my understanding, MODE currently uses a form of component wiring based on JSON configuration that is unique to MODE (or perhaps a few other projects). Are there any plans to switch to use a wiring scheme that is more common? (For example, CDI.)

[rhauch]

Yes, ModeShape instantiates and initializes the sequencers, connectors, and other components internally using reflection, based upon the JSON configuration. The idea of this is to have the repository create all of the necessary instances, to ensure that the repository owns all its component instances, and to manage the lifecycle of all of the components. This works well for the basic cases, when the components are independent of any non-ModeShape services. It does start to show its limitations when a component needs to have references to other services/components; right now JNDI is probably our only straightforward solution.

CDI is a more accepted and widely-used standard now, but my understanding is that CDI is really for wiring together a series of objects that are manually instantiated. Perhaps I don't understand your goal; can you give me an example of what you describe might look like?

[ajs6f]

I guess we're coming from different assumptions-- my understanding of CDI is that it is most definitely not limited to wiring manually created beans. In fact, it is fully capable of auto-instantiating beans (or collections of beans) to fill dependency slots based on types, annotation qualifiers, etc. As well, it offers excellent JSR-defined lifecycle services, eventing, etc.

Given that understanding, what I was imagining was a MODE in which there would exist much less explicit wiring configuration (especially because it comes in a form distinct to MODE) and instead MODE components would simply feature type information for their dependencies (which, of course, they already do) and these dependencies could be automatically wired at runtime by the bean container. This would make MODE a more open system and enable projects writing on top of MODE (like the one I represent) to extend and expand it more easily, without having to learn or support a MODE-specific framework.

Does that clarify the question?

[rhauch]

I understand that CDI can auto-instantiate and can wire together. What I'm struggling with is that the different components really can rarely be constructed in a default form, and instead you almost always have to specify at least some properties. How do you do that in CDI? My understanding is that you'd have to manually instantiate these components and call setters. Or, if ModeShape instantiates them (via CDI and possibly auto-instantiation), then you'd have to be able to get the objects from ModeShape. If that's the case, what value does CDI have?

My impression of CDI is that it does a great job for wiring components together based upon annotations in your code. I just don't understand how that can help ModeShape, when the annotations in the components will rarely be sufficient.

[hchiorean]

An additional question: ModeShape has to be able to run in embedded (standlone) mode, outside of any container. On the other hand, any DI I'm aware of is handled by various container/3rd party libraries. How would this work ?

[ajs6f]

Keeping in mind that CDI was just an example I was giving, there are at least three patterns answering that question that I have seen (and there are probably many more ).

One is to factor configuration for a component out of the component itself. This can be cumbersome for a small component, but it is often nice because it keeps changes in the impl of configuration away from its use. Basically it's just the introduction of a contract between a component and its configuration. I think this tends to be more suitable for large components.

Another is to enrich the annotations. Keeping in mind that an annotation can pack values into itself, you can use them to configure:

http://www.java-tutorial.ch/context-and-dependency-injection/property-file-configuration-using-injection-cdi

A third is demonstrated here:

https://weblogs.java.net/blog/jjviana/archive/2010/05/18/applicaction-configuration-java-ee-6-using-cdi-simple-example

It shows a simple configuration annotation being used to signify that a field is to be configured, and then a producer method is provided elsewhere that can fill that slot. If there is are a lot of different slots, they can be differentiated by type or qualifier.

[ajs6f]

Here I would turn the question around: MODE is currently instantiating and wiring dependencies and managing lifecycles for itself. Isn't that exactly what a 3rd party framework would do for MODE, without MODE having to maintain its own codebase to do that? Except instead of having people learn a framework specific to MODE, MODE would learn a framework (CDI or other: my current project is using Spring) that is more widely used.

As far as runtime, it doesn't seem to me to be unusual for a project to provide options for integrators who want to use it inside a container, or with a container pre-packaged. For example, when I'm working on a project intended to run in a web container, I often provide a WAR artifact with an integrated Jetty so that it can be executed standalone. I find that's typically just a few lines of Maven.

[rhauch]

Well, the first step might be registering the manually-constructed objects with CDI, so that a component can use CDI annotations. Seems like we'd be able to go a long a way with just that. Anything else would be internal.

My concerns with using something like CDI outside of a container is that it adds yet another required dependency for something that we get by pretty well with just reflection. CDI can get awfully complicated, especially when you start separating out configuration from the component. (Yes, I understand there are benefits; the drawback is added complexity in areas that we've really strived for some time to make things as easy as possible.)

[rhauch]

Here I would turn the question around: MODE is currently instantiating and wiring dependencies and managing lifecycles for itself. Isn't that exactly what a 3rd party framework would do for MODE, without MODE having to maintain its own codebase to do that? Except instead of having people learn a framework specific to MODE, MODE would learn a framework (CDI or other: my current project is using Spring) that is more widely used.

As soon as we pick a framework, lots of people will not like that framework. (For example, I absolutely hate Spring and will never use it again in my life. Sorry, it's just horrible IMO.) So, do we have to support multiple frameworks now? Sometimes, frameworks just aren't worth it.

Now, you have to realize that the amount of code that actually manages and wires components together is quite small. Yes, developers of ModeShape may have to learn that. But users of ModeShape that want to write components should not have to learn that (or any other framework), and instead should just have to learn how to write the custom component. (Sequencers are pretty easy, connectors are harder simply because they are more complicated and involved. No DI framework is going to help with that.)

What is the actually problem that we need to address?

[ajs6f]

I guess we're seeing this from two points of view. As an integrator, it's precisely because we haven't found MODE's internal system to be as easy as possible that I raise the question. The simplicity of the system seems to be outweighed by its uniqueness and limitations. But distinguishing between internal and external contracts seems like a good weigh to avoid these kinds of "opinion-based" conversations.

[ajs6f]

Don't worry, rhauch, I couldn't possibly care less about a given framework. I only introduced CDI as an example since it is clear that JBoss has invested heavily in it.

I guess, again, it's a question of POV. As you ask, "do we have to support multiple frameworks now?" I ask "Why do we have to support MODE's unique framework now?" (See below for an explanation of why we do.)

About the other points you raise I can speak more usefully. You suggest that people (like the project with which I am working) who want to implement MODE components do not need to concern themselves with the MODE framework. We simply haven't found that to be the case when we did some examples of custom federation connectors (and contributed some results back to MODE). And as we integrate MODE as a component into a larger system, we haven't found it easy to let other frameworks manage the lifecycle of MODE-as-a-component. This matters because MODE is only one piece of our platform. We don't want one piece (no matter how important) to control the lifecycle of the framework we are building.

Here's another more interesting problem: we are not writing an application using MODE as a "black box". We are writing a framework with MODE as a sub-component. MODE's unique method of combining configuration and wiring in one serialized form is a real obstacle for us. We would like to be able to offer our users abilities to configure and reconfigure the workings of such components as persistence, sequencers, etc., but to offer them anything other than a JSON string, we find that we must parse and operate over MODE's unique system of editable JSON documents. (That's why I say we find ourselves in the position of having to "support" MODE's internal dependency management.)

I think some of these issues that we experience really are rooted in the fact (as I note above) that we aren't writing an application that just uses MODE to manage content. We are writing a framework for a certain audience to use to create repositories with some complex services and forms of management in them and the needs are quite diverse and extensive. It may be that we are mistakenly trying to push MODE to a level of dynamicity outside its intended use, and that's why we are experiencing issues that don't seem to come up for other users?

[rhauch]

I completely concede that ModeShape is currently focused on the "black-box" case, such as embedding an engine inside an application or managing repositories within EAP. With this approach, simply provide a JSON configuration that declares what the repository is supposed to do, and the ModeShapeEngine takes care of bringing it up. The engine does have some basic lifecycle functionality that makes sense from a repository perspective; basically just "start" and "stop" either gracefully or immediately.

What we haven't really attempted is to provide the "component-based" approach, where we provide a series of components that can be managed and wired together along with other systems and components. This is much harder and far more complicated, so it's something we've definitely considered but so far hesitated to make possible. First, we never really had good use cases, requirements or expectations. Secondly, this opens up a huge potential for problems and variation that makes it difficult for us to help diagnose and solve problems.

However, if you have a need for ModeShape to better fit into your environment, lets discuss that. For example, you mention that working with ModeShape's Documents is challenging, so how much of your challenges might be solved by a far-easier-to-use API to programmatically define a configuration? That's actually quite feasible on top of RepositoryConfiguration, and it may even allow us to rethink how RepositoryConfiguration is implemented under the covers.

What other problems are you having? Do you really want to manage a whole bunch of ModeShape components? If so, what are the benefits?

[ajs6f]

I think we've really gotten to the heart of the matter now. I completely understand that offering a suite of components is a very different proposition than offering a pre-integrated (so-to-speak) component, and I certainly don't want to sound like I'm making demands on you. We're getting tremendous value from MODE and it's a great product, so please don't take my comments above as criticism.

As to the specifics, let me answer your questions in reverse order:

It's not that we want to manage a crowd of MODE components as much as that we would like to make sure we can do a good job at fitting a MODE block into a larger crowd of components. As long as MODE's lifecycle can "slot into" a larger framework, that's the most important thing. For us, because we're targeting a highly-dynamic run-time experience, we'd ideally like MODE's lifecycle to support that. That is one of the reasons I was raising the question of wiring to begin with: it would be fantastic to be able to alter components of MODE at run-time without having to restart the entire repository.

On the other question, I think a really flexible and easy-to-use configuration API would be HUGE for us. (Yes, boldface and all-capitals; that's how great it would be! ) Based on preexisting configuration mechanisms, I could imagine something like a builder API for repositories that would allow me to start from a serialized configuration, the configuration of a pre-existing repository, or from scratch, then make additions or emendations as desired by adding or mutating repository components, then call a "finalize" or "build" kind of method to engage those changes on a running repository or actually start a repository from that configuration. That's just a first idea, but with a powerful configuration API would let my project much more feasibly expose some of the power of MODE's configuration to our users without making them engage with the JSON serialization.

Our absolute ideal? Storing repository configuration in the repository. We'd love to be able to access a privileged portion of the repository that represented configuration and use the ordinary good old JCR API to make changes. In fact, that's what we'll be doing for our own configuration. I realize there can be some serious bootstrapping issues with that kind of design, but I just thought I'd mention it because it is our "golden dream".

[rhauch]

I think we've really gotten to the heart of the matter now. I completely understand that offering a suite of components is a very different proposition than offering a pre-integrated (so-to-speak) component, and I certainly don't want to sound like I'm making demands on you. We're getting tremendous value from MODE and it's a great product, so please don't take my comments above as criticism.

I'm definitely not taking it as criticism. But if ModeShape doesn't fit seamlessly into your environment, let's figure out why and see how we can improve it.

It's not that we want to manage a crowd of MODE components as much as that we would like to make sure we can do a good job at fitting a MODE block into a larger crowd of components. As long as MODE's lifecycle can "slot into" a larger framework, that's the most important thing. For us, because we're targeting a highly-dynamic run-time experience, we'd ideally like MODE's lifecycle to support that. That is one of the reasons I was raising the question of wiring to begin with: it would be fantastic to be able to alter components of MODE at run-time without having to restart the entire repository.

This is actually possible now with most components, using the (very unique) editable document mechanism. Some changes will require restarting the repository, but most should be able to be applied while the repository is in use (without starting). Granted, the editable document mechanism is not very easy, but one really nice thing is that it records changes to the configuration, and then those changes are applied to the running repository's configuration. Thus, there's an element of concurrent safety there. Can this be improved? Probably, if just by hiding it ...

On the other question, I think a really flexible and easy-to-use configuration API would be HUGE for us. (Yes, boldface and all-capitals; that's how great it would be! ) Based on preexisting configuration mechanisms, I could imagine something like a builder API for repositories that would allow me to start from a serialized configuration, the configuration of a pre-existing repository, or from scratch, then make additions or emendations as desired by adding or mutating repository components, then call a "finalize" or "build" kind of method to engage those changes on a running repository or actually start a repository from that configuration. That's just a first idea, but with a powerful configuration API would let my project much more feasibly expose some of the power of MODE's configuration to our users without making them engage with the JSON serialization.

Okay, this is definitely possible and something I agree is very valuable, especially now that we have users driving the requirements! The challenge will be having something that can also be used to modify a running repository. But that might be pretty easy by putting it all on top of the existing (nicely-serializable) JSON configuration representation and the editable documents. Do you want to open a JIRA issue and add some requirements/thoughts? We'll target this to 4.0.

Our absolute ideal? Storing repository configuration in the repository. We'd love to be able to access a privileged portion of the repository that represented configuration and use the ordinary good old JCR API to make changes. In fact, that's what we'll be doing for our own configuration. I realize there can be some serious bootstrapping issues with that kind of design, but I just thought I'd mention it because it is our "golden dream".

That's actually a lot harder than it sounds. It's something we did way (way) back pre 1.0, but we ran into challenges/problems in clustered scenarios. Some configurations in a cluster may be different, and we'd have to keep track of all of the configurations.

Another approach we thought of back then was actually using a separate repository to manage the (presumably many) configuration files. Add a JSON sequencer, and you get some pretty nice functionality in that repository. And if this is served by a service that exposes the files via a URL, then it's cake to create a RepositoryConfiguration from that URL to have the JSON file downloaded automatically.

[ajs6f]

I'll definitely open a ticket and start getting some details in there.

We're getting familiar with the difficulties of self-hosted configuration. We hadn't thought about the implications of a "privileged" configuration repository, and I wonder if that solves the bootstrapping problems or just moves them around...

As to clustering, it's good to hear that you guys investigated that. My understanding currently is that two repositories can cluster even if they have (say) differing federation setups, but that mutations propagating between incompatible sections of repository will cause run-time errors. Is that not the case? Do repositories do some kind of validation when they connect to cluster? We thought the great bulk of cluster communication is going on in Infinispan, which wouldn't be aware of MODE's configuration... is that wrong?