ModeShape and concurrent Infinispan usage

[kbachl]

Hi,

I want to use modeshape together with infinispan. Idea is that I have 1 infinispan cluster on 1..n machines. This cluster is then connected to by about 3 app's using each a jcr that is connected using modeshape-infinispan connector. Now, if those 3 app's access the same JCR-repos, is there any danger that may happen?

Some time ago Randy sugessted to share modeshape by setting upa central one, so I'm not sure my idea would work? Reason I'd like to do is that I use infinispan for other purposes (caching), too and I need to save the whole repo on the file-system and with the FileConnector it wont work as my master-node's are of typ nt:unstructured; As infinispan supports a cache store I expected this to be a better solution than fiddling around with a own "save to file" solution (plus I would get benefits of clustering for free when needed);

[rhauch]

You absolutely can set up ModeShape to do this, and this is perhaps the highest performing configuration there is.

First of all, set up your Infinispan data grid (which like you said will be comprised of multiple process nodes). Then, how you set up ModeShape will depend on how you are deploying your application. Note that each source in ModeShape's configuration will use a single named cache; if you want to use other named caches for other purposes, feel free.

If all of your applications will be deployed to a single process (e.g., an application server instance), then you'll want a single ModeShape engine in that process. If deploying to JBoss AS, you can use our kit and change its configuration; otherwise, you'll probably use the RepositoryFactory in your applications, with a repository URL that points to the configuration file.

If your applications will be deployed in multiple processes, you'll want a ModeShape engine in each of those processes and you'll want these ModeShape engines to be clustered. Clustering the ModeShape engines means they'll all share a distributed JGroups channel for their events, meaning each engine will see all of the events regardless in which engine they originate. Clustering ModeShape is also completely independent of how Infinispan is clustered - both use JGroups, but they'll use their own channels.

Hope this helps. Best regards,

Randall

[kbachl]

Thank you for your post Randall, helped me really much.

Regarding the "single process": if I deploy 3 webapps but within 1 instance of e.g. Glassfish are they seen as "single processed" or not more seen as "3 independent processes"?

One thing that puzzles me is that you say I can configure JGroups from ModeShape different than Infinispan, because afaik it is so that with limit to important settings like transport properties these only can be set on the JVM level (at least this was told me: http://community.jboss.org/thread/166095 ); Is there an easy way to tell modeshape just to use the while infinispan grid and not care for "own" settings?

Best Regards,

Korbinian

PS: sorry for making you to "Randy" in the post before

[rhauch]

Regarding the "single process": if I deploy 3 webapps but within 1 instance of e.g. Glassfish are they seen as "single processed" or not more seen as "3 independent processes"?

Well, it all depends on how you configure the web apps and where you put the ModeShape jars. ModeShape's RepositoryFactory uses static fields to cache the ModeShape engine instances, keyed by the value of the "org.modeshape.jcr.URL" property supplied to the factory. So as long as one classloader is used to load the 'modeshape-jcr-<version>.jar' file (where our RepositoryFactory implementation is), looking up a Repository multiple times via the RepositoryFactory will return the same Repository instance.

However, if each app uses a different classloader to load the ModeShape RepositoryFactory implementation (e.g., the ModeShape JARs are deployed with each web app), there will be multiple RepositoryFactory implementation classes that do not share static state, and each application will get a unique/distinct/separate Repository instance.

So, with Glassfish, you'll want to put the ModeShape jars on the global classpath (or at least higher than the app-specific classpath). Our JBoss AS kit installs ModeShape as a service, effectively doing the same thing (though only the JCR and ModeShape JCR API jars on on the global classpath.)

Another (perhaps easier) way to do this is to create a separate WAR whose sole job is to instantiate the ModeShape engine, specifying in ModeShape's configuration the JNDI name where ModeShape should register the Repository instance. Then, your other applications can use a JNDI-based "org.modeshape.jcr.URL" (or even look directly in JNDI for the 'javax.jcr.Repository' instance). But this approach has two other issues: 1) you still have to have the JCR and ModeShape JCR API jars on the global classpath, and 2) startup order is important.

One thing that puzzles me is that you say I can configure JGroups from ModeShape different than Infinispan, because afaik it is so that with limit to important settings like transport properties these only can be set on the JVM level (at least this was told me: http://community.jboss.org/thread/166095 ); Is there an easy way to tell modeshape just to use the while infinispan grid and not care for "own" settings?

Yes, technically I misspoke. The JGroups configuration will be the same, because it has more to do with your network and how you want to configure the transport. With JGroups, multiple channels can use the same transport, and this is what you'd want to do. Basically, ModeShape configuration and Infinispan configuration will both contain the same JGroups configuration specification, which should include the 'singleton_name' property. ModeShape and Infinispan each need their own dedicated channels, but these are configured in the ModeShape configuration and Infinispan configuration (outside of the JGroups config).

Hope this helps!

PS: sorry for making you to "Randy" in the post before

No problem. I'll let it slide just this once.

[kbachl]

I tried to go the route with the war for initialization as I also need to boot-up infinispan.

Another (perhaps easier) way to do this is to create a separate WAR whose sole job is to instantiate the ModeShape engine, specifying in ModeShape's configuration the JNDI name where ModeShape should register the Repository instance. Then, your other applications can use a JNDI-based "org.modeshape.jcr.URL" (or even look directly in JNDI for the 'javax.jcr.Repository' instance). But this approach has two other issues: 1) you still have to have the JCR and ModeShape JCR API jars on the global classpath, and 2) startup order is important.

However, I'm somehow stuck. First my idea was to let the webapp do sth. like this:

public void contextInitialized(ServletContextEvent servletContextEvent) {

        repository = createRepository();

        try {

            InitialContext initCtx = new InitialContext();

            Context envCtx = (Context) initCtx.lookup("java:comp/env");

            envCtx.addToEnvironment("jcr/repo", repository);

just to find out that this is not possible as other applications can't access the JNDI. Then I tried to get it going using the JNDI-based "org.modeshape.jcr.URL" but I weren't really able to find any useful way to play with this, as all my paths are ending at the border of the webapp.

Can you please tell me how I can get ModeShape configured that it puts the factory up so I can simply grab a repo from it?

[rhauch]

Sounds like your JNDI scope is limited to the web application, but you need server-scope to share JNDI entries between web applications. Again, I'm not familiar enough with Glassfish to know how to configure the JNDI scopes.

[kbachl]

yes, your right. However, I was able to deploy modeshape as a "service" to glassfish (by shamelessly spying into the jboss-service-files ). It took me quite a while to discover all the class not founds but it works (as soon as it works 100% I'll put a wiki entry up how to do that). The last thing I don't know yet how I can put infinispan into that "service" mode. Do you know how I could put infinispan into that?

[rhauch]

yes, your right. However, I was able to deploy modeshape as a "service" to glassfish (by shamelessly spying into the jboss-service-files ). It took me quite a while to discover all the class not founds but it works (as soon as it works 100% I'll put a wiki entry up how to do that).

Or, if you're willing, contribute it as another deployable service implementation (under "deploy/glassfish/modeshape-glassfish-service")??

The last thing I don't know yet how I can put infinispan into that "service" mode. Do you know how I could put infinispan into that?

Sorry, I don't know the best way to do that for Infinispan. There are a lot of options - for example, you could even run Infinispan separately from glassfish. That'd be a great question to ask in the Infinispan forum, or even ask in their IRC room.

[kbachl]

I've found out what I did wrong with JNDI, see the JEE 6 spec:

Java EE 6 introduces application component environment namespaces, which includes the following scopes.

• java:comp. Names in this namespace have per-component visibility.
• java:module. Names in this namespace are shared by all components in a module, for example, the EJB componentsdefined in an a ejb-jar.xml file.
• java:app. Names in this namespace are shared by all components and modules in an application, for example,the application-client, web, and EJB components in an .ear file.
• java:global. Names in this namespace are shared by all the applications in the server.

As soon as I did initCtx.bind("java:global/jcr/myRepo", repository); it worked. This should also be true now for JBoss 6.

Regarding Infinispan as a Service I didnt find anything useful so far;

[rhauch]

Ah, that's great. It's much more obvious how to control the JNDI scope.  BTW, I was looking at some Glassfish documentation this morning, and I was confused why they listed the different scopes, but used "java:comp" for the namespace of all the scopes. I bet that was a typo.

[kbachl]

Hi,

I got it running so far as expected, but had the problem that my app(s) need to react to underlying changes, meaning if a node is saved the JCR save event needs to be catched - this won't work if I just share my engine as allways the first started app gains all notifications.

The solution according to the docs would be to let modeshape run clustered. I tried it, but I just can't get a simple config running. The problem is that jgroups offers a ton of options and I hardly can figure out what is needed - plus I have no clue about jgroups (and docs are not that easy to understand ). Could you point me to a simple config that allows modeshape to just form a network on localhost (with say maximum of 5 nodes)? - What is better for modeshape, TCP, TCP-NIO, UDP? What other services are required?

Best Regards

[rhauch]

I got it running so far as expected, but had the problem that my app(s) need to react to underlying changes, meaning if a node is saved the JCR save event needs to be catched - this won't work if I just share my engine as allways the first started app gains all notifications.

I don't understand. If all of your apps are sharing a single ModeShape engine instance, they should each be able to register a listener and each receive all the events. If this does not work, you don't actually have a shared single instance of ModeShape. If this is what you want, you should try and figure out why.

The solution according to the docs would be to let modeshape run clustered. I tried it, but I just can't get a simple config running. The problem is that jgroups offers a ton of options and I hardly can figure out what is needed - plus I have no clue about jgroups (and docs are not that easy to understand ). Could you point me to a simple config that allows modeshape to just form a network on localhost (with say maximum of 5 nodes)? - What is better for modeshape, TCP, TCP-NIO, UDP? What other services are required?

Actually, the JGroups configuration completely depends on your networking environment, and unfortunately it can get very complicated. The default JGroups configuration often works pretty well, especially on a local host. To use the default JGroups configuration with ModeShape, simply specify clustering section but do not specify a configuration attribute:

<clustering clusterName="modeshape-cluster" />

Of course, you can use a different cluster name - just make sure all of your configurations use the exact same cluster name, or your processes will start in their own separate clusters. For more details, see our Reference Guide.

Clustering can, however, get tricky when you support IPv6 and IPv4. The easiest thing to try then is to temporarily disable IPv6 (or IPv4) on your computer and get things working, then re-enable the networking and adjust your environment to support the correct IP version.

[kbachl]

I don't understand. If all of your apps are sharing a single ModeShape engine instance, they should each be able to register a listener and each receive all the events. If this does not work, you don't actually have a shared single instance of ModeShape. If this is what you want, you should try and figure out why.

Thats a point I'm not sure yet why. Im investigating but it seems that the "JcrNodeWrapper"s used by BRIX to determine changes, only get registered by the first instance.

Situation is:

WebApp: A (first deployed)

WebApp: B

both share same repo's; Now if I changed something in A or B the underlying JcrRepo got the changes. However, only in case of change from WebApp A the listener calls are reacting (both A + B). While if I change within B none of the listener got called (neither A nor B); However, during this testing i stumbled over a mjaor problem with this scenario for me: I can't change the underlying JNDI part without need to restart the whole server or in case I use the JcrWebApp approach I can't even guarantee the server will start up correctly as I cant guarantee startup ordering; So this now brings me to full clustering as result.

Your idea with:

<clustering clusterName="modeshape-cluster" />

Of course, you can use a different cluster name - just make sure all of your configurations use the exact same cluster name, or your processes will start in their own separate clusters. For more details, see our Reference Guide.

Clustering can, however, get tricky when you support IPv6 and IPv4. The easiest thing to try then is to temporarily disable IPv6 (or IPv4) on your computer and get things working, then re-enable the networking and adjust your environment to support the correct IP version.

was fantastic! - This really should be mentioned in doc's at chapter 6! I wasted a whole day on this.

With regards to the cluster it now gets up and notices it, but it seems that I need to now redeploy *all* workspaces on all cluster nodes? Is this true or is my simple test-config bogus?

<?xml version="1.0" encoding="UTF-8"?>
<configuration xmlns:mode="http://www.modeshape.org/1.0" xmlns:jcr="http://www.jcp.org/jcr/1.0">
    <mode:clustering clusterName="modeshape-cluster"/>
    <mode:repositories>
        <!--
        Define a JCR repository that accesses the 'Brix' source directly.
        -->
        <mode:repository jcr:name="Brix repository" mode:source="Brix">
            <mode:options>
                <mode:option jcr:name="anonymousUserRoles" mode:value="admin"/>
                <mode:option jcr:name="useAnonymousAccessOnFailedLogin" mode:value="true"/>
                <mode:option jcr:name="systemSourceName" mode:value="system@Brix"/>
            </mode:options>
        </mode:repository>
    </mode:repositories>

    <mode:sources jcr:primaryType="nt:folder">
        <mode:source jcr:name="Brix"
                     mode:classname="org.modeshape.graph.connector.inmemory.InMemoryRepositorySource"
                     mode:retryLimit="3" mode:defaultWorkspaceName="default">
            <predefinedWorkspaceNames>system</predefinedWorkspaceNames>
        </mode:source>
    </mode:sources>

</configuration>

Best Regards and many thanks so far

[kbachl]

PS: I've jsut noticed that the calls over jgroups create following errors:

ERROR - QueryManager$SelfContained - Error updating the query indexes: The workspace "bx_ZHXHfPpikBKMZbfpHlSHMHLQR" does not exist in the "Brix" repositoryorg.modeshape.graph.request.InvalidWorkspaceException: The workspace "bx_ZHXHfPpikBKMZbfpHlSHMHLQR" does not exist in the "Brix" repository

    at org.modeshape.graph.connector.base.Processor.getWorkspace(Processor.java:629)

It seems as the two different Brix repos get different workspace names even they came from same import file. Interesting, the workspaces are defined as:

bx_bxfyNBebtTMQZbbeKYupkrcyg in A

bx_ZHXHfPpikBKMZbfpHlSHMHLQR in B

[kbachl]

Follow up:

I've been away last days but finally could pinpoint the problem I have. The CMS I use creates workspaces using a random UUID as its identifiers. This is no problem usually as on startup it scans all the repos if it belongs to it. The thing I am now faced with is that it seems that the underlying Infinispan is not DIST_SYNC, at least not in the way it behaves.

I also saw in doc that there is mentioned, but I'm not sure how to apply it and what this is good for; 

The rootNodeUuid property must be set to the same value for all Infinispan sources in the cluster.

So I kindly wanted to ask where my error could be. I use a quite simple config and were told that modeShape uses sensable default, so why wont this work then?

<?xml version="1.0" encoding="UTF-8"?>

<configuration xmlns:mode="http://www.modeshape.org/1.0" xmlns:jcr="http://www.jcp.org/jcr/1.0">

    <mode:clustering clusterName="demoCluster"/>

<mode:repositories>

        <!--

        Define a JCR repository that accesses the 'Brix' source directly.

        -->

        <mode:repository jcr:name="wwRepo" mode:source="sourceRepo">

            <mode:options>

                <mode:option jcr:name="anonymousUserRoles" mode:value="admin"/>

                <mode:option jcr:name="useAnonymousAccessOnFailedLogin" mode:value="true"/>

                <mode:option jcr:name="systemSourceName" mode:value="system@sourceRepo"/>

            </mode:options>

        </mode:repository>

    </mode:repositories>

   <mode:sources jcr:primaryType="nt:unstructured">

        <mode:source jcr:name="sourceRepo"

                     mode:classname="org.modeshape.connector.infinispan.InfinispanSource"

                     mode:description="The repository for our content"

                     mode:defaultworkspaceName="default">

            <mode:predefinedWorkspaceNames>system</mode:predefinedWorkspaceNames>

        </mode:source>

    </mode:sources>

</configuration>