Scalability problems in multi client environment

[nilian]

Hi,

we are trying to build a proof of concept for a special use case of a customer. This are the requirements and steps we have done so far

1.) Target platform is a multi client environment. The customer really want to have seperate repositories for each client. The estimated of supported clients are about maximum of 10000.

2.) We are using the EAP6.1.1 with the embedded approach to use modeshape, but using all the libraries from the installed modeshape and infinispan subsystem. So we are starting the modeshape engine inside our web-application and creating programmatically new repositories for a new client with the help of the modeshape API. Furthermore we are also creating programmatically infinispan cachestores (there a two implementation right now, one with postgres and one for the javadb).

Since we are heading to do performance and load tests I'm not quite sure about this approach at all. On my dev-machine it was not possible to create up to 200 different repositories with some example nodes inserted. The EAP6.1 is running out of threads. Each repository is spawning about 8 threads which are not get released even when the repository is undeployed (via modeshape api). So the virtual machine is running out of threads after allocating 2048 threads (Java 7, MacOSX) with a "java.lang.OutOfMemoryError: unable to create new native thread" Exception.

At this point there are no threadpools defined (we are reading default configuration from a json file). Is there a default threadpool setting or are no threadpools used at all, if they are not specified ?

Does make it sense to go further with this approach or is it not feasible at all ? What other approach should be taken then ?

Thank you very much

Best regards,

Daniel

[rhauch]

Each repository does have some overhead, including thread pools, caches, and state management. Obviously we never envisioned ModeShape to manage 1000s of repositories (even 100s is probably stretching it, as you seen in testing). I don't think your approach is really practical and faces a lot of challenges.

I'm not sure what to suggest if the customer requires separate repositories, other than to scale out by partitioning much smaller groups of repositories onto separate servers. This approach is not without disadvantages, primarily around the effort require to manage it.

If the customer is willing to relax the requirement and allow multiple customers into a single repository, then I would suggest looking at the new ACL feature to help ensure that each client's data is isolated and accessible only by that client. How many repositories (or workspaces in multiple repositories) is up to you.

[hchiorean]

We are using the EAP6.1.1 with the embedded approach to use modeshape, but using all the libraries from the installed modeshape and infinispan subsystem

What exactly do you mean by this ? You're not using the ModeShape EAP subsystem but rather have your own WAR/EAR instance deployed which starts a ModeShape engine & deploys repositories via the API ?

If this is the case, I'm not really sure how things will look like from an EAP class loading perspective: for example in the case of Infinispan and JGroups they are both integrated (i.e. start/stop/monitor etc) in the container via their own subsystems. However, I'm pretty sure that none of those subsystems will be available directly from another deployable unit and therefore your custom code will trigger the loading of Infinispan & JGroups in embedded mode, outside of the EAP system CL.

This is one of the main reasons we recommend using the ModeShape EAP subsystem and not using ModeShape in embedded mode.

The EAP6.1 is running out of threads. Each repository is spawning about 8 threads which are not get released even when the repository is undeployed (via modeshape api).

I'm really curious what these threads are: if you attach a monitoring app to EAP e.g. VisualVM, you should be able to see the names of these threads because ModeShape uses a NamedThreadFactory when spawning new threads.

Code wise, assuming you're using ModeShapeEngine#shutdownRepository, this in turn calls shutdown() on a repository which terminates all threads that belong to the repository in an asynchronous manner. The result of JcrRepository#shutdown is a Future, so when testing this you should really wait for it to complete before checking live threads.

[nilian]

Hi,

thank you for your feedback.

A little more details and some new investigations on my side ;-)

• You are right, we are only using the libraries from the subsystem and does not have any jar files inside the war-file. I expect that modeshape, infinispan and jgroups will live "outside" the EAP managed area. But this approach should work for other applicationserver as well (ok in that case the war file must include all the jar-files). So thats why we didn't see any problems with this approach in the first place. Our concrete installation is EAP 6.1.1 + Modeshape Kit 3.5.0.
• For the thread problem, after more investigation and your hint regarding really checking the ModeShapeEngine#shutdownRepository behavior, it shows that indeed the shutdown is not running successfully, so all the created repositories are still running and so are their threads.
• Trying to shutdown a repository we are facing the following exception:

Trying to shutdown repository 'a': java.util.concurrent.ExecutionException: java.lang.UnsupportedOperationException: JBAS011859: Naming-Kontext ist schreibgesch?tzt
  at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) [rt.jar:1.7.0_25]
  at java.util.concurrent.FutureTask.get(FutureTask.java:111) [rt.jar:1.7.0_25]
  at de.akquinet.modeshape.demo.backend.modeshape.ModeShapeManager.shutdownRepository(ModeShapeManager.java:77) [classes:]
  at de.akquinet.modeshape.demo.backend.modeshape.ModeShapeManager$Proxy$_$$_WeldClientProxy.shutdownRepository(ModeShapeManager$Proxy$_$$_WeldClientProxy.java) [classes:]
  at de.akquinet.modeshape.demo.backend.modeshape.JCRSessionProducer.logoutSession(JCRSessionProducer.java:43) [classes:]
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_25]
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_25]
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25]
  at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25]
  at org.jboss.weld.util.reflection.SecureReflections$13.work(SecureReflections.java:267) [weld-core-1.1.13.Final-redhat-1.jar:1.1.13.Final-redhat-1]
  at org.jboss.weld.util.reflection.SecureReflectionAccess.run(SecureReflectionAccess.java:52) [weld-core-1.1.13.Final-redhat-1.jar:1.1.13.Final-redhat-1]
  at org.jboss.weld.util.reflection.SecureReflectionAccess.runAsInvocation(SecureReflectionAccess.java:137) [weld-core-1.1.13.Final-redhat-1.jar:1.1.13.Final-redhat-1]
  at org.jboss.weld.util.reflection.SecureReflections.invoke(SecureReflections.java:263) [weld-core-1.1.13.Final-redhat-1.jar:1.1.13.Final-redhat-1]
  at org.jboss.weld.introspector.jlr.WeldMethodImpl.invokeOnInstance(WeldMethodImpl.java:170) [weld-core-1.1.13.Final-redhat-1.jar:1.1.13.Final-redhat-1]
  at org.jboss.weld.introspector.ForwardingWeldMethod.invokeOnInstance(ForwardingWeldMethod.java:51) [weld-core-1.1.13.Final-redhat-1.jar:1.1.13.Final-redhat-1]
  at org.jboss.weld.injection.MethodInjectionPoint.invokeOnInstanceWithSpecialValue(MethodInjectionPoint.java:154) [weld-core-1.1.13.Final-redhat-1.jar:1.1.13.Final-redhat-1]
  at org.jboss.weld.bean.DisposalMethod.invokeDisposeMethod(DisposalMethod.java:159) [weld-core-1.1.13.Final-redhat-1.jar:1.1.13.Final-redhat-1]
  at org.jboss.weld.bean.ProducerMethod$ProducerMethodProducer.dispose(ProducerMethod.java:126) [weld-core-1.1.13.Final-redhat-1.jar:1.1.13.Final-redhat-1]
  at org.jboss.weld.bean.ProducerMethod.destroy(ProducerMethod.java:197) [weld-core-1.1.13.Final-redhat-1.jar:1.1.13.Final-redhat-1]
  at org.jboss.weld.context.ForwardingContextual.destroy(ForwardingContextual.java:31) [weld-core-1.1.13.Final-redhat-1.jar:1.1.13.Final-redhat-1]
  at org.jboss.weld.context.AbstractContext.destroy(AbstractContext.java:126) [weld-core-1.1.13.Final-redhat-1.jar:1.1.13.Final-redhat-1]
  at org.jboss.weld.context.AbstractContext.destroy(AbstractContext.java:140) [weld-core-1.1.13.Final-redhat-1.jar:1.1.13.Final-redhat-1]
  at org.jboss.weld.context.AbstractManagedContext.deactivate(AbstractManagedContext.java:41) [weld-core-1.1.13.Final-redhat-1.jar:1.1.13.Final-redhat-1]
  at org.jboss.weld.context.AbstractBoundContext.deactivate(AbstractBoundContext.java:72) [weld-core-1.1.13.Final-redhat-1.jar:1.1.13.Final-redhat-1]
  at org.jboss.weld.context.http.HttpRequestContextImpl.deactivate(HttpRequestContextImpl.java:86) [weld-core-1.1.13.Final-redhat-1.jar:1.1.13.Final-redhat-1]
  at org.jboss.weld.servlet.WeldListener.requestDestroyed(WeldListener.java:103) [weld-core-1.1.13.Final-redhat-1.jar:1.1.13.Final-redhat-1]
  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:175) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
  at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
  at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
  at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.2.Final-redhat-1.jar:7.2.2.Final-redhat-1]
  at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
Caused by: java.lang.UnsupportedOperationException: JBAS011859: Naming-Kontext ist schreibgesch?tzt
  at org.jboss.as.naming.WritableServiceBasedNamingStore.requireOwner(WritableServiceBasedNamingStore.java:144)
  at org.jboss.as.naming.WritableServiceBasedNamingStore.unbind(WritableServiceBasedNamingStore.java:109)
  at org.jboss.as.naming.NamingContext.unbind(NamingContext.java:286)
  at org.jboss.as.naming.InitialContext.unbind(InitialContext.java:165)
  at org.jboss.as.naming.NamingContext.unbind(NamingContext.java:294)
  at javax.naming.InitialContext.unbind(InitialContext.java:435) [rt.jar:1.7.0_25]
  at org.modeshape.jcr.JcrRepository$RunningState.unbindFromJndi(JcrRepository.java:1645) [modeshape-jcr-3.5.0.Final.jar:3.5.0.Final]
  at org.modeshape.jcr.JcrRepository$RunningState.shutdown(JcrRepository.java:1566) [modeshape-jcr-3.5.0.Final.jar:3.5.0.Final]
  at org.modeshape.jcr.JcrRepository.doShutdown(JcrRepository.java:392) [modeshape-jcr-3.5.0.Final.jar:3.5.0.Final]
  at org.modeshape.jcr.JcrRepository$1.call(JcrRepository.java:301) [modeshape-jcr-3.5.0.Final.jar:3.5.0.Final]
  at org.modeshape.jcr.JcrRepository$1.call(JcrRepository.java:298) [modeshape-jcr-3.5.0.Final.jar:3.5.0.Final]
  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_25]
  at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_25]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
  ... 1 more

So its now quite clear, why the vm is running out of threads. But why the shutdown is failing ? Do you have any idea ?

Best regards,

Daniel

[hchiorean]

When a repository starts/stops in embedded mode, it attempts to register/deregister itself from a local JNDI context. In the case when such a context is readonly (which apparently is the case of a deployable unit inside of EAP), this will throw an UnssuportedOperationException

This is an issue which we fixed for the startup sequence - and we're logging a warning which you should see when the repository starts up, but it seems is still an issue on shutdown.

When running via the ModeShape subsystem, this doesn't occur because ModeShape integrates with EAP's JNDI subsystem (as opposed to using a local initial context). I've opened [MODE-2058] Repository shutdown doesn't complete if unregistering from JNDI fails - JBoss Issue Tracker, so we'll handle this case and log a warning.

Unfortunately I can't think of a workaround except trying to search and find out if it's possible from a webapp deployed in EAP to use a local initial context - i.e. make the InitialContext writable.

[rhauch]

nilian, thanks for reporting the exception on shutdown. This is simply not really a setup that we test, since we encourage people to use the EAP subsystem as much as possible. As hchiorean mentioned, he'll shortly have a fix for the shutdown problem.

Other readers contemplating not using the ModeShape subsystem for EAP to manage the repository instances should please read the earlier related thread by Daniel Bremer-Tonn: ModeShape as a EAP6.1 Subsystem but programmatically repository creation is required.