How to get more then 10 concurrent instances to run on a single VM?
iwitham Mar 28, 2013 3:56 PMI have 2 VM's that each have the following resources:
4 CPU's
16GB RAM
JBos AS 6
Each jboss instance has its own virtual IP address
Each jboss instance has a dedicated peer ID
I have 14 instances of Jboss on one and 15 on the other that I want to run concurrently. The instances are configured with the following: Jboss has a max heap size of 3.5GB and each instance have max heap sizes of 1GB. I launched Jconsole to watch the instances memory usage during and after loading. Once I launched the 10 instance the committed memory was at 4+GB and I could not launch additional instances. The log shows the following information:
2013-03-28 15:15:23,191 INFO [org.jboss.system.server.jmx.JMXKernel] (Thread-21) Server exit called, exiting the JVM now!
2013-03-28 15:15:23,195 INFO [STDOUT] (Thread-1) Posting Shutdown Request to the server...
2013-03-28 15:15:23,196 INFO [org.jboss.bootstrap.impl.base.server.AbstractServer] (Thread-22) Stopping: JBossAS [6.1.0.Final "Neo"]
2013-03-28 15:15:23,197 INFO [org.apache.coyote.http11.Http11Protocol] (Thread-22) Pausing Coyote HTTP/1.1 on http-jcmshowell%2F10.3.2.125-8080
2013-03-28 15:15:23,198 INFO [org.apache.coyote.http11.Http11Protocol] (Thread-22) Stopping Coyote HTTP/1.1 on http-jcmshowell%2F10.3.2.125-8080
2013-03-28 15:15:23,201 INFO [org.apache.coyote.ajp.AjpProtocol] (Thread-22) Pausing Coyote AJP/1.3 on ajp-jcmshowell%2F10.3.2.125-8009
2013-03-28 15:15:23,202 INFO [org.apache.coyote.ajp.AjpProtocol] (Thread-22) Stopping Coyote AJP/1.3 on ajp-jcmshowell%2F10.3.2.125-8009
2013-03-28 15:15:23,205 INFO [org.apache.coyote.http11.Http11Protocol] (Thread-22) Pausing Coyote HTTP/1.1 on http-jcmshowell%2F10.3.2.125-8443
2013-03-28 15:15:23,207 INFO [org.apache.coyote.http11.Http11Protocol] (Thread-22) Stopping Coyote HTTP/1.1 on http-jcmshowell%2F10.3.2.125-8443
2013-03-28 15:15:23,227 INFO [org.jboss.web.tomcat.service.deployers.TomcatDeployment] (Thread-22) undeploy, ctxPath=/jcms
2013-03-28 15:18:06,089 ERROR [ProfileServiceBootstrap] (Thread-2) Failed to load profile:: org.jboss.deployers.client.spi.IncompleteDeploymentException: Summary of incomplete deployments (SEE PREVIOUS ERRORS FOR DETAILS):
DEPLOYMENTS MISSING DEPENDENCIES:
Deployment "AdminConsoleActivator" is missing the following dependencies:
Dependency "WebServer" (should be in state "Installed", but is actually in state "Configured")
Deployment "CachedConnectionManager" is missing the following dependencies:
Dependency "TransactionManager" (should be in state "Installed", but is actually in state "PreInstall")
Deployment "DefaultUserTransactionprovider" is missing the following dependencies:
Dependency "<UNKNOWN DefaultUserTransactionprovider>" (should be in state "Installed", but is actually in state "** UNRESOLVED Demands 'TransactionManager' **")
Deployment "JBossTransactionServices" is missing the following dependencies:
Dependency "TransactionManager" (should be in state "Installed", but is actually in state "PreInstall")
Deployment "JBossWSConsoleActivator" is missing the following dependencies:
Dependency "WebServer" (should be in state "Installed", but is actually in state "Configured")
Deployment "JmxConsoleActivator" is missing the following dependencies:
Dependency "WebServer" (should be in state "Installed", but is actually in state "Configured")
Deployment "MK2TimerServiceFactory" is missing the following dependencies:
Dependency "persistence.unit:unitName=jboss-ejb3-timerservice-mk2.jar#timerdb" (should be in state "Installed", but is actually in state "Configured")
Dependency "interface javax.transaction.TransactionManager" (should be in state "Installed", but is actually in state "** NOT FOUND Depends on 'interface javax.transaction.TransactionManager' **")
Deployment "ModClusterListener" is missing the following dependencies:
Dependency "WebServer" (should be in state "Create", but is actually in state "Configured")
Deployment "ObjectStoreBrowserBean" is missing the following dependencies:
Dependency "<UNKNOWN ObjectStoreBrowserBean>" (should be in state "Installed", but is actually in state "** UNRESOLVED Demands 'TransactionManager' **")
Deployment "RealTransactionManager" is missing the following dependencies:
Dependency "TransactionManager" (should be in state "Installed", but is actually in state "PreInstall")
Deployment "TransactionManager" is missing the following dependencies:
Dependency "<UNKNOWN TransactionManager>" (should be in state "Installed", but is actually in state "** UNRESOLVED Demands 'RecoveryManager' **")
Deployment "TxControl" is missing the following dependencies:
Dependency "<UNKNOWN TxControl>" (should be in state "Installed", but is actually in state "** UNRESOLVED Demands 'RecoveryManager' **")
Deployment "UnifiedInvoker" is missing the following dependencies:
Dependency "TransactionManager" (should be in state "Create", but is actually in state "PreInstall")
Deployment "WebServer" is missing the following dependencies:
Dependency "jboss.jca:service=CachedConnectionManager" (should be in state "Create", but is actually in state "** NOT FOUND Depends on 'jboss.jca:service=CachedConnectionManager' **")
Dependency "jboss:service=TransactionManager" (should be in state "Create", but is actually in state "** NOT FOUND Depends on 'jboss:service=TransactionManager' **")
Deployment "WorkManager" is missing the following dependencies:
Dependency "TransactionManager" (should be in state "Installed", but is actually in state "PreInstall")
Deployment "jboss-switchboard:appName=JCMSWebTier,module=JCMSWebTier" is missing the following dependencies:
Dependency "java:global/cdi/JCMSWebTier/JCMSWebTier/BeanManager" (should be in state "Installed", but is actually in state "** NOT FOUND Depends on 'java:global/cdi/JCMSWebTier/JCMSWebTier/BeanManager' **")
Deployment "jboss.ejb:persistencePolicy=database,service=EJBTimerService" is missing the following dependencies:
Dependency "jboss.jca:name=DefaultDS,service=DataSourceBinding" (should be in state "Create", but is actually in state "Configured")
Deployment "jboss.ejb:service=EJBTimerService" is missing the following dependencies:
Dependency "jboss:service=TransactionManager" (should be in state "Create", but is actually in state "** NOT FOUND Depends on 'jboss:service=TransactionManager' **")
Dependency "jboss.ejb:persistencePolicy=database,service=EJBTimerService" (should be in state "Create", but is actually in state "Configured")
Deployment "jboss.jca:name='jboss-local-jdbc.rar',service=RARDeployment" is missing the following dependencies:
Dependency "jboss.jca:service=WorkManager" (should be in state "Installed", but is actually in state "** NOT FOUND Depends on 'jboss.jca:service=WorkManager' **")
Dependency "jboss:service=TransactionManager" (should be in state "Installed", but is actually in state "** NOT FOUND Depends on 'jboss:service=TransactionManager' **")
Deployment "jboss.jca:name='jboss-xa-jdbc.rar',service=RARDeployment" is missing the following dependencies:
Dependency "jboss.jca:service=WorkManager" (should be in state "Installed", but is actually in state "** NOT FOUND Depends on 'jboss.jca:service=WorkManager' **")
Dependency "jboss:service=TransactionManager" (should be in state "Installed", but is actually in state "** NOT FOUND Depends on 'jboss:service=TransactionManager' **")
Deployment "jboss.jca:name='jms-ra.rar',service=RARDeployment" is missing the following dependencies:
Dependency "jboss.jca:service=WorkManager" (should be in state "Installed", but is actually in state "** NOT FOUND Depends on 'jboss.jca:service=WorkManager' **")
Dependency "jboss:service=TransactionManager" (should be in state "Installed", but is actually in state "** NOT FOUND Depends on 'jboss:service=TransactionManager' **")
Deployment "jboss.jca:name='mail-ra.rar',service=RARDeployment" is missing the following dependencies:
Dependency "jboss:service=TransactionManager" (should be in state "Installed", but is actually in state "** NOT FOUND Depends on 'jboss:service=TransactionManager' **")
Dependency "jboss.jca:service=WorkManager" (should be in state "Installed", but is actually in state "** NOT FOUND Depends on 'jboss.jca:service=WorkManager' **")
Deployment "jboss.jca:name='quartz-ra.rar',service=RARDeployment" is missing the following dependencies:
Dependency "jboss:service=TransactionManager" (should be in state "Installed", but is actually in state "** NOT FOUND Depends on 'jboss:service=TransactionManager' **")
Dependency "jboss.jca:service=WorkManager" (should be in state "Installed", but is actually in state "** NOT FOUND Depends on 'jboss.jca:service=WorkManager' **")
Deployment "jboss.jca:name=DefaultDS,service=DataSourceBinding" is missing the following dependencies:
Dependency "jboss.jca:name=DefaultDS,service=LocalTxCM" (should be in state "Create", but is actually in state "Configured")
Deployment "jboss.jca:name=DefaultDS,service=LocalTxCM" is missing the following dependencies:
Dependency "jboss:service=TransactionManager" (should be in state "Create", but is actually in state "** NOT FOUND Depends on 'jboss:service=TransactionManager' **")
Dependency "jboss.jca:name=DefaultDS,service=ManagedConnectionPool" (should be in state "Create", but is actually in state "Configured")
Dependency "jboss.jca:service=CachedConnectionManager" (should be in state "Create", but is actually in state "** NOT FOUND Depends on 'jboss.jca:service=CachedConnectionManager' **")
Deployment "jboss.jca:name=DefaultDS,service=ManagedConnectionFactory" is missing the following dependencies:
Dependency "RecoveryManager" (should be in state "Installed", but is actually in state "**ERROR**")
Deployment "jboss.jca:name=DefaultDS,service=ManagedConnectionPool" is missing the following dependencies:
Dependency "jboss.jca:name=DefaultDS,service=ManagedConnectionFactory" (should be in state "Create", but is actually in state "Instantiated")
Deployment "jboss.jca:name=JCMSWebDBDS,service=DataSourceBinding" is missing the following dependencies:
Dependency "jboss.jca:name=JCMSWebDBDS,service=LocalTxCM" (should be in state "Create", but is actually in state "Configured")
Deployment "jboss.jca:name=JCMSWebDBDS,service=LocalTxCM" is missing the following dependencies:
Dependency "jboss:service=TransactionManager" (should be in state "Create", but is actually in state "** NOT FOUND Depends on 'jboss:service=TransactionManager' **")
Dependency "jboss.jca:service=CachedConnectionManager" (should be in state "Create", but is actually in state "** NOT FOUND Depends on 'jboss.jca:service=CachedConnectionManager' **")
Dependency "jboss.jca:name=JCMSWebDBDS,service=ManagedConnectionPool" (should be in state "Create", but is actually in state "Configured")
Deployment "jboss.jca:name=JCMSWebDBDS,service=ManagedConnectionFactory" is missing the following dependencies:
Dependency "RecoveryManager" (should be in state "Installed", but is actually in state "**ERROR**")
Deployment "jboss.jca:name=JCMSWebDBDS,service=ManagedConnectionPool" is missing the following dependencies:
Dependency "jboss.jca:name=JCMSWebDBDS,service=ManagedConnectionFactory" (should be in state "Create", but is actually in state "Instantiated")
Deployment "jboss.jca:name=JCMSWebDS,service=DataSourceBinding" is missing the following dependencies:
Dependency "jboss.jca:name=JCMSWebDS,service=LocalTxCM" (should be in state "Create", but is actually in state "Configured")
Deployment "jboss.jca:name=JCMSWebDS,service=LocalTxCM" is missing the following dependencies:
Dependency "jboss.jca:service=CachedConnectionManager" (should be in state "Create", but is actually in state "** NOT FOUND Depends on 'jboss.jca:service=CachedConnectionManager' **")
Dependency "jboss.jca:name=JCMSWebDS,service=ManagedConnectionPool" (should be in state "Create", but is actually in state "Configured")
Dependency "jboss:service=TransactionManager" (should be in state "Create", but is actually in state "** NOT FOUND Depends on 'jboss:service=TransactionManager' **")
Deployment "jboss.jca:name=JCMSWebDS,service=ManagedConnectionFactory" is missing the following dependencies:
Dependency "RecoveryManager" (should be in state "Installed", but is actually in state "**ERROR**")
Deployment "jboss.jca:name=JCMSWebDS,service=ManagedConnectionPool" is missing the following dependencies:
Dependency "jboss.jca:name=JCMSWebDS,service=ManagedConnectionFactory" (should be in state "Create", but is actually in state "Instantiated")
Deployment "jboss.jca:name=JmsXA,service=ConnectionFactoryBinding" is missing the following dependencies:
Dependency "jboss.jca:name=JmsXA,service=TxCM" (should be in state "Create", but is actually in state "Configured")
Deployment "jboss.jca:name=JmsXA,service=ManagedConnectionFactory" is missing the following dependencies:
Dependency "RecoveryManager" (should be in state "Installed", but is actually in state "**ERROR**")
Deployment "jboss.jca:name=JmsXA,service=ManagedConnectionPool" is missing the following dependencies:
Dependency "jboss.jca:name=JmsXA,service=ManagedConnectionFactory" (should be in state "Create", but is actually in state "Instantiated")
Deployment "jboss.jca:name=JmsXA,service=TxCM" is missing the following dependencies:
Dependency "jboss:service=TransactionManager" (should be in state "Create", but is actually in state "** NOT FOUND Depends on 'jboss:service=TransactionManager' **")
Dependency "jboss.jca:name=JmsXA,service=ManagedConnectionPool" (should be in state "Create", but is actually in state "Configured")
Dependency "jboss.jca:service=CachedConnectionManager" (should be in state "Create", but is actually in state "** NOT FOUND Depends on 'jboss.jca:service=CachedConnectionManager' **")
Deployment "jboss.jca:name=RSLIMSDS,service=DataSourceBinding" is missing the following dependencies:
Dependency "jboss.jca:name=RSLIMSDS,service=LocalTxCM" (should be in state "Create", but is actually in state "Configured")
Deployment "jboss.jca:name=RSLIMSDS,service=LocalTxCM" is missing the following dependencies:
Dependency "jboss.jca:service=CachedConnectionManager" (should be in state "Create", but is actually in state "** NOT FOUND Depends on 'jboss.jca:service=CachedConnectionManager' **")
Dependency "jboss:service=TransactionManager" (should be in state "Create", but is actually in state "** NOT FOUND Depends on 'jboss:service=TransactionManager' **")
Dependency "jboss.jca:name=RSLIMSDS,service=ManagedConnectionPool" (should be in state "Create", but is actually in state "Configured")
Deployment "jboss.jca:name=RSLIMSDS,service=ManagedConnectionFactory" is missing the following dependencies:
Dependency "RecoveryManager" (should be in state "Installed", but is actually in state "**ERROR**")
Deployment "jboss.jca:name=RSLIMSDS,service=ManagedConnectionPool" is missing the following dependencies:
Dependency "jboss.jca:name=RSLIMSDS,service=ManagedConnectionFactory" (should be in state "Create", but is actually in state "Instantiated")
Deployment "jboss.web.deployment:war=/ROOT" is missing the following dependencies:
Dependency "jboss.web:service=WebServer" (should be in state "Create", but is actually in state "Configured")
Deployment "jboss.web.deployment:war=/invoker" is missing the following dependencies:
Dependency "jboss.web:service=WebServer" (should be in state "Create", but is actually in state "Configured")
Deployment "jboss.web.deployment:war=/jcms" is missing the following dependencies:
Dependency "jboss.web:service=WebServer" (should be in state "Create", but is actually in state "Configured")
Dependency "vfs:///usr/local/jbosshowell/server/default/deploy/JCMSWebTier.war_WeldBootstrapBean" (should be in state "Create", but is actually in state "Instantiated")
Deployment "jboss:service=ClientUserTransaction" is missing the following dependencies:
Dependency "jboss:service=proxyFactory,target=ClientUserTransactionFactory" (should be in state "Create", but is actually in state "Configured")
Dependency "jboss:service=proxyFactory,target=ClientUserTransaction" (should be in state "Create", but is actually in state "Configured")
Deployment "jboss:service=KeyGeneratorFactory,type=HiLo" is missing the following dependencies:
Dependency "jboss.jca:name=DefaultDS,service=DataSourceBinding" (should be in state "Create", but is actually in state "Configured")
Dependency "jboss:service=TransactionManager" (should be in state "Create", but is actually in state "** NOT FOUND Depends on 'jboss:service=TransactionManager' **")
Deployment "jboss:service=invoker,type=local" is missing the following dependencies:
Dependency "jboss:service=TransactionManager" (should be in state "Create", but is actually in state "** NOT FOUND Depends on 'jboss:service=TransactionManager' **")
Deployment "jboss:service=proxyFactory,target=ClientUserTransaction" is missing the following dependencies:
Dependency "jboss:service=invoker,type=unified" (should be in state "Create", but is actually in state "Configured")
Deployment "jboss:service=proxyFactory,target=ClientUserTransactionFactory" is missing the following dependencies:
Dependency "jboss:service=invoker,type=unified" (should be in state "Create", but is actually in state "Configured")
Deployment "persistence.unit:unitName=jboss-ejb3-timerservice-mk2.jar#timerdb" is missing the following dependencies:
Dependency "jboss.jca:name=DefaultDS,service=DataSourceBinding" (should be in state "Create", but is actually in state "Configured")
Deployment "vfs:///usr/local/jbosshowell/server/default/deploy/JCMSIntegrationTier.jar" is missing the following dependencies:
Dependency "interface javax.transaction.TransactionManager" (should be in state "Installed", but is actually in state "** NOT FOUND Depends on 'interface javax.transaction.TransactionManager' **")
Dependency "interface javax.transaction.TransactionManager" (should be in state "Installed", but is actually in state "** NOT FOUND Depends on 'interface javax.transaction.TransactionManager' **")
Deployment "vfs:///usr/local/jbosshowell/server/default/deploy/JCMSMiddleTier.jar" is missing the following dependencies:
Dependency "interface javax.transaction.TransactionManager" (should be in state "Installed", but is actually in state "** NOT FOUND Depends on 'interface javax.transaction.TransactionManager' **")
Dependency "interface javax.transaction.TransactionManager" (should be in state "Installed", but is actually in state "** NOT FOUND Depends on 'interface javax.transaction.TransactionManager' **")
Deployment "vfs:///usr/local/jbosshowell/server/default/deploy/JCMSWebTier.war_WeldBootstrapBean" is missing the following dependencies:
Dependency "JBossTransactionServices" (should be in state "Installed", but is actually in state "Described")
DEPLOYMENTS IN ERROR:
Deployment "interface javax.transaction.TransactionManager" is in error due to the following reason(s): ** NOT FOUND Depends on 'interface javax.transaction.TransactionManager' **
Deployment "<UNKNOWN ObjectStoreBrowserBean>" is in error due to the following reason(s): ** UNRESOLVED Demands 'TransactionManager' **
Deployment "jboss.jca:service=WorkManager" is in error due to the following reason(s): ** NOT FOUND Depends on 'jboss.jca:service=WorkManager' **
Deployment "<UNKNOWN TxControl>" is in error due to the following reason(s): ** UNRESOLVED Demands 'RecoveryManager' **
Deployment "jboss:service=invoker,type=unified" is in error due to the following reason(s): Configured
Deployment "<UNKNOWN TransactionManager>" is in error due to the following reason(s): ** UNRESOLVED Demands 'RecoveryManager' **
Deployment "java:global/cdi/JCMSWebTier/JCMSWebTier/BeanManager" is in error due to the following reason(s): ** NOT FOUND Depends on 'java:global/cdi/JCMSWebTier/JCMSWebTier/BeanManager' **
Deployment "<UNKNOWN DefaultUserTransactionprovider>" is in error due to the following reason(s): ** UNRESOLVED Demands 'TransactionManager' **
Deployment "jboss:service=TransactionManager" is in error due to the following reason(s): ** NOT FOUND Depends on 'jboss:service=TransactionManager' **
Deployment "RecoveryManager" is in error due to the following reason(s): com.arjuna.ats.arjuna.exceptions.FatalError: ARJUNA-12359 SocketProcessId.getpid could not get unique port., **ERROR**
Deployment "jboss.jca:service=CachedConnectionManager" is in error due to the following reason(s): ** NOT FOUND Depends on 'jboss.jca:service=CachedConnectionManager' **
Deployment "jboss.web:service=WebServer" is in error due to the following reason(s): Configured
at org.jboss.deployers.plugins.deployers.DeployersImpl.checkComplete(DeployersImpl.java:1228) [:2.2.2.GA]
at org.jboss.deployers.plugins.main.MainDeployerImpl.checkComplete(MainDeployerImpl.java:905) [:2.2.2.GA]
at org.jboss.system.server.profileservice.deployers.MainDeployerPlugin.checkComplete(MainDeployerPlugin.java:87) [:6.1.0.Final]
at org.jboss.profileservice.deployment.ProfileDeployerPluginRegistry.checkAllComplete(ProfileDeployerPluginRegistry.java:107) [:0.2.2]
at org.jboss.system.server.profileservice.bootstrap.BasicProfileServiceBootstrap.start(BasicProfileServiceBootstrap.java:135) [:6.1.0.Final]
at org.jboss.system.server.profileservice.bootstrap.BasicProfileServiceBootstrap.start(BasicProfileServiceBootstrap.java:56) [:6.1.0.Final]
at org.jboss.bootstrap.impl.base.server.AbstractServer.startBootstraps(AbstractServer.java:827) [jboss-bootstrap-impl-base.jar:2.1.0-alpha-6]
at org.jboss.bootstrap.impl.base.server.AbstractServer$StartServerTask.run(AbstractServer.java:417) [jboss-bootstrap-impl-base.jar:2.1.0-alpha-6]
at java.lang.Thread.run(Thread.java:662) [:1.6.0_30]
2013-03-28 15:18:06,094 INFO [org.jboss.bootstrap.impl.base.server.AbstractServer] (Thread-2) JBossAS [6.1.0.Final "Neo"] Started in 20s:920ms
I am confused by this since I would believe this is configured correctly. Is there a limit on the number of concurrent instances that can be run on a single server? Is there a way to make this work? Anyone run into this issue and have been able to resolve it?
Thanks,
Iry Witham
Scientific Applications Administrator
Scientific Computing Group
Computational Sciences Dept.
The Jackson Laboratory
600 Main Street
Bar Harbor, ME 04609
Phone: 207-288-6744
email: iry.witham@jax.org