2 Replies Latest reply on Oct 31, 2012 4:30 AM by cramond

    startSingleton blocked in ClassLoader.defineClass1

    cramond

      Hi,

       

      I use Jboss 5.1.0 in a cluster of 2 nodes. Under deploy-hasingleton, i have  a sar ( size 4300k) , a jar (26k) and a datasource definition.

      Most of the times, when i restart the master node, the slave node redeploy the 3 components.

      Sometimes, the 3 components are not redeployed on the slave node , and Jboss on the slave node is taking 100% of the CPU.

       

      Doing several jstack on the jboss of the slave node, i saw that the startSingleton method is blocked :

      Thread 1944: (state = IN_VM)

      - java.lang.ClassLoader.defineClass1(java.lang.String, byte[], int, int, java.security.ProtectionDomain, java.lang.String) @bci=0 (Compiled frame; information may be imprecise)

      - java.lang.ClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.ProtectionDomain) @bci=34, line=620 (Compiled frame)

      - org.jboss.classloader.spi.base.BaseClassLoader.access$200(org.jboss.classloader.spi.base.BaseClassLoader, java.lang.String, byte[], int, int, java.security.ProtectionDomain) @bci=8, line=63 (Compiled frame)

      - org.jboss.classloader.spi.base.BaseClassLoader$2.run() @bci=1, line=532 (Compiled frame)

      - java.security.AccessController.doPrivileged(java.security.PrivilegedAction, java.security.AccessControlContext) @bci=0 (Compiled frame)

      - org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(java.lang.String, boolean) @bci=71, line=530 (Compiled frame)

      - org.jboss.classloader.spi.base.ClassLoaderManager.process(java.lang.Thread, org.jboss.classloader.spi.base.ClassLoadingTask) @bci=9, line=150 (Compiled frame)

      - org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(org.jboss.classloader.spi.base.BaseClassLoader, java.lang.String, boolean) @bci=117, line=265 (Compiled frame)

      - org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(org.jboss.classloader.spi.base.BaseClassLoader, java.lang.String) @bci=4, line=1119 (Compiled frame)

      - java.lang.ClassLoader.loadClass(java.lang.String) @bci=3, line=251 (Compiled frame)

      - java.lang.ClassLoader.loadClassInternal(java.lang.String) @bci=2, line=319 (Compiled frame)

      - java.lang.Class.forName0(java.lang.String, boolean, java.lang.ClassLoader) @bci=0 (Compiled frame)

      - java.lang.Class.forName(java.lang.String, boolean, java.lang.ClassLoader) @bci=32, line=242 (Compiled frame)

      - javassist.CtClassType.getAnnotations(boolean) @bci=34, line=458 (Compiled frame)

      - org.jboss.classloading.plugins.vfs.VFSResourceVisitor.visit(org.jboss.virtual.VirtualFile) @bci=95, line=264 (Compiled frame)

      - org.jboss.virtual.plugins.vfs.helpers.WrappingVirtualFileHandlerVisitor.visit(org.jboss.virtual.spi.VirtualFileHandler) @bci=12, line=62 (Compiled frame)

      - org.jboss.virtual.plugins.context.AbstractVFSContext.visit(org.jboss.virtual.spi.VirtualFileHandler, org.jboss.virtual.spi.VirtualFileHandlerVisitor, boolean, boolean, boolean, boolean, org.jboss.virtual.VirtualFileFilter) @bci=287, line=376 (Compiled frame)

      - org.jboss.virtual.plugins.context.AbstractVFSContext.visit(org.jboss.virtual.spi.VirtualFileHandler, org.jboss.virtual.spi.VirtualFileHandlerVisitor, boolean, boolean, boolean, boolean, org.jboss.virtual.VirtualFileFilter) @bci=287, line=376 (Compiled frame)

      - org.jboss.virtual.plugins.context.AbstractVFSContext.visit(org.jboss.virtual.spi.VirtualFileHandler, org.jboss.virtual.spi.VirtualFileHandlerVisitor, boolean, boolean, boolean, boolean, org.jboss.virtual.VirtualFileFilter) @bci=287, line=376 (Compiled frame)

      - org.jboss.virtual.plugins.context.AbstractVFSContext.visit(org.jboss.virtual.spi.VirtualFileHandler, org.jboss.virtual.spi.VirtualFileHandlerVisitor, boolean, boolean, boolean, boolean, org.jboss.virtual.VirtualFileFilter) @bci=287, line=376 (Compiled frame)

      - org.jboss.virtual.plugins.context.AbstractVFSContext.visit(org.jboss.virtual.spi.VirtualFileHandler, org.jboss.virtual.spi.VirtualFileHandlerVisitor) @bci=78, line=306 (Compiled frame)

      - org.jboss.virtual.plugins.context.AbstractVFSContext.visit(org.jboss.virtual.spi.VirtualFileHandler, org.jboss.virtual.spi.VirtualFileHandlerVisitor) @bci=78, line=306 (Interpreted frame)

      - org.jboss.virtual.VFS.visit(org.jboss.virtual.VirtualFile, org.jboss.virtual.VirtualFileVisitor) @bci=42, line=421 (Compiled frame)

      - org.jboss.virtual.VirtualFile.visit(org.jboss.virtual.VirtualFileVisitor) @bci=13, line=437 (Compiled frame)

      - org.jboss.classloading.plugins.vfs.VFSResourceVisitor.visit(org.jboss.virtual.VirtualFile[], org.jboss.virtual.VirtualFile[], org.jboss.classloader.spi.filter.ClassFilter, org.jboss.classloader.spi.filter.ClassFilter, java.lang.ClassLoader, org.jboss.classloading.spi.visitor.ResourceVisitor, org.jboss.classloading.spi.visitor.ResourceFilter, org.jboss.classloading.spi.visitor.ResourceFilter, java.net.URL[]) @bci=78, line=101 (Interpreted frame)

      - org.jboss.deployers.vfs.plugins.annotations.FilteredAnnotationEnvironmentDeployer.visitModule(org.jboss.deployers.vfs.spi.structure.VFSDeploymentUnit, org.jboss.classloading.spi.dependency.Module, org.jboss.deployers.plugins.annotations.GenericAnnotationResourceVisitor) @bci=60, line=88 (Interpreted frame)

      - org.jboss.deployers.vfs.plugins.annotations.AnnotationEnvironmentDeployer.deploy(org.jboss.deployers.vfs.spi.structure.VFSDeploymentUnit, org.jboss.classloading.spi.dependency.Module) @bci=205, line=244 (Interpreted frame)

      - org.jboss.deployers.vfs.plugins.annotations.AnnotationEnvironmentDeployer.deploy(org.jboss.deployers.vfs.spi.structure.VFSDeploymentUnit, java.lang.Object) @bci=6, line=48 (Interpreted frame)

      - org.jboss.deployers.vfs.spi.deployer.AbstractOptionalVFSRealDeployer.deploy(org.jboss.deployers.structure.spi.DeploymentUnit, java.lang.Object) @bci=16, line=57 (Interpreted frame)

      - org.jboss.deployers.spi.deployer.helpers.AbstractOptionalRealDeployer.internalDeploy(org.jboss.deployers.structure.spi.DeploymentUnit) @bci=25, line=74 (Interpreted frame)

      - org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(org.jboss.deployers.structure.spi.DeploymentUnit) @bci=2, line=50 (Interpreted frame)

      - org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(org.jboss.deployers.structure.spi.DeploymentUnit) @bci=68, line=171 (Compiled frame)

      - org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(org.jboss.deployers.spi.deployer.Deployer, org.jboss.deployers.structure.spi.DeploymentUnit) @bci=19, line=1439 (Interpreted frame)

      - org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(org.jboss.deployers.spi.deployer.Deployer, org.jboss.deployers.structure.spi.DeploymentContext) @bci=67, line=1157 (Interpreted frame)

      - org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(org.jboss.deployers.spi.deployer.Deployer, org.jboss.deployers.structure.spi.DeploymentContext) @bci=302, line=1210 (Interpreted frame)

      - org.jboss.deployers.plugins.deployers.DeployersImpl.install(org.jboss.dependency.spi.ControllerContext, org.jboss.dependency.spi.ControllerState, org.jboss.dependency.spi.ControllerState) @bci=128, line=1098 (Interpreted frame)

      - org.jboss.dependency.plugins.AbstractControllerContext.install(org.jboss.dependency.spi.ControllerState, org.jboss.dependency.spi.ControllerState) @bci=16, line=348 (Interpreted frame)

      - org.jboss.dependency.plugins.AbstractController.install(org.jboss.dependency.spi.ControllerContext, org.jboss.dependency.spi.ControllerState, org.jboss.dependency.spi.ControllerState) @bci=22, line=1631 (Compiled frame)

      - org.jboss.dependency.plugins.AbstractController.incrementState(org.jboss.dependency.spi.ControllerContext, boolean) @bci=410, line=934 (Compiled frame)

      - org.jboss.dependency.plugins.AbstractController.resolveContexts(org.jboss.dependency.spi.ControllerState, org.jboss.dependency.spi.ControllerState, boolean) @bci=384, line=1082 (Compiled frame)

      - org.jboss.dependency.plugins.AbstractController.resolveContexts(boolean) @bci=75, line=984 (Compiled frame)

      - org.jboss.dependency.plugins.AbstractController.change(org.jboss.dependency.spi.ControllerContext, org.jboss.dependency.spi.ControllerState, boolean) @bci=196, line=822 (Interpreted frame)

      - org.jboss.dependency.plugins.AbstractController.change(org.jboss.dependency.spi.ControllerContext, org.jboss.dependency.spi.ControllerState) @bci=40, line=553 (Interpreted frame)

      - org.jboss.deployers.plugins.deployers.DeployersImpl.process(java.util.List, java.util.List) @bci=945, line=781 (Interpreted frame)

      - org.jboss.deployers.plugins.main.MainDeployerImpl.process() @bci=264, line=702 (Interpreted frame)

      - org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process() @bci=4, line=117 (Interpreted frame)

      - org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(org.jboss.profileservice.spi.Profile) @bci=102, line=70 (Interpreted frame)

      - org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(org.jboss.system.server.profileservice.repository.ProfileContext) @bci=12, line=53 (Interpreted frame)

      - org.jboss.system.server.profileservice.repository.AbstractProfileService.install(org.jboss.dependency.spi.ControllerContext, org.jboss.dependency.spi.ControllerState, org.jboss.dependency.spi.ControllerState) @bci=34, line=361 (Interpreted frame)

      - org.jboss.dependency.plugins.AbstractControllerContext.install(org.jboss.dependency.spi.ControllerState, org.jboss.dependency.spi.ControllerState) @bci=16, line=348 (Interpreted frame)

      - org.jboss.dependency.plugins.AbstractController.install(org.jboss.dependency.spi.ControllerContext, org.jboss.dependency.spi.ControllerState, org.jboss.dependency.spi.ControllerState) @bci=22, line=1631 (Compiled frame)

      - org.jboss.dependency.plugins.AbstractController.incrementState(org.jboss.dependency.spi.ControllerContext, boolean) @bci=410, line=934 (Compiled frame)

      - org.jboss.dependency.plugins.AbstractController.resolveContexts(org.jboss.dependency.spi.ControllerState, org.jboss.dependency.spi.ControllerState, boolean) @bci=384, line=1082 (Compiled frame)

      - org.jboss.dependency.plugins.AbstractController.resolveContexts(boolean) @bci=75, line=984 (Compiled frame)

      - org.jboss.dependency.plugins.AbstractController.change(org.jboss.dependency.spi.ControllerContext, org.jboss.dependency.spi.ControllerState, boolean) @bci=196, line=822 (Interpreted frame)

      - org.jboss.dependency.plugins.AbstractController.change(org.jboss.dependency.spi.ControllerContext, org.jboss.dependency.spi.ControllerState) @bci=40, line=553 (Interpreted frame)

      - org.jboss.system.server.profileservice.repository.AbstractProfileService.activateProfile(org.jboss.profileservice.spi.ProfileKey) @bci=132, line=306 (Interpreted frame)

      - org.jboss.ha.singleton.HASingletonProfileActivator.activateProfile() @bci=32, line=189 (Interpreted frame)

      - sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame)

      - sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=87, line=39 (Interpreted frame)

      - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=25 (Compiled frame)

      - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=111, line=585 (Compiled frame)

      - org.jboss.ha.singleton.HASingletonController.invokeSingletonMethod(java.lang.Object, java.lang.String, java.lang.Object) @bci=151, line=229 (Interpreted frame)

      - org.jboss.ha.singleton.HASingletonController.startSingleton() @bci=24, line=153 (Interpreted frame)

      - org.jboss.ha.framework.server.HASingletonImpl.startNewMaster() @bci=47, line=263 (Interpreted frame)

      - org.jboss.ha.framework.server.HASingletonImpl.makeThisNodeMaster() @bci=5, line=243 (Interpreted frame)

      - org.jboss.ha.framework.server.HASingletonImpl.partitionTopologyChanged(java.util.List, int, boolean) @bci=127, line=182 (Interpreted frame)

      - org.jboss.ha.framework.server.HAServiceImpl.replicantsChanged(java.lang.String, java.util.List, int, boolean) @bci=21, line=295 (Interpreted frame)

      - org.jboss.ha.framework.server.DistributedReplicantManagerImpl.notifyKeyListeners(java.lang.String, java.util.List, boolean) @bci=160, line=882 (Interpreted frame)

      - org.jboss.ha.framework.server.DistributedReplicantManagerImpl.processEvent(java.lang.Object) @bci=18, line=403 (Interpreted frame)

      - org.jboss.ha.framework.server.AsynchEventHandler.run() @bci=78, line=108 (Interpreted frame)

      - java.lang.Thread.run() @bci=11, line=595 (Interpreted frame)

       

      The slave jboss is then blocked intinitly.

      It is not JVM memory configuration problem , because there is plenty of space in the JVM : ( i did a jmap -heap on jboss )

      Attaching to process ID 22455, please wait...
      Debugger attached successfully.
      Server compiler detected.
      JVM version is 1.5.0_15-b04

       

      using thread-local object allocation.
      Parallel GC with 8 thread(s)

       

      Heap Configuration:
         MinHeapFreeRatio = 40
         MaxHeapFreeRatio = 70
         MaxHeapSize      = 1610612736 (1536.0MB)
         NewSize          = 655360 (0.625MB)
         MaxNewSize       = 4294901760 (4095.9375MB)
         OldSize          = 1441792 (1.375MB)
         NewRatio         = 1
         SurvivorRatio    = 8
         PermSize         = 16777216 (16.0MB)
         MaxPermSize      = 134217728 (128.0MB)

       

      Heap Usage:
      PS Young Generation
      Eden Space:
         capacity = 32112640 (30.625MB)
         used     = 31032464 (29.594863891601562MB)
         free     = 1080176 (1.0301361083984375MB)
         96.63629025829081% used
      From Space:
         capacity = 458752 (0.4375MB)
         used     = 360448 (0.34375MB)
         free     = 98304 (0.09375MB)
         78.57142857142857% used
      To Space:
         capacity = 458752 (0.4375MB)
         used     = 0 (0.0MB)
         free     = 458752 (0.4375MB)
         0.0% used
      PS Old Generation
         capacity = 805306368 (768.0MB)
         used     = 548190120 (522.7948379516602MB)
         free     = 257116248 (245.20516204833984MB)
         68.07224452495575% used
      PS Perm Generation
         capacity = 134217728 (128.0MB)
         used     = 86341728 (82.34188842773438MB)
         free     = 47876000 (45.658111572265625MB)
         64.32960033416748% used

       

      Does somebody got a similar jboss blocked deploying components under deploy-hasingleton ?

      Is it not recommended to deploy singleton using this way ?

       

      Christophe

      Thread 1944: (state = IN_VM)
      - java.lang.ClassLoader.defineClass1(java.lang.String, byte[], int, int, java.security.ProtectionDomain, java.lang.String) @bci=0 (Compiled frame; information may be imprecise)
      - java.lang.ClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.ProtectionDomain) @bci=34, line=620 (Compiled frame)
      - org.jboss.classloader.spi.base.BaseClassLoader.access$200(org.jboss.classloader.spi.base.BaseClassLoader, java.lang.String, byte[], int, int, java.security.ProtectionDomain) @bci=8, line=63 (Compiled frame)
      - org.jboss.classloader.spi.base.BaseClassLoader$2.run() @bci=1, line=532 (Compiled frame)
      - java.security.AccessController.doPrivileged(java.security.PrivilegedAction, java.security.AccessControlContext) @bci=0 (Compiled frame)
      - org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(java.lang.String, boolean) @bci=71, line=530 (Compiled frame)
      - org.jboss.classloader.spi.base.ClassLoaderManager.process(java.lang.Thread, org.jboss.classloader.spi.base.ClassLoadingTask) @bci=9, line=150 (Compiled frame)
      - org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(org.jboss.classloader.spi.base.BaseClassLoader, java.lang.String, boolean) @bci=117, line=265 (Compiled frame)
      - org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(org.jboss.classloader.spi.base.BaseClassLoader, java.lang.String) @bci=4, line=1119 (Compiled frame)
      - java.lang.ClassLoader.loadClass(java.lang.String) @bci=3, line=251 (Compiled frame)
      - java.lang.ClassLoader.loadClassInternal(java.lang.String) @bci=2, line=319 (Compiled frame)
      - java.lang.Class.forName0(java.lang.String, boolean, java.lang.ClassLoader) @bci=0 (Compiled frame)
      - java.lang.Class.forName(java.lang.String, boolean, java.lang.ClassLoader) @bci=32, line=242 (Compiled frame)
      - javassist.CtClassType.getAnnotations(boolean) @bci=34, line=458 (Compiled frame)
      - org.jboss.classloading.plugins.vfs.VFSResourceVisitor.visit(org.jboss.virtual.VirtualFile) @bci=95, line=264 (Compiled frame)
      - org.jboss.virtual.plugins.vfs.helpers.WrappingVirtualFileHandlerVisitor.visit(org.jboss.virtual.spi.VirtualFileHandler) @bci=12, line=62 (Compiled frame)
      - org.jboss.virtual.plugins.context.AbstractVFSContext.visit(org.jboss.virtual.spi.VirtualFileHandler, org.jboss.virtual.spi.VirtualFileHandlerVisitor, boolean, boolean, boolean, boolean, org.jboss.virtual.VirtualFileFilter) @bci=287, line=376 (Compiled frame)
      - org.jboss.virtual.plugins.context.AbstractVFSContext.visit(org.jboss.virtual.spi.VirtualFileHandler, org.jboss.virtual.spi.VirtualFileHandlerVisitor, boolean, boolean, boolean, boolean, org.jboss.virtual.VirtualFileFilter) @bci=287, line=376 (Compiled frame)
      - org.jboss.virtual.plugins.context.AbstractVFSContext.visit(org.jboss.virtual.spi.VirtualFileHandler, org.jboss.virtual.spi.VirtualFileHandlerVisitor, boolean, boolean, boolean, boolean, org.jboss.virtual.VirtualFileFilter) @bci=287, line=376 (Compiled frame)
      - org.jboss.virtual.plugins.context.AbstractVFSContext.visit(org.jboss.virtual.spi.VirtualFileHandler, org.jboss.virtual.spi.VirtualFileHandlerVisitor, boolean, boolean, boolean, boolean, org.jboss.virtual.VirtualFileFilter) @bci=287, line=376 (Compiled frame)
      - org.jboss.virtual.plugins.context.AbstractVFSContext.visit(org.jboss.virtual.spi.VirtualFileHandler, org.jboss.virtual.spi.VirtualFileHandlerVisitor) @bci=78, line=306 (Compiled frame)
      - org.jboss.virtual.plugins.context.AbstractVFSContext.visit(org.jboss.virtual.spi.VirtualFileHandler, org.jboss.virtual.spi.VirtualFileHandlerVisitor) @bci=78, line=306 (Interpreted frame)
      - org.jboss.virtual.VFS.visit(org.jboss.virtual.VirtualFile, org.jboss.virtual.VirtualFileVisitor) @bci=42, line=421 (Compiled frame)
      - org.jboss.virtual.VirtualFile.visit(org.jboss.virtual.VirtualFileVisitor) @bci=13, line=437 (Compiled frame)
      - org.jboss.classloading.plugins.vfs.VFSResourceVisitor.visit(org.jboss.virtual.VirtualFile[], org.jboss.virtual.VirtualFile[], org.jboss.classloader.spi.filter.ClassFilter, org.jboss.classloader.spi.filter.ClassFilter, java.lang.ClassLoader, org.jboss.classloading.spi.visitor.ResourceVisitor, org.jboss.classloading.spi.visitor.ResourceFilter, org.jboss.classloading.spi.visitor.ResourceFilter, java.net.URL[]) @bci=78, line=101 (Interpreted frame)
      - org.jboss.deployers.vfs.plugins.annotations.FilteredAnnotationEnvironmentDeployer.visitModule(org.jboss.deployers.vfs.spi.structure.VFSDeploymentUnit, org.jboss.classloading.spi.dependency.Module, org.jboss.deployers.plugins.annotations.GenericAnnotationResourceVisitor) @bci=60, line=88 (Interpreted frame)
      - org.jboss.deployers.vfs.plugins.annotations.AnnotationEnvironmentDeployer.deploy(org.jboss.deployers.vfs.spi.structure.VFSDeploymentUnit, org.jboss.classloading.spi.dependency.Module) @bci=205, line=244 (Interpreted frame)
      - org.jboss.deployers.vfs.plugins.annotations.AnnotationEnvironmentDeployer.deploy(org.jboss.deployers.vfs.spi.structure.VFSDeploymentUnit, java.lang.Object) @bci=6, line=48 (Interpreted frame)
      - org.jboss.deployers.vfs.spi.deployer.AbstractOptionalVFSRealDeployer.deploy(org.jboss.deployers.structure.spi.DeploymentUnit, java.lang.Object) @bci=16, line=57 (Interpreted frame)
      - org.jboss.deployers.spi.deployer.helpers.AbstractOptionalRealDeployer.internalDeploy(org.jboss.deployers.structure.spi.DeploymentUnit) @bci=25, line=74 (Interpreted frame)
      - org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(org.jboss.deployers.structure.spi.DeploymentUnit) @bci=2, line=50 (Interpreted frame)
      - org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(org.jboss.deployers.structure.spi.DeploymentUnit) @bci=68, line=171 (Compiled frame)
      - org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(org.jboss.deployers.spi.deployer.Deployer, org.jboss.deployers.structure.spi.DeploymentUnit) @bci=19, line=1439 (Interpreted frame)
      - org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(org.jboss.deployers.spi.deployer.Deployer, org.jboss.deployers.structure.spi.DeploymentContext) @bci=67, line=1157 (Interpreted frame)
      - org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(org.jboss.deployers.spi.deployer.Deployer, org.jboss.deployers.structure.spi.DeploymentContext) @bci=302, line=1210 (Interpreted frame)
      - org.jboss.deployers.plugins.deployers.DeployersImpl.install(org.jboss.dependency.spi.ControllerContext, org.jboss.dependency.spi.ControllerState, org.jboss.dependency.spi.ControllerState) @bci=128, line=1098 (Interpreted frame)
      - org.jboss.dependency.plugins.AbstractControllerContext.install(org.jboss.dependency.spi.ControllerState, org.jboss.dependency.spi.ControllerState) @bci=16, line=348 (Interpreted frame)
      - org.jboss.dependency.plugins.AbstractController.install(org.jboss.dependency.spi.ControllerContext, org.jboss.dependency.spi.ControllerState, org.jboss.dependency.spi.ControllerState) @bci=22, line=1631 (Compiled frame)
      - org.jboss.dependency.plugins.AbstractController.incrementState(org.jboss.dependency.spi.ControllerContext, boolean) @bci=410, line=934 (Compiled frame)
      - org.jboss.dependency.plugins.AbstractController.resolveContexts(org.jboss.dependency.spi.ControllerState, org.jboss.dependency.spi.ControllerState, boolean) @bci=384, line=1082 (Compiled frame)
      - org.jboss.dependency.plugins.AbstractController.resolveContexts(boolean) @bci=75, line=984 (Compiled frame)
      - org.jboss.dependency.plugins.AbstractController.change(org.jboss.dependency.spi.ControllerContext, org.jboss.dependency.spi.ControllerState, boolean) @bci=196, line=822 (Interpreted frame)
      - org.jboss.dependency.plugins.AbstractController.change(org.jboss.dependency.spi.ControllerContext, org.jboss.dependency.spi.ControllerState) @bci=40, line=553 (Interpreted frame)
      - org.jboss.deployers.plugins.deployers.DeployersImpl.process(java.util.List, java.util.List) @bci=945, line=781 (Interpreted frame)
      - org.jboss.deployers.plugins.main.MainDeployerImpl.process() @bci=264, line=702 (Interpreted frame)
      - org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process() @bci=4, line=117 (Interpreted frame)
      - org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(org.jboss.profileservice.spi.Profile) @bci=102, line=70 (Interpreted frame)
      - org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(org.jboss.system.server.profileservice.repository.ProfileContext) @bci=12, line=53 (Interpreted frame)
      - org.jboss.system.server.profileservice.repository.AbstractProfileService.install(org.jboss.dependency.spi.ControllerContext, org.jboss.dependency.spi.ControllerState, org.jboss.dependency.spi.ControllerState) @bci=34, line=361 (Interpreted frame)
      - org.jboss.dependency.plugins.AbstractControllerContext.install(org.jboss.dependency.spi.ControllerState, org.jboss.dependency.spi.ControllerState) @bci=16, line=348 (Interpreted frame)
      - org.jboss.dependency.plugins.AbstractController.install(org.jboss.dependency.spi.ControllerContext, org.jboss.dependency.spi.ControllerState, org.jboss.dependency.spi.ControllerState) @bci=22, line=1631 (Compiled frame)
      - org.jboss.dependency.plugins.AbstractController.incrementState(org.jboss.dependency.spi.ControllerContext, boolean) @bci=410, line=934 (Compiled frame)
      - org.jboss.dependency.plugins.AbstractController.resolveContexts(org.jboss.dependency.spi.ControllerState, org.jboss.dependency.spi.ControllerState, boolean) @bci=384, line=1082 (Compiled frame)
      - org.jboss.dependency.plugins.AbstractController.resolveContexts(boolean) @bci=75, line=984 (Compiled frame)
      - org.jboss.dependency.plugins.AbstractController.change(org.jboss.dependency.spi.ControllerContext, org.jboss.dependency.spi.ControllerState, boolean) @bci=196, line=822 (Interpreted frame)
      - org.jboss.dependency.plugins.AbstractController.change(org.jboss.dependency.spi.ControllerContext, org.jboss.dependency.spi.ControllerState) @bci=40, line=553 (Interpreted frame)
      - org.jboss.system.server.profileservice.repository.AbstractProfileService.activateProfile(org.jboss.profileservice.spi.ProfileKey) @bci=132, line=306 (Interpreted frame)
      - org.jboss.ha.singleton.HASingletonProfileActivator.activateProfile() @bci=32, line=189 (Interpreted frame)
      - sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame)
      - sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=87, line=39 (Interpreted frame)
      - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=25 (Compiled frame)
      - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=111, line=585 (Compiled frame)
      - org.jboss.ha.singleton.HASingletonController.invokeSingletonMethod(java.lang.Object, java.lang.String, java.lang.Object) @bci=151, line=229 (Interpreted frame)
      - org.jboss.ha.singleton.HASingletonController.startSingleton() @bci=24, line=153 (Interpreted frame)
      - org.jboss.ha.framework.server.HASingletonImpl.startNewMaster() @bci=47, line=263 (Interpreted frame)
      - org.jboss.ha.framework.server.HASingletonImpl.makeThisNodeMaster() @bci=5, line=243 (Interpreted frame)
      - org.jboss.ha.framework.server.HASingletonImpl.partitionTopologyChanged(java.util.List, int, boolean) @bci=127, line=182 (Interpreted frame)
      - org.jboss.ha.framework.server.HAServiceImpl.replicantsChanged(java.lang.String, java.util.List, int, boolean) @bci=21, line=295 (Interpreted frame)
      - org.jboss.ha.framework.server.DistributedReplicantManagerImpl.notifyKeyListeners(java.lang.String, java.util.List, boolean) @bci=160, line=882 (Interpreted frame)
      - org.jboss.ha.framework.server.DistributedReplicantManagerImpl.processEvent(java.lang.Object) @bci=18, line=403 (Interpreted frame)
      - org.jboss.ha.framework.server.AsynchEventHandler.run() @bci=78, line=108 (Interpreted frame)
      - java.lang.Thread.run() @bci=11, line=595 (Interpreted frame)

       

        • 1. Re: startSingleton blocked in ClassLoader.defineClass1
          bondchan921

          any updates?

           

          I have a very similar issue,

           

           

          one thread seem calling  java.lang.ClassLoader.defineClass1, and never gets return from this method, causing other threads waiting monitors lock buy this thread.

           

          part of threaddump


          Version:0.9 StartHTML:-1 EndHTML:-1 StartFragment:0000000111 EndFragment:0000001999  

          "http-0.0.0.0-8080-10" daemon prio=10 tid=0x00002aabc1b80400 nid=0x2639 runnable [0x000000004a5ca000..0x000000004a5cdc20] java.lang.Thread.State: RUNNABLE at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:621) at org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:63) at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:572) at org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:532) at java.security.AccessController.doPrivileged(Native Method) at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:530) - locked <0x00002aaac6c94778> (a org.jboss.classloader.spi.base.BaseClassLoader) at org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:507) at org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134) at org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131) at org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452) at org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:251) at org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:150) at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:265) at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1119) at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:798) at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:441)
          • 2. Re: startSingleton blocked in ClassLoader.defineClass1
            cramond

            after posting these post , i tried to reduce the size of the sar .

            The sar was build by maven , then it was filled with a lot of external library.

            With the library , i created an ear by maven containing all the external library and renamed it .jar . I put the jar under lib

            and the sar was containing only the mbean needed to be in singleton.

            This workaround is running in production since one year and a half and the customer does not complain.