12 Replies Latest reply on Dec 31, 2014 4:21 AM by Eduardo Martins

    Help Needed: Deadlock in JNDI binding (trying to migrate from JBoss AS 6.2 to Wildfly 8.2)

    Luke Meyers Newbie

      I'm trying to upgrade a large legacy application from JBoss AS 6.2 to Wildfly 8.2.  Unfortunately, I'm getting a deadlock when trying to deploy my application EAR.  I'm pretty new to tweaking JBoss mechanics and would really appreciate some help (including identifying what additional information might be helpful to post).

       

      Here's what I'm seeing.  Most of my MSC threads are waiting to acquire a lock:

       

      "MSC service thread 1-7" #21 prio=5 os_prio=0 tid=0x00007fb8cc66a800 nid=0x11c1 waiting for monitor entry [0x00007fb8af6e7000]

         java.lang.Thread.State: BLOCKED (on object monitor)

        at org.jboss.util.naming.NonSerializableFactory.rebind(NonSerializableFactory.java:200)

        - waiting to lock <0x00000000f8929370> (a java.lang.Class for org.jboss.util.naming.NonSerializableFactory)

        at com.mycompany.delegates.RemoteServiceDelegateFactory.rebind(RemoteServiceDelegateFactory.java:89)

        at com.mycompany.delegates.RemoteServiceDelegateFactory.start(RemoteServiceDelegateFactory.java:28)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:483)

        at org.jboss.as.service.AbstractService.invokeLifecycleMethod(AbstractService.java:71)

        at org.jboss.as.service.StartStopService.start(StartStopService.java:56)

        at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)

        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

       

      One MSC thread holds that lock, but is stuck in a call to StabilityMonitor.awaitStability:

       

      "MSC service thread 1-5" #19 prio=5 os_prio=0 tid=0x00007fb86c002000 nid=0x11bf in Object.wait() [0x00007fb8af8e9000]

         java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        at java.lang.Object.wait(Object.java:502)

        at org.jboss.msc.service.StabilityMonitor.awaitStability(StabilityMonitor.java:306)

        - locked <0x00000000f87edf00> (a java.lang.Object)

        at org.jboss.msc.service.StabilityMonitor.awaitStability(StabilityMonitor.java:221)

        at org.jboss.as.naming.WritableServiceBasedNamingStore.bind(WritableServiceBasedNamingStore.java:90)

        at org.jboss.as.naming.WritableServiceBasedNamingStore.rebind(WritableServiceBasedNamingStore.java:114)

        at org.jboss.as.naming.NamingContext.rebind(NamingContext.java:274)

        at org.jboss.as.naming.InitialContext$DefaultInitialContext.rebind(InitialContext.java:269)

        at org.jboss.as.naming.NamingContext.rebind(NamingContext.java:282)

        at javax.naming.InitialContext.rebind(InitialContext.java:433)

        at javax.naming.InitialContext.rebind(InitialContext.java:433)

        at org.jboss.util.naming.NonSerializableFactory.rebind(NonSerializableFactory.java:166)

        - locked <0x00000000f8929370> (a java.lang.Class for org.jboss.util.naming.NonSerializableFactory)

        at org.jboss.util.naming.NonSerializableFactory.rebind(NonSerializableFactory.java:207)

        - locked <0x00000000f8929370> (a java.lang.Class for org.jboss.util.naming.NonSerializableFactory)

        at com.mycompany.delegates.LocalServiceDelegateFactory.rebind(LocalServiceDelegateFactory.java:104)

        at com.mycompany.delegates.LocalServiceDelegateFactory.start(LocalServiceDelegateFactory.java:39)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:483)

        at org.jboss.as.service.AbstractService.invokeLifecycleMethod(AbstractService.java:71)

        at org.jboss.as.service.StartStopService.start(StartStopService.java:56)

        at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)

        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

       

      I also see a "management handler" thread doing the same wait:

       

      "management-handler-thread - 18" #173 prio=5 os_prio=0 tid=0x00007fb858330800 nid=0x138a in Object.wait() [0x00007fb8a71f0000]

         java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        at java.lang.Object.wait(Object.java:502)

        at org.jboss.msc.service.StabilityMonitor.awaitStability(StabilityMonitor.java:306)

        - locked <0x00000000e051fab8> (a java.lang.Object)

        at org.jboss.msc.service.StabilityMonitor.awaitStability(StabilityMonitor.java:273)

        at org.jboss.as.controller.ContainerStateMonitor.awaitContainerStateChangeReport(ContainerStateMonitor.java:102)

        at org.jboss.as.controller.ModelControllerImpl.awaitContainerStateChangeReport(ModelControllerImpl.java:633)

        at org.jboss.as.controller.OperationContextImpl.awaitModelControllerContainerMonitor(OperationContextImpl.java:207)

        at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:475)

        at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:298)

        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:293)

        at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:276)

        at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:150)

        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:199)

        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.access$300(ModelControllerClientOperationHandler.java:130)

        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:150)

        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:146)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:422)

        at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:94)

        at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:146)

        at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:283)

        at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:504)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

        at org.jboss.threads.JBossThread.run(JBossThread.java:122)

       

      I did find this old issue that mentions similar symptoms:

      [LIVEOAK-290] LiveOak on WildFly deadlocks starting up in some environments - JBoss Issue Tracker

       

      However, the recommended workaround there is not to call StabilityMonitor at all, or at least not from MSC threads.  I don't see how to apply that in this case, since all those calls are happening within JBoss code.  I have noticed that the problem seems to crop up when I hit a certain critical number of MBeans trying to register themselves with JNDI, which seems consistent with Marko's comment there about auto-sizing of MSC thread pools.  I've got a quad-core and the problem starts when I hit 8 mbeans (doesn't seem to matter much which ones), so with hyperthreading that seems to add up.

       

      Please advise.  I'm happy to provide any additional information that could help.  My deployment consists of an EAR archive that contains a single SAR; the SAR uses a jboss-service.xml to specify the MBeans it wants (if I understand right, this is the "old way" but is supposed to work -- please correct me if I'm misunderstanding that).  There are no MBean interdependencies or anything that I can see.  I've got a working deployment of a smaller service that's set up along similar lines, so I think my general approach is sound apart from this deadlock issue.

       

      Thanks in advance for any help.