11 Replies Latest reply on Feb 10, 2012 6:36 AM by jbrow

    Deployment gets stuck during EJB instance creation

    porobd

      I am trying to deploy EJBCA 4.0.3 (www.ejbca.org) on Jboss AS 7. The EJBCA application has a number of web archives and a couple of EJB modules. There are 2 servlets (each in a separate web archive) which are loaded at startup. Each of these servlets injects one/more Stateless Session Beans and invokes methods on these beans inside the init function. Both these servlets seem to get stuck in their respective init functions and so deployment does not proceed further. In fact Jboss goes into a state where I have to hard kill it. The thread dump at this point shows the 'MSC Service Threads' in the following state:

       

      "MSC service thread 1-2" prio=10 tid=0x0867d000 nid=0x5c8c in Object.wait() [0xc418d000..0xc418e030]

         java.lang.Thread.State: WAITING (on object monitor)

              at java.lang.Object.wait(Native Method)

              - waiting on <0xcb0b3d50> (a org.jboss.as.ejb3.component.stateless.StatelessSessionComponent)

              at java.lang.Object.wait(Object.java:485)

              at org.jboss.as.ee.component.BasicComponent.waitForComponentStart(BasicComponent.java:102)

              - locked <0xcb0b3d50> (a org.jboss.as.ejb3.component.stateless.StatelessSessionComponent)

              at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:76)

              at org.jboss.as.ejb3.component.stateless.StatelessSessionComponent$1.create(StatelessSessionComponent.java:70)

              at org.jboss.as.ejb3.component.stateless.StatelessSessionComponent$1.create(StatelessSessionComponent.java:67)

              at org.jboss.ejb3.pool.AbstractPool.create(AbstractPool.java:65)

              at org.jboss.ejb3.pool.strictmax.StrictMaxPool.get(StrictMaxPool.java:145)

              at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:41)

              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:287)

              at org.jboss.as.ejb3.component.session.SessionInvocationContextInterceptor$CustomSessionInvocationContext.proceed(SessionInvocationContextInterceptor.java:126)

              at org.jboss.ejb3.tx2.impl.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:257)

              at org.jboss.ejb3.tx2.impl.CMTTxInterceptor.required(CMTTxInterceptor.java:359)

              at org.jboss.ejb3.tx2.impl.CMTTxInterceptor.invoke(CMTTxInterceptor.java:219)

              at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:35)

              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:287)

              at org.jboss.as.ejb3.component.session.SessionInvocationContextInterceptor.processInvocation(SessionInvocationContextInterceptor.java:71)

              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:287)

              at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)

              at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:146)

              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:287)

              at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)

              at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:76)

              at org.ejbca.core.ejb.ca.caadmin.CAAdminSessionLocal$$$view39.initializeAndUpgradeCAs(Unknown Source)

              at org.ejbca.ui.web.admin.configuration.StartServicesServlet.ejbcaInit(StartServicesServlet.java:169)

              at org.ejbca.ui.web.admin.configuration.StartServicesServlet.init(StartServicesServlet.java:97)

              at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1202)

              - locked <0xcb216220> (a org.apache.catalina.core.StandardWrapper)

              at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1102)

              - locked <0xcb216220> (a org.apache.catalina.core.StandardWrapper)

              at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:3631)

              at org.apache.catalina.core.StandardContext.start(StandardContext.java:3844)

              - locked <0xca7b6548> (a org.apache.catalina.core.StandardContext)

              at org.jboss.as.web.deployment.WebDeploymentService.start(WebDeploymentService.java:70)

              - locked <0xca7b05e8> (a org.jboss.as.web.deployment.WebDeploymentService)

              at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1765)

              at org.jboss.msc.service.ServiceControllerImpl$ClearTCCLTask.run(ServiceControllerImpl.java:2291)

              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

              at java.lang.Thread.run(Thread.java:619)

       

      "MSC service thread 1-1" prio=10 tid=0x0867f000 nid=0x5c8b in Object.wait() [0xc41de000..0xc41deeb0]

         java.lang.Thread.State: WAITING (on object monitor)

              at java.lang.Object.wait(Native Method)

              - waiting on <0xcb0b01e8> (a org.jboss.as.ejb3.component.stateless.StatelessSessionComponent)

              at java.lang.Object.wait(Object.java:485)

              at org.jboss.as.ee.component.BasicComponent.waitForComponentStart(BasicComponent.java:102)

              - locked <0xcb0b01e8> (a org.jboss.as.ejb3.component.stateless.StatelessSessionComponent)

              at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:76)

              at org.jboss.as.ejb3.component.stateless.StatelessSessionComponent$1.create(StatelessSessionComponent.java:70)

              at org.jboss.as.ejb3.component.stateless.StatelessSessionComponent$1.create(StatelessSessionComponent.java:67)

              at org.jboss.ejb3.pool.AbstractPool.create(AbstractPool.java:65)

              at org.jboss.ejb3.pool.strictmax.StrictMaxPool.get(StrictMaxPool.java:145)

              at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:41)

              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:287)

              at org.jboss.as.ejb3.component.session.SessionInvocationContextInterceptor$CustomSessionInvocationContext.proceed(SessionInvocationContextInterceptor.java:126)

              at org.jboss.ejb3.tx2.impl.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:244)

              at org.jboss.ejb3.tx2.impl.CMTTxInterceptor.supports(CMTTxInterceptor.java:448)

              at org.jboss.ejb3.tx2.impl.CMTTxInterceptor.invoke(CMTTxInterceptor.java:223)

              at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:35)

              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:287)

              at org.jboss.as.ejb3.component.session.SessionInvocationContextInterceptor.processInvocation(SessionInvocationContextInterceptor.java:71)

              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:287)

              at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)

              at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:146)

              at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:287)

              at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)

              at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:76)

              at org.ejbca.core.ejb.ca.store.CertificateStoreSessionLocal$$$view63.findCertificatesByType(Unknown Source)

              at org.ejbca.core.protocol.certificatestore.CertificateCache.findCertificatesByType(CertificateCache.java:355)

              at org.ejbca.core.protocol.certificatestore.CertificateCache.loadCertificates(CertificateCache.java:237)

              at org.ejbca.core.protocol.certificatestore.CertificateCache.<init>(CertificateCache.java:96)

              at org.ejbca.core.protocol.certificatestore.CertificateCacheFactory.getInstance(CertificateCacheFactory.java:40)

              at org.ejbca.ui.web.protocol.OCSPServlet.createCertificateCache(OCSPServlet.java:60)

              at org.ejbca.ui.web.protocol.OCSPServletBase.init(OCSPServletBase.java:305)

              at org.ejbca.ui.web.protocol.OCSPServlet.init(OCSPServlet.java:50)

              at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1202)

              - locked <0xc4e79f18> (a org.apache.catalina.core.StandardWrapper)

              at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1102)

              - locked <0xc4e79f18> (a org.apache.catalina.core.StandardWrapper)

              at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:3631)

              at org.apache.catalina.core.StandardContext.start(StandardContext.java:3844)

              - locked <0xcabd2130> (a org.apache.catalina.core.StandardContext)

              at org.jboss.as.web.deployment.WebDeploymentService.start(WebDeploymentService.java:70)

              - locked <0xcabb30c8> (a org.jboss.as.web.deployment.WebDeploymentService)

              at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1765)

              at org.jboss.msc.service.ServiceControllerImpl$ClearTCCLTask.run(ServiceControllerImpl.java:2291)

              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

              at java.lang.Thread.run(Thread.java:619)

       

      Any idea why the component has not got started at this point ? When does the component get started ? If you want to have a look at the code, the EJBCA source code is available @ http://sourceforge.net/projects/ejbca/files/ejbca4/ejbca_4_0_3/ejbca_4_0_3.zip/download. Any hints on why the deployment is getting stuck ? If I remove the 'load-on-startup' option for 'OCSPServlet', the deployment does go ahead.

        • 1. Re: Deployment gets stuck during EJB instance creation
          adrianzuo

          Guys

           

          I met the same issue, any update on it?

           

          Regards,

          Candice.

          • 2. Re: Deployment gets stuck during EJB instance creation
            jaikiran

            Please create a JIRA and if possible attach a sample application.

            • 3. Re: Deployment gets stuck during EJB instance creation
              jbrow

              I'm having the same issue on 7.1.0.CR1b.  Has anyone solved this????

              • 4. Re: Deployment gets stuck during EJB instance creation
                jbrow

                I just downloaded the latest jboss 7.1.0.Final-SNAPSHOT from github, and it still has the same problem.

                • 5. Re: Deployment gets stuck during EJB instance creation
                  jaikiran

                  Jim Brownfield wrote:

                   

                  I just downloaded the latest jboss 7.1.0.Final-SNAPSHOT from github, and it still has the same problem.

                  The JIRA with a reproducible application was never created. Please create one now so that it gets addressed in 7.1.0.Final.

                  • 6. Re: Deployment gets stuck during EJB instance creation
                    kpiwko

                    I wasn't able to do a reproducer JBPAPP-7388 which will trigger this issue regularly and I hit this rarely lately.

                     

                    My opinion on that is that either Arquillian or jboss-as-maven-plugin triggers a deadlock condition during deployement and service thread on server makes server stuck. If you are able to do a reproducer, that would be awesome!

                    • 7. Re: Deployment gets stuck during EJB instance creation
                      jbrow

                      Unfortunately, I can't release my software that generates the issue.  This is what I've found so far from debuggin a github snapshot. I'm about to retire for the day, but I thought I'd post this in case it creates an "aha" moment.

                       

                      JBoss is hanging in an EJB that has the @Singleton and @Startup, and @PostConstruct annotations.  In the @PostConstruct method, it is hanging when it tries to create the first instance of a stateless session bean that exists in a separate jar file.  It hangs at an Object.wait() after the creation of the SSB.  Presumably it is waiting on a notify() that never comes.

                       

                      I tried to use the maven-ear-plugin to create a jboss-app.xml with <module-order>strict</module-order> to see if that made a difference, but when I do that, I get a weird XML parsing error, "

                      Caused by: com.ctc.wstx.exc.WstxParsingException: Received non-all-whitespace CHARACTERS or CDATA event in nextTag()." I get this even though the jboss-app.xml file looks perfectly normal.

                       

                      I don't know if this helps or if I'm even on the right track.  I will continue looking tomorrow.

                      • 8. Re: Deployment gets stuck during EJB instance creation
                        swd847

                        When it is stuck can you please connect with jconsole and get a dump of the current state of all services from the MSC MBean?

                        • 9. Re: Deployment gets stuck during EJB instance creation
                          jbrow

                          Here's a dump of the MSC MBean state.

                           

                          This is a port from AS4 to AS7 and I'm refactoring a lot of the code, so I know there are lots of problems still, but JBoss just hanging makes it difficult to debug them.

                           

                          Thank you for your time, Stuart!

                          • 10. Re: Deployment gets stuck during EJB instance creation
                            swd847

                            Do you have initalize-in-order set in application.xml ?

                             

                            Also can you attach the thread dump that corresponds with the MSC service dump.

                            • 11. Re: Deployment gets stuck during EJB instance creation
                              jbrow

                              Hi Stuart,

                               

                              Thanks for the suggestion of setting "initialize-in-order"!  That set me on the path to getting it working.  I had initialize-in-order set previously, and I'd gotten errors, but I don't know exactly what changed, but I started getting errors instead of JBoss just hanging.

                               

                              My problem ended up being that some of the jndi lookup strings for the @EJB annotation were incorrect.  Correcting them allowed the ear to fully deploy without hanging JBoss.

                               

                              I do think JBoss needs some way to error out instead of just hanging in these circumstances.

                               

                              Thanks again for your help!