1 2 Previous Next 15 Replies Latest reply on Sep 2, 2011 9:07 PM by atobe3

    All deployments failing after continuous deployment (JBoss7)

    rnaodm Newbie

      I'm using ant to build and deploy applications to the server for me by copying them into standalone\deployments, and when it is done with them it undeploys them by deleting the war file. I can succesfully deploy between 25-30 applications before they begin to fail, after this the applications stay at ".isdeploying" state, and then switch to ".failed" state. Whats the problem here? If I restart the server and start over it works fine again, but always fails after 25-30 applications have been deployed and undeployed.

       

      The failed files consisntetly have this text in them:

      Did not receive a response to the deployment operation within the allowed timeout period [120 seconds]. Check the server configurationfile and the server logs to find more about the status of the deployment.

        • 1. Re: All deployments failing after continued deployment (JBoss7)
          jaikiran pai Master

          Not sure what the issue is. Can you get a thread dump (you can use jstack <processid>) from the server when it starts failing?

          • 2. Re: All deployments failing after continued deployment (JBoss7)
            rnaodm Newbie

            Thanks for the reply!

             

            I don't have access to the server console (if thats what you mean?) I can however use the command from a new prompt...Which process (image name) do I run it for? Java.exe?

            • 3. Re: All deployments failing after continued deployment (JBoss7)
              rnaodm Newbie

              This is the jstack from one of the two java.exes running...(the one using more memory)

               

               

              2011-07-20 23:01:55

              Full thread dump Java HotSpot(TM) Server VM (17.0-b16 mixed mode):

               

              "ServerDeploymentRepository-temp-threads - 2" prio=6 tid=0x34327c00 nid=0x780 wa

              iting on condition [0x368ef000]

                 java.lang.Thread.State: WAITING (parking)

                      at sun.misc.Unsafe.park(Native Method)

                      - parking to wait for  <0x13b06d68> (a java.util.concurrent.locks.Abstra

              ctQueuedSynchronizer$ConditionObject)

                      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject

              .await(AbstractQueuedSynchronizer.java:1987)

                      at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak

              e(ScheduledThreadPoolExecutor.java:583)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak

              e(ScheduledThreadPoolExecutor.java:576)

                      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.ja

              va:947)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

              .java:907)

                      at java.lang.Thread.run(Thread.java:619)

                      at org.jboss.threads.JBossThread.run(JBossThread.java:122)

               

              "http--127.0.0.1-8080-6" daemon prio=6 tid=0x35b1f800 nid=0xfc8 in Object.wait()

              [0x36a7f000]

                 java.lang.Thread.State: WAITING (on object monitor)

                      at java.lang.Object.wait(Native Method)

                      - waiting on <0x162294e8> (a org.apache.tomcat.util.net.JIoEndpoint$Work

              er)

                      at java.lang.Object.wait(Object.java:485)

                      at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:

              906)

                      - locked <0x162294e8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

                      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:93

              2)

                      at java.lang.Thread.run(Thread.java:619)

               

              "http--127.0.0.1-8080-5" daemon prio=6 tid=0x34b00400 nid=0xae0 waiting for moni

              tor entry [0x36a2f000]

                 java.lang.Thread.State: BLOCKED (on object monitor)

                      at java.net.Socket.setSoTimeout(Socket.java:1017)

                      - locked <0x30caa2d8> (a java.net.Socket)

                      at org.apache.tomcat.util.net.JIoEndpoint.setSocketOptions(JIoEndpoint.j

              ava:1138)

                      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:95

              1)

                      at java.lang.Thread.run(Thread.java:619)

               

              "http--127.0.0.1-8080-4" daemon prio=6 tid=0x363e9400 nid=0xc80 in Object.wait()

              [0x369df000]

                 java.lang.Thread.State: WAITING (on object monitor)

                      at java.lang.Object.wait(Native Method)

                      - waiting on <0x16229f60> (a org.apache.tomcat.util.net.JIoEndpoint$Work

              er)

                      at java.lang.Object.wait(Object.java:485)

                      at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:

              906)

                      - locked <0x16229f60> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

                      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:93

              2)

                      at java.lang.Thread.run(Thread.java:619)

               

              "http--127.0.0.1-8080-3" daemon prio=6 tid=0x3584c400 nid=0xc08 in Object.wait()

              [0x3698f000]

                 java.lang.Thread.State: WAITING (on object monitor)

                      at java.lang.Object.wait(Native Method)

                      - waiting on <0x1623f2b8> (a org.apache.tomcat.util.net.JIoEndpoint$Work

              er)

                      at java.lang.Object.wait(Object.java:485)

                      at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:

              906)

                      - locked <0x1623f2b8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

                      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:93

              2)

                      at java.lang.Thread.run(Thread.java:619)

               

              "http--127.0.0.1-8080-2" daemon prio=6 tid=0x35cecc00 nid=0xb50 in Object.wait()

              [0x3693f000]

                 java.lang.Thread.State: WAITING (on object monitor)

                      at java.lang.Object.wait(Native Method)

                      - waiting on <0x16243670> (a org.apache.tomcat.util.net.JIoEndpoint$Work

              er)

                      at java.lang.Object.wait(Object.java:485)

                      at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:

              906)

                      - locked <0x16243670> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

                      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:93

              2)

                      at java.lang.Thread.run(Thread.java:619)

               

              "ServerDeploymentRepository-temp-threads - 1" prio=6 tid=0x34565400 nid=0x554 wa

              iting on condition [0x3689f000]

                 java.lang.Thread.State: WAITING (parking)

                      at sun.misc.Unsafe.park(Native Method)

                      - parking to wait for  <0x13b06d68> (a java.util.concurrent.locks.Abstra

              ctQueuedSynchronizer$ConditionObject)

                      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject

              .await(AbstractQueuedSynchronizer.java:1987)

                      at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak

              e(ScheduledThreadPoolExecutor.java:583)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak

              e(ScheduledThreadPoolExecutor.java:576)

                      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.ja

              va:947)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

              .java:907)

                      at java.lang.Thread.run(Thread.java:619)

                      at org.jboss.threads.JBossThread.run(JBossThread.java:122)

               

              "http--127.0.0.1-8080-1" daemon prio=6 tid=0x35c10400 nid=0x2f8 waiting for moni

              tor entry [0x3684f000]

                 java.lang.Thread.State: BLOCKED (on object monitor)

                      at sun.util.resources.TimeZoneNames.getContents(TimeZoneNames.java:58)

                      at sun.util.resources.OpenListResourceBundle.loadLookup(OpenListResource

              Bundle.java:109)

                      - locked <0x30cae6c8> (a sun.util.resources.TimeZoneNames)

                      at sun.util.resources.OpenListResourceBundle.loadLookupTablesIfNecessary

              (OpenListResourceBundle.java:97)

                      at sun.util.resources.OpenListResourceBundle.handleGetObject(OpenListRes

              ourceBundle.java:58)

                      at sun.util.resources.TimeZoneNamesBundle.handleGetObject(TimeZoneNamesB

              undle.java:59)

                      at java.util.ResourceBundle.getObject(ResourceBundle.java:368)

                      at java.util.ResourceBundle.getObject(ResourceBundle.java:371)

                      at java.util.ResourceBundle.getStringArray(ResourceBundle.java:351)

                      at sun.util.TimeZoneNameUtility.retrieveDisplayNames(TimeZoneNameUtility

              .java:100)

                      at sun.util.TimeZoneNameUtility.retrieveDisplayNames(TimeZoneNameUtility

              .java:81)

                      at java.util.TimeZone.getDisplayNames(TimeZone.java:399)

                      at java.util.TimeZone.getDisplayName(TimeZone.java:350)

                      at java.text.SimpleDateFormat.subFormat(SimpleDateFormat.java:1110)

                      at java.text.SimpleDateFormat.format(SimpleDateFormat.java:899)

                      at java.text.SimpleDateFormat.format(SimpleDateFormat.java:869)

                      at java.text.DateFormat.format(DateFormat.java:316)

                      at org.apache.tomcat.util.http.FastHttpDateFormat.getCurrentDate(FastHtt

              pDateFormat.java:115)

                      - locked <0x1522d0c8> (a java.text.SimpleDateFormat)

                      at org.apache.coyote.http11.Http11Processor.prepareResponse(Http11Proces

              sor.java:1617)

                      at org.apache.coyote.http11.Http11Processor.action(Http11Processor.java:

              998)

                      at org.apache.coyote.Response.action(Response.java:186)

                      at org.apache.coyote.Response.sendHeaders(Response.java:386)

                      at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:

              334)

                      at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:30

              0)

                      at org.apache.catalina.connector.Response.finishResponse(Response.java:4

              76)

                      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav

              a:389)

                      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java

              :877)

                      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proce

              ss(Http11Protocol.java:667)

                      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:95

              1)

                      at java.lang.Thread.run(Thread.java:619)

               

              "DeploymentScanner-threads - 2" prio=6 tid=0x34730400 nid=0x844 runnable [0x349f

              e000]

                 java.lang.Thread.State: RUNNABLE

                      at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:99)

                      - locked <0x13b68580> (a java.io.OutputStreamWriter)

                      at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:116)

                      at java.io.OutputStreamWriter.write(OutputStreamWriter.java:203)

                      at java.io.Writer.write(Writer.java:140)

                      at org.jboss.logmanager.handlers.WriterHandler.doPublish(WriterHandler.j

              ava:68)

                      - locked <0x13b68578> (a java.lang.Object)

                      at org.jboss.logmanager.ExtHandler.publish(ExtHandler.java:64)

                      at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:283)

                      at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:291)

                      at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:291)

                      at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:291)

                      at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:291)

                      at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:291)

                      at org.jboss.logmanager.Logger.logRaw(Logger.java:649)

                      at org.jboss.logmanager.Logger.log(Logger.java:600)

                      at org.jboss.logging.JBossLogManagerLogger.doLogf(JBossLogManagerLogger.

              java:54)

                      at org.jboss.logging.Logger.infof(Logger.java:856)

                      at org.jboss.as.server.deployment.DeploymentHandlerUtil$1.execute(Deploy

              mentHandlerUtil.java:120)

                      at org.jboss.as.controller.OperationContextImpl.executeStep(OperationCon

              textImpl.java:351)

                      at org.jboss.as.controller.OperationContextImpl.doCompleteStep(Operation

              ContextImpl.java:298)

                      at org.jboss.as.controller.OperationContextImpl.completeStep(OperationCo

              ntextImpl.java:223)

                      at org.jboss.as.server.deployment.DeploymentDeployHandler.execute(Deploy

              mentDeployHandler.java:68)

                      at org.jboss.as.controller.OperationContextImpl.executeStep(OperationCon

              textImpl.java:351)

                      at org.jboss.as.controller.OperationContextImpl.doCompleteStep(Operation

              ContextImpl.java:298)

                      at org.jboss.as.controller.OperationContextImpl.completeStep(OperationCo

              ntextImpl.java:223)

                      at org.jboss.as.server.deployment.DeploymentAddHandler.execute(Deploymen

              tAddHandler.java:181)

                      at org.jboss.as.controller.OperationContextImpl.executeStep(OperationCon

              textImpl.java:351)

                      at org.jboss.as.controller.OperationContextImpl.doCompleteStep(Operation

              ContextImpl.java:298)

                      at org.jboss.as.controller.OperationContextImpl.completeStep(OperationCo

              ntextImpl.java:223)

                      at org.jboss.as.controller.CompositeOperationHandler.execute(CompositeOp

              erationHandler.java:83)

                      at org.jboss.as.controller.OperationContextImpl.executeStep(OperationCon

              textImpl.java:351)

                      at org.jboss.as.controller.OperationContextImpl.doCompleteStep(Operation

              ContextImpl.java:298)

                      at org.jboss.as.controller.OperationContextImpl.completeStep(OperationCo

              ntextImpl.java:223)

                      at org.jboss.as.controller.CompositeOperationHandler.execute(CompositeOp

              erationHandler.java:83)

                      at org.jboss.as.controller.OperationContextImpl.executeStep(OperationCon

              textImpl.java:351)

                      at org.jboss.as.controller.OperationContextImpl.doCompleteStep(Operation

              ContextImpl.java:298)

                      at org.jboss.as.controller.OperationContextImpl.completeStep(OperationCo

              ntextImpl.java:223)

                      at org.jboss.as.controller.ModelControllerImpl$DefaultPrepareStepHandler

              .execute(ModelControllerImpl.java:350)

                      at org.jboss.as.controller.OperationContextImpl.executeStep(OperationCon

              textImpl.java:351)

                      at org.jboss.as.controller.OperationContextImpl.doCompleteStep(Operation

              ContextImpl.java:298)

                      at org.jboss.as.controller.OperationContextImpl.completeStep(OperationCo

              ntextImpl.java:223)

                      at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerIm

              pl.java:119)

                      at org.jboss.as.controller.ModelControllerImpl$1.execute(ModelController

              Impl.java:209)

                      at org.jboss.as.controller.ModelControllerImpl$1.execute(ModelController

              Impl.java:199)

                      at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$De

              ploymentTask.call(FileSystemDeploymentService.java:1190)

                      at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$De

              ploymentTask.call(FileSystemDeploymentService.java:1180)

                      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

                      at java.util.concurrent.FutureTask.run(FutureTask.java:138)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.

              access$301(ScheduledThreadPoolExecutor.java:98)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.

              run(ScheduledThreadPoolExecutor.java:207)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec

              utor.java:886)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

              .java:908)

                      at java.lang.Thread.run(Thread.java:619)

                      at org.jboss.threads.JBossThread.run(JBossThread.java:122)

               

              "DeploymentScanner-threads - 1" prio=6 tid=0x34e56c00 nid=0x968 waiting on condi

              tion [0x357ff000]

                 java.lang.Thread.State: WAITING (parking)

                      at sun.misc.Unsafe.park(Native Method)

                      - parking to wait for  <0x13bca8a0> (a java.util.concurrent.locks.Abstra

              ctQueuedSynchronizer$ConditionObject)

                      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject

              .await(AbstractQueuedSynchronizer.java:1987)

                      at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak

              e(ScheduledThreadPoolExecutor.java:583)

                      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak

              e(ScheduledThreadPoolExecutor.java:576)

                      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.ja

              va:947)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

              .java:907)

                      at java.lang.Thread.run(Thread.java:619)

                      at org.jboss.threads.JBossThread.run(JBossThread.java:122)

              "GC Daemon" daemon prio=2 tid=0x34ef2400 nid=0xfec in Object.wait() [0x357af000]

                 java.lang.Thread.State: TIMED_WAITING (on object monitor)

                      at java.lang.Object.wait(Native Method)

                      - waiting on <0x13f0ccc0> (a sun.misc.GC$LatencyLock)

                      at sun.misc.GC$Daemon.run(GC.java:100)

                      - locked <0x13f0ccc0> (a sun.misc.GC$LatencyLock)

               

              "RMI Reaper" prio=6 tid=0x347d7800 nid=0xbb8 in Object.wait() [0x3575f000]

                 java.lang.Thread.State: WAITING (on object monitor)

                      at java.lang.Object.wait(Native Method)

                      - waiting on <0x13f0ccc8> (a java.lang.ref.ReferenceQueue$Lock)

                      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

                      - locked <0x13f0ccc8> (a java.lang.ref.ReferenceQueue$Lock)

                      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

                      at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)

                      at java.lang.Thread.run(Thread.java:619)

               

              "RMI TCP Accept-1091" daemon prio=6 tid=0x34e7a800 nid=0xcc0 runnable [0x3570f00

              0]

                 java.lang.Thread.State: RUNNABLE

                      at java.net.PlainSocketImpl.socketAccept(Native Method)

                      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                      - locked <0x13ed2940> (a java.net.SocksSocketImpl)

                      at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                      at org.jboss.as.network.ManagedServerSocketBinding.accept(ManagedServerS

              ocketBinding.java:84)

                      at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTr

              ansport.java:369)

                      at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:3

              41)

                      at java.lang.Thread.run(Thread.java:619)

               

              "RMI TCP Accept-1090" daemon prio=6 tid=0x342f3c00 nid=0xec0 runnable [0x356bf00

              0]

                 java.lang.Thread.State: RUNNABLE

                      at java.net.PlainSocketImpl.socketAccept(Native Method)

                      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                      - locked <0x13ed2a28> (a java.net.SocksSocketImpl)

                      at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                      at org.jboss.as.network.ManagedServerSocketBinding.accept(ManagedServerS

              ocketBinding.java:84)

                      at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTr

              ansport.java:369)

                      at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:3

              41)

                      at java.lang.Thread.run(Thread.java:619)

               

              "Transaction Reaper Worker 0" daemon prio=6 tid=0x347d3800 nid=0x694 in Object.w

              ait() [0x3566f000]

                 java.lang.Thread.State: WAITING (on object monitor)

                      at java.lang.Object.wait(Native Method)

                      - waiting on <0x13f053e0> (a java.util.LinkedList)

                      at java.lang.Object.wait(Object.java:485)

                      at com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellati

              ons(TransactionReaper.java:318)

                      - locked <0x13f053e0> (a java.util.LinkedList)

                      at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(Rea

              perWorkerThread.java:66)

               

              "Transaction Reaper" daemon prio=6 tid=0x347d3400 nid=0x95c in Object.wait() [0x

              3561f000]

                 java.lang.Thread.State: TIMED_WAITING (on object monitor)

                      at java.lang.Object.wait(Native Method)

                      - waiting on <0x13f04950> (a com.arjuna.ats.arjuna.coordinator.Transacti

              onReaper)

                      at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThr

              ead.java:91)

                      - locked <0x13f04950> (a com.arjuna.ats.arjuna.coordinator.TransactionRe

              aper)

               

              "Periodic Recovery" prio=6 tid=0x34e06400 nid=0x89c runnable [0x355cf000]

                 java.lang.Thread.State: RUNNABLE

                      at com.arjuna.ats.internal.arjuna.common.UidHelper.unpackFrom(UidHelper.

              java:47)

                      at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.tran

              sactionInitiatedRecovery(XARecoveryModule.java:224)

                      at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.peri

              odicWorkSecondPass(XARecoveryModule.java:141)

                      at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkIntern

              al(PeriodicRecovery.java:789)

                      at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(Periodic

              Recovery.java:371)

               

              "Transaction Expired Entry Monitor" daemon prio=6 tid=0x347bfc00 nid=0x314 in Ob

              ject.wait() [0x3552f000]

                 java.lang.Thread.State: TIMED_WAITING (on object monitor)

                      at java.lang.Object.wait(Native Method)

                      - waiting on <0x13deb068> (a com.arjuna.ats.internal.arjuna.recovery.Exp

              iredEntryMonitor)

                      at com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor.run(Expir

              edEntryMonitor.java:191)

                      - locked <0x13deb068> (a com.arjuna.ats.internal.arjuna.recovery.Expired

              EntryMonitor)

               

              "http--127.0.0.1-8080-Acceptor-0" daemon prio=6 tid=0x34e5ec00 nid=0x788 runnabl

              e [0x3548f000]

                 java.lang.Thread.State: RUNNABLE

                      at java.net.PlainSocketImpl.socketAccept(Native Method)

                      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                      - locked <0x13d19c18> (a java.net.SocksSocketImpl)

                      at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                      at java.net.ServerSocket.accept(ServerSocket.java:421)

                      at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(De

              faultServerSocketFactory.java:61)

                      at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:

              343)

                      at java.lang.Thread.run(Thread.java:619)

               

              "http--127.0.0.1-8080-Poller" daemon prio=6 tid=0x34cb4800 nid=0xba0 runnable [0

              x3543f000]

                 java.lang.Thread.State: RUNNABLE

                      at org.apache.tomcat.util.net.JIoEndpoint$Poller.run(JIoEndpoint.java:76

              4)

                      - locked <0x13d19d78> (a org.apache.tomcat.util.net.JIoEndpoint$Poller)

                      at java.lang.Thread.run(Thread.java:619)

               

              "XNIO NIO Write 1" prio=6 tid=0x34fae400 nid=0x8c8 runnable [0x3539f000]

                 java.lang.Thread.State: RUNNABLE

                      at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

                      at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.j

              ava:273)

                      at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelector

              Impl.java:255)

                      at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:136)

                      at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

                      - locked <0x13e14e40> (a sun.nio.ch.Util$1)

                      - locked <0x13e14e30> (a java.util.Collections$UnmodifiableSet)

                      - locked <0x13e0e790> (a sun.nio.ch.WindowsSelectorImpl)

                      at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

                      at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)

                      at org.xnio.nio.AbstractNioChannelThread$1.run(AbstractNioChannelThread.

              java:124)

                      at java.lang.Thread.run(Thread.java:619)

               

              "XNIO NIO Read 1" prio=6 tid=0x34eb6800 nid=0x870 runnable [0x3534f000]

                 java.lang.Thread.State: RUNNABLE

                      at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

                      at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.j

              ava:273)

                      at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelector

              Impl.java:255)

                      at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:136)

                      at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

                      - locked <0x13e0f4a8> (a sun.nio.ch.Util$1)

                      - locked <0x13e0f498> (a java.util.Collections$UnmodifiableSet)

                      - locked <0x13e0f258> (a sun.nio.ch.WindowsSelectorImpl)

                      at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

                      at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)

                      at org.xnio.nio.AbstractNioChannelThread$1.run(AbstractNioChannelThread.

              java:124)

                      at java.lang.Thread.run(Thread.java:619)

               

              "NamingEventCoordinator-threads - 1" prio=6 tid=0x34690000 nid=0x210 waiting on

              condition [0x3525f000]

                 java.lang.Thread.State: WAITING (parking)

                      at sun.misc.Unsafe.park(Native Method)

                      - parking to wait for  <0x13b8bc90> (a java.util.concurrent.locks.Abstra

              ctQueuedSynchronizer$ConditionObject)

                      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject

              .await(AbstractQueuedSynchronizer.java:1987)

                      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.jav

              a:399)

                      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.ja

              va:947)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

              .java:907)

                      at java.lang.Thread.run(Thread.java:619)

                      at org.jboss.threads.JBossThread.run(JBossThread.java:122)

               

              "DestroyJavaVM" prio=6 tid=0x003b7800 nid=0xf64 waiting on condition [0x00000000

              ]

                 java.lang.Thread.State: RUNNABLE

               

              "MSC service thread 1-4" prio=6 tid=0x34605c00 nid=0x7c0 waiting for monitor ent

              ry [0x349af000]

                 java.lang.Thread.State: BLOCKED (on object monitor)

                      at org.jboss.msc.service.ServiceControllerImpl.invokeListener(ServiceCon

              trollerImpl.java:1486)

                      at org.jboss.msc.service.ServiceControllerImpl.access$2500(ServiceContro

              llerImpl.java:49)

                      at org.jboss.msc.service.ServiceControllerImpl$ListenerTask.run(ServiceC

              ontrollerImpl.java:1940)

                      at org.jboss.msc.service.ServiceControllerImpl$ClearTCCLTask.run(Service

              ControllerImpl.java:2291)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec

              utor.java:886)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

              .java:908)

                      at java.lang.Thread.run(Thread.java:619)

               

              "MSC service thread 1-3" prio=6 tid=0x3432f400 nid=0xf90 waiting for monitor ent

              ry [0x3495f000]

                 java.lang.Thread.State: BLOCKED (on object monitor)

                      at org.apache.xerces.dom.DeferredDocumentImpl.getNodeObject(DeferredDocu

              mentImpl.java:1014)

                      at org.apache.xerces.dom.DeferredDocumentImpl.synchronizeChildren(Deferr

              edDocumentImpl.java:1752)

                      at org.apache.xerces.dom.DeferredElementNSImpl.synchronizeChildren(Defer

              redElementNSImpl.java:153)

                      at org.apache.xerces.dom.ParentNode.hasChildNodes(ParentNode.java:193)

                      at org.apache.xerces.dom.DeepNodeListImpl.nextMatchingElementAfter(DeepN

              odeListImpl.java:169)

                      at org.apache.xerces.dom.DeepNodeListImpl.item(DeepNodeListImpl.java:144

              )

                      at org.apache.xerces.dom.DeepNodeListImpl.getLength(DeepNodeListImpl.jav

              a:115)

                      at com.sun.faces.config.processor.FaceletTaglibConfigProcessor.process(F

              aceletTaglibConfigProcessor.java:260)

                      at com.sun.faces.config.ConfigManager.initialize(ConfigManager.java:341)

                      at com.sun.faces.config.ConfigureListener.contextInitialized(ConfigureLi

              stener.java:226)

                      at org.apache.catalina.core.StandardContext.contextListenerStart(Standar

              dContext.java:3368)

                      at org.apache.catalina.core.StandardContext.start(StandardContext.java:3

              821)

                      - locked <0x305f2e60> (a org.apache.catalina.core.StandardContext)

                      at org.jboss.as.web.deployment.WebDeploymentService.start(WebDeploymentS

              ervice.java:70)

                      - locked <0x305f7640> (a org.jboss.as.web.deployment.WebDeploymentServic

              e)

                      at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceCont

              rollerImpl.java:1765)

                      at org.jboss.msc.service.ServiceControllerImpl$ClearTCCLTask.run(Service

              ControllerImpl.java:2291)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec

              utor.java:886)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

              .java:908)

                      at java.lang.Thread.run(Thread.java:619)

               

              "MSC service thread 1-2" prio=6 tid=0x3455c400 nid=0xe7c runnable [0x3490f000]

                 java.lang.Thread.State: RUNNABLE

                      at org.jboss.msc.service.ServiceControllerImpl.getListenerTasks(ServiceC

              ontrollerImpl.java:670)

                      at org.jboss.msc.service.ServiceControllerImpl.transition(ServiceControl

              lerImpl.java:547)

                      at org.jboss.msc.service.ServiceControllerImpl.immediateDependencyDown(S

              erviceControllerImpl.java:1032)

                      - locked <0x305e8eb8> (a org.jboss.msc.service.ServiceControllerImpl)

                      at org.jboss.msc.service.ServiceControllerImpl$DependencyStoppedTask.run

              (ServiceControllerImpl.java:1986)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec

              utor.java:886)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

              .java:908)

                      at java.lang.Thread.run(Thread.java:619)

               

              "MSC service thread 1-1" prio=6 tid=0x34715400 nid=0xc4c waiting for monitor ent

              ry [0x348bf000]

                 java.lang.Thread.State: BLOCKED (on object monitor)

                      at java.util.AbstractList.iterator(AbstractList.java:273)

                      at org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControll

              erImpl.java:695)

                      at org.jboss.msc.service.ServiceControllerImpl.invokeListener(ServiceCon

              trollerImpl.java:1493)

                      at org.jboss.msc.service.ServiceControllerImpl.access$2500(ServiceContro

              llerImpl.java:49)

                      at org.jboss.msc.service.ServiceControllerImpl$ListenerTask.run(ServiceC

              ontrollerImpl.java:1940)

                      at org.jboss.msc.service.ServiceControllerImpl$ClearTCCLTask.run(Service

              ControllerImpl.java:2291)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec

              utor.java:886)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

              .java:908)

                      at java.lang.Thread.run(Thread.java:619)

               

              "Reference Reaper" daemon prio=6 tid=0x34296800 nid=0x618 in Object.wait() [0x34

              44f000]

                 java.lang.Thread.State: WAITING (on object monitor)

                      at java.lang.Object.wait(Native Method)

                      - waiting on <0x13b97cb8> (a java.lang.ref.ReferenceQueue$Lock)

                      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

                      - locked <0x13b97cb8> (a java.lang.ref.ReferenceQueue$Lock)

                      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

                      at org.jboss.modules.ref.References$ReaperThread.run(References.java:68)

               

              "Low Memory Detector" daemon prio=6 tid=0x33ea6400 nid=0xa50 runnable [0x0000000

              0]

                 java.lang.Thread.State: RUNNABLE

               

              "CompilerThread1" daemon prio=10 tid=0x33ea3c00 nid=0x358 waiting on condition [

              0x00000000]

                 java.lang.Thread.State: RUNNABLE

               

              "CompilerThread0" daemon prio=10 tid=0x33e9e000 nid=0x93c waiting on condition [

              0x00000000]

                 java.lang.Thread.State: RUNNABLE

               

              "Attach Listener" daemon prio=10 tid=0x33e9c400 nid=0xd50 waiting on condition [

              0x00000000]

                 java.lang.Thread.State: RUNNABLE

               

              "Signal Dispatcher" daemon prio=10 tid=0x33e9b000 nid=0xde4 runnable [0x00000000

              ]

                 java.lang.Thread.State: RUNNABLE

               

              "Finalizer" daemon prio=8 tid=0x33e8c000 nid=0x17c in Object.wait() [0x3401f000]

                 java.lang.Thread.State: BLOCKED (on object monitor)

                      at java.lang.Object.wait(Native Method)

                      - waiting on <0x13b26450> (a java.lang.ref.ReferenceQueue$Lock)

                      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

                      - locked <0x13b26450> (a java.lang.ref.ReferenceQueue$Lock)

                      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

                      at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

               

              "Reference Handler" daemon prio=10 tid=0x33e87800 nid=0xc60 in Object.wait() [0x

              33fcf000]

                 java.lang.Thread.State: WAITING (on object monitor)

                      at java.lang.Object.wait(Native Method)

                      - waiting on <0x13bed260> (a java.lang.ref.Reference$Lock)

                      at java.lang.Object.wait(Object.java:485)

                      at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

                      - locked <0x13bed260> (a java.lang.ref.Reference$Lock)

               

                      at org.jboss.msc.service.ServiceControllerImpl.immediateDependencyDown(S

              erviceControllerImpl.java:1032)

                      - locked <0x305e8eb8> (a org.jboss.msc.service.ServiceControllerImpl)

                      at org.jboss.msc.service.ServiceControllerImpl$DependencyStoppedTask.run

              (ServiceControllerImpl.java:1986)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec

              utor.java:886)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

              .java:908)

                      at java.lang.Thread.run(Thread.java:619)

               

              "MSC service thread 1-1" prio=6 tid=0x34715400 nid=0xc4c waiting for monitor ent

              ry [0x348bf000]

                 java.lang.Thread.State: BLOCKED (on object monitor)

                      at java.util.AbstractList.iterator(AbstractList.java:273)

                      at org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControll

              erImpl.java:695)

                      at org.jboss.msc.service.ServiceControllerImpl.invokeListener(ServiceCon

              trollerImpl.java:1493)

                      at org.jboss.msc.service.ServiceControllerImpl.access$2500(ServiceContro

              llerImpl.java:49)

                      at org.jboss.msc.service.ServiceControllerImpl$ListenerTask.run(ServiceC

              ontrollerImpl.java:1940)

                      at org.jboss.msc.service.ServiceControllerImpl$ClearTCCLTask.run(Service

              ControllerImpl.java:2291)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec

              utor.java:886)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

              .java:908)

                      at java.lang.Thread.run(Thread.java:619)

               

              "Reference Reaper" daemon prio=6 tid=0x34296800 nid=0x618 in Object.wait() [0x34

              44f000]

                 java.lang.Thread.State: WAITING (on object monitor)

                      at java.lang.Object.wait(Native Method)

                      - waiting on <0x13b97cb8> (a java.lang.ref.ReferenceQueue$Lock)

                      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

                      - locked <0x13b97cb8> (a java.lang.ref.ReferenceQueue$Lock)

                      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

                      at org.jboss.modules.ref.References$ReaperThread.run(References.java:68)

               

              "Low Memory Detector" daemon prio=6 tid=0x33ea6400 nid=0xa50 runnable [0x0000000

              0]

                 java.lang.Thread.State: RUNNABLE

               

              "CompilerThread1" daemon prio=10 tid=0x33ea3c00 nid=0x358 waiting on condition [

              0x00000000]

                 java.lang.Thread.State: RUNNABLE

               

              "CompilerThread0" daemon prio=10 tid=0x33e9e000 nid=0x93c waiting on condition [

              0x00000000]

                 java.lang.Thread.State: RUNNABLE

               

              "Attach Listener" daemon prio=10 tid=0x33e9c400 nid=0xd50 waiting on condition [

              0x00000000]

                 java.lang.Thread.State: RUNNABLE

               

              "Signal Dispatcher" daemon prio=10 tid=0x33e9b000 nid=0xde4 runnable [0x00000000

              ]

                 java.lang.Thread.State: RUNNABLE

               

              "Finalizer" daemon prio=8 tid=0x33e8c000 nid=0x17c in Object.wait() [0x3401f000]

                 java.lang.Thread.State: BLOCKED (on object monitor)

                      at java.lang.Object.wait(Native Method)

                      - waiting on <0x13b26450> (a java.lang.ref.ReferenceQueue$Lock)

                      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

                      - locked <0x13b26450> (a java.lang.ref.ReferenceQueue$Lock)

                      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

                      at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

               

              "Reference Handler" daemon prio=10 tid=0x33e87800 nid=0xc60 in Object.wait() [0x

              33fcf000]

                 java.lang.Thread.State: WAITING (on object monitor)

                      at java.lang.Object.wait(Native Method)

                      - waiting on <0x13bed260> (a java.lang.ref.Reference$Lock)

                      at java.lang.Object.wait(Object.java:485)

                      at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

                      - locked <0x13bed260> (a java.lang.ref.Reference$Lock)

               

              "VM Thread" prio=10 tid=0x33e84c00 nid=0x92c runnable

               

              "GC task thread#0 (ParallelGC)" prio=6 tid=0x003bec00 nid=0x200 runnable

               

              "GC task thread#1 (ParallelGC)" prio=6 tid=0x33de0400 nid=0xed0 runnable

               

              "VM Periodic Task Thread" prio=10 tid=0x33eb0400 nid=0xbbc waiting on condition

               

              JNI global references: 1789

              • 4. Re: All deployments failing after continued deployment (JBoss7)
                jaikiran pai Master

                Please get a couple of more thread dumps with a interval of around 5 seconds and attach them in separate files to this thread.

                • 5. Re: All deployments failing after continued deployment (JBoss7)
                  rnaodm Newbie

                  Here you go, I did a thread dump every 5 seconds between the time it was at ".isdeploying" to the time where it was at ".failed". I'm sure you don't need them all but just in case it helps!

                   

                  Thanks!

                  • 6. Re: All deployments failing after continued deployment (JBoss7)
                    rnaodm Newbie

                    I recieved a different message this time in .failed file:

                     

                    {"Composite operation failed and was rolled back. Steps that failed:" => {"Operation step-2" => {"Failed services" => {"jboss.deployment.unit.\"1910.war\".INSTALL" => "org.jboss.msc.service.StartException in service jboss.deployment.unit.\"1910.war\".INSTALL: Failed to process phase INSTALL of deployment \"1910.war\""},"Services with missing/unavailable dependencies" => ["jboss.naming.context.java.comp.1910.1910.1910.Validator missing [ jboss.naming.context.java.module.1910.1910 ]","jboss.naming.context.java.comp.1910.1910.1910.ValidatorFactory missing [ jboss.naming.context.java.module.1910.1910 ]"]}}}

                    • 7. Re: All deployments failing after continued deployment (JBoss7)
                      rnaodm Newbie

                      Here is the server.log to go along with it

                      • 10. Re: All deployments failing after continued deployment (JBoss7)
                        jaikiran pai Master

                        Let us know how it goes. Also if possible please attach (or share privately if you prefer) the application which reproduces this. We would like to see if there is some leak somewhere.

                        • 11. Re: All deployments failing after continued deployment (JBoss7)
                          rnaodm Newbie

                          The applications get deployed in a random sequence each time, and each time the application causing the error is different from the last time...running the tests now. Will post back here when I know if it fails or not

                          • 12. Re: All deployments failing after continued deployment (JBoss7)
                            rnaodm Newbie

                            The applications began failing again, except this time there is no file being created.

                             

                            Initially there was a .isdeploying file for the first application that failed. However, now the applications don't even get a .isdeploying file assocatted with it. They are just copied into the deployments directory and sit there as if nothing is happening.

                             

                            Server log shows:

                             

                            11:05:44,046 ERROR [org.jboss.as.deployment] (DeploymentScanner-threads - 1) Did not receive a response to the deployment operation within the allowed timeout period [120 seconds]. Check the server configurationfile and the server logs to find more about the status of the deployment.

                            11:05:53,421 ERROR [stderr] (Periodic Recovery) Exception in thread "Periodic Recovery" java.lang.OutOfMemoryError: Java heap space

                             

                            followed shortly after by:

                             

                            11:05:58,671 ERROR [stderr] (Periodic Recovery)     at java.util.Arrays.copyOf(Arrays.java:2882)

                            11:06:21,609 ERROR [stderr] (Periodic Recovery) Exception in thread "server-timer"     at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)

                            11:06:56,500 ERROR [stderr] (Periodic Recovery)     at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)

                            11:07:08,093 ERROR [stderr] (Periodic Recovery)     at java.lang.StringBuilder.append(StringBuilder.java:119)

                            11:08:24,375 ERROR [stderr] (Periodic Recovery)     at com.arjuna.ats.internal.arjuna.objectstore.FileSystemStore.allObjUids(FileSystemStore.java:170)

                            11:08:29,218 ERROR [stderr] (Periodic Recovery)     at com.arjuna.ats.arjuna.objectstore.ObjectStore.allObjUids(ObjectStore.java:70)

                            11:08:29,218 ERROR [stderr] (Periodic Recovery)     at com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.periodicWorkFirstPass(AtomicActionRecoveryModule.java:91)

                            11:08:29,218 ERROR [stderr] (Periodic Recovery)     at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:743)

                            11:08:33,953 ERROR [stderr] (Periodic Recovery)     at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371)

                            • 13. Re: All deployments failing after continued deployment (JBoss7)
                              Scott Marlow Master

                              Sounds like a deployment memory leak to me.  Try running the app server with -XX:+HeapDumpOnOutOfMemoryError, so that a heap dump file with the AS memory contents will be created that can be read with http://www.eclipse.org/mat/.

                               

                              Add the following to the run.conf file (after line containing "byteman").  Or after line containing %PROGNAME% in standalone.bat

                              {quote}

                              JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError"

                              {quote}

                               

                              After the heap dump gets generated, open a jira with the steps to recreate.  And attach the .hprof file.

                               

                              Some screen shots for the eclipse memory analyzer tool (mat) are here http://community.jboss.org/blogs/smarlow/2010/10/07/as-6-memoryboot-speed-improvements

                              • 14. Re: All deployments failing after continued deployment (JBoss7)
                                rnaodm Newbie

                                What location does this heap dump file get created at?

                                1 2 Previous Next