All deployments failing after continuous deployment (JBoss7)

[rnaodm]

I'm using ant to build and deploy applications to the server for me by copying them into standalone\deployments, and when it is done with them it undeploys them by deleting the war file. I can succesfully deploy between 25-30 applications before they begin to fail, after this the applications stay at ".isdeploying" state, and then switch to ".failed" state. Whats the problem here? If I restart the server and start over it works fine again, but always fails after 25-30 applications have been deployed and undeployed.

The failed files consisntetly have this text in them:

Did not receive a response to the deployment operation within the allowed timeout period [120 seconds]. Check the server configurationfile and the server logs to find more about the status of the deployment.

[jaikiran]

Not sure what the issue is. Can you get a thread dump (you can use jstack <processid>) from the server when it starts failing?

[rnaodm]

Thanks for the reply!

I don't have access to the server console (if thats what you mean?) I can however use the command from a new prompt...Which process (image name) do I run it for? Java.exe?

[rnaodm]

This is the jstack from one of the two java.exes running...(the one using more memory)

2011-07-20 23:01:55

Full thread dump Java HotSpot(TM) Server VM (17.0-b16 mixed mode):

"ServerDeploymentRepository-temp-threads - 2" prio=6 tid=0x34327c00 nid=0x780 wa

iting on condition [0x368ef000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x13b06d68> (a java.util.concurrent.locks.Abstra

ctQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject

.await(AbstractQueuedSynchronizer.java:1987)

        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak

e(ScheduledThreadPoolExecutor.java:583)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak

e(ScheduledThreadPoolExecutor.java:576)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.ja

va:947)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

.java:907)

        at java.lang.Thread.run(Thread.java:619)

        at org.jboss.threads.JBossThread.run(JBossThread.java:122)

"http--127.0.0.1-8080-6" daemon prio=6 tid=0x35b1f800 nid=0xfc8 in Object.wait()

[0x36a7f000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x162294e8> (a org.apache.tomcat.util.net.JIoEndpoint$Work

er)

        at java.lang.Object.wait(Object.java:485)

        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:

906)

        - locked <0x162294e8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:93

2)

        at java.lang.Thread.run(Thread.java:619)

"http--127.0.0.1-8080-5" daemon prio=6 tid=0x34b00400 nid=0xae0 waiting for moni

tor entry [0x36a2f000]

   java.lang.Thread.State: BLOCKED (on object monitor)

        at java.net.Socket.setSoTimeout(Socket.java:1017)

        - locked <0x30caa2d8> (a java.net.Socket)

        at org.apache.tomcat.util.net.JIoEndpoint.setSocketOptions(JIoEndpoint.j

ava:1138)

        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:95

1)

        at java.lang.Thread.run(Thread.java:619)

"http--127.0.0.1-8080-4" daemon prio=6 tid=0x363e9400 nid=0xc80 in Object.wait()

[0x369df000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x16229f60> (a org.apache.tomcat.util.net.JIoEndpoint$Work

er)

        at java.lang.Object.wait(Object.java:485)

        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:

906)

        - locked <0x16229f60> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:93

2)

        at java.lang.Thread.run(Thread.java:619)

"http--127.0.0.1-8080-3" daemon prio=6 tid=0x3584c400 nid=0xc08 in Object.wait()

[0x3698f000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x1623f2b8> (a org.apache.tomcat.util.net.JIoEndpoint$Work

er)

        at java.lang.Object.wait(Object.java:485)

        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:

906)

        - locked <0x1623f2b8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:93

2)

        at java.lang.Thread.run(Thread.java:619)

"http--127.0.0.1-8080-2" daemon prio=6 tid=0x35cecc00 nid=0xb50 in Object.wait()

[0x3693f000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x16243670> (a org.apache.tomcat.util.net.JIoEndpoint$Work

er)

        at java.lang.Object.wait(Object.java:485)

        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:

906)

        - locked <0x16243670> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)

        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:93

2)

        at java.lang.Thread.run(Thread.java:619)

"ServerDeploymentRepository-temp-threads - 1" prio=6 tid=0x34565400 nid=0x554 wa

iting on condition [0x3689f000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x13b06d68> (a java.util.concurrent.locks.Abstra

ctQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject

.await(AbstractQueuedSynchronizer.java:1987)

        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak

e(ScheduledThreadPoolExecutor.java:583)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak

e(ScheduledThreadPoolExecutor.java:576)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.ja

va:947)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

.java:907)

        at java.lang.Thread.run(Thread.java:619)

        at org.jboss.threads.JBossThread.run(JBossThread.java:122)

"http--127.0.0.1-8080-1" daemon prio=6 tid=0x35c10400 nid=0x2f8 waiting for moni

tor entry [0x3684f000]

   java.lang.Thread.State: BLOCKED (on object monitor)

        at sun.util.resources.TimeZoneNames.getContents(TimeZoneNames.java:58)

        at sun.util.resources.OpenListResourceBundle.loadLookup(OpenListResource

Bundle.java:109)

        - locked <0x30cae6c8> (a sun.util.resources.TimeZoneNames)

        at sun.util.resources.OpenListResourceBundle.loadLookupTablesIfNecessary

(OpenListResourceBundle.java:97)

        at sun.util.resources.OpenListResourceBundle.handleGetObject(OpenListRes

ourceBundle.java:58)

        at sun.util.resources.TimeZoneNamesBundle.handleGetObject(TimeZoneNamesB

undle.java:59)

        at java.util.ResourceBundle.getObject(ResourceBundle.java:368)

        at java.util.ResourceBundle.getObject(ResourceBundle.java:371)

        at java.util.ResourceBundle.getStringArray(ResourceBundle.java:351)

        at sun.util.TimeZoneNameUtility.retrieveDisplayNames(TimeZoneNameUtility

.java:100)

        at sun.util.TimeZoneNameUtility.retrieveDisplayNames(TimeZoneNameUtility

.java:81)

        at java.util.TimeZone.getDisplayNames(TimeZone.java:399)

        at java.util.TimeZone.getDisplayName(TimeZone.java:350)

        at java.text.SimpleDateFormat.subFormat(SimpleDateFormat.java:1110)

        at java.text.SimpleDateFormat.format(SimpleDateFormat.java:899)

        at java.text.SimpleDateFormat.format(SimpleDateFormat.java:869)

        at java.text.DateFormat.format(DateFormat.java:316)

        at org.apache.tomcat.util.http.FastHttpDateFormat.getCurrentDate(FastHtt

pDateFormat.java:115)

        - locked <0x1522d0c8> (a java.text.SimpleDateFormat)

        at org.apache.coyote.http11.Http11Processor.prepareResponse(Http11Proces

sor.java:1617)

        at org.apache.coyote.http11.Http11Processor.action(Http11Processor.java:

998)

        at org.apache.coyote.Response.action(Response.java:186)

        at org.apache.coyote.Response.sendHeaders(Response.java:386)

        at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:

334)

        at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:30

0)

        at org.apache.catalina.connector.Response.finishResponse(Response.java:4

76)

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav

a:389)

        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java

:877)

        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proce

ss(Http11Protocol.java:667)

        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:95

1)

        at java.lang.Thread.run(Thread.java:619)

"DeploymentScanner-threads - 2" prio=6 tid=0x34730400 nid=0x844 runnable [0x349f

e000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:99)

        - locked <0x13b68580> (a java.io.OutputStreamWriter)

        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:116)

        at java.io.OutputStreamWriter.write(OutputStreamWriter.java:203)

        at java.io.Writer.write(Writer.java:140)

        at org.jboss.logmanager.handlers.WriterHandler.doPublish(WriterHandler.j

ava:68)

        - locked <0x13b68578> (a java.lang.Object)

        at org.jboss.logmanager.ExtHandler.publish(ExtHandler.java:64)

        at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:283)

        at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:291)

        at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:291)

        at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:291)

        at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:291)

        at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:291)

        at org.jboss.logmanager.Logger.logRaw(Logger.java:649)

        at org.jboss.logmanager.Logger.log(Logger.java:600)

        at org.jboss.logging.JBossLogManagerLogger.doLogf(JBossLogManagerLogger.

java:54)

        at org.jboss.logging.Logger.infof(Logger.java:856)

        at org.jboss.as.server.deployment.DeploymentHandlerUtil$1.execute(Deploy

mentHandlerUtil.java:120)

        at org.jboss.as.controller.OperationContextImpl.executeStep(OperationCon

textImpl.java:351)

        at org.jboss.as.controller.OperationContextImpl.doCompleteStep(Operation

ContextImpl.java:298)

        at org.jboss.as.controller.OperationContextImpl.completeStep(OperationCo

ntextImpl.java:223)

        at org.jboss.as.server.deployment.DeploymentDeployHandler.execute(Deploy

mentDeployHandler.java:68)

        at org.jboss.as.controller.OperationContextImpl.executeStep(OperationCon

textImpl.java:351)

        at org.jboss.as.controller.OperationContextImpl.doCompleteStep(Operation

ContextImpl.java:298)

        at org.jboss.as.controller.OperationContextImpl.completeStep(OperationCo

ntextImpl.java:223)

        at org.jboss.as.server.deployment.DeploymentAddHandler.execute(Deploymen

tAddHandler.java:181)

        at org.jboss.as.controller.OperationContextImpl.executeStep(OperationCon

textImpl.java:351)

        at org.jboss.as.controller.OperationContextImpl.doCompleteStep(Operation

ContextImpl.java:298)

        at org.jboss.as.controller.OperationContextImpl.completeStep(OperationCo

ntextImpl.java:223)

        at org.jboss.as.controller.CompositeOperationHandler.execute(CompositeOp

erationHandler.java:83)

        at org.jboss.as.controller.OperationContextImpl.executeStep(OperationCon

textImpl.java:351)

        at org.jboss.as.controller.OperationContextImpl.doCompleteStep(Operation

ContextImpl.java:298)

        at org.jboss.as.controller.OperationContextImpl.completeStep(OperationCo

ntextImpl.java:223)

        at org.jboss.as.controller.CompositeOperationHandler.execute(CompositeOp

erationHandler.java:83)

        at org.jboss.as.controller.OperationContextImpl.executeStep(OperationCon

textImpl.java:351)

        at org.jboss.as.controller.OperationContextImpl.doCompleteStep(Operation

ContextImpl.java:298)

        at org.jboss.as.controller.OperationContextImpl.completeStep(OperationCo

ntextImpl.java:223)

        at org.jboss.as.controller.ModelControllerImpl$DefaultPrepareStepHandler

.execute(ModelControllerImpl.java:350)

        at org.jboss.as.controller.OperationContextImpl.executeStep(OperationCon

textImpl.java:351)

        at org.jboss.as.controller.OperationContextImpl.doCompleteStep(Operation

ContextImpl.java:298)

        at org.jboss.as.controller.OperationContextImpl.completeStep(OperationCo

ntextImpl.java:223)

        at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerIm

pl.java:119)

        at org.jboss.as.controller.ModelControllerImpl$1.execute(ModelController

Impl.java:209)

        at org.jboss.as.controller.ModelControllerImpl$1.execute(ModelController

Impl.java:199)

        at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$De

ploymentTask.call(FileSystemDeploymentService.java:1190)

        at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$De

ploymentTask.call(FileSystemDeploymentService.java:1180)

        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

        at java.util.concurrent.FutureTask.run(FutureTask.java:138)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.

access$301(ScheduledThreadPoolExecutor.java:98)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.

run(ScheduledThreadPoolExecutor.java:207)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec

utor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

.java:908)

        at java.lang.Thread.run(Thread.java:619)

        at org.jboss.threads.JBossThread.run(JBossThread.java:122)

"DeploymentScanner-threads - 1" prio=6 tid=0x34e56c00 nid=0x968 waiting on condi

tion [0x357ff000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x13bca8a0> (a java.util.concurrent.locks.Abstra

ctQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject

.await(AbstractQueuedSynchronizer.java:1987)

        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak

e(ScheduledThreadPoolExecutor.java:583)

        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak

e(ScheduledThreadPoolExecutor.java:576)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.ja

va:947)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

.java:907)

        at java.lang.Thread.run(Thread.java:619)

        at org.jboss.threads.JBossThread.run(JBossThread.java:122)

"GC Daemon" daemon prio=2 tid=0x34ef2400 nid=0xfec in Object.wait() [0x357af000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x13f0ccc0> (a sun.misc.GC$LatencyLock)

        at sun.misc.GC$Daemon.run(GC.java:100)

        - locked <0x13f0ccc0> (a sun.misc.GC$LatencyLock)

"RMI Reaper" prio=6 tid=0x347d7800 nid=0xbb8 in Object.wait() [0x3575f000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x13f0ccc8> (a java.lang.ref.ReferenceQueue$Lock)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

        - locked <0x13f0ccc8> (a java.lang.ref.ReferenceQueue$Lock)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

        at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)

        at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-1091" daemon prio=6 tid=0x34e7a800 nid=0xcc0 runnable [0x3570f00

0]

   java.lang.Thread.State: RUNNABLE

        at java.net.PlainSocketImpl.socketAccept(Native Method)

        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

        - locked <0x13ed2940> (a java.net.SocksSocketImpl)

        at java.net.ServerSocket.implAccept(ServerSocket.java:453)

        at org.jboss.as.network.ManagedServerSocketBinding.accept(ManagedServerS

ocketBinding.java:84)

        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTr

ansport.java:369)

        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:3

41)

        at java.lang.Thread.run(Thread.java:619)

"RMI TCP Accept-1090" daemon prio=6 tid=0x342f3c00 nid=0xec0 runnable [0x356bf00

0]

   java.lang.Thread.State: RUNNABLE

        at java.net.PlainSocketImpl.socketAccept(Native Method)

        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

        - locked <0x13ed2a28> (a java.net.SocksSocketImpl)

        at java.net.ServerSocket.implAccept(ServerSocket.java:453)

        at org.jboss.as.network.ManagedServerSocketBinding.accept(ManagedServerS

ocketBinding.java:84)

        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTr

ansport.java:369)

        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:3

41)

        at java.lang.Thread.run(Thread.java:619)

"Transaction Reaper Worker 0" daemon prio=6 tid=0x347d3800 nid=0x694 in Object.w

ait() [0x3566f000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x13f053e0> (a java.util.LinkedList)

        at java.lang.Object.wait(Object.java:485)

        at com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellati

ons(TransactionReaper.java:318)

        - locked <0x13f053e0> (a java.util.LinkedList)

        at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(Rea

perWorkerThread.java:66)

"Transaction Reaper" daemon prio=6 tid=0x347d3400 nid=0x95c in Object.wait() [0x

3561f000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x13f04950> (a com.arjuna.ats.arjuna.coordinator.Transacti

onReaper)

        at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThr

ead.java:91)

        - locked <0x13f04950> (a com.arjuna.ats.arjuna.coordinator.TransactionRe

aper)

"Periodic Recovery" prio=6 tid=0x34e06400 nid=0x89c runnable [0x355cf000]

   java.lang.Thread.State: RUNNABLE

        at com.arjuna.ats.internal.arjuna.common.UidHelper.unpackFrom(UidHelper.

java:47)

        at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.tran

sactionInitiatedRecovery(XARecoveryModule.java:224)

        at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.peri

odicWorkSecondPass(XARecoveryModule.java:141)

        at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkIntern

al(PeriodicRecovery.java:789)

        at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(Periodic

Recovery.java:371)

"Transaction Expired Entry Monitor" daemon prio=6 tid=0x347bfc00 nid=0x314 in Ob

ject.wait() [0x3552f000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x13deb068> (a com.arjuna.ats.internal.arjuna.recovery.Exp

iredEntryMonitor)

        at com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor.run(Expir

edEntryMonitor.java:191)

        - locked <0x13deb068> (a com.arjuna.ats.internal.arjuna.recovery.Expired

EntryMonitor)

"http--127.0.0.1-8080-Acceptor-0" daemon prio=6 tid=0x34e5ec00 nid=0x788 runnabl

e [0x3548f000]

   java.lang.Thread.State: RUNNABLE

        at java.net.PlainSocketImpl.socketAccept(Native Method)

        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

        - locked <0x13d19c18> (a java.net.SocksSocketImpl)

        at java.net.ServerSocket.implAccept(ServerSocket.java:453)

        at java.net.ServerSocket.accept(ServerSocket.java:421)

        at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(De

faultServerSocketFactory.java:61)

        at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:

343)

        at java.lang.Thread.run(Thread.java:619)

"http--127.0.0.1-8080-Poller" daemon prio=6 tid=0x34cb4800 nid=0xba0 runnable [0

x3543f000]

   java.lang.Thread.State: RUNNABLE

        at org.apache.tomcat.util.net.JIoEndpoint$Poller.run(JIoEndpoint.java:76

4)

        - locked <0x13d19d78> (a org.apache.tomcat.util.net.JIoEndpoint$Poller)

        at java.lang.Thread.run(Thread.java:619)

"XNIO NIO Write 1" prio=6 tid=0x34fae400 nid=0x8c8 runnable [0x3539f000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.j

ava:273)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelector

Impl.java:255)

        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:136)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

        - locked <0x13e14e40> (a sun.nio.ch.Util$1)

        - locked <0x13e14e30> (a java.util.Collections$UnmodifiableSet)

        - locked <0x13e0e790> (a sun.nio.ch.WindowsSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)

        at org.xnio.nio.AbstractNioChannelThread$1.run(AbstractNioChannelThread.

java:124)

        at java.lang.Thread.run(Thread.java:619)

"XNIO NIO Read 1" prio=6 tid=0x34eb6800 nid=0x870 runnable [0x3534f000]

   java.lang.Thread.State: RUNNABLE

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.j

ava:273)

        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelector

Impl.java:255)

        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:136)

        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

        - locked <0x13e0f4a8> (a sun.nio.ch.Util$1)

        - locked <0x13e0f498> (a java.util.Collections$UnmodifiableSet)

        - locked <0x13e0f258> (a sun.nio.ch.WindowsSelectorImpl)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)

        at org.xnio.nio.AbstractNioChannelThread$1.run(AbstractNioChannelThread.

java:124)

        at java.lang.Thread.run(Thread.java:619)

"NamingEventCoordinator-threads - 1" prio=6 tid=0x34690000 nid=0x210 waiting on

condition [0x3525f000]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x13b8bc90> (a java.util.concurrent.locks.Abstra

ctQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject

.await(AbstractQueuedSynchronizer.java:1987)

        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.jav

a:399)

        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.ja

va:947)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

.java:907)

        at java.lang.Thread.run(Thread.java:619)

        at org.jboss.threads.JBossThread.run(JBossThread.java:122)

"DestroyJavaVM" prio=6 tid=0x003b7800 nid=0xf64 waiting on condition [0x00000000

]

   java.lang.Thread.State: RUNNABLE

"MSC service thread 1-4" prio=6 tid=0x34605c00 nid=0x7c0 waiting for monitor ent

ry [0x349af000]

   java.lang.Thread.State: BLOCKED (on object monitor)

        at org.jboss.msc.service.ServiceControllerImpl.invokeListener(ServiceCon

trollerImpl.java:1486)

        at org.jboss.msc.service.ServiceControllerImpl.access$2500(ServiceContro

llerImpl.java:49)

        at org.jboss.msc.service.ServiceControllerImpl$ListenerTask.run(ServiceC

ontrollerImpl.java:1940)

        at org.jboss.msc.service.ServiceControllerImpl$ClearTCCLTask.run(Service

ControllerImpl.java:2291)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec

utor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

.java:908)

        at java.lang.Thread.run(Thread.java:619)

"MSC service thread 1-3" prio=6 tid=0x3432f400 nid=0xf90 waiting for monitor ent

ry [0x3495f000]

   java.lang.Thread.State: BLOCKED (on object monitor)

        at org.apache.xerces.dom.DeferredDocumentImpl.getNodeObject(DeferredDocu

mentImpl.java:1014)

        at org.apache.xerces.dom.DeferredDocumentImpl.synchronizeChildren(Deferr

edDocumentImpl.java:1752)

        at org.apache.xerces.dom.DeferredElementNSImpl.synchronizeChildren(Defer

redElementNSImpl.java:153)

        at org.apache.xerces.dom.ParentNode.hasChildNodes(ParentNode.java:193)

        at org.apache.xerces.dom.DeepNodeListImpl.nextMatchingElementAfter(DeepN

odeListImpl.java:169)

        at org.apache.xerces.dom.DeepNodeListImpl.item(DeepNodeListImpl.java:144

)

        at org.apache.xerces.dom.DeepNodeListImpl.getLength(DeepNodeListImpl.jav

a:115)

        at com.sun.faces.config.processor.FaceletTaglibConfigProcessor.process(F

aceletTaglibConfigProcessor.java:260)

        at com.sun.faces.config.ConfigManager.initialize(ConfigManager.java:341)

        at com.sun.faces.config.ConfigureListener.contextInitialized(ConfigureLi

stener.java:226)

        at org.apache.catalina.core.StandardContext.contextListenerStart(Standar

dContext.java:3368)

        at org.apache.catalina.core.StandardContext.start(StandardContext.java:3

821)

        - locked <0x305f2e60> (a org.apache.catalina.core.StandardContext)

        at org.jboss.as.web.deployment.WebDeploymentService.start(WebDeploymentS

ervice.java:70)

        - locked <0x305f7640> (a org.jboss.as.web.deployment.WebDeploymentServic

e)

        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceCont

rollerImpl.java:1765)

        at org.jboss.msc.service.ServiceControllerImpl$ClearTCCLTask.run(Service

ControllerImpl.java:2291)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec

utor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

.java:908)

        at java.lang.Thread.run(Thread.java:619)

"MSC service thread 1-2" prio=6 tid=0x3455c400 nid=0xe7c runnable [0x3490f000]

   java.lang.Thread.State: RUNNABLE

        at org.jboss.msc.service.ServiceControllerImpl.getListenerTasks(ServiceC

ontrollerImpl.java:670)

        at org.jboss.msc.service.ServiceControllerImpl.transition(ServiceControl

lerImpl.java:547)

        at org.jboss.msc.service.ServiceControllerImpl.immediateDependencyDown(S

erviceControllerImpl.java:1032)

        - locked <0x305e8eb8> (a org.jboss.msc.service.ServiceControllerImpl)

        at org.jboss.msc.service.ServiceControllerImpl$DependencyStoppedTask.run

(ServiceControllerImpl.java:1986)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec

utor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

.java:908)

        at java.lang.Thread.run(Thread.java:619)

"MSC service thread 1-1" prio=6 tid=0x34715400 nid=0xc4c waiting for monitor ent

ry [0x348bf000]

   java.lang.Thread.State: BLOCKED (on object monitor)

        at java.util.AbstractList.iterator(AbstractList.java:273)

        at org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControll

erImpl.java:695)

        at org.jboss.msc.service.ServiceControllerImpl.invokeListener(ServiceCon

trollerImpl.java:1493)

        at org.jboss.msc.service.ServiceControllerImpl.access$2500(ServiceContro

llerImpl.java:49)

        at org.jboss.msc.service.ServiceControllerImpl$ListenerTask.run(ServiceC

ontrollerImpl.java:1940)

        at org.jboss.msc.service.ServiceControllerImpl$ClearTCCLTask.run(Service

ControllerImpl.java:2291)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec

utor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

.java:908)

        at java.lang.Thread.run(Thread.java:619)

"Reference Reaper" daemon prio=6 tid=0x34296800 nid=0x618 in Object.wait() [0x34

44f000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x13b97cb8> (a java.lang.ref.ReferenceQueue$Lock)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

        - locked <0x13b97cb8> (a java.lang.ref.ReferenceQueue$Lock)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

        at org.jboss.modules.ref.References$ReaperThread.run(References.java:68)

"Low Memory Detector" daemon prio=6 tid=0x33ea6400 nid=0xa50 runnable [0x0000000

0]

   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x33ea3c00 nid=0x358 waiting on condition [

0x00000000]

   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x33e9e000 nid=0x93c waiting on condition [

0x00000000]

   java.lang.Thread.State: RUNNABLE

"Attach Listener" daemon prio=10 tid=0x33e9c400 nid=0xd50 waiting on condition [

0x00000000]

   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x33e9b000 nid=0xde4 runnable [0x00000000

]

   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=8 tid=0x33e8c000 nid=0x17c in Object.wait() [0x3401f000]

   java.lang.Thread.State: BLOCKED (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x13b26450> (a java.lang.ref.ReferenceQueue$Lock)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

        - locked <0x13b26450> (a java.lang.ref.ReferenceQueue$Lock)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x33e87800 nid=0xc60 in Object.wait() [0x

33fcf000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x13bed260> (a java.lang.ref.Reference$Lock)

        at java.lang.Object.wait(Object.java:485)

        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

        - locked <0x13bed260> (a java.lang.ref.Reference$Lock)

        at org.jboss.msc.service.ServiceControllerImpl.immediateDependencyDown(S

erviceControllerImpl.java:1032)

        - locked <0x305e8eb8> (a org.jboss.msc.service.ServiceControllerImpl)

        at org.jboss.msc.service.ServiceControllerImpl$DependencyStoppedTask.run

(ServiceControllerImpl.java:1986)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec

utor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

.java:908)

        at java.lang.Thread.run(Thread.java:619)

"MSC service thread 1-1" prio=6 tid=0x34715400 nid=0xc4c waiting for monitor ent

ry [0x348bf000]

   java.lang.Thread.State: BLOCKED (on object monitor)

        at java.util.AbstractList.iterator(AbstractList.java:273)

        at org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControll

erImpl.java:695)

        at org.jboss.msc.service.ServiceControllerImpl.invokeListener(ServiceCon

trollerImpl.java:1493)

        at org.jboss.msc.service.ServiceControllerImpl.access$2500(ServiceContro

llerImpl.java:49)

        at org.jboss.msc.service.ServiceControllerImpl$ListenerTask.run(ServiceC

ontrollerImpl.java:1940)

        at org.jboss.msc.service.ServiceControllerImpl$ClearTCCLTask.run(Service

ControllerImpl.java:2291)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec

utor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor

.java:908)

        at java.lang.Thread.run(Thread.java:619)

"Reference Reaper" daemon prio=6 tid=0x34296800 nid=0x618 in Object.wait() [0x34

44f000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x13b97cb8> (a java.lang.ref.ReferenceQueue$Lock)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

        - locked <0x13b97cb8> (a java.lang.ref.ReferenceQueue$Lock)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

        at org.jboss.modules.ref.References$ReaperThread.run(References.java:68)

"Low Memory Detector" daemon prio=6 tid=0x33ea6400 nid=0xa50 runnable [0x0000000

0]

   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x33ea3c00 nid=0x358 waiting on condition [

0x00000000]

   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x33e9e000 nid=0x93c waiting on condition [

0x00000000]

   java.lang.Thread.State: RUNNABLE

"Attach Listener" daemon prio=10 tid=0x33e9c400 nid=0xd50 waiting on condition [

0x00000000]

   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x33e9b000 nid=0xde4 runnable [0x00000000

]

   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=8 tid=0x33e8c000 nid=0x17c in Object.wait() [0x3401f000]

   java.lang.Thread.State: BLOCKED (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x13b26450> (a java.lang.ref.ReferenceQueue$Lock)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

        - locked <0x13b26450> (a java.lang.ref.ReferenceQueue$Lock)

        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x33e87800 nid=0xc60 in Object.wait() [0x

33fcf000]

   java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0x13bed260> (a java.lang.ref.Reference$Lock)

        at java.lang.Object.wait(Object.java:485)

        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

        - locked <0x13bed260> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=10 tid=0x33e84c00 nid=0x92c runnable

"GC task thread#0 (ParallelGC)" prio=6 tid=0x003bec00 nid=0x200 runnable

"GC task thread#1 (ParallelGC)" prio=6 tid=0x33de0400 nid=0xed0 runnable

"VM Periodic Task Thread" prio=10 tid=0x33eb0400 nid=0xbbc waiting on condition

JNI global references: 1789

[jaikiran]

Please get a couple of more thread dumps with a interval of around 5 seconds and attach them in separate files to this thread.

[rnaodm]

Here you go, I did a thread dump every 5 seconds between the time it was at ".isdeploying" to the time where it was at ".failed". I'm sure you don't need them all but just in case it helps!

Thanks!

[rnaodm]

I recieved a different message this time in .failed file:

{"Composite operation failed and was rolled back. Steps that failed:" => {"Operation step-2" => {"Failed services" => {"jboss.deployment.unit.\"1910.war\".INSTALL" => "org.jboss.msc.service.StartException in service jboss.deployment.unit.\"1910.war\".INSTALL: Failed to process phase INSTALL of deployment \"1910.war\""},"Services with missing/unavailable dependencies" => ["jboss.naming.context.java.comp.1910.1910.1910.Validator missing [ jboss.naming.context.java.module.1910.1910 ]","jboss.naming.context.java.comp.1910.1910.1910.ValidatorFactory missing [ jboss.naming.context.java.module.1910.1910 ]"]}}}

[rnaodm]

Here is the server.log to go along with it

[rnaodm]

I will give that a try now, thank you.

[jaikiran]

Let us know how it goes. Also if possible please attach (or share privately if you prefer) the application which reproduces this. We would like to see if there is some leak somewhere.

[rnaodm]

The applications get deployed in a random sequence each time, and each time the application causing the error is different from the last time...running the tests now. Will post back here when I know if it fails or not

[rnaodm]

The applications began failing again, except this time there is no file being created.

Initially there was a .isdeploying file for the first application that failed. However, now the applications don't even get a .isdeploying file assocatted with it. They are just copied into the deployments directory and sit there as if nothing is happening.

Server log shows:

11:05:44,046 ERROR [org.jboss.as.deployment] (DeploymentScanner-threads - 1) Did not receive a response to the deployment operation within the allowed timeout period [120 seconds]. Check the server configurationfile and the server logs to find more about the status of the deployment.

11:05:53,421 ERROR [stderr] (Periodic Recovery) Exception in thread "Periodic Recovery" java.lang.OutOfMemoryError: Java heap space

followed shortly after by:

11:05:58,671 ERROR [stderr] (Periodic Recovery)     at java.util.Arrays.copyOf(Arrays.java:2882)

11:06:21,609 ERROR [stderr] (Periodic Recovery) Exception in thread "server-timer"     at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)

11:06:56,500 ERROR [stderr] (Periodic Recovery)     at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)

11:07:08,093 ERROR [stderr] (Periodic Recovery)     at java.lang.StringBuilder.append(StringBuilder.java:119)

11:08:24,375 ERROR [stderr] (Periodic Recovery)     at com.arjuna.ats.internal.arjuna.objectstore.FileSystemStore.allObjUids(FileSystemStore.java:170)

11:08:29,218 ERROR [stderr] (Periodic Recovery)     at com.arjuna.ats.arjuna.objectstore.ObjectStore.allObjUids(ObjectStore.java:70)

11:08:29,218 ERROR [stderr] (Periodic Recovery)     at com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.periodicWorkFirstPass(AtomicActionRecoveryModule.java:91)

11:08:29,218 ERROR [stderr] (Periodic Recovery)     at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:743)

11:08:33,953 ERROR [stderr] (Periodic Recovery)     at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371)

[smarlow]

Sounds like a deployment memory leak to me.  Try running the app server with -XX:+HeapDumpOnOutOfMemoryError, so that a heap dump file with the AS memory contents will be created that can be read with http://www.eclipse.org/mat/.

Add the following to the run.conf file (after line containing "byteman").  Or after line containing %PROGNAME% in standalone.bat

{quote}

JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError"

{quote}

After the heap dump gets generated, open a jira with the steps to recreate.  And attach the .hprof file.

Some screen shots for the eclipse memory analyzer tool (mat) are here http://community.jboss.org/blogs/smarlow/2010/10/07/as-6-memoryboot-speed-improvements

[rnaodm]

What location does this heap dump file get created at?