-
1. Re: All deployments failing after continued deployment (JBoss7)
jaikiran Jul 21, 2011 12:49 AM (in response to rnaodm)Not sure what the issue is. Can you get a thread dump (you can use jstack <processid>) from the server when it starts failing?
-
2. Re: All deployments failing after continued deployment (JBoss7)
rnaodm Jul 21, 2011 1:08 AM (in response to jaikiran)Thanks for the reply!
I don't have access to the server console (if thats what you mean?) I can however use the command from a new prompt...Which process (image name) do I run it for? Java.exe?
-
3. Re: All deployments failing after continued deployment (JBoss7)
rnaodm Jul 21, 2011 1:09 AM (in response to rnaodm)This is the jstack from one of the two java.exes running...(the one using more memory)
2011-07-20 23:01:55
Full thread dump Java HotSpot(TM) Server VM (17.0-b16 mixed mode):
"ServerDeploymentRepository-temp-threads - 2" prio=6 tid=0x34327c00 nid=0x780 wa
iting on condition [0x368ef000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x13b06d68> (a java.util.concurrent.locks.Abstra
ctQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak
e(ScheduledThreadPoolExecutor.java:583)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak
e(ScheduledThreadPoolExecutor.java:576)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.ja
va:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:907)
at java.lang.Thread.run(Thread.java:619)
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
"http--127.0.0.1-8080-6" daemon prio=6 tid=0x35b1f800 nid=0xfc8 in Object.wait()
[0x36a7f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x162294e8> (a org.apache.tomcat.util.net.JIoEndpoint$Work
er)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:
906)
- locked <0x162294e8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:93
2)
at java.lang.Thread.run(Thread.java:619)
"http--127.0.0.1-8080-5" daemon prio=6 tid=0x34b00400 nid=0xae0 waiting for moni
tor entry [0x36a2f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.net.Socket.setSoTimeout(Socket.java:1017)
- locked <0x30caa2d8> (a java.net.Socket)
at org.apache.tomcat.util.net.JIoEndpoint.setSocketOptions(JIoEndpoint.j
ava:1138)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:95
1)
at java.lang.Thread.run(Thread.java:619)
"http--127.0.0.1-8080-4" daemon prio=6 tid=0x363e9400 nid=0xc80 in Object.wait()
[0x369df000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x16229f60> (a org.apache.tomcat.util.net.JIoEndpoint$Work
er)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:
906)
- locked <0x16229f60> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:93
2)
at java.lang.Thread.run(Thread.java:619)
"http--127.0.0.1-8080-3" daemon prio=6 tid=0x3584c400 nid=0xc08 in Object.wait()
[0x3698f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x1623f2b8> (a org.apache.tomcat.util.net.JIoEndpoint$Work
er)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:
906)
- locked <0x1623f2b8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:93
2)
at java.lang.Thread.run(Thread.java:619)
"http--127.0.0.1-8080-2" daemon prio=6 tid=0x35cecc00 nid=0xb50 in Object.wait()
[0x3693f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x16243670> (a org.apache.tomcat.util.net.JIoEndpoint$Work
er)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:
906)
- locked <0x16243670> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:93
2)
at java.lang.Thread.run(Thread.java:619)
"ServerDeploymentRepository-temp-threads - 1" prio=6 tid=0x34565400 nid=0x554 wa
iting on condition [0x3689f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x13b06d68> (a java.util.concurrent.locks.Abstra
ctQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak
e(ScheduledThreadPoolExecutor.java:583)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak
e(ScheduledThreadPoolExecutor.java:576)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.ja
va:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:907)
at java.lang.Thread.run(Thread.java:619)
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
"http--127.0.0.1-8080-1" daemon prio=6 tid=0x35c10400 nid=0x2f8 waiting for moni
tor entry [0x3684f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at sun.util.resources.TimeZoneNames.getContents(TimeZoneNames.java:58)
at sun.util.resources.OpenListResourceBundle.loadLookup(OpenListResource
Bundle.java:109)
- locked <0x30cae6c8> (a sun.util.resources.TimeZoneNames)
at sun.util.resources.OpenListResourceBundle.loadLookupTablesIfNecessary
(OpenListResourceBundle.java:97)
at sun.util.resources.OpenListResourceBundle.handleGetObject(OpenListRes
ourceBundle.java:58)
at sun.util.resources.TimeZoneNamesBundle.handleGetObject(TimeZoneNamesB
undle.java:59)
at java.util.ResourceBundle.getObject(ResourceBundle.java:368)
at java.util.ResourceBundle.getObject(ResourceBundle.java:371)
at java.util.ResourceBundle.getStringArray(ResourceBundle.java:351)
at sun.util.TimeZoneNameUtility.retrieveDisplayNames(TimeZoneNameUtility
.java:100)
at sun.util.TimeZoneNameUtility.retrieveDisplayNames(TimeZoneNameUtility
.java:81)
at java.util.TimeZone.getDisplayNames(TimeZone.java:399)
at java.util.TimeZone.getDisplayName(TimeZone.java:350)
at java.text.SimpleDateFormat.subFormat(SimpleDateFormat.java:1110)
at java.text.SimpleDateFormat.format(SimpleDateFormat.java:899)
at java.text.SimpleDateFormat.format(SimpleDateFormat.java:869)
at java.text.DateFormat.format(DateFormat.java:316)
at org.apache.tomcat.util.http.FastHttpDateFormat.getCurrentDate(FastHtt
pDateFormat.java:115)
- locked <0x1522d0c8> (a java.text.SimpleDateFormat)
at org.apache.coyote.http11.Http11Processor.prepareResponse(Http11Proces
sor.java:1617)
at org.apache.coyote.http11.Http11Processor.action(Http11Processor.java:
998)
at org.apache.coyote.Response.action(Response.java:186)
at org.apache.coyote.Response.sendHeaders(Response.java:386)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:
334)
at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:30
0)
at org.apache.catalina.connector.Response.finishResponse(Response.java:4
76)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav
a:389)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java
:877)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proce
ss(Http11Protocol.java:667)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:95
1)
at java.lang.Thread.run(Thread.java:619)
"DeploymentScanner-threads - 2" prio=6 tid=0x34730400 nid=0x844 runnable [0x349f
e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:99)
- locked <0x13b68580> (a java.io.OutputStreamWriter)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:116)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:203)
at java.io.Writer.write(Writer.java:140)
at org.jboss.logmanager.handlers.WriterHandler.doPublish(WriterHandler.j
ava:68)
- locked <0x13b68578> (a java.lang.Object)
at org.jboss.logmanager.ExtHandler.publish(ExtHandler.java:64)
at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:283)
at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:291)
at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:291)
at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:291)
at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:291)
at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:291)
at org.jboss.logmanager.Logger.logRaw(Logger.java:649)
at org.jboss.logmanager.Logger.log(Logger.java:600)
at org.jboss.logging.JBossLogManagerLogger.doLogf(JBossLogManagerLogger.
java:54)
at org.jboss.logging.Logger.infof(Logger.java:856)
at org.jboss.as.server.deployment.DeploymentHandlerUtil$1.execute(Deploy
mentHandlerUtil.java:120)
at org.jboss.as.controller.OperationContextImpl.executeStep(OperationCon
textImpl.java:351)
at org.jboss.as.controller.OperationContextImpl.doCompleteStep(Operation
ContextImpl.java:298)
at org.jboss.as.controller.OperationContextImpl.completeStep(OperationCo
ntextImpl.java:223)
at org.jboss.as.server.deployment.DeploymentDeployHandler.execute(Deploy
mentDeployHandler.java:68)
at org.jboss.as.controller.OperationContextImpl.executeStep(OperationCon
textImpl.java:351)
at org.jboss.as.controller.OperationContextImpl.doCompleteStep(Operation
ContextImpl.java:298)
at org.jboss.as.controller.OperationContextImpl.completeStep(OperationCo
ntextImpl.java:223)
at org.jboss.as.server.deployment.DeploymentAddHandler.execute(Deploymen
tAddHandler.java:181)
at org.jboss.as.controller.OperationContextImpl.executeStep(OperationCon
textImpl.java:351)
at org.jboss.as.controller.OperationContextImpl.doCompleteStep(Operation
ContextImpl.java:298)
at org.jboss.as.controller.OperationContextImpl.completeStep(OperationCo
ntextImpl.java:223)
at org.jboss.as.controller.CompositeOperationHandler.execute(CompositeOp
erationHandler.java:83)
at org.jboss.as.controller.OperationContextImpl.executeStep(OperationCon
textImpl.java:351)
at org.jboss.as.controller.OperationContextImpl.doCompleteStep(Operation
ContextImpl.java:298)
at org.jboss.as.controller.OperationContextImpl.completeStep(OperationCo
ntextImpl.java:223)
at org.jboss.as.controller.CompositeOperationHandler.execute(CompositeOp
erationHandler.java:83)
at org.jboss.as.controller.OperationContextImpl.executeStep(OperationCon
textImpl.java:351)
at org.jboss.as.controller.OperationContextImpl.doCompleteStep(Operation
ContextImpl.java:298)
at org.jboss.as.controller.OperationContextImpl.completeStep(OperationCo
ntextImpl.java:223)
at org.jboss.as.controller.ModelControllerImpl$DefaultPrepareStepHandler
.execute(ModelControllerImpl.java:350)
at org.jboss.as.controller.OperationContextImpl.executeStep(OperationCon
textImpl.java:351)
at org.jboss.as.controller.OperationContextImpl.doCompleteStep(Operation
ContextImpl.java:298)
at org.jboss.as.controller.OperationContextImpl.completeStep(OperationCo
ntextImpl.java:223)
at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerIm
pl.java:119)
at org.jboss.as.controller.ModelControllerImpl$1.execute(ModelController
Impl.java:209)
at org.jboss.as.controller.ModelControllerImpl$1.execute(ModelController
Impl.java:199)
at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$De
ploymentTask.call(FileSystemDeploymentService.java:1190)
at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$De
ploymentTask.call(FileSystemDeploymentService.java:1180)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
run(ScheduledThreadPoolExecutor.java:207)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
utor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:908)
at java.lang.Thread.run(Thread.java:619)
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
"DeploymentScanner-threads - 1" prio=6 tid=0x34e56c00 nid=0x968 waiting on condi
tion [0x357ff000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x13bca8a0> (a java.util.concurrent.locks.Abstra
ctQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak
e(ScheduledThreadPoolExecutor.java:583)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.tak
e(ScheduledThreadPoolExecutor.java:576)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.ja
va:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:907)
at java.lang.Thread.run(Thread.java:619)
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
"GC Daemon" daemon prio=2 tid=0x34ef2400 nid=0xfec in Object.wait() [0x357af000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x13f0ccc0> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:100)
- locked <0x13f0ccc0> (a sun.misc.GC$LatencyLock)
"RMI Reaper" prio=6 tid=0x347d7800 nid=0xbb8 in Object.wait() [0x3575f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x13f0ccc8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x13f0ccc8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
at java.lang.Thread.run(Thread.java:619)
"RMI TCP Accept-1091" daemon prio=6 tid=0x34e7a800 nid=0xcc0 runnable [0x3570f00
0]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked <0x13ed2940> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at org.jboss.as.network.ManagedServerSocketBinding.accept(ManagedServerS
ocketBinding.java:84)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTr
ansport.java:369)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:3
41)
at java.lang.Thread.run(Thread.java:619)
"RMI TCP Accept-1090" daemon prio=6 tid=0x342f3c00 nid=0xec0 runnable [0x356bf00
0]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked <0x13ed2a28> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at org.jboss.as.network.ManagedServerSocketBinding.accept(ManagedServerS
ocketBinding.java:84)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTr
ansport.java:369)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:3
41)
at java.lang.Thread.run(Thread.java:619)
"Transaction Reaper Worker 0" daemon prio=6 tid=0x347d3800 nid=0x694 in Object.w
ait() [0x3566f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x13f053e0> (a java.util.LinkedList)
at java.lang.Object.wait(Object.java:485)
at com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellati
ons(TransactionReaper.java:318)
- locked <0x13f053e0> (a java.util.LinkedList)
at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(Rea
perWorkerThread.java:66)
"Transaction Reaper" daemon prio=6 tid=0x347d3400 nid=0x95c in Object.wait() [0x
3561f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x13f04950> (a com.arjuna.ats.arjuna.coordinator.Transacti
onReaper)
at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThr
ead.java:91)
- locked <0x13f04950> (a com.arjuna.ats.arjuna.coordinator.TransactionRe
aper)
"Periodic Recovery" prio=6 tid=0x34e06400 nid=0x89c runnable [0x355cf000]
java.lang.Thread.State: RUNNABLE
at com.arjuna.ats.internal.arjuna.common.UidHelper.unpackFrom(UidHelper.
java:47)
at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.tran
sactionInitiatedRecovery(XARecoveryModule.java:224)
at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.peri
odicWorkSecondPass(XARecoveryModule.java:141)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkIntern
al(PeriodicRecovery.java:789)
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(Periodic
Recovery.java:371)
"Transaction Expired Entry Monitor" daemon prio=6 tid=0x347bfc00 nid=0x314 in Ob
ject.wait() [0x3552f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x13deb068> (a com.arjuna.ats.internal.arjuna.recovery.Exp
iredEntryMonitor)
at com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor.run(Expir
edEntryMonitor.java:191)
- locked <0x13deb068> (a com.arjuna.ats.internal.arjuna.recovery.Expired
EntryMonitor)
"http--127.0.0.1-8080-Acceptor-0" daemon prio=6 tid=0x34e5ec00 nid=0x788 runnabl
e [0x3548f000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
- locked <0x13d19c18> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(De
faultServerSocketFactory.java:61)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:
343)
at java.lang.Thread.run(Thread.java:619)
"http--127.0.0.1-8080-Poller" daemon prio=6 tid=0x34cb4800 nid=0xba0 runnable [0
x3543f000]
java.lang.Thread.State: RUNNABLE
at org.apache.tomcat.util.net.JIoEndpoint$Poller.run(JIoEndpoint.java:76
4)
- locked <0x13d19d78> (a org.apache.tomcat.util.net.JIoEndpoint$Poller)
at java.lang.Thread.run(Thread.java:619)
"XNIO NIO Write 1" prio=6 tid=0x34fae400 nid=0x8c8 runnable [0x3539f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.j
ava:273)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelector
Impl.java:255)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:136)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x13e14e40> (a sun.nio.ch.Util$1)
- locked <0x13e14e30> (a java.util.Collections$UnmodifiableSet)
- locked <0x13e0e790> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at org.xnio.nio.AbstractNioChannelThread$1.run(AbstractNioChannelThread.
java:124)
at java.lang.Thread.run(Thread.java:619)
"XNIO NIO Read 1" prio=6 tid=0x34eb6800 nid=0x870 runnable [0x3534f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.j
ava:273)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelector
Impl.java:255)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:136)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x13e0f4a8> (a sun.nio.ch.Util$1)
- locked <0x13e0f498> (a java.util.Collections$UnmodifiableSet)
- locked <0x13e0f258> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
at org.xnio.nio.AbstractNioChannelThread$1.run(AbstractNioChannelThread.
java:124)
at java.lang.Thread.run(Thread.java:619)
"NamingEventCoordinator-threads - 1" prio=6 tid=0x34690000 nid=0x210 waiting on
condition [0x3525f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x13b8bc90> (a java.util.concurrent.locks.Abstra
ctQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.jav
a:399)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.ja
va:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:907)
at java.lang.Thread.run(Thread.java:619)
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
"DestroyJavaVM" prio=6 tid=0x003b7800 nid=0xf64 waiting on condition [0x00000000
]
java.lang.Thread.State: RUNNABLE
"MSC service thread 1-4" prio=6 tid=0x34605c00 nid=0x7c0 waiting for monitor ent
ry [0x349af000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.jboss.msc.service.ServiceControllerImpl.invokeListener(ServiceCon
trollerImpl.java:1486)
at org.jboss.msc.service.ServiceControllerImpl.access$2500(ServiceContro
llerImpl.java:49)
at org.jboss.msc.service.ServiceControllerImpl$ListenerTask.run(ServiceC
ontrollerImpl.java:1940)
at org.jboss.msc.service.ServiceControllerImpl$ClearTCCLTask.run(Service
ControllerImpl.java:2291)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
utor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:908)
at java.lang.Thread.run(Thread.java:619)
"MSC service thread 1-3" prio=6 tid=0x3432f400 nid=0xf90 waiting for monitor ent
ry [0x3495f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.xerces.dom.DeferredDocumentImpl.getNodeObject(DeferredDocu
mentImpl.java:1014)
at org.apache.xerces.dom.DeferredDocumentImpl.synchronizeChildren(Deferr
edDocumentImpl.java:1752)
at org.apache.xerces.dom.DeferredElementNSImpl.synchronizeChildren(Defer
redElementNSImpl.java:153)
at org.apache.xerces.dom.ParentNode.hasChildNodes(ParentNode.java:193)
at org.apache.xerces.dom.DeepNodeListImpl.nextMatchingElementAfter(DeepN
odeListImpl.java:169)
at org.apache.xerces.dom.DeepNodeListImpl.item(DeepNodeListImpl.java:144
)
at org.apache.xerces.dom.DeepNodeListImpl.getLength(DeepNodeListImpl.jav
a:115)
at com.sun.faces.config.processor.FaceletTaglibConfigProcessor.process(F
aceletTaglibConfigProcessor.java:260)
at com.sun.faces.config.ConfigManager.initialize(ConfigManager.java:341)
at com.sun.faces.config.ConfigureListener.contextInitialized(ConfigureLi
stener.java:226)
at org.apache.catalina.core.StandardContext.contextListenerStart(Standar
dContext.java:3368)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:3
821)
- locked <0x305f2e60> (a org.apache.catalina.core.StandardContext)
at org.jboss.as.web.deployment.WebDeploymentService.start(WebDeploymentS
ervice.java:70)
- locked <0x305f7640> (a org.jboss.as.web.deployment.WebDeploymentServic
e)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceCont
rollerImpl.java:1765)
at org.jboss.msc.service.ServiceControllerImpl$ClearTCCLTask.run(Service
ControllerImpl.java:2291)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
utor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:908)
at java.lang.Thread.run(Thread.java:619)
"MSC service thread 1-2" prio=6 tid=0x3455c400 nid=0xe7c runnable [0x3490f000]
java.lang.Thread.State: RUNNABLE
at org.jboss.msc.service.ServiceControllerImpl.getListenerTasks(ServiceC
ontrollerImpl.java:670)
at org.jboss.msc.service.ServiceControllerImpl.transition(ServiceControl
lerImpl.java:547)
at org.jboss.msc.service.ServiceControllerImpl.immediateDependencyDown(S
erviceControllerImpl.java:1032)
- locked <0x305e8eb8> (a org.jboss.msc.service.ServiceControllerImpl)
at org.jboss.msc.service.ServiceControllerImpl$DependencyStoppedTask.run
(ServiceControllerImpl.java:1986)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
utor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:908)
at java.lang.Thread.run(Thread.java:619)
"MSC service thread 1-1" prio=6 tid=0x34715400 nid=0xc4c waiting for monitor ent
ry [0x348bf000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.AbstractList.iterator(AbstractList.java:273)
at org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControll
erImpl.java:695)
at org.jboss.msc.service.ServiceControllerImpl.invokeListener(ServiceCon
trollerImpl.java:1493)
at org.jboss.msc.service.ServiceControllerImpl.access$2500(ServiceContro
llerImpl.java:49)
at org.jboss.msc.service.ServiceControllerImpl$ListenerTask.run(ServiceC
ontrollerImpl.java:1940)
at org.jboss.msc.service.ServiceControllerImpl$ClearTCCLTask.run(Service
ControllerImpl.java:2291)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
utor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:908)
at java.lang.Thread.run(Thread.java:619)
"Reference Reaper" daemon prio=6 tid=0x34296800 nid=0x618 in Object.wait() [0x34
44f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x13b97cb8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x13b97cb8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at org.jboss.modules.ref.References$ReaperThread.run(References.java:68)
"Low Memory Detector" daemon prio=6 tid=0x33ea6400 nid=0xa50 runnable [0x0000000
0]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x33ea3c00 nid=0x358 waiting on condition [
0x00000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x33e9e000 nid=0x93c waiting on condition [
0x00000000]
java.lang.Thread.State: RUNNABLE
"Attach Listener" daemon prio=10 tid=0x33e9c400 nid=0xd50 waiting on condition [
0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x33e9b000 nid=0xde4 runnable [0x00000000
]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=8 tid=0x33e8c000 nid=0x17c in Object.wait() [0x3401f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x13b26450> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x13b26450> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x33e87800 nid=0xc60 in Object.wait() [0x
33fcf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x13bed260> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x13bed260> (a java.lang.ref.Reference$Lock)
at org.jboss.msc.service.ServiceControllerImpl.immediateDependencyDown(S
erviceControllerImpl.java:1032)
- locked <0x305e8eb8> (a org.jboss.msc.service.ServiceControllerImpl)
at org.jboss.msc.service.ServiceControllerImpl$DependencyStoppedTask.run
(ServiceControllerImpl.java:1986)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
utor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:908)
at java.lang.Thread.run(Thread.java:619)
"MSC service thread 1-1" prio=6 tid=0x34715400 nid=0xc4c waiting for monitor ent
ry [0x348bf000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.AbstractList.iterator(AbstractList.java:273)
at org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControll
erImpl.java:695)
at org.jboss.msc.service.ServiceControllerImpl.invokeListener(ServiceCon
trollerImpl.java:1493)
at org.jboss.msc.service.ServiceControllerImpl.access$2500(ServiceContro
llerImpl.java:49)
at org.jboss.msc.service.ServiceControllerImpl$ListenerTask.run(ServiceC
ontrollerImpl.java:1940)
at org.jboss.msc.service.ServiceControllerImpl$ClearTCCLTask.run(Service
ControllerImpl.java:2291)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
utor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:908)
at java.lang.Thread.run(Thread.java:619)
"Reference Reaper" daemon prio=6 tid=0x34296800 nid=0x618 in Object.wait() [0x34
44f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x13b97cb8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x13b97cb8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at org.jboss.modules.ref.References$ReaperThread.run(References.java:68)
"Low Memory Detector" daemon prio=6 tid=0x33ea6400 nid=0xa50 runnable [0x0000000
0]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x33ea3c00 nid=0x358 waiting on condition [
0x00000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x33e9e000 nid=0x93c waiting on condition [
0x00000000]
java.lang.Thread.State: RUNNABLE
"Attach Listener" daemon prio=10 tid=0x33e9c400 nid=0xd50 waiting on condition [
0x00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x33e9b000 nid=0xde4 runnable [0x00000000
]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=8 tid=0x33e8c000 nid=0x17c in Object.wait() [0x3401f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x13b26450> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x13b26450> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x33e87800 nid=0xc60 in Object.wait() [0x
33fcf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x13bed260> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x13bed260> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=10 tid=0x33e84c00 nid=0x92c runnable
"GC task thread#0 (ParallelGC)" prio=6 tid=0x003bec00 nid=0x200 runnable
"GC task thread#1 (ParallelGC)" prio=6 tid=0x33de0400 nid=0xed0 runnable
"VM Periodic Task Thread" prio=10 tid=0x33eb0400 nid=0xbbc waiting on condition
JNI global references: 1789
-
4. Re: All deployments failing after continued deployment (JBoss7)
jaikiran Jul 21, 2011 1:13 AM (in response to rnaodm)Please get a couple of more thread dumps with a interval of around 5 seconds and attach them in separate files to this thread.
-
5. Re: All deployments failing after continued deployment (JBoss7)
rnaodm Jul 21, 2011 11:24 AM (in response to jaikiran)Here you go, I did a thread dump every 5 seconds between the time it was at ".isdeploying" to the time where it was at ".failed". I'm sure you don't need them all but just in case it helps!
Thanks!
-
jstack11.txt.zip 3.4 KB
-
jstack10.txt.zip 2.9 KB
-
jstack9.txt.zip 3.8 KB
-
jstack8.txt.zip 3.5 KB
-
jstack7.txt.zip 3.7 KB
-
jstack6.txt.zip 3.5 KB
-
jstack5.txt.zip 3.5 KB
-
jstack4.txt.zip 2.9 KB
-
-
6. Re: All deployments failing after continued deployment (JBoss7)
rnaodm Jul 21, 2011 11:39 AM (in response to rnaodm)I recieved a different message this time in .failed file:
{"Composite operation failed and was rolled back. Steps that failed:" => {"Operation step-2" => {"Failed services" => {"jboss.deployment.unit.\"1910.war\".INSTALL" => "org.jboss.msc.service.StartException in service jboss.deployment.unit.\"1910.war\".INSTALL: Failed to process phase INSTALL of deployment \"1910.war\""},"Services with missing/unavailable dependencies" => ["jboss.naming.context.java.comp.1910.1910.1910.Validator missing [ jboss.naming.context.java.module.1910.1910 ]","jboss.naming.context.java.comp.1910.1910.1910.ValidatorFactory missing [ jboss.naming.context.java.module.1910.1910 ]"]}}}
-
7. Re: All deployments failing after continued deployment (JBoss7)
rnaodm Jul 21, 2011 11:42 AM (in response to rnaodm)Here is the server.log to go along with it
-
server.log.zip 8.1 KB
-
-
9. Re: All deployments failing after continued deployment (JBoss7)
rnaodm Jul 21, 2011 12:13 PM (in response to jaikiran)I will give that a try now, thank you.
-
10. Re: All deployments failing after continued deployment (JBoss7)
jaikiran Jul 21, 2011 12:22 PM (in response to rnaodm)Let us know how it goes. Also if possible please attach (or share privately if you prefer) the application which reproduces this. We would like to see if there is some leak somewhere.
-
11. Re: All deployments failing after continued deployment (JBoss7)
rnaodm Jul 21, 2011 12:28 PM (in response to jaikiran)The applications get deployed in a random sequence each time, and each time the application causing the error is different from the last time...running the tests now. Will post back here when I know if it fails or not
-
12. Re: All deployments failing after continued deployment (JBoss7)
rnaodm Jul 21, 2011 1:27 PM (in response to jaikiran)The applications began failing again, except this time there is no file being created.
Initially there was a .isdeploying file for the first application that failed. However, now the applications don't even get a .isdeploying file assocatted with it. They are just copied into the deployments directory and sit there as if nothing is happening.
Server log shows:
11:05:44,046 ERROR [org.jboss.as.deployment] (DeploymentScanner-threads - 1) Did not receive a response to the deployment operation within the allowed timeout period [120 seconds]. Check the server configurationfile and the server logs to find more about the status of the deployment.
11:05:53,421 ERROR [stderr] (Periodic Recovery) Exception in thread "Periodic Recovery" java.lang.OutOfMemoryError: Java heap space
followed shortly after by:
11:05:58,671 ERROR [stderr] (Periodic Recovery) at java.util.Arrays.copyOf(Arrays.java:2882)
11:06:21,609 ERROR [stderr] (Periodic Recovery) Exception in thread "server-timer" at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
11:06:56,500 ERROR [stderr] (Periodic Recovery) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
11:07:08,093 ERROR [stderr] (Periodic Recovery) at java.lang.StringBuilder.append(StringBuilder.java:119)
11:08:24,375 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.arjuna.objectstore.FileSystemStore.allObjUids(FileSystemStore.java:170)
11:08:29,218 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.arjuna.objectstore.ObjectStore.allObjUids(ObjectStore.java:70)
11:08:29,218 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.periodicWorkFirstPass(AtomicActionRecoveryModule.java:91)
11:08:29,218 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:743)
11:08:33,953 ERROR [stderr] (Periodic Recovery) at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371)
-
13. Re: All deployments failing after continued deployment (JBoss7)
smarlow Jul 21, 2011 1:46 PM (in response to rnaodm)Sounds like a deployment memory leak to me. Try running the app server with -XX:+HeapDumpOnOutOfMemoryError, so that a heap dump file with the AS memory contents will be created that can be read with http://www.eclipse.org/mat/.
Add the following to the run.conf file (after line containing "byteman"). Or after line containing %PROGNAME% in standalone.bat
{quote}
JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError"
{quote}
After the heap dump gets generated, open a jira with the steps to recreate. And attach the .hprof file.
Some screen shots for the eclipse memory analyzer tool (mat) are here http://community.jboss.org/blogs/smarlow/2010/10/07/as-6-memoryboot-speed-improvements
-
14. Re: All deployments failing after continued deployment (JBoss7)
rnaodm Jul 21, 2011 4:17 PM (in response to smarlow)What location does this heap dump file get created at?