10 Replies Latest reply on May 27, 2009 12:43 PM by dmr1122

JBoss AS 5.0.0.CR2 hangs indefinitely with CPU at 100% when

d-rock Oct 9, 2008 4:59 PM

Everything is loading up fine until it goes to deploy my WAR file. Here is the tail of the log file:

2008-10-09 15:38:39,739 DEBUG [org.jboss.web.tomcat.service.deployers.TomcatDepl
oyment] (main) Linked java:comp/TransactionSynchronizationRegistry to JNDI name:
 java:TransactionSynchronizationRegistry
2008-10-09 15:38:39,739 DEBUG [org.jboss.web.tomcat.service.deployers.TomcatDepl
oyment] (main) Linked java:comp/UserTransaction to JNDI name: UserTransaction
2008-10-09 15:38:39,739 DEBUG [org.jboss.web.tomcat.service.deployers.TomcatDepl
oyment] (main) linkSecurityDomain
2008-10-09 15:38:39,739 DEBUG [org.jboss.web.tomcat.service.deployers.TomcatDepl
oyment] (main) Linking security/securityMgr to JNDI name: java:/jaas/ADSRealm
2008-10-09 15:38:39,740 DEBUG [org.jboss.web.tomcat.service.deployers.TomcatDepl
oyment] (main) injectionContainer enabled and processing beginning
2008-10-09 15:38:39,792 DEBUG [org.jboss.web.tomcat.filters.ReplyHeaderFilter] (
main) Adding header name: X-Powered-By='Servlet 2.5; JBoss-5.0/JBossWeb-2.1'

I turned on the HPROF agent to try and figure out what was going on and it looks like it's spinning in TCP socket connections:

THREAD START (obj=5000013e, id = 200001, name="main", group="main")
THREAD END (id = 200001)
THREAD START (obj=5000013e, id = 200004, name="DestroyJavaVM", group="main")
THREAD START (obj=5000013e, id = 200005, name="main", group="jboss")
THREAD START (obj=500008b0, id = 200007, name="ZipFile Lock Reaper", group="jboss")
THREAD START (obj=5000013e, id = 200008, name="HDScanner", group="jboss")
THREAD START (obj=500008b0, id = 200009, name="Timer-Log4jService", group="jboss")
THREAD START (obj=5000013e, id = 200010, name="JBoss System Threads(1)-1", group="System Threads")
THREAD START (obj=5000013e, id = 200011, name="RMI TCP Accept-1098", group="system")
THREAD START (obj=5000013e, id = 200012, name="RMI Reaper", group="system")
THREAD START (obj=5000013e, id = 200013, name="GC Daemon", group="system")
THREAD START (obj=5000013e, id = 200014, name="JBoss System Threads(1)-2", group="System Threads")
THREAD START (obj=500008b0, id = 200015, name="ServerSocketRefresh", group="jboss")
THREAD START (obj=50000eca, id = 200016, name="AcceptorThread[ServerSocket[addr=/127.0.0.1,port=0,localport=4446]]", group="jboss")
THREAD START (obj=5000013e, id = 200017, name="SubscriptionWatchDog", group="jboss")
THREAD START (obj=5000013e, id = 200018, name="RMI TCP Accept-1090", group="system")
THREAD START (obj=5000013e, id = 200019, name="RMI TCP Accept-0", group="system")
THREAD START (obj=5000013e, id = 200020, name="RMI TCP Connection(idle)", group="RMI Runtime")
THREAD START (obj=5000013e, id = 200021, name="RMI Scheduler(0)", group="system")
THREAD START (obj=5000013e, id = 200022, name="RMI RenewClean-[127.0.0.1:49029]", group="system")
THREAD START (obj=5000013e, id = 200023, name="RMI TCP Connection(idle)", group="RMI Runtime")
THREAD START (obj=500008b0, id = 200024, name="ServerSocketRefresh", group="jboss")
THREAD START (obj=50000eca, id = 200025, name="AcceptorThread[ServerSocket[addr=/127.0.0.1,port=0,localport=4457]]", group="jboss")
THREAD START (obj=50000eca, id = 200026, name="secondaryServerSocketThread[0]", group="jboss")
THREAD START (obj=500008b0, id = 200027, name="ServerSocketRefresh", group="jboss")
THREAD START (obj=50000eca, id = 200028, name="AcceptorThread[ServerSocket[addr=/127.0.0.1,port=0,localport=3873]]", group="jboss")
THREAD START (obj=50001a50, id = 200029, name="Listener:55822", group="jboss")
THREAD START (obj=50000eca, id = 200030, name="Thread-8", group="jboss")
THREAD START (obj=500008b0, id = 200031, name="Thread-9", group="jboss")
THREAD START (obj=5000013e, id = 200032, name="Thread-10", group="jboss")
THREAD START (obj=500008b0, id = 200033, name="Server.Connection:127.0.0.1:43619", group="jboss")
THREAD END (id = 200033)
THREAD START (obj=50001a50, id = 200034, name="Listener:4712", group="jboss")
THREAD START (obj=500008b0, id = 200035, name="Thread-11", group="jboss")
THREAD START (obj=5000013e, id = 200037, name="RMI TCP Accept-4444", group="system")
THREAD START (obj=5000013e, id = 200038, name="PooledInvokerAcceptor#0-4445", group="jboss")
THREAD START (obj=5000013e, id = 200039, name="ContainerBackgroundProcessor[StandardEngine[jboss.web]]", group="jboss")
THREAD START (obj=500008b0, id = 200040, name="AOPListner", group="jboss")
THREAD START (obj=5000013e, id = 200041, name="WorkManager(2)-1", group="JBoss Pooled Threads")
THREAD START (obj=50000eca, id = 200042, name="DefaultQuartzScheduler_Worker-0", group="jboss")
THREAD START (obj=50000eca, id = 200043, name="DefaultQuartzScheduler_Worker-1", group="jboss")
THREAD START (obj=50000eca, id = 200044, name="DefaultQuartzScheduler_Worker-2", group="jboss")
THREAD START (obj=50000eca, id = 200045, name="DefaultQuartzScheduler_Worker-3", group="jboss")
THREAD START (obj=50000eca, id = 200046, name="DefaultQuartzScheduler_Worker-4", group="jboss")
THREAD START (obj=50000eca, id = 200047, name="DefaultQuartzScheduler_Worker-5", group="jboss")
THREAD START (obj=50000eca, id = 200048, name="DefaultQuartzScheduler_Worker-6", group="jboss")
THREAD START (obj=50000eca, id = 200049, name="DefaultQuartzScheduler_Worker-7", group="jboss")
THREAD START (obj=50000eca, id = 200050, name="DefaultQuartzScheduler_Worker-8", group="jboss")
THREAD START (obj=50000eca, id = 200051, name="DefaultQuartzScheduler_Worker-9", group="jboss")
THREAD START (obj=50001d4d, id = 200052, name="DefaultQuartzScheduler_QuartzSchedulerThread", group="QuartzScheduler:DefaultQuartzScheduler")
THREAD START (obj=5000013e, id = 200053, name="HSQLDB Timer @430f4", group="jboss")
THREAD START (obj=500008b0, id = 200054, name="Timer-0", group="jboss")
THREAD START (obj=5000013e, id = 200055, name="JCA PoolFiller", group="jboss")
THREAD START (obj=50000eca, id = 200056, name="JBossEJB3QuartzScheduler_Worker-0", group="jboss")
THREAD START (obj=50000eca, id = 200057, name="JBossEJB3QuartzScheduler_Worker-1", group="jboss")
THREAD START (obj=50000eca, id = 200059, name="JBossEJB3QuartzScheduler_Worker-3", group="jboss")
THREAD START (obj=50000eca, id = 200061, name="JBossEJB3QuartzScheduler_Worker-5", group="jboss")
THREAD START (obj=50000eca, id = 200060, name="JBossEJB3QuartzScheduler_Worker-4", group="jboss")
THREAD START (obj=50000eca, id = 200062, name="JBossEJB3QuartzScheduler_Worker-6", group="jboss")
THREAD START (obj=50000eca, id = 200063, name="JBossEJB3QuartzScheduler_Worker-7", group="jboss")
THREAD START (obj=50000eca, id = 200058, name="JBossEJB3QuartzScheduler_Worker-2", group="jboss")
THREAD START (obj=50000eca, id = 200064, name="JBossEJB3QuartzScheduler_Worker-8", group="jboss")
THREAD START (obj=50000eca, id = 200065, name="JBossEJB3QuartzScheduler_Worker-9", group="jboss")
THREAD START (obj=50001d4d, id = 200066, name="JBossEJB3QuartzScheduler_QuartzSchedulerThread", group="QuartzScheduler:JBossEJB3QuartzScheduler")
THREAD START (obj=50000eca, id = 200067, name="QuartzScheduler_JBossEJB3QuartzScheduler-NON_CLUSTERED_MisfireHandler", group="jboss")
THREAD START (obj=5000013e, id = 200068, name="jbm-cf-jndimapper", group="JBM-threads")
THREAD START (obj=5000013e, id = 200069, name="Thread-16", group="jboss")
THREAD START (obj=5000013e, id = 200070, name="IdleRemover", group="jboss")
THREAD END (id = 200020)
THREAD END (id = 200023)
THREAD START (obj=5000013e, id = 200071, name="RMI TCP Connection(idle)", group="RMI Runtime")
THREAD END (id = 200071)
THREAD START (obj=5000013e, id = 200072, name="RMI TCP Connection(idle)", group="RMI Runtime")
THREAD END (id = 200072)
TRACE 300956:
 java.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java:Unknown line)
 java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
 java.net.ServerSocket.implAccept(ServerSocket.java:453)
 java.net.ServerSocket.accept(ServerSocket.java:421)
 sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
 sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
 java.lang.Thread.run(Thread.java:619)
TRACE 300993:
 java.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java:Unknown line)
 java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
 java.net.ServerSocket.implAccept(ServerSocket.java:453)
 java.net.ServerSocket.accept(ServerSocket.java:421)
 org.jboss.remoting.util.SecurityUtility.accept(SecurityUtility.java:863)
 org.jboss.remoting.transport.socket.SocketServerInvoker$AcceptThread.run(SocketServerInvoker.java:1003)
TRACE 301962:
 java.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java:Unknown line)
 java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
 java.net.ServerSocket.implAccept(ServerSocket.java:453)
 java.net.ServerSocket.accept(ServerSocket.java:421)
 com.arjuna.ats.internal.arjuna.recovery.Listener.run(Listener.java:119)
TRACE 300882:
 java.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java:Unknown line)
 java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
 java.net.ServerSocket.implAccept(ServerSocket.java:453)
 java.net.ServerSocket.accept(ServerSocket.java:421)
 org.jboss.web.WebServer.run(WebServer.java:320)
 org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:147)
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
 java.lang.Thread.run(Thread.java:619)
TRACE 300982:
 java.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java:Unknown line)
 java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
 java.net.ServerSocket.implAccept(ServerSocket.java:453)
 java.net.ServerSocket.accept(ServerSocket.java:421)
 org.jnp.server.Main$AcceptHandler.run(Main.java:482)
 org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:147)
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
 java.lang.Thread.run(Thread.java:619)
TRACE 301927:
 java.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java:Unknown line)
 java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
 java.net.ServerSocket.implAccept(ServerSocket.java:453)
 java.net.ServerSocket.accept(ServerSocket.java:421)
 org.jboss.remoting.transport.bisocket.BisocketServerInvoker$SecondaryServerSocketThread$1.run(BisocketServerInvoker.java:1110)
 java.security.AccessController.doPrivileged(AccessController.java:Unknown line)
 org.jboss.remoting.transport.bisocket.BisocketServerInvoker$SecondaryServerSocketThread.run(BisocketServerInvoker.java:1106)
TRACE 301974:
 java.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java:Unknown line)
 java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
 java.net.ServerSocket.implAccept(ServerSocket.java:453)
 java.net.ServerSocket.accept(ServerSocket.java:421)
 org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:262)
 java.lang.Thread.run(Thread.java:619)
TRACE 300608:
 java.lang.ClassLoader.defineClass1(ClassLoader.java:Unknown line)
 java.lang.ClassLoader.defineClass(ClassLoader.java:620)
 org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:62)
 org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:504)
 org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:464)
 java.security.AccessController.doPrivileged(AccessController.java:Unknown line)
 org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:462)
 org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:439)
 org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
TRACE 301921:
 java.net.SocketInputStream.socketRead0(SocketInputStream.java:Unknown line)
 java.net.SocketInputStream.read(SocketInputStream.java:129)
 java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
 java.io.BufferedInputStream.read(BufferedInputStream.java:237)
 java.io.FilterInputStream.read(FilterInputStream.java:66)
 sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
 sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
 sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
 java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
TRACE 302247:
 java.lang.ClassLoader.defineClass(ClassLoader.java:620)
 org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:62)
 org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:504)
 org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:464)
 java.security.AccessController.doPrivileged(AccessController.java:Unknown line)
 org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:462)
 org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:439)
 org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
 org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
TRACE 302241:
 org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:462)
 org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:439)
 org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
 org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
 org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
 org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:254)
 org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:148)
 org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:256)
 org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1099)
TRACE 300737:
 java.util.zip.Inflater.inflateBytes(Inflater.java:Unknown line)
 java.util.zip.Inflater.inflate(Inflater.java:223)
 java.util.zip.InflaterInputStream.read(InflaterInputStream.java:135)
 java.io.FilterInputStream.read(FilterInputStream.java:90)
 org.jboss.virtual.plugins.context.zip.ZipEntryInputStream.read(ZipEntryInputStream.java:112)
 org.jboss.classloader.plugins.ClassLoaderUtils.loadByteCode(ClassLoaderUtils.java:150)
 org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:475)
 org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:464)
 java.security.AccessController.doPrivileged(AccessController.java:Unknown line)
TRACE 302226:
 java.io.UnixFileSystem.getBooleanAttributes0(UnixFileSystem.java:Unknown line)
 java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228)
 java.io.File.exists(File.java:733)
 org.jboss.virtual.protocol.AbstractVFSHandler.openConnection(AbstractVFSHandler.java:87)
 java.net.URL.openConnection(URL.java:945)
 sun.security.provider.PolicyFile.canonicalizeCodebase(PolicyFile.java:1797)
 sun.security.provider.PolicyFile.access$700(PolicyFile.java:263)
 sun.security.provider.PolicyFile$7.run(PolicyFile.java:1253)
 java.security.AccessController.doPrivileged(AccessController.java:Unknown line)
TRACE 301423:
 java.io.UnixFileSystem.getBooleanAttributes0(UnixFileSystem.java:Unknown line)
 java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228)
 java.io.File.exists(File.java:733)
 org.jboss.virtual.protocol.AbstractVFSHandler.openConnection(AbstractVFSHandler.java:76)
 java.net.URL.openConnection(URL.java:945)
 sun.security.provider.PolicyFile.canonicalizeCodebase(PolicyFile.java:1797)
 sun.security.provider.PolicyFile.access$700(PolicyFile.java:263)
 sun.security.provider.PolicyFile$7.run(PolicyFile.java:1253)
 java.security.AccessController.doPrivileged(AccessController.java:Unknown line)
TRACE 302224:
 java.lang.Throwable.fillInStackTrace(Throwable.java:Unknown line)
 java.lang.Throwable.<init>(Throwable.java:196)
 java.lang.Error.<init>(Error.java:49)
 java.lang.LinkageError.<init>(LinkageError.java:36)
 java.lang.ClassLoader.defineClass1(ClassLoader.java:Unknown line)
 java.lang.ClassLoader.defineClass(ClassLoader.java:620)
 org.jboss.classloader.spi.base.BaseClassLoader.access$200(BaseClassLoader.java:62)
 org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:504)
 org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:464)
TRACE 300770:
 java.util.zip.Inflater.init(Inflater.java:Unknown line)
 java.util.zip.Inflater.<init>(Inflater.java:83)
 java.util.zip.ZipFile.getInflater(ZipFile.java:266)
 java.util.zip.ZipFile.getInputStream(ZipFile.java:212)
 java.util.zip.ZipFile.getInputStream(ZipFile.java:180)
 org.jboss.virtual.plugins.context.zip.ZipFileWrapper.openStream(ZipFileWrapper.java:214)
 org.jboss.virtual.plugins.context.zip.ZipEntryContext.openStream(ZipEntryContext.java:899)
 org.jboss.virtual.plugins.context.zip.ZipEntryHandler.openStream(ZipEntryHandler.java:142)
 org.jboss.virtual.VirtualFile.openStream(VirtualFile.java:216)
TRACE 302088:
 java.security.AccessController.doPrivileged(AccessController.java:Unknown line)
 sun.security.provider.PolicyFile.getPermissions(PolicyFile.java:1249)
 sun.security.provider.PolicyFile.getPermissions(PolicyFile.java:1196)
 org.jboss.classloading.spi.vfs.policy.VFSClassLoaderPolicy.getProtectionDomain(VFSClassLoaderPolicy.java:625)
 org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:479)
 org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:464)
 java.security.AccessController.doPrivileged(AccessController.java:Unknown line)
 org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:462)
 org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:439)
TRACE 302252:
 java.lang.Thread.sleep(Thread.java:Unknown line)
 org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:394)
TRACE 302223:
 java.security.AccessController.doPrivileged(AccessController.java:Unknown line)
 org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:462)
 org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally(BaseClassLoader.java:439)
 org.jboss.classloader.spi.base.BaseDelegateLoader.loadClass(BaseDelegateLoader.java:134)
 org.jboss.classloader.spi.filter.FilteredDelegateLoader.loadClass(FilteredDelegateLoader.java:131)
 org.jboss.classloader.spi.base.ClassLoadingTask$ThreadTask.run(ClassLoadingTask.java:452)
 org.jboss.classloader.spi.base.ClassLoaderManager.nextTask(ClassLoaderManager.java:254)
 org.jboss.classloader.spi.base.ClassLoaderManager.process(ClassLoaderManager.java:148)
 org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:256)
TRACE 302230:
 java.io.UnixFileSystem.getBooleanAttributes0(UnixFileSystem.java:Unknown line)
 java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228)
 java.io.File.isDirectory(File.java:754)
 java.io.File.toURL(File.java:623)
 org.jboss.virtual.protocol.AbstractVFSHandler.openConnection(AbstractVFSHandler.java:89)
 java.net.URL.openConnection(URL.java:945)
 sun.security.provider.PolicyFile.canonicalizeCodebase(PolicyFile.java:1797)
 sun.security.provider.PolicyFile.access$700(PolicyFile.java:263)
 sun.security.provider.PolicyFile$7.run(PolicyFile.java:1253)
TRACE 301199:
 java.util.zip.ZipFile.read(ZipFile.java:Unknown line)
 java.util.zip.ZipFile.access$1200(ZipFile.java:29)
 java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:447)
 java.util.zip.ZipFile$1.fill(ZipFile.java:230)
 java.util.zip.InflaterInputStream.read(InflaterInputStream.java:141)
 java.io.FilterInputStream.read(FilterInputStream.java:90)
 org.jboss.virtual.plugins.context.zip.ZipEntryInputStream.read(ZipEntryInputStream.java:112)
 org.jboss.classloader.plugins.ClassLoaderUtils.loadByteCode(ClassLoaderUtils.java:150)
 org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:475)
TRACE 301109:
 java.util.zip.ZipFile.getEntry(ZipFile.java:Unknown line)
 java.util.zip.ZipFile.getInputStream(ZipFile.java:195)
 java.util.zip.ZipFile.getInputStream(ZipFile.java:180)
 org.jboss.virtual.plugins.context.zip.ZipFileWrapper.openStream(ZipFileWrapper.java:214)
 org.jboss.virtual.plugins.context.zip.ZipEntryContext.openStream(ZipEntryContext.java:899)
 org.jboss.virtual.plugins.context.zip.ZipEntryHandler.openStream(ZipEntryHandler.java:142)
 org.jboss.virtual.VirtualFile.openStream(VirtualFile.java:216)
 org.jboss.classloading.spi.vfs.policy.VFSClassLoaderPolicy.getResourceAsStream(VFSClassLoaderPolicy.java:476)
 org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:466)
TRACE 300652:
 java.security.AccessController.doPrivileged(AccessController.java:Unknown line)
 sun.security.provider.PolicyFile.addPermissions(PolicyFile.java:1316)
 sun.security.provider.PolicyFile.getPermissions(PolicyFile.java:1268)
 sun.security.provider.PolicyFile.getPermissions(PolicyFile.java:1257)
 sun.security.provider.PolicyFile.getPermissions(PolicyFile.java:1196)
 org.jboss.classloading.spi.vfs.policy.VFSClassLoaderPolicy.getProtectionDomain(VFSClassLoaderPolicy.java:625)
 org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:479)
 org.jboss.classloader.spi.base.BaseClassLoader$2.run(BaseClassLoader.java:464)
 java.security.AccessController.doPrivileged(AccessController.java:Unknown line)
CPU SAMPLES BEGIN (total = 455355) Thu Oct 9 15:50:31 2008
rank self accum count trace method
 1 28.32% 28.32% 128942 300956 java.net.PlainSocketImpl.socketAccept
 2 21.28% 49.59% 96890 300993 java.net.PlainSocketImpl.socketAccept
 3 14.06% 63.66% 64025 301962 java.net.PlainSocketImpl.socketAccept
 4 7.20% 70.85% 32782 300882 java.net.PlainSocketImpl.socketAccept
 5 7.20% 78.05% 32775 300982 java.net.PlainSocketImpl.socketAccept
 6 7.04% 85.09% 32066 301927 java.net.PlainSocketImpl.socketAccept
 7 7.03% 92.12% 32004 301974 java.net.PlainSocketImpl.socketAccept
 8 5.87% 97.99% 26720 300608 java.lang.ClassLoader.defineClass1
 9 0.57% 98.56% 2590 301921 java.net.SocketInputStream.socketRead0
 10 0.36% 98.91% 1620 302247 java.lang.ClassLoader.defineClass
 11 0.27% 99.19% 1239 302241 org.jboss.classloader.spi.base.BaseClassLoader.loadClassLocally
 12 0.07% 99.26% 323 300737 java.util.zip.Inflater.inflateBytes
 13 0.06% 99.32% 290 302226 java.io.UnixFileSystem.getBooleanAttributes0
 14 0.05% 99.37% 205 301423 java.io.UnixFileSystem.getBooleanAttributes0
 15 0.04% 99.41% 196 302224 java.lang.Throwable.fillInStackTrace
 16 0.03% 99.44% 146 300770 java.util.zip.Inflater.init
 17 0.02% 99.47% 106 302088 java.security.AccessController.doPrivileged
 18 0.02% 99.49% 102 302252 java.lang.Thread.sleep
 19 0.02% 99.50% 76 302223 java.security.AccessController.doPrivileged
 20 0.02% 99.52% 74 302230 java.io.UnixFileSystem.getBooleanAttributes0
 21 0.01% 99.53% 62 301199 java.util.zip.ZipFile.read
 22 0.01% 99.55% 56 301109 java.util.zip.ZipFile.getEntry
 23 0.01% 99.56% 47 300652 java.security.AccessController.doPrivileged
CPU SAMPLES END

I'm at a loss...

Derek

1. Re: JBoss AS 5.0.0.CR2 hangs indefinitely with CPU at 100% w

alesj Oct 10, 2008 5:02 AM (in response to d-rock)

Can you provide any more details of your app.
Since hprof log is not very useful.

Don't really see how simple war could hang AS.
What about if you deploy some other simple war/ear+war?
Actions

2. Re: JBoss AS 5.0.0.CR2 hangs indefinitely with CPU at 100% w

d-rock Oct 10, 2008 10:16 AM (in response to d-rock)

The app is packaged as an EAR. The contents of the EAR are

 0 Tue Oct 07 17:00:44 CDT 2008 META-INF/
 126 Tue Oct 07 17:00:42 CDT 2008 META-INF/MANIFEST.MF
 0 Tue Oct 07 17:00:42 CDT 2008 lib/
 20341 Tue Oct 07 17:00:42 CDT 2008 JPA-2.0b2.jar
4821456 Tue Oct 07 17:00:42 CDT 2008 Web-2.0b2.war
 5811 Tue Oct 07 17:00:42 CDT 2008 EJB-2.0b2.jar
 556 Tue Oct 07 17:00:42 CDT 2008 META-INF/application.xml
 5620 Tue Oct 07 17:00:42 CDT 2008 lib/EJB-Interfaces-2.0b2.jar
 12453 Tue Oct 07 17:00:42 CDT 2008 lib/Logic-2.0b2.jar

application.xml looks like:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE application PUBLIC
 "-//Sun Microsystems, Inc.//DTD J2EE Application 1.3//EN"
 "http://java.sun.com/dtd/application_1_3.dtd">
<application>
 <display-name>EAR-Build</display-name>
 <description>A system for handling rosters for event notification</description>
 <module>
 <ejb>JPA-2.0b2.jar</ejb>
 </module>
 <module>
 <ejb>EJB-2.0b2.jar</ejb>
 </module>
 <module>
 <web>
 <web-uri>Web-2.0b2.war</web-uri>
 <context-root>/roster</context-root>
 </web>
 </module>
</application>

The JPA jar is a JPA persistence module, Web is the WAR that's hanging, EJB is a set of EJB impls for the EJB interfaces in lib/EJB-Interfaces-2.0b2.jar, lib/Logic-2.0b2.jar is a set of business logic classes shared by the EJBs and the WAR classes. The WAR is comprised of Struts Actions and some JSP. The WAR contains the following libraries:

 90503 Tue Oct 07 15:41:56 CDT 2008 WEB-INF/lib/commons-collections-2.0.jar
 65261 Tue Oct 07 13:09:54 CDT 2008 WEB-INF/lib/oro-2.0.8.jar
 20962 Tue Oct 07 15:41:52 CDT 2008 WEB-INF/lib/jstl-1.0.2.jar
549876 Tue Oct 07 15:42:02 CDT 2008 WEB-INF/lib/struts-1.2.9.jar
109096 Tue Oct 07 15:41:56 CDT 2008 WEB-INF/lib/commons-digester-1.5.jar
109318 Tue Oct 07 13:09:58 CDT 2008 WEB-INF/lib/xml-apis-1.0.b2.jar
358273 Tue Oct 07 15:41:54 CDT 2008 WEB-INF/lib/antlr-2.7.2.jar
515008 Tue Oct 07 15:42:04 CDT 2008 WEB-INF/lib/standard-1.0.4.jar
2907319 Tue Oct 07 15:42:00 CDT 2008 WEB-INF/lib/xalan-2.5.1.jar
118726 Tue Oct 07 15:42:02 CDT 2008 WEB-INF/lib/commons-beanutils-1.6.1.jar
 84260 Tue Oct 07 15:42:00 CDT 2008 WEB-INF/lib/commons-validator-1.1.3.jar
169763 Tue Oct 07 15:41:54 CDT 2008 WEB-INF/lib/commons-lang-2.0.jar
 22379 Tue Oct 07 15:41:56 CDT 2008 WEB-INF/lib/commons-fileupload-1.0.jar
 38015 Tue Oct 07 12:08:12 CDT 2008 WEB-INF/lib/commons-logging-1.0.4.jar

I know some of those are redundant but I haven't had time to prune them. The JPA and EJB modules load just fine, but when the WAR goes to load the CPU goes to 100% and it goes no further. I do see some Arjuna messages in the log that look like normal periodic sweeps, but otherwise the AS is unresponsive and I can't get into the web interface.

Other WARs appear to deploy fine, although the one I tested with isn't bundled in an EAR.

Thanks,

Derek

3. Re: JBoss AS 5.0.0.CR2 hangs indefinitely with CPU at 100% w

kconner Oct 14, 2008 7:11 AM (in response to d-rock)

The socketAccepts are unlikely to be the issue as they will be waiting for connections. They will, therefore, show up as a high count in a sampling hprof but this does not mean that they are using CPU.

You may get a better idea if you use the 'times' setting as this will inject monitoring code into the classes. It is more invasive though.
Actions

4. Re: JBoss AS 5.0.0.CR2 hangs indefinitely with CPU at 100% w

d-rock Oct 14, 2008 3:42 PM (in response to d-rock)

Well, using "times", I get the following output:

CPU TIME (ms) BEGIN (total = 265564231) Tue Oct 14 14:38:08 2008
rank self accum count trace method
 1 62.18% 62.18% 164745 470417 org.quartz.simpl.SimpleThreadPool.getNextRunna
ble
 2 7.97% 70.15% 2557 355984 java.util.concurrent.locks.LockSupport.parkNan
os
 3 4.75% 74.89% 961 315597 java.lang.Object.wait
 4 4.75% 79.64% 594335 315608 java.lang.ref.ReferenceQueue.remove
 5 3.28% 82.92% 31 433617 java.net.PlainSocketImpl.accept
 6 3.28% 86.19% 53 435759 java.lang.ref.ReferenceQueue.remove
 7 3.11% 89.31% 1654 480075 org.hsqldb.lib.HsqlTimer$TaskQueue.park
 8 2.89% 92.20% 64 471109 com.arjuna.ats.internal.arjuna.recovery.Period
icRecovery.doPeriodicWait
 9 1.79% 93.99% 2 384580 java.util.TimerThread.mainLoop
 10 0.80% 94.79% 282394 336742 java.lang.ClassLoader.defineClass
 11 0.70% 95.49% 31 437610 java.util.concurrent.locks.LockSupport.parkNan
os
 12 0.24% 95.73% 65 449168 com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doBackoffWait
 13 0.18% 95.91% 61 434142 java.net.SocketInputStream.read
 14 0.11% 96.02% 282393 336228 java.net.URLDecoder.decode
...

This is after firing it up and letting it run for a few hours. It still isn't meaningful to me. I'm afraid I may have to download the source and do some hardcore digging with a debugger :(

Derek

5. Re: JBoss AS 5.0.0.CR2 hangs indefinitely with CPU at 100% w

jaikiran Oct 15, 2008 7:30 AM (in response to d-rock)

How big is your Web-2.0b2.war? And do you deploy it as a zipped archive or in an exploded format? If you are deploying it as a zipped archive then try deploying the entire application (including the EAR and the WAR) in a exploded format. See this for details https://www.jboss.org/community/docs/DOC-9719. When you deploy an application as a zipped archive, JBoss extracts the archive to a temp folder. Probably that is causing the spike in CPU?
Actions
6. Re: JBoss AS 5.0.0.CR2 hangs indefinitely with CPU at 100% w

d-rock Oct 16, 2008 5:44 PM (in response to d-rock)

The war file is 4.8MB, mostly because of the bundled struts, xalan and JSTL jars. The EAR file is only slightly larger since the EJBs and PAR are pretty small. I tried deploying with an exploded EAR and exploded WAR inside and it made no difference.

Derek
Actions
7. Re: JBoss AS 5.0.0.CR2 hangs indefinitely with CPU at 100% w

dmr1122 May 25, 2009 3:10 PM (in response to d-rock)

Was there any reslution on this. I'm having a similar concern with WAS 5.0.1 GA. It runs for several hours without a problem and then all of a sudden it goes to 100% CPU and stays there. I try restarting the server and it quickly goes to 100% CPU utilization. I then start of the JBoss 4.2.2 reference with the same app and configuration and it works fine. If I start the Jboss 5 instance bound to a different IP the problem seems to go away. Any ideas?

I have an exploded WAR Increasing the time limits on the Hot Deployment seemed to help, but didn't solve it completely.
Actions
8. Re: JBoss AS 5.0.0.CR2 hangs indefinitely with CPU at 100% w

alesj May 25, 2009 3:23 PM (in response to d-rock)

Can you try the latest version, 5.1.0.GA.
As we fixed some of the CL issues that could be potential blockers.
Actions
9. Re: JBoss AS 5.0.0.CR2 hangs indefinitely with CPU at 100% w

dmr1122 May 27, 2009 12:14 PM (in response to d-rock)

We installed 5.1.0GA and are still seeing the same problem. I have managed to capture a threaddumb and a heapdump as well as the application snapshot using jvisualvm.

Memory usuage is actually fine. I don't see any problems with the memory.

When looking at the thread dump I see a lot of threads waiting on object monitor.

java.lang.Thread.State: WAITING (on object monitor)
and
java.lang.Thread.State: TIMED_WAITING (on object monitor)

I'm not sure if it's significant. I'd be happy to post the thread dump and application snapshot if it would be useful to anyone.

The application server doesn't crash completely, it just run slower than expected.
Actions

10. Re: JBoss AS 5.0.0.CR2 hangs indefinitely with CPU at 100% w

dmr1122 May 27, 2009 12:43 PM (in response to d-rock)

2009-05-27 10:06:33
Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.3-b02 mixed mode):

"ZipFile Lock Reaper" daemon prio=6 tid=0x000000002fd78400 nid=0x19d0 in Object.wait() [0x000000000861f000..0x000000000861f8e0]
 java.lang.Thread.State: TIMED_WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x00000000f5aae250> (a java.util.TaskQueue)
 at java.util.TimerThread.mainLoop(Timer.java:509)
 - locked <0x00000000f5aae250> (a java.util.TaskQueue)
 at java.util.TimerThread.run(Timer.java:462)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-87" daemon prio=6 tid=0x000000002fd79c00 nid=0x1d8c in Object.wait() [0x000000005069f000..0x000000005069f960]
 java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x00000000f010eaa8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
 at java.lang.Object.wait(Object.java:485)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
 - locked <0x00000000f010eaa8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-86" daemon prio=6 tid=0x000000002fd79400 nid=0x1e88 runnable [0x000000005059f000..0x000000005059f9e0]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x00000000efcd0398> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000000efcd0460> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-85" daemon prio=6 tid=0x000000002fd78c00 nid=0x15ec runnable [0x000000004fe9f000..0x000000004fe9fa60]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x00000000efa68fe0> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000000efa690a8> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-84" daemon prio=6 tid=0x000000002fd78000 nid=0x14f8 runnable [0x000000004ab9f000..0x000000004ab9fae0]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x00000000ef0fe198> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000000ef0fe260> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-83" daemon prio=6 tid=0x000000002fd77800 nid=0x142c runnable [0x000000005049f000..0x000000005049fae0]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x00000000ff3a4668> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000000ff3a4730> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-82" daemon prio=6 tid=0x000000002fd77000 nid=0x1f10 runnable [0x000000005039f000..0x000000005039f760]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x00000000ff72ed78> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000000ff72ee40> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-81" daemon prio=6 tid=0x000000002fd76800 nid=0x204 runnable [0x000000005029f000..0x000000005029f7e0]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x0000000101f28d40> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x0000000101f28e08> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-80" daemon prio=6 tid=0x000000002fd76400 nid=0x1b28 runnable [0x000000005019e000..0x000000005019f860]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketOutputStream.socketWrite0(Native Method)
 at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
 at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
 at com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(OutputRecord.java:295)
 at com.sun.net.ssl.internal.ssl.OutputRecord.write(OutputRecord.java:284)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:734)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:722)
 at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
 - locked <0x00000000fae5be80> (a com.sun.net.ssl.internal.ssl.AppOutputStream)
 at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:724)
 at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:449)
 at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)
 at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:748)
 at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:126)
 at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:557)
 at org.apache.coyote.Response.doWrite(Response.java:567)
 at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:394)
 at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:449)
 at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:338)
 at org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:299)
 at org.apache.catalina.connector.Response.finishResponse(Response.java:493)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - <0x00000000fae5bcf0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"http-10.0.100.151-443-79" daemon prio=6 tid=0x000000003167b400 nid=0x1a4c runnable [0x000000005009f000..0x000000005009f8e0]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x0000000104812f40> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x000000010483cd60> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-78" daemon prio=6 tid=0x000000000c551000 nid=0x1e2c runnable [0x000000000762f000..0x000000000762f960]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x00000000f137be60> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000000f137bf28> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"AWT-Windows" daemon prio=6 tid=0x000000003167a400 nid=0x1fb8 runnable [0x000000004ff9f000..0x000000004ff9fae0]
 java.lang.Thread.State: RUNNABLE
 at sun.awt.windows.WToolkit.eventLoop(Native Method)
 at sun.awt.windows.WToolkit.run(WToolkit.java:291)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"Java2D Disposer" daemon prio=10 tid=0x000000000c552800 nid=0x1874 in Object.wait() [0x000000004824f000..0x000000004824f7e0]
 java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x00000000a284f0f0> (a java.lang.ref.ReferenceQueue$Lock)
 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
 - locked <0x00000000a284f0f0> (a java.lang.ref.ReferenceQueue$Lock)
 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
 at sun.java2d.Disposer.run(Disposer.java:125)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-77" daemon prio=6 tid=0x0000000016612400 nid=0xe64 runnable [0x000000004a89f000..0x000000004a89f860]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x0000000100a29af8> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x0000000100a29bc0> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-76" daemon prio=6 tid=0x000000003167a000 nid=0x105c runnable [0x000000004fd9f000..0x000000004fd9f8e0]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x00000000ff72d840> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000000ff72d908> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-75" daemon prio=6 tid=0x0000000031679800 nid=0x1194 runnable [0x000000004fc9f000..0x000000004fc9f960]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x00000000f8d74230> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000000f8d742f8> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-74" daemon prio=6 tid=0x0000000031679000 nid=0x1974 in Object.wait() [0x000000004fb9f000..0x000000004fb9f9e0]
 java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x00000000a0799cd0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
 at java.lang.Object.wait(Object.java:485)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
 - locked <0x00000000a0799cd0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-73" daemon prio=6 tid=0x0000000031678800 nid=0x1bf0 runnable [0x000000004fa9f000..0x000000004fa9fa60]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x00000000efa67a80> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000000efa67b48> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-72" daemon prio=6 tid=0x0000000031678400 nid=0x1b4 runnable [0x000000004f99e000..0x000000004f99fae0]
 java.lang.Thread.State: RUNNABLE
 at org.apache.catalina.connector.CoyoteReader.readLine(CoyoteReader.java:158)
 at com.hometracker.integration.gateway.app.eAppraiseIT.AppraisalReceiver.doPost(Unknown Source)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
 at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)
 at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
 at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
 at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
 at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
 at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
 at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
 at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:383)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-71" daemon prio=6 tid=0x000000003098cc00 nid=0x1ec4 runnable [0x000000004f89d000..0x000000004f89f760]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketOutputStream.socketWrite0(Native Method)
 at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
 at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
 at com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(OutputRecord.java:295)
 at com.sun.net.ssl.internal.ssl.OutputRecord.write(OutputRecord.java:284)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:734)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:722)
 at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
 - locked <0x0000000102896158> (a com.sun.net.ssl.internal.ssl.AppOutputStream)
 at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:724)
 at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:449)
 at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)
 at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:748)
 at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:126)
 at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:557)
 at org.apache.coyote.Response.doWrite(Response.java:567)
 at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:394)
 at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:449)
 at org.apache.catalina.connector.OutputBuffer.realWriteChars(OutputBuffer.java:465)
 at org.apache.tomcat.util.buf.CharChunk.append(CharChunk.java:305)
 at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:497)
 at org.apache.catalina.connector.CoyoteWriter.write(CoyoteWriter.java:143)
 at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:119)
 at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:326)
 at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:342)
 at org.apache.jsp.ac.forms.CaseContact_jsp._jspService(CaseContact_jsp.java:2809)
 at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:369)
 at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:322)
 at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:249)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:638)
 at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:444)
 at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:382)
 at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:310)
 at com.hometracker.common.framework.RouterServlet.forwardToPage(Unknown Source)
 at com.hometracker.common.framework.RouterServlet.processRequest(Unknown Source)
 at com.hometracker.common.framework.RouterServlet.service(Unknown Source)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
 at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)
 at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
 at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
 at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
 at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
 at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
 at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
 at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:383)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - <0x0000000102895fc8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"http-10.0.100.151-443-70" daemon prio=6 tid=0x000000003098c400 nid=0x2cc runnable [0x000000004f79e000..0x000000004f79f7e0]
 java.lang.Thread.State: RUNNABLE
 at org.apache.catalina.connector.CoyoteReader.readLine(CoyoteReader.java:158)
 at com.hometracker.integration.gateway.app.eAppraiseIT.AppraisalReceiver.doPost(Unknown Source)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
 at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)
 at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
 at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
 at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
 at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
 at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
 at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
 at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:383)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-69" daemon prio=6 tid=0x000000003098bc00 nid=0x1140 runnable [0x000000004f69f000..0x000000004f69f860]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x00000001048be2d8> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000001048bfe38> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-68" daemon prio=6 tid=0x000000003098b400 nid=0x10c runnable [0x000000004f59f000..0x000000004f59f8e0]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x0000000100a2b058> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x0000000100a2b120> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-67" daemon prio=6 tid=0x00000000289e2c00 nid=0x48c runnable [0x000000004f09f000..0x000000004f09f960]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x00000000ff1e4588> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000000ff1e4650> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-66" daemon prio=6 tid=0x000000003098b000 nid=0x1998 in Object.wait() [0x000000004ef9f000..0x000000004ef9f9e0]
 java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x00000000a087f640> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
 at java.lang.Object.wait(Object.java:485)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
 - locked <0x00000000a087f640> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-65" daemon prio=6 tid=0x000000003098a800 nid=0x1f1c runnable [0x000000004f49f000..0x000000004f49fa60]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x00000000ea33a760> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000000ea33a828> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-64" daemon prio=6 tid=0x000000003098a000 nid=0x1934 runnable [0x000000004f39e000..0x000000004f39fae0]
 java.lang.Thread.State: RUNNABLE
 at org.apache.catalina.connector.CoyoteReader.readLine(CoyoteReader.java:158)
 at com.hometracker.integration.gateway.app.eAppraiseIT.AppraisalReceiver.doPost(Unknown Source)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
 at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)
 at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
 at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
 at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
 at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
 at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
 at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
 at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:383)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-63" daemon prio=6 tid=0x0000000030989800 nid=0xd9c runnable [0x000000004f29f000..0x000000004f29f760]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x00000000ef3b44c0> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000000ef3b4588> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-62" daemon prio=6 tid=0x0000000030989400 nid=0x1d28 in Object.wait() [0x000000004f19f000..0x000000004f19f7e0]
 java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x00000000a091d210> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
 at java.lang.Object.wait(Object.java:485)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
 - locked <0x00000000a091d210> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-61" daemon prio=6 tid=0x00000000289e2400 nid=0xde0 runnable [0x000000004ed9f000..0x000000004ed9f960]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x00000000f92d2110> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000000f92d21d8> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-60" daemon prio=6 tid=0x00000000289e2000 nid=0x1d3c in Object.wait() [0x000000004ec9f000..0x000000004ec9f9e0]
 java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x00000000a0935f68> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
 at java.lang.Object.wait(Object.java:485)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
 - locked <0x00000000a0935f68> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-59" daemon prio=6 tid=0x00000000289e1800 nid=0xc98 runnable [0x000000004eb9f000..0x000000004eb9fa60]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x0000000104972318> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x0000000104997fb8> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-58" daemon prio=6 tid=0x00000000289e1000 nid=0x1924 runnable [0x000000004ea9f000..0x000000004ea9fae0]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x00000000f87f6170> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000000f87f6238> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-57" daemon prio=6 tid=0x00000000289e0800 nid=0x1e44 runnable [0x000000004e99f000..0x000000004e99f760]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x00000000ff9a3138> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000000ff9a3200> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-56" daemon prio=6 tid=0x00000000289e0400 nid=0x16e8 runnable [0x000000004e89f000..0x000000004e89f7e0]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x00000000f7a846a0> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000000f7a84768> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-55" daemon prio=6 tid=0x0000000028e9ec00 nid=0x1228 runnable [0x000000004e79f000..0x000000004e79f860]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x00000000e64dd180> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000000e64dd248> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-54" daemon prio=6 tid=0x0000000028e9e400 nid=0x1a50 runnable [0x000000004e69f000..0x000000004e69f8e0]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x0000000102052420> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000001020524e8> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-53" daemon prio=6 tid=0x0000000028e9dc00 nid=0x1ce4 in Object.wait() [0x000000004e59f000..0x000000004e59f960]
 java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x00000000a0a95758> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
 at java.lang.Object.wait(Object.java:485)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
 - locked <0x00000000a0a95758> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-52" daemon prio=6 tid=0x0000000028e9d400 nid=0x1fa0 runnable [0x000000004e49f000..0x000000004e49f9e0]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x00000000ff1e5ac0> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000000ff1e5b88> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-51" daemon prio=6 tid=0x0000000028e9d000 nid=0xc10 runnable [0x000000004e39f000..0x000000004e39fa60]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x0000000104a80e40> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x0000000104a80ac8> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-50" daemon prio=6 tid=0x0000000028e9c800 nid=0x238 runnable [0x000000004e29f000..0x000000004e29fae0]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x00000000a0b09d70> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000000a0b196e0> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-49" daemon prio=6 tid=0x0000000028e9c000 nid=0x1d7c in Object.wait() [0x000000004e19f000..0x000000004e19f760]
 java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x00000000a0b393d8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
 at java.lang.Object.wait(Object.java:485)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
 - locked <0x00000000a0b393d8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-48" daemon prio=6 tid=0x0000000028e9b800 nid=0x1d00 runnable [0x000000004e09f000..0x000000004e09f7e0]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x0000000104a4f4b8> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x0000000104a4f140> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-47" daemon prio=6 tid=0x0000000028e9b400 nid=0xb38 in Object.wait() [0x000000004df9f000..0x000000004df9f860]
 java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x00000000a0b9a070> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
 at java.lang.Object.wait(Object.java:485)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
 - locked <0x00000000a0b9a070> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-46" daemon prio=6 tid=0x000000002c5fbc00 nid=0xa34 runnable [0x000000004de9f000..0x000000004de9f8e0]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x00000000f23793a0> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000000f2379468> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-45" daemon prio=6 tid=0x000000002c5fb400 nid=0x18f8 runnable [0x000000004dd9f000..0x000000004dd9f960]
 java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
 at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
 - locked <0x00000000f87f4c38> (a java.lang.Object)
 at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:746)
 at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
 - locked <0x00000000f87f4d00> (a com.sun.net.ssl.internal.ssl.AppInputStream)
 at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
 at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:619)

 Locked ownable synchronizers:
 - None

"http-10.0.100.151-443-44" daemon prio=6 tid=0x000000002c5fac00 nid=0x16d4 runnable [0x000000004dc9e000..0x000000004dc9f9e0]
 java.lang.Thread.State: RUNNABLE
 at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:309)
 - locked <0x00000001054fb7f8> (a org.apache.jasper.servlet.JspServletWrapper)
 at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:322)
 at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:249)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:638)
 at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:444)
 at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:382)
 at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:310)
 at com.hometracker.common.framework.RouterServlet.forwardToPage(Unknown Source)
 at com.hometracker.common.framework.RouterServlet.processRequest(Unknown Source)
 at com.hometracker.common.framework.RouterServlet.service(Unknown Source)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
 at org.apache.catalina.core.Stand

Go to original post