4 Replies Latest reply on May 27, 2010 8:21 AM by mariqn nenchev

    Jboss out of memory problem

    mariqn nenchev Novice

      Hi,  i have serious problem with jboss out of memory. Please see the log and advice me how to fix it.

      <code>

      2010-05-24 05:45:42,649 ERROR [org.jgroups.protocols.TCP] (OOB-11043,10.35.12.2:7900) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
          at java.util.HashMap.entrySet0(HashMap.java:945)
          at java.util.HashMap.entrySet(HashMap.java:940)
          at java.util.concurrent.ConcurrentHashMap.putAll(ConcurrentHashMap.java:908)
          at org.jgroups.util.Digest.readFrom(Digest.java:379)
          at org.jgroups.util.Util.readStreamable(Util.java:645)
          at org.jgroups.protocols.pbcast.STABLE$StableHeader.readFrom(STABLE.java:771)
          at org.jgroups.Message.readHeader(Message.java:779)
          at org.jgroups.Message.readFrom(Message.java:673)
          at org.jgroups.protocols.TP.readMessage(TP.java:1514)
          at org.jgroups.protocols.TP.access$700(TP.java:49)
          at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1816)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:619)
      2010-05-24 05:46:10,264 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/clairews].[AccountService]] (http-10.35.12.2-8080-1) Servlet.service() for servlet AccountService threw exception
      java.lang.OutOfMemoryError: Java heap space
          at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:45)
          at java.lang.StringBuilder.<init>(StringBuilder.java:68)
          at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.getPathName(AbstractVirtualFileHandler.java:219)
          at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.readLocalPathName(AbstractVirtualFileHandler.java:260)
          at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.getLocalPathName(AbstractVirtualFileHandler.java:244)
          at org.jboss.virtual.plugins.context.zip.ZipEntryContext.initEntries(ZipEntryContext.java:583)
          at org.jboss.virtual.plugins.context.zip.ZipEntryContext.ensureEntries(ZipEntryContext.java:619)
          at org.jboss.virtual.plugins.context.zip.ZipEntryContext.checkIfModified(ZipEntryContext.java:773)
          at org.jboss.virtual.plugins.context.zip.ZipEntryContext.getChild(ZipEntryContext.java:817)
          at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.createChildHandler(ZipEntryHandler.java:191)
          at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(AbstractVirtualFileHandler.java:684)
          at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getChild(ZipEntryHandler.java:165)
          at org.jboss.virtual.plugins.context.DelegatingHandler.getChild(DelegatingHandler.java:107)
          at org.jboss.virtual.VirtualFile.findChild(VirtualFile.java:457)
          at org.jboss.virtual.plugins.vfs.VirtualFileURLConnection.resolveVirtualFile(VirtualFileURLConnection.java:106)
          at org.jboss.virtual.plugins.vfs.VirtualFileURLConnection.getVirtualFile(VirtualFileURLConnection.java:118)
          at org.jboss.virtual.plugins.vfs.VirtualFileURLConnection.getInputStream(VirtualFileURLConnection.java:93)
          at java.net.URL.openStream(URL.java:1010)
          at java.lang.ClassLoader.getResourceAsStream(ClassLoader.java:1170)
          at org.jboss.ws.core.utils.JBossWSEntityResolver$1.run(JBossWSEntityResolver.java:80)
          at org.jboss.ws.core.utils.JBossWSEntityResolver$1.run(JBossWSEntityResolver.java:78)
          at java.security.AccessController.doPrivileged(Native Method)
          at org.jboss.ws.core.utils.JBossWSEntityResolver.loadEntitiesMappingFromClasspath(JBossWSEntityResolver.java:76)
          at org.jboss.ws.core.utils.JBossWSEntityResolver.<init>(JBossWSEntityResolver.java:59)
          at org.jboss.ws.core.utils.JBossWSEntityResolver.<init>(JBossWSEntityResolver.java:51)
          at sun.reflect.GeneratedConstructorAccessor365.newInstance(Unknown Source)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
          at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
          at java.lang.Class.newInstance0(Class.java:355)
          at java.lang.Class.newInstance(Class.java:308)
          at org.jboss.wsf.common.DOMUtils$1.setEntityResolver(DOMUtils.java:121)
          at org.jboss.wsf.common.DOMUtils$1.initialValue(DOMUtils.java:101)
      2010-05-24 08:33:58,387 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/clairews].[AccountService]] (http-10.35.12.2-8080-1) Servlet.service() for servlet AccountService threw exception
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 08:36:12,916 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/clairews].[AccountService]] (http-10.35.12.2-8080-1) Servlet.service() for servlet AccountService threw exception
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 08:38:57,558 ERROR [STDERR] (ZipFile Lock Reaper) Exception in thread "ZipFile Lock Reaper"
      2010-05-24 08:38:57,558 ERROR [STDERR] (ZipFile Lock Reaper) java.lang.OutOfMemoryError: Java heap space
      2010-05-24 08:38:57,558 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/clairews].[AccountService]] (http-10.35.12.2-8080-6) Servlet.service() for servlet AccountService threw exception
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 08:40:05,175 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/clairews].[AccountService]] (http-10.35.12.2-8080-10) Servlet.service() for servlet AccountService threw exception
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 08:41:21,031 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/clairews].[AccountService]] (http-10.35.12.2-8080-2) Servlet.service() for servlet AccountService threw exception
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 08:43:49,700 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/clairews].[AccountService]] (http-10.35.12.2-8080-10) Servlet.service() for servlet AccountService threw exception
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 08:46:10,133 ERROR [org.jgroups] (OOB-11369,10.35.12.2:7900) uncaught exception in Thread[OOB-11369,10.35.12.2:7900,5,Thread Pools] (thread group=org.jgroups.util.Util$1[name=JGroups,maxpri=10] )
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 08:46:18,313 ERROR [STDERR] (DefaultQuartzScheduler_QuartzSchedulerThread) Exception in thread "DefaultQuartzScheduler_QuartzSchedulerThread"
      2010-05-24 08:46:18,313 ERROR [org.jgroups.protocols.TCP] (OOB-11370,10.35.12.2:7900) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 08:46:22,306 ERROR [STDERR] (DefaultQuartzScheduler_QuartzSchedulerThread) java.lang.OutOfMemoryError: Java heap space
      2010-05-24 08:50:45,047 ERROR [STDERR] (AOPListner) Exception in thread "AOPListner"
      2010-05-24 08:50:45,047 ERROR [STDERR] (AOPListner) java.lang.OutOfMemoryError: Java heap space
      2010-05-24 08:51:37,051 ERROR [org.jgroups.util.TimeScheduler] (Timer-2,10.35.12.2:7900) failed running task org.jgroups.protocols.pbcast.STABLE$StableTask@971e07
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 08:51:37,051 WARN  [org.jgroups.protocols.pbcast.NAKACK] (Timer-1,10.35.12.2:7900) failure passing message down
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 08:51:37,051 WARN  [org.jgroups.protocols.pbcast.NAKACK] (Timer-3,10.35.12.2:7900) failure passing message down
      java.lang.reflect.UndeclaredThrowableException
          at org.apache.commons.logging.impl.Log4jProxy.log(Log4jProxy.java:309)
          at org.apache.commons.logging.impl.Log4jProxy.error(Log4jProxy.java:283)
          at org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:208)
          at org.jgroups.protocols.TP.down(TP.java:1205)
          at org.jgroups.protocols.TP$ProtocolAdapter.down(TP.java:2316)
          at org.jgroups.protocols.Discovery.down(Discovery.java:374)
          at org.jgroups.protocols.MERGE2.down(MERGE2.java:175)
          at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:362)
          at org.jgroups.protocols.FD.down(FD.java:315)
          at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:95)
          at org.jgroups.protocols.BARRIER.down(BARRIER.java:107)
          at org.jgroups.protocols.pbcast.NAKACK.send(NAKACK.java:810)
          at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:612)
          at org.jgroups.protocols.UNICAST.down(UNICAST.java:462)
          at org.jgroups.protocols.pbcast.STABLE$1.run(STABLE.java:644)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:619)
      Caused by: java.lang.OutOfMemoryError: Java heap space
      2010-05-24 08:51:37,054 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/clairews].[AccountService]] (http-10.35.12.2-8080-1) Servlet.service() for servlet AccountService threw exception
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 08:53:56,807 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/clairews].[AccountService]] (http-10.35.12.2-8080-2) Servlet.service() for servlet AccountService threw exception
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 09:27:05,114 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/clairews].[AccountService]] (http-10.35.12.2-8080-2) Servlet.service() for servlet AccountService threw exception
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 09:32:00,768 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/clairews].[AccountService]] (http-10.35.12.2-8080-2) Servlet.service() for servlet AccountService threw exception
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 09:52:40,557 ERROR [STDERR] (main-FastReceiver) Exception in thread "main-FastReceiver"
      2010-05-24 09:52:40,557 ERROR [STDERR] (main-FastReceiver) java.lang.OutOfMemoryError: Java heap space
      2010-05-24 09:52:40,557 WARN  [org.jgroups.protocols.pbcast.NAKACK] (Timer-11,10.35.12.2:36017) failure passing message down
      java.lang.reflect.UndeclaredThrowableException
          at org.apache.commons.logging.impl.Log4jProxy.log(Log4jProxy.java:309)
          at org.apache.commons.logging.impl.Log4jProxy.error(Log4jProxy.java:283)
          at org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:208)
          at org.jgroups.protocols.TP.down(TP.java:1205)
          at org.jgroups.protocols.TP$ProtocolAdapter.down(TP.java:2316)
          at org.jgroups.protocols.Discovery.down(Discovery.java:374)
          at org.jgroups.protocols.MERGE2.down(MERGE2.java:175)
          at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:362)
          at org.jgroups.protocols.FD.down(FD.java:315)
          at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:95)
          at org.jgroups.protocols.pbcast.NAKACK.send(NAKACK.java:810)
          at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:612)
          at org.jgroups.protocols.UNICAST.down(UNICAST.java:462)
          at org.jgroups.protocols.pbcast.STABLE$StabilitySendTask.run(STABLE.java:855)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:619)
      Caused by: java.lang.OutOfMemoryError: Java heap space
      2010-05-24 09:52:40,558 WARN  [org.jboss.system.server.profileservice.hotdeploy.HDScanner] (HDScanner) Scan failed
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 09:52:46,653 ERROR [STDERR] (SubscriptionWatchDog) Exception in thread "SubscriptionWatchDog"
      2010-05-24 09:52:46,653 ERROR [STDERR] (SubscriptionWatchDog) java.lang.OutOfMemoryError: Java heap space
      2010-05-24 09:53:12,414 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/clairews].[AccountService]] (http-10.35.12.2-8080-1) Servlet.service() for servlet AccountService threw exception
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 09:53:12,415 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/clairews].[AccountService]] (http-10.35.12.2-8080-2) Servlet.service() for servlet AccountService threw exception
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 09:53:12,416 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/clairews].[AccountService]] (http-10.35.12.2-8080-10) Servlet.service() for servlet AccountService threw exception
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 09:58:12,185 ERROR [org.jgroups] (OOB--28175,10.35.12.2:36017) uncaught exception in Thread[OOB--28175,10.35.12.2:36017,5,Thread Pools] (thread group=org.jgroups.util.Util$1[name=JGroups,maxpri=10] )
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 11:18:01,497 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/clairews].[AccountService]] (http-10.35.12.2-8080-2) Servlet.service() for servlet AccountService threw exception
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 11:19:40,988 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/clairews].[AccountService]] (http-10.35.12.2-8080-10) Servlet.service() for servlet AccountService threw exception
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 13:47:27,892 ERROR [org.jgroups.protocols.UDP] (Timer-6,10.35.12.2:36017) failed sending message to null (162 bytes)
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 13:48:03,810 ERROR [org.jgroups.util.TimeScheduler] (Timer-1,10.35.12.2:36017) failed running task org.jgroups.protocols.pbcast.STABLE$StableTask@5b54e6
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 13:47:55,635 ERROR [STDERR] (RMI TCP Connection(idle)) Exception in thread "RMI TCP Connection(idle)"
      2010-05-24 13:48:23,481 ERROR [org.jgroups.util.TimeScheduler] (Timer-1,10.35.12.2:7900) failed running task org.jgroups.protocols.pbcast.STABLE$StableTask@971e07
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 13:57:44,626 ERROR [STDERR] (RMI TCP Connection(idle)) java.lang.OutOfMemoryError: Java heap space
      2010-05-24 13:58:04,391 ERROR [STDERR] (RMI RenewClean-[10.35.12.2:1101]) Exception in thread "RMI RenewClean-[10.35.12.2:1101]"
      2010-05-24 14:00:58,286 ERROR [STDERR] (RMI RenewClean-[10.35.12.2:1101]) java.lang.OutOfMemoryError: Java heap space
      2010-05-24 14:01:02,788 ERROR [STDERR] (Thread-19) Exception in thread "Thread-19"
      2010-05-24 14:01:02,788 ERROR [STDERR] (Thread-19) java.lang.OutOfMemoryError: Java heap space
      2010-05-24 14:11:26,037 ERROR [org.jgroups] (OOB--27597,10.35.12.2:36017) uncaught exception in Thread[OOB--27597,10.35.12.2:36017,5,Thread Pools] (thread group=org.jgroups.util.Util$1[name=JGroups,maxpri=10] )
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 14:37:32,020 ERROR [org.jgroups.protocols.UDP] (Timer-4,10.35.12.2:36017) failed sending message to null (162 bytes)
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 14:40:17,091 ERROR [org.jgroups.protocols.UDP] (OOB-1,PojoCacheCluster,10.35.12.2:36454) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 14:40:17,091 ERROR [org.jgroups.util.TimeScheduler] (Timer-1,PojoCacheCluster,10.35.12.2:36454) failed running task org.jgroups.protocols.pbcast.STABLE$StableTask@102eca7
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 14:39:10,051 WARN  [org.jgroups.protocols.pbcast.NAKACK] (Timer-10,10.35.12.2:36017) failure passing message down
      java.lang.reflect.UndeclaredThrowableException
          at org.apache.commons.logging.impl.Log4jProxy.log(Log4jProxy.java:309)
          at org.apache.commons.logging.impl.Log4jProxy.error(Log4jProxy.java:283)
          at org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:208)
          at org.jgroups.protocols.TP.down(TP.java:1205)
          at org.jgroups.protocols.TP$ProtocolAdapter.down(TP.java:2316)
          at org.jgroups.protocols.Discovery.down(Discovery.java:374)
          at org.jgroups.protocols.MERGE2.down(MERGE2.java:175)
          at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:362)
          at org.jgroups.protocols.FD.down(FD.java:315)
          at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:95)
          at org.jgroups.protocols.BARRIER.down(BARRIER.java:107)
          at org.jgroups.protocols.pbcast.NAKACK.send(NAKACK.java:810)
          at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:612)
          at org.jgroups.protocols.UNICAST.down(UNICAST.java:462)
          at org.jgroups.protocols.pbcast.STABLE$1.run(STABLE.java:644)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:207)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:619)
      Caused by: java.lang.OutOfMemoryError: Java heap space
      2010-05-24 14:38:11,297 WARN  [org.jboss.system.server.profileservice.hotdeploy.HDScanner] (HDScanner) Scan failed
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 14:40:24,878 ERROR [STDERR] (main-FastReceiver) Exception in thread "main-FastReceiver"
      2010-05-24 14:40:24,878 ERROR [STDERR] (main-FastReceiver) java.lang.OutOfMemoryError: Java heap space
      2010-05-24 14:40:24,878 ERROR [org.jgroups.protocols.UDP] (Timer-1,10.35.12.2:36017) failed sending message to null (159 bytes)
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 15:01:05,206 ERROR [org.jgroups] (OOB--27554,10.35.12.2:36017) uncaught exception in Thread[OOB--27554,10.35.12.2:36017,5,Thread Pools] (thread group=org.jgroups.util.Util$1[name=JGroups,maxpri=10] )
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 15:22:13,935 ERROR [org.jgroups.protocols.UDP] (Timer-3,PojoCacheCluster,10.35.12.2:36454) failed sending message to null (115 bytes)
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 15:23:40,386 WARN  [org.jboss.system.server.profileservice.hotdeploy.HDScanner] (HDScanner) Scan failed
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 15:22:57,050 ERROR [org.jgroups.protocols.TCP] (Timer-4,10.35.12.2:7900) failed handling data from 10.35.12.2:7900
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 15:23:56,089 WARN  [org.jgroups.protocols.pbcast.NAKACK] (Timer-3,10.35.12.2:7900) failure passing message down
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 15:23:56,089 ERROR [org.apache.catalina.core.ContainerBase] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Exception invoking periodic operation:
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 15:25:23,135 ERROR [org.jgroups.util.TimeScheduler] (Timer-2,10.35.12.2:7900) failed running task org.jgroups.protocols.pbcast.STABLE$StableTask@971e07
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 15:40:25,476 ERROR [STDERR] (main-FastReceiver) Exception in thread "main-FastReceiver"
      2010-05-24 15:40:25,477 ERROR [STDERR] (main-FastReceiver) java.lang.OutOfMemoryError: Java heap space
      2010-05-24 15:40:29,681 WARN  [org.jboss.system.server.profileservice.hotdeploy.HDScanner] (HDScanner) Scan failed
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 15:40:29,681 ERROR [org.jgroups.protocols.UDP] (OOB-1,PojoCacheCluster,10.35.12.2:36454) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 16:15:02,031 WARN  [org.jboss.system.server.profileservice.hotdeploy.HDScanner] (HDScanner) Scan failed
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 16:18:53,850 ERROR [org.jgroups.protocols.TCP] (Timer-3,10.35.12.2:7900) failed sending message to null (157 bytes)
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 16:23:06,322 ERROR [org.jgroups.protocols.UDP] (Timer-2,10.35.12.2:36017) failed sending message to null (162 bytes)
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 16:45:29,456 ERROR [STDERR] (MaintThread) Exception in thread "MaintThread"
      2010-05-24 16:51:13,905 ERROR [org.jgroups.protocols.UDP] (OOB-2,PojoCacheCluster,10.35.12.2:36454) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 16:50:20,769 WARN  [org.jgroups.protocols.pbcast.NAKACK] (Timer-1,10.35.12.2:36017) failure passing message down
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 16:50:05,042 ERROR [org.jgroups.protocols.UDP] (Timer-6,10.35.12.2:36017) failed sending message to null (159 bytes)
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 16:52:41,962 ERROR [STDERR] (MaintThread) java.lang.OutOfMemoryError: Java heap space
      2010-05-24 16:51:50,978 ERROR [org.jgroups.protocols.UDP] (UDP mcast,10.35.12.2:36017) failure in multicast receive()
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 16:51:27,529 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/clairews].[AccountService]] (http-10.35.12.2-8080-10) Servlet.service() for servlet AccountService threw exception
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:01:36,224 ERROR [STDERR] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Exception in thread "ContainerBackgroundProcessor[StandardEngine[jboss.web]]"
      2010-05-24 17:01:08,522 WARN  [org.jgroups.protocols.pbcast.NAKACK] (Timer-10,10.35.12.2:36017) failure passing message down
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:01:04,625 WARN  [org.jgroups.protocols.pbcast.NAKACK] (Timer-12,10.35.12.2:36017) failure passing message down
      java.lang.reflect.UndeclaredThrowableException
      Caused by: java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:00:01,752 ERROR [org.jgroups.protocols.UDP] (Timer-3,10.35.12.2:36017) failed sending message to null (166 bytes)
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 16:59:38,208 ERROR [org.jgroups.util.TimeScheduler] (Timer-1,10.35.12.2:36017) failed running task org.jgroups.protocols.pbcast.STABLE$StableTask@16243cc
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 16:59:22,557 ERROR [org.jgroups.util.TimeScheduler] (Timer-7,10.35.12.2:36017) failed running task org.jgroups.protocols.pbcast.STABLE$StableTask@4ae89c
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 16:56:02,315 ERROR [org.jgroups.protocols.UDP] (Timer-8,10.35.12.2:36017) failed sending message to null (149 bytes)
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 16:55:46,480 ERROR [org.jgroups.util.TimeScheduler] (Timer-2,10.35.12.2:7900) failed running task org.jgroups.protocols.pbcast.STABLE$StableTask@971e07
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 16:54:55,295 WARN  [org.jgroups.protocols.pbcast.NAKACK] (Timer-9,10.35.12.2:36017) failure passing message down
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:04:06,370 ERROR [org.jgroups] (OOB--27477,10.35.12.2:36017) uncaught exception in Thread[OOB--27477,10.35.12.2:36017,5,Thread Pools] (thread group=org.jgroups.util.Util$1[name=JGroups,maxpri=10] )
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:03:38,521 ERROR [STDERR] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:03:34,589 ERROR [org.apache.catalina.connector.CoyoteAdapter] (http-10.35.12.2-8080-10) An exception or error occurred in the container during the request processing
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:03:10,945 ERROR [org.jgroups] (OOB--27476,10.35.12.2:36017) uncaught exception in Thread[OOB--27476,10.35.12.2:36017,5,Thread Pools] (thread group=org.jgroups.util.Util$1[name=JGroups,maxpri=10] )
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:02:39,344 WARN  [org.jgroups.protocols.pbcast.NAKACK] (Timer-6,10.35.12.2:36017) failure passing message down
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:05:21,532 ERROR [org.apache.coyote.http11.Http11Processor] (http-10.35.12.2-8080-10) Error finishing response
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:05:25,520 ERROR [org.jgroups] (OOB--27481,10.35.12.2:36017) uncaught exception in Thread[OOB--27481,10.35.12.2:36017,5,Thread Pools] (thread group=org.jgroups.util.Util$1[name=JGroups,maxpri=10] )
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:09:53,916 ERROR [org.jgroups.protocols.UDP] (OOB--27444,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:09:06,621 ERROR [org.jgroups.protocols.UDP] (OOB--27452,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:10:53,122 ERROR [org.jgroups.protocols.UDP] (OOB--27445,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:10:49,204 ERROR [org.jgroups.protocols.UDP] (UDP mcast,10.35.12.2:36017) failed handling data from 10.35.12.2:45688
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:10:23,686 ERROR [org.jgroups.protocols.UDP] (OOB--27447,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:15:59,459 ERROR [org.jgroups.protocols.UDP] (Timer-13,10.35.12.2:36017) failed sending message to null (149 bytes)
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:14:45,996 ERROR [org.jgroups.protocols.UDP] (OOB--27441,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:14:20,096 ERROR [org.jgroups.protocols.UDP] (UDP mcast,10.35.12.2:36017) failure in multicast receive()
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:13:32,725 ERROR [org.jgroups.protocols.UDP] (Timer-8,10.35.12.2:36017) failed sending message to null (149 bytes)
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:13:04,946 ERROR [org.jgroups.util.TimeScheduler] (Timer-5,10.35.12.2:7900) failed running task org.jgroups.protocols.pbcast.STABLE$StableTask@971e07
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:17:03,015 ERROR [org.jgroups.protocols.UDP] (Timer-12,10.35.12.2:36017) failed sending message to null (159 bytes)
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:18:45,255 ERROR [org.jgroups.protocols.UDP] (OOB--27434,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:19:17,081 ERROR [org.jgroups.protocols.UDP] (OOB--27438,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:19:52,439 ERROR [org.jgroups.protocols.UDP] (OOB--27437,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:21:23,758 ERROR [org.jgroups.protocols.UDP] (OOB--27438,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:23:06,921 ERROR [org.jgroups.protocols.UDP] (OOB--27433,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:22:43,100 ERROR [org.jgroups.protocols.UDP] (UDP mcast,10.35.12.2:36017) failed handling data from 10.35.12.2:45688
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:24:24,145 ERROR [org.jgroups.protocols.UDP] (OOB--27431,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:25:51,226 ERROR [org.jgroups.protocols.UDP] (UDP mcast,10.35.12.2:36017) failure in multicast receive()
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:27:55,549 ERROR [org.jgroups.protocols.UDP] (UDP mcast,10.35.12.2:36017) failure in multicast receive()
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:28:23,154 ERROR [org.jgroups] (Timer-5,10.35.12.2:7900) uncaught exception in Thread[Timer-5,10.35.12.2:7900,5,JGroups] (thread group=org.jgroups.util.Util$1[name=JGroups,maxpri=10] )
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:31:24,198 ERROR [org.jgroups.protocols.UDP] (OOB--27413,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:34:17,547 ERROR [org.jgroups.protocols.UDP] (OOB--27410,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:34:33,301 ERROR [org.jgroups.protocols.UDP] (OOB--27409,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:36:31,858 ERROR [org.jgroups.protocols.UDP] (OOB--27400,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:36:55,345 ERROR [org.jgroups.protocols.UDP] (OOB--27408,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:42:10,395 ERROR [org.jgroups] (OOB--27402,10.35.12.2:36017) uncaught exception in Thread[OOB--27402,10.35.12.2:36017,10,Thread Pools] (thread group=org.jgroups.util.Util$1[name=JGroups,maxpri=10] )
      java.lang.reflect.UndeclaredThrowableException
          at org.apache.commons.logging.impl.Log4jProxy.log(Log4jProxy.java:309)
          at org.apache.commons.logging.impl.Log4jProxy.error(Log4jProxy.java:283)
          at org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:208)
          at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1822)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:619)
      Caused by: java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:42:34,103 ERROR [org.jgroups] (OOB--27401,10.35.12.2:36017) uncaught exception in Thread[OOB--27401,10.35.12.2:36017,10,Thread Pools] (thread group=org.jgroups.util.Util$1[name=JGroups,maxpri=10] )
      java.lang.reflect.UndeclaredThrowableException
      Caused by: java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:47:02,382 ERROR [org.jgroups.protocols.UDP] (OOB--27383,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:48:09,072 ERROR [org.jgroups.protocols.UDP] (OOB--27378,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:50:00,694 ERROR [org.jgroups.protocols.UDP] (OOB--27381,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:52:12,690 ERROR [org.jgroups.protocols.UDP] (OOB--27373,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:52:20,540 ERROR [org.jgroups.protocols.UDP] (OOB--27375,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:52:36,204 ERROR [org.jgroups.protocols.UDP] (UDP mcast,10.35.12.2:36017) failed handling data from 10.35.12.2:45688
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:52:44,023 ERROR [org.jgroups] (OOB--27374,10.35.12.2:36017) uncaught exception in Thread[OOB--27374,10.35.12.2:36017,10,Thread Pools] (thread group=org.jgroups.util.Util$1[name=JGroups,maxpri=10] )
      java.lang.reflect.UndeclaredThrowableException
          at org.apache.commons.logging.impl.Log4jProxy.log(Log4jProxy.java:309)
          at org.apache.commons.logging.impl.Log4jProxy.error(Log4jProxy.java:283)
          at org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:208)
          at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1822)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:619)
      Caused by: java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:58:10,635 ERROR [org.jgroups.protocols.UDP] (OOB--27343,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:58:24,413 ERROR [org.jgroups.protocols.UDP] (UDP mcast,10.35.12.2:36017) failed handling data from 10.35.12.2:45688
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 17:58:32,252 ERROR [org.jgroups.protocols.UDP] (OOB--27342,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:04:04,192 ERROR [org.jgroups.protocols.UDP] (OOB--27303,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:04:19,853 ERROR [org.jgroups.protocols.UDP] (OOB--27301,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:08:17,667 ERROR [org.apache.coyote.ajp.AjpProtocol] (ajp-10.35.12.2-8009-119) Error reading request, ignored
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:08:21,572 ERROR [STDERR] (ajp-10.35.12.2-8009-119) Exception in thread "ajp-10.35.12.2-8009-119"
      2010-05-24 18:08:23,539 ERROR [STDERR] (ajp-10.35.12.2-8009-119) java.lang.NullPointerException
      2010-05-24 18:08:23,540 ERROR [STDERR] (ajp-10.35.12.2-8009-119)     at java.util.concurrent.ConcurrentLinkedQueue.offer(ConcurrentLinkedQueue.java:190)
      2010-05-24 18:08:31,350 ERROR [STDERR] (ajp-10.35.12.2-8009-119)     at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler$1.offer(AjpProtocol.java:340)
      2010-05-24 18:08:35,240 ERROR [STDERR] (ajp-10.35.12.2-8009-119)     at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler$1.offer(AjpProtocol.java:357)
      2010-05-24 18:08:39,150 ERROR [STDERR] (ajp-10.35.12.2-8009-119)     at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:411)
      2010-05-24 18:08:43,037 ERROR [STDERR] (ajp-10.35.12.2-8009-119)     at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
      2010-05-24 18:08:46,923 ERROR [STDERR] (ajp-10.35.12.2-8009-119)     at java.lang.Thread.run(Thread.java:619)
      2010-05-24 18:14:06,936 ERROR [org.jgroups.protocols.UDP] (OOB--27245,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:14:38,295 ERROR [org.jgroups.protocols.UDP] (OOB--27247,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:14:53,904 ERROR [org.jgroups.protocols.UDP] (OOB--27244,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:15:23,779 ERROR [org.jgroups.protocols.UDP] (OOB--27253,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:15:35,530 ERROR [org.jgroups.protocols.UDP] (OOB--27243,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:19:47,259 ERROR [org.jgroups.protocols.UDP] (OOB--27222,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:19:43,334 ERROR [org.jgroups.protocols.UDP] (OOB--27225,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:19:23,525 ERROR [org.jgroups.protocols.UDP] (OOB--27224,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:18:39,965 ERROR [org.jgroups.protocols.UDP] (OOB--27227,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:20:26,717 ERROR [org.jgroups.protocols.UDP] (UDP mcast,10.35.12.2:36017) failure in multicast receive()
      java.lang.reflect.UndeclaredThrowableException
          at org.apache.commons.logging.impl.Log4jProxy.log(Log4jProxy.java:309)
          at org.apache.commons.logging.impl.Log4jProxy.error(Log4jProxy.java:283)
          at org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:208)
          at org.jgroups.protocols.TP.receive(TP.java:1352)
          at org.jgroups.protocols.UDP.run(UDP.java:277)
          at java.lang.Thread.run(Thread.java:619)
      Caused by: java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:20:03,093 ERROR [org.jgroups.protocols.UDP] (OOB--27223,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:22:13,444 ERROR [org.jgroups.protocols.UDP] (OOB--27221,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:22:09,512 ERROR [org.jgroups.protocols.UDP] (UDP mcast,10.35.12.2:36017) failed handling data from 10.35.12.2:45688
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:23:31,990 ERROR [org.jgroups.protocols.UDP] (OOB--27221,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:24:11,474 ERROR [org.jgroups.protocols.UDP] (UDP mcast,10.35.12.2:36017) failed handling data from 10.35.12.2:45688
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:24:37,003 ERROR [org.jgroups.protocols.UDP] (OOB--27217,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:26:45,395 ERROR [org.jgroups.protocols.UDP] (OOB--27215,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:26:21,830 ERROR [org.jgroups.protocols.UDP] (OOB--27217,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:25:58,054 ERROR [org.jgroups.protocols.UDP] (OOB--27212,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:25:40,400 ERROR [org.jgroups.protocols.UDP] (OOB--27214,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:28:09,591 ERROR [org.jgroups.protocols.UDP] (UDP mcast,10.35.12.2:36017) failure in multicast receive()
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:32:00,601 ERROR [org.jgroups.protocols.UDP] (OOB--27191,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:32:49,716 ERROR [org.jgroups.protocols.UDP] (OOB--27192,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:33:33,170 ERROR [org.jgroups.protocols.UDP] (OOB--27190,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:33:17,408 ERROR [org.jgroups.protocols.UDP] (OOB--27191,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:34:28,134 ERROR [org.jgroups.protocols.UDP] (OOB--27188,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:34:47,695 ERROR [org.jgroups.protocols.UDP] (OOB--27189,10.35.12.2:36017) failed handling incoming message
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:54:02,004 ERROR [org.apache.coyote.ajp.AjpProtocol] (ajp-10.35.12.2-8009-120) Error reading request, ignored
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 18:54:05,926 ERROR [STDERR] (ajp-10.35.12.2-8009-120) Exception in thread "ajp-10.35.12.2-8009-120"
      2010-05-24 18:54:05,926 ERROR [STDERR] (ajp-10.35.12.2-8009-120) java.lang.NullPointerException
      2010-05-24 18:54:09,819 ERROR [STDERR] (ajp-10.35.12.2-8009-120)     at java.util.concurrent.ConcurrentLinkedQueue.offer(ConcurrentLinkedQueue.java:190)
      2010-05-24 18:54:15,695 ERROR [STDERR] (ajp-10.35.12.2-8009-120)     at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler$1.offer(AjpProtocol.java:340)
      2010-05-24 18:54:17,642 ERROR [STDERR] (ajp-10.35.12.2-8009-120)     at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler$1.offer(AjpProtocol.java:357)
      2010-05-24 18:54:25,463 ERROR [STDERR] (ajp-10.35.12.2-8009-120)     at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:411)
      2010-05-24 18:54:29,374 ERROR [STDERR] (ajp-10.35.12.2-8009-120)     at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
      2010-05-24 18:54:29,374 ERROR [STDERR] (ajp-10.35.12.2-8009-120)     at java.lang.Thread.run(Thread.java:619)
      2010-05-24 19:43:20,437 ERROR [org.apache.coyote.ajp.AjpProtocol] (ajp-10.35.12.2-8009-121) Error reading request, ignored
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 19:43:24,339 ERROR [STDERR] (ajp-10.35.12.2-8009-121) Exception in thread "ajp-10.35.12.2-8009-121"
      2010-05-24 19:43:24,339 ERROR [STDERR] (ajp-10.35.12.2-8009-121) java.lang.NullPointerException
      2010-05-24 19:43:28,273 ERROR [STDERR] (ajp-10.35.12.2-8009-121)     at java.util.concurrent.ConcurrentLinkedQueue.offer(ConcurrentLinkedQueue.java:190)
      2010-05-24 19:43:34,183 ERROR [STDERR] (ajp-10.35.12.2-8009-121)     at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler$1.offer(AjpProtocol.java:340)
      2010-05-24 19:43:36,140 ERROR [STDERR] (ajp-10.35.12.2-8009-121)     at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler$1.offer(AjpProtocol.java:357)
      2010-05-24 19:43:43,994 ERROR [STDERR] (ajp-10.35.12.2-8009-121)     at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:411)
      2010-05-24 19:43:47,909 ERROR [STDERR] (ajp-10.35.12.2-8009-121)     at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
      2010-05-24 19:43:47,909 ERROR [STDERR] (ajp-10.35.12.2-8009-121)     at java.lang.Thread.run(Thread.java:619)
      2010-05-24 20:27:58,265 ERROR [org.apache.coyote.ajp.AjpProtocol] (ajp-10.35.12.2-8009-122) Error reading request, ignored
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 20:28:02,352 ERROR [STDERR] (ajp-10.35.12.2-8009-122) Exception in thread "ajp-10.35.12.2-8009-122"
      2010-05-24 20:28:02,352 ERROR [STDERR] (ajp-10.35.12.2-8009-122) java.lang.NullPointerException
      2010-05-24 20:28:06,295 ERROR [STDERR] (ajp-10.35.12.2-8009-122)     at java.util.concurrent.ConcurrentLinkedQueue.offer(ConcurrentLinkedQueue.java:190)
      2010-05-24 20:28:12,153 ERROR [STDERR] (ajp-10.35.12.2-8009-122)     at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler$1.offer(AjpProtocol.java:340)
      2010-05-24 20:28:14,091 ERROR [STDERR] (ajp-10.35.12.2-8009-122)     at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler$1.offer(AjpProtocol.java:357)
      2010-05-24 20:28:21,995 ERROR [STDERR] (ajp-10.35.12.2-8009-122)     at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:411)
      2010-05-24 20:28:25,916 ERROR [STDERR] (ajp-10.35.12.2-8009-122)     at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
      2010-05-24 20:28:25,916 ERROR [STDERR] (ajp-10.35.12.2-8009-122)     at java.lang.Thread.run(Thread.java:619)
      2010-05-24 21:05:46,185 ERROR [org.apache.coyote.ajp.AjpProtocol] (ajp-10.35.12.2-8009-123) Error reading request, ignored
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 21:05:50,081 ERROR [STDERR] (ajp-10.35.12.2-8009-123) Exception in thread "ajp-10.35.12.2-8009-123"
      2010-05-24 21:05:50,081 ERROR [STDERR] (ajp-10.35.12.2-8009-123) java.lang.NullPointerException
      2010-05-24 21:05:53,955 ERROR [STDERR] (ajp-10.35.12.2-8009-123)     at java.util.concurrent.ConcurrentLinkedQueue.offer(ConcurrentLinkedQueue.java:190)
      2010-05-24 21:05:59,773 ERROR [STDERR] (ajp-10.35.12.2-8009-123)     at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler$1.offer(AjpProtocol.java:340)
      2010-05-24 21:06:01,741 ERROR [STDERR] (ajp-10.35.12.2-8009-123)     at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler$1.offer(AjpProtocol.java:357)
      2010-05-24 21:06:09,585 ERROR [STDERR] (ajp-10.35.12.2-8009-123)     at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:411)
      2010-05-24 21:06:13,509 ERROR [STDERR] (ajp-10.35.12.2-8009-123)     at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
      2010-05-24 21:06:13,509 ERROR [STDERR] (ajp-10.35.12.2-8009-123)     at java.lang.Thread.run(Thread.java:619)
      2010-05-24 22:48:59,264 ERROR [org.apache.coyote.ajp.AjpProtocol] (ajp-10.35.12.2-8009-124) Error reading request, ignored
      java.lang.OutOfMemoryError: Java heap space
      2010-05-24 22:49:03,360 ERROR [STDERR] (ajp-10.35.12.2-8009-124) Exception in thread "ajp-10.35.12.2-8009-124"
      2010-05-24 22:49:03,360 ERROR [STDERR] (ajp-10.35.12.2-8009-124) java.lang.NullPointerException
      2010-05-24 22:49:05,342 ERROR [STDERR] (ajp-10.35.12.2-8009-124)     at java.util.concurrent.ConcurrentLinkedQueue.offer(ConcurrentLinkedQueue.java:190)
      2010-05-24 22:49:09,224 ERROR [STDERR] (ajp-10.35.12.2-8009-124)     at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler$1.offer(AjpProtocol.java:340)
      2010-05-24 22:49:13,150 ERROR [STDERR] (ajp-10.35.12.2-8009-124)     at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler$1.offer(AjpProtocol.java:357)
      2010-05-24 22:49:20,948 ERROR [STDERR] (ajp-10.35.12.2-8009-124)     at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:411)
      2010-05-24 22:49:20,948 ERROR [STDERR] (ajp-10.35.12.2-8009-124)     at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
      2010-05-24 22:49:26,821 ERROR [STDERR] (ajp-10.35.12.2-8009-124)     at java.lang.Thread.run(Thread.java:619)

      </code>