1 Reply Latest reply on Jul 12, 2016 10:17 AM by jbertram

java.lang.OutOfMemoryError: Java heap space- Hornet Queue.

alif Jul 12, 2016 5:12 AM

Hi All,

I am using Jboss 7.1.1. I have deployed EAR project in Jboss. The jboss is running quite fine around 10 days. After that I am getting some Hornet queue Java out of memory issue and server is getting down.

Please find the below log details and please help.

2016-07-09 15:59:03,148 INFO [com.optum.cm2.crypto.EncryptedDataSource] (MyScheduler_Worker-1) Loaded JDBC driver: oracle.jdbc.driver.OracleDriver
2016-07-09 15:59:07,032 INFO [com.optum.cm2.crypto.EncryptedDataSource] (MyScheduler_Worker-1) Loaded JDBC driver: oracle.jdbc.driver.OracleDriver
2016-07-09 16:00:03,256 INFO [com.optum.cm2.crypto.EncryptedDataSource] (MyScheduler_Worker-3) Loaded JDBC driver: oracle.jdbc.driver.OracleDriver
2016-07-09 16:00:03,256 INFO [com.optum.cm2.crypto.EncryptedDataSource] (MyScheduler_QuartzSchedulerThread) Loaded JDBC driver: oracle.jdbc.driver.OracleDriver
2016-07-09 16:00:03,256 INFO [com.optum.cm2.crypto.EncryptedDataSource] (MyScheduler_Worker-2) Loaded JDBC driver: oracle.jdbc.driver.OracleDriver
2016-07-09 16:00:18,623 INFO [com.optum.cm2.crypto.EncryptedDataSource] (MyScheduler_Worker-2) Loaded JDBC driver: oracle.jdbc.driver.OracleDriver
2016-07-09 16:15:00,059 INFO [com.optum.cm2.crypto.EncryptedDataSource] (MyScheduler_QuartzSchedulerThread) Loaded JDBC driver: oracle.jdbc.driver.OracleDriver
2016-07-09 16:30:00,044 INFO [com.optum.cm2.crypto.EncryptedDataSource] (MyScheduler_QuartzSchedulerThread) Loaded JDBC driver: oracle.jdbc.driver.OracleDriver
2016-07-09 16:45:00,045 INFO [com.optum.cm2.crypto.EncryptedDataSource] (MyScheduler_QuartzSchedulerThread) Loaded JDBC driver: oracle.jdbc.driver.OracleDriver
2016-07-09 16:59:03,253 INFO [com.optum.cm2.crypto.EncryptedDataSource] (MyScheduler_Worker-1) Loaded JDBC driver: oracle.jdbc.driver.OracleDriver
2016-07-09 16:59:18,605 INFO [com.optum.cm2.crypto.EncryptedDataSource] (MyScheduler_Worker-1) Loaded JDBC driver: oracle.jdbc.driver.OracleDriver
2016-07-09 17:00:17,222 INFO [com.optum.cm2.crypto.EncryptedDataSource] (MyScheduler_Worker-2) Loaded JDBC driver: oracle.jdbc.driver.OracleDriver
2016-07-09 17:00:17,222 INFO [com.optum.cm2.crypto.EncryptedDataSource] (MyScheduler_QuartzSchedulerThread) Loaded JDBC driver: oracle.jdbc.driver.OracleDriver
2016-07-09 17:00:17,222 INFO [com.optum.cm2.crypto.EncryptedDataSource] (MyScheduler_Worker-3) Loaded JDBC driver: oracle.jdbc.driver.OracleDriver
2016-07-09 17:06:41,141 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-8) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:05:53,293 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-9) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space
at org.apache.coyote.http11.InternalOutputBuffer.<init>(InternalOutputBuffer.java:67)
at org.apache.coyote.http11.Http11Processor.<init>(Http11Processor.java:100)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.createProcessor(Http11Protocol.java:710)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:661)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930)
at java.lang.Thread.run(Thread.java:744)

2016-07-09 17:13:19,955 ERROR [org.hornetq.utils.OrderedExecutorFactory] (Thread-15640 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=8079e472-428e-11e6-aff2-005056b602c9-13290713-5175690)) Caught unexpected Throwable: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:28:57,913 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (Thread-14552 (HornetQ-client-global-threads-9524183)) Connection failure has been detected: Did not receive data from server for InVMConnection [serverID=0, id=842fc0c9-428e-11e6-aff2-005056b602c9] [code=3]
2016-07-09 17:07:03,575 ERROR [org.hornetq.utils.OrderedExecutorFactory] (Thread-15583 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=8079e472-428e-11e6-aff2-005056b602c9-13290713-5175690)) Caught unexpected Throwable: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:08:05,542 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-1) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:11:16,009 ERROR [org.hornetq.utils.OrderedExecutorFactory] (Thread-16 (HornetQ-server-HornetQServerImpl::serverUUID=8079e472-428e-11e6-aff2-005056b602c9-10244918)) Caught unexpected Throwable: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:19:32,135 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/].[default]] (http--0.0.0.0-8080-2) Servlet.service() for servlet default threw exception: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:17:21,747 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-6) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:28:57,913 ERROR [org.quartz.core.JobRunShell] (MyScheduler_Worker-2) Job DEFAULT.6da64b5bd2ee-deb3815a-f199-4205-b89a-a36658dd30b7 threw an unhandled Exception: : java.lang.NoClassDefFoundError: Could not initialize class org.springframework.beans.factory.BeanCreationException

2016-07-09 17:29:00,799 ERROR [org.quartz.core.ErrorLogger] (MyScheduler_Worker-2) Job (DEFAULT.6da64b5bd2ee-deb3815a-f199-4205-b89a-a36658dd30b7 threw an exception.: org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.NoClassDefFoundError: Could not initialize class org.springframework.beans.factory.BeanCreationException]
at org.quartz.core.JobRunShell.run(JobRunShell.java:224)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.springframework.beans.factory.BeanCreationException

2016-07-09 17:25:17,917 ERROR [org.hornetq.utils.OrderedExecutorFactory] (Thread-15644 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=8079e472-428e-11e6-aff2-005056b602c9-13290713-5175690)) Caught unexpected Throwable: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:30:16,087 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-39) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:29:44,324 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-27) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:29:27,351 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-46) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:30:24,527 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-31) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:28:57,929 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (Thread-14555 (HornetQ-client-global-threads-9524183)) Connection failure has been detected: Did not receive data from server for InVMConnection [serverID=0, id=842d4fbf-428e-11e6-aff2-005056b602c9] [code=3]
2016-07-09 17:17:21,747 ERROR [org.hornetq.utils.OrderedExecutorFactory] (Thread-15632 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=8079e472-428e-11e6-aff2-005056b602c9-13290713-5175690)) Caught unexpected Throwable: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:28:57,929 ERROR [org.apache.catalina.connector.CoyoteAdapter] (http--0.0.0.0-8080-5) An exception or error occurred in the container during the request processing: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:22:55,486 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-13) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:26:50,490 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (Thread-14550 (HornetQ-client-global-threads-9524183)) Connection failure has been detected: Did not receive data from server for InVMConnection [serverID=0, id=842d4fb3-428e-11e6-aff2-005056b602c9] [code=3]
2016-07-09 17:26:33,579 ERROR [org.hornetq.utils.OrderedExecutorFactory] (Thread-15645 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=8079e472-428e-11e6-aff2-005056b602c9-13290713-5175690)) Caught unexpected Throwable: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:28:57,913 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (Thread-14553 (HornetQ-client-global-threads-9524183)) Connection failure has been detected: Did not receive data from server for InVMConnection [serverID=0, id=842b05af-428e-11e6-aff2-005056b602c9] [code=3]
2016-07-09 17:36:53,834 ERROR [org.apache.coyote.http11.Http11Processor] (http--0.0.0.0-8080-29) Error processing request: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:33:33,494 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-49) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:39:41,959 ERROR [org.apache.catalina.connector.CoyoteAdapter] (http--0.0.0.0-8080-53) An exception or error occurred in the container during the request processing: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:38:45,782 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-15) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:28:57,913 ERROR [org.quartz.core.JobRunShell] (MyScheduler_Worker-3) Job DEFAULT.6da64b5bd2ee-7b815b65-b724-4a7a-a1c7-0a7277bb2903 threw an unhandled Exception: : java.lang.NoClassDefFoundError: Could not initialize class org.springframework.beans.factory.BeanCreationException
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:547)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:304)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:300)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:195)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:681)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:760)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)
at com.optum.cm2.scheduler.HttpRequestSchedular.execute(HttpRequestSchedular.java:24)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)

2016-07-09 17:33:25,038 ERROR [org.apache.coyote.http11.Http11Processor] (http--0.0.0.0-8080-30) Error processing request: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:34:15,708 ERROR [org.apache.coyote.http11.Http11Processor] (http--0.0.0.0-8080-48) Error processing request: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:35:34,787 ERROR [org.apache.coyote.http11.Http11Processor] (http--0.0.0.0-8080-25) Error processing request: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:36:05,738 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-45) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:34:44,023 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-51) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:34:41,121 ERROR [org.apache.coyote.http11.Http11Processor] (http--0.0.0.0-8080-47) Error processing request: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:34:29,718 ERROR [org.apache.coyote.http11.Http11Processor] (http--0.0.0.0-8080-17) Error processing request: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:33:05,273 ERROR [org.apache.coyote.http11.Http11Processor] (http--0.0.0.0-8080-42) Error processing request: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:30:49,971 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-32) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:30:58,333 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-4) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:35:23,617 ERROR [org.apache.coyote.http11.Http11Processor] (http--0.0.0.0-8080-41) Error processing request: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:31:18,098 ERROR [org.apache.coyote.http11.Http11Processor] (http--0.0.0.0-8080-19) Error processing request: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:36:28,343 ERROR [org.apache.coyote.http11.Http11Processor] (http--0.0.0.0-8080-44) Error processing request: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:29:35,822 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-14) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 18:01:00,503 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/].[default]] (http--0.0.0.0-8080-63) Servlet.service() for servlet default threw exception: java.lang.OutOfMemoryError: Java heap space

2016-07-09 18:02:16,289 ERROR [org.apache.coyote.http11.Http11Processor] (http--0.0.0.0-8080-47) Error processing request: java.lang.OutOfMemoryError: Java heap space

2016-07-09 18:02:27,490 ERROR [org.apache.coyote.http11.Http11Processor] (http--0.0.0.0-8080-48) Error processing request: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:58:17,182 ERROR [org.apache.catalina.connector.CoyoteAdapter] (http--0.0.0.0-8080-62) An exception or error occurred in the container during the request processing: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:54:20,150 ERROR [org.hornetq.utils.OrderedExecutorFactory] (Thread-8 (HornetQ-server-HornetQServerImpl::serverUUID=8079e472-428e-11e6-aff2-005056b602c9-10244918)) Caught unexpected Throwable: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:32:33,776 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-35) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:33:13,744 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-52) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:49:24,461 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-36) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:49:41,247 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-56) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:50:37,642 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-68) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:50:54,694 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-2) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:57:26,154 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-34) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 18:00:52,094 ERROR [org.quartz.simpl.SimpleThreadPool] (MyScheduler_Worker-3) Error while executing the Runnable: : java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:56:09,868 ERROR [org.apache.coyote.http11.Http11Processor] (http--0.0.0.0-8080-57) Error processing request: java.lang.OutOfMemoryError: Java heap space

2016-07-09 18:01:48,318 ERROR [org.apache.coyote.http11.Http11Processor] (http--0.0.0.0-8080-53) Error processing request: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:48:45,054 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-22) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:48:53,557 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-26) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:48:02,138 ERROR [org.apache.catalina.connector.CoyoteAdapter] (http--0.0.0.0-8080-21) An exception or error occurred in the container during the request processing: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:49:15,834 ERROR [org.apache.coyote.http11.Http11Processor] (http--0.0.0.0-8080-40) Error processing request: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:46:28,832 ERROR [org.apache.coyote.http11.Http11Processor] (http--0.0.0.0-8080-54) Error processing request: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:45:35,338 ERROR [org.apache.catalina.connector.CoyoteAdapter] (http--0.0.0.0-8080-16) An exception or error occurred in the container during the request processing: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:42:27,635 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-60) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:47:50,921 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-28) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:47:50,921 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-37) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:42:27,635 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-12) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:41:00,600 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-55) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 17:40:29,602 ERROR [org.apache.coyote.http11.Http11Protocol] (http--0.0.0.0-8080-59) Error reading request, ignored: java.lang.OutOfMemoryError: Java heap space

2016-07-09 18:07:36,705 ERROR [stderr] (Thread-0 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=8079e472-428e-11e6-aff2-005056b602c9-13290713-5175690)) Exception in thread "http--0.0.0.0-8080-8" Exception in thread "Timer-1" Exception in thread "Periodic Recovery" 8" Exception in thread "Thread-1 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=8079e472-428e-11e6-aff2-005056b602c9-13290713-5175690)" java.lang.NullPointerException

1. Re: java.lang.OutOfMemoryError: Java heap space- Hornet Queue.

jbertram Jul 12, 2016 10:17 AM (in response to alif)

Based on what you've provided there's no way of knowing whether or not the OOME has been caused by something related to HornetQ. You need to follow the normal trouble-shooting procedure for an OOME (e.g. analyze heap dumps, enable GC logging, set up JVM monitoring, look over your code to make sure you're closing all your resources, etc.). Once you've got some solid data indicating what is causing the OOME or at least where the accumulation is happening then you'll be more likely to get further help.
Actions