JBoss 4.2.3 hangs or sometime stops
prabhun Dec 7, 2009 2:07 AMHi,
We have JBoss 4.2 3 server that is configured with JBoss MQ. We have some components which communicate by JBoss MQ.
Frequently this server either hangs or stops. Sometime we get outofmemoryexceptions, I am not sure what may be the reason for this.
I have pasted the log below.
2009-10-16 12:56:08,231 ERROR [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Exception in JMSCI message listener javax.ejb.TransactionRolledbackLocalException: Unexpected Error java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:574) at sun.rmi.transport.DGCClient$EndpointEntry.<init>(DGCClient.java:223) at sun.rmi.transport.DGCClient$EndpointEntry.lookup(DGCClient.java:193) at sun.rmi.transport.DGCClient.registerRefs(DGCClient.java:111) at sun.rmi.transport.LiveRef.read(LiveRef.java:277) at sun.rmi.server.UnicastRef2.readExternal(UnicastRef2.java:54) at java.rmi.server.RemoteObject.readObject(RemoteObject.java:438) at sun.reflect.GeneratedMethodAccessor177.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:946) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1818) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1718) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1304) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:349) at java.rmi.MarshalledObject.get(MarshalledObject.java:135) at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:257) at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1533) at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:634) at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627) at javax.naming.InitialContext.lookup(InitialContext.java:351) at uk.co.firstchoice.cyrus.server.jmx.locator.RemoteJBossServerLocator.locate(Unknown Source) at uk.co.firstchoice.cyrus.server.seasoncontroller.SeasonControlFacade.getRow(SeasonControlFacade.java:148) at uk.co.firstchoice.cyrus.server.seasoncontroller.SeasonControlFacade.getOnSaleDate(SeasonControlFacade.java:396) at uk.co.firstchoice.cyrus.server.consensus.InventoryItemFacade.getOnSaleDay(InventoryItemFacade.java:76) at uk.co.firstchoice.cyrus.server.consensus.InventoryItemFacade.getOnSalePeriod(InventoryItemFacade.java:54) at uk.co.firstchoice.cyrus.server.consensus.ConsensusBean.createConsensus(ConsensusBean.java:298) at uk.co.firstchoice.cyrus.server.consensusengine.ConsensusEngineConsumerBean.process(ConsensusEngineConsumerBean.java:204) at uk.co.firstchoice.cyrus.framework.engine.AbstractJmsEngine.onMessage(AbstractJmsEngine.java:251) at sun.reflect.GeneratedMethodAccessor297.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.invocation.Invocation.performCall(Invocation.java:359) at org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:495) at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158) at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:116) at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63) at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121) at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350) at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181) at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:109) at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205) at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138) at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:402) at org.jboss.ejb.Container.invoke(Container.java:960) at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:1092) at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onMessage(JMSContainerInvoker.java:1392) at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:266) at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:906) at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:170) at org.jboss.mq.SpySession.run(SpySession.java:323) at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761) at java.lang.Thread.run(Thread.java:595) at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:262) at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350) at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181) at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:109) at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205) at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138) at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:402) at org.jboss.ejb.Container.invoke(Container.java:960) at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:1092) at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onMessage(JMSContainerInvoker.java:1392) at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:266) at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:906) at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:170) at org.jboss.mq.SpySession.run(SpySession.java:323) at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761) at java.lang.Thread.run(Thread.java:595) Caused by: javax.ejb.EJBException: Unexpected Error java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:574) at sun.rmi.transport.DGCClient$EndpointEntry.<init>(DGCClient.java:223) at sun.rmi.transport.DGCClient$EndpointEntry.lookup(DGCClient.java:193) at sun.rmi.transport.DGCClient.registerRefs(DGCClient.java:111) at sun.rmi.transport.LiveRef.read(LiveRef.java:277) at sun.rmi.server.UnicastRef2.readExternal(UnicastRef2.java:54) at java.rmi.server.RemoteObject.readObject(RemoteObject.java:438) at sun.reflect.GeneratedMethodAccessor177.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:946) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1818) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1718) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1304) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:349) at java.rmi.MarshalledObject.get(MarshalledObject.java:135) at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:257) at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1533) at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:634) at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627) at javax.naming.InitialContext.lookup(InitialContext.java:351) at uk.co.firstchoice.cyrus.server.jmx.locator.RemoteJBossServerLocator.locate(Unknown Source) at uk.co.firstchoice.cyrus.server.seasoncontroller.SeasonControlFacade.getRow(SeasonControlFacade.java:148) at uk.co.firstchoice.cyrus.server.seasoncontroller.SeasonControlFacade.getOnSaleDate(SeasonControlFacade.java:396) at uk.co.firstchoice.cyrus.server.consensus.InventoryItemFacade.getOnSaleDay(InventoryItemFacade.java:76) at uk.co.firstchoice.cyrus.server.consensus.InventoryItemFacade.getOnSalePeriod(InventoryItemFacade.java:54) at uk.co.firstchoice.cyrus.server.consensus.ConsensusBean.createConsensus(ConsensusBean.java:298) at uk.co.firstchoice.cyrus.server.consensusengine.ConsensusEngineConsumerBean.process(ConsensusEngineConsumerBean.java:204) at uk.co.firstchoice.cyrus.framework.engine.AbstractJmsEngine.onMessage(AbstractJmsEngine.java:251) at sun.reflect.GeneratedMethodAccessor297.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.invocation.Invocation.performCall(Invocation.java:359) at org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:495) at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158) at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:116) at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63) at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121) at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350) at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181) at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:109) at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205) at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138) at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:402) at org.jboss.ejb.Container.invoke(Container.java:960) at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:1092) at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onMessage(JMSContainerInvoker.java:1392) at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:266) at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:906) at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:170) at org.jboss.mq.SpySession.run(SpySession.java:323) at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761) at java.lang.Thread.run(Thread.java:595) at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:237) ... 16 more
Also sometimes we get the below socket time out exception:
2009-10-16 12:57:08,365 ERROR [uk.co.firstchoice.cyrus.server.consensus.ConsensusBean] Couldn't look up consensus weights. java.lang.reflect.UndeclaredThrowableException at uk.co.firstchoice.cyrus.server.jmx.locator.RemoteJBossServerLocator.locate(Unknown Source) at uk.co.firstchoice.cyrus.server.seasoncontroller.SeasonControlFacade.getRow(SeasonControlFacade.java:148) at uk.co.firstchoice.cyrus.server.seasoncontroller.SeasonControlFacade.getOnSaleDate(SeasonControlFacade.java:396) at uk.co.firstchoice.cyrus.server.consensus.InventoryItemFacade.getOnSaleDay(InventoryItemFacade.java:76) at uk.co.firstchoice.cyrus.server.consensus.InventoryItemFacade.getOnSalePeriod(InventoryItemFacade.java:54) at uk.co.firstchoice.cyrus.server.consensus.ConsensusBean.createConsensus(ConsensusBean.java:298) at uk.co.firstchoice.cyrus.server.consensusengine.ConsensusEngineConsumerBean.process(ConsensusEngineConsumerBean.java:204) at uk.co.firstchoice.cyrus.framework.engine.AbstractJmsEngine.onMessage(AbstractJmsEngine.java:251) at sun.reflect.GeneratedMethodAccessor297.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.invocation.Invocation.performCall(Invocation.java:359) at org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:495) at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158) at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:116) at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63) at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121) at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350) at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181) at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:109) at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205) at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138) at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:402) at org.jboss.ejb.Container.invoke(Container.java:960) at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:1092) at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onMessage(JMSContainerInvoker.java:1392) at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:266) at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:906) at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:170) at org.jboss.mq.SpySession.run(SpySession.java:323) at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761) at java.lang.Thread.run(Thread.java:595) Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: java.net.SocketTimeoutException: Read timed out] at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:780) at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627) at javax.naming.InitialContext.lookup(InitialContext.java:351) ... 33 more Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: java.net.SocketTimeoutException: Read timed out at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:273) at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:171) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:94) at org.jnp.server.NamingServer_Stub.lookup(Unknown Source) at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:667) ... 35 more Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:235) at java.io.DataInputStream.readByte(DataInputStream.java:241) at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:215) ... 39 more
Please let me know what may the reason for this failure.
I have also taken trace details when the server was hangling. I have used pstack and jstack commands.
Based on the review of this I will attach those files in the next post.
How to attach a file in this forum?
Thanks,
Regards,
Prabhu