0 Replies Latest reply on Dec 7, 2009 2:07 AM by prabhun

    JBoss 4.2.3 hangs or sometime stops

    prabhun
    prabhun Dec 7, 2009 2:07 AM

      Hi,

      We have JBoss 4.2 3 server that is configured with JBoss MQ. We have some components which communicate by JBoss MQ.

      Frequently this server either hangs or stops. Sometime we get outofmemoryexceptions, I am not sure what may be the reason for this.

      I have pasted the log below.

      2009-10-16 12:56:08,231 ERROR [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Exception in JMSCI message listener
javax.ejb.TransactionRolledbackLocalException: Unexpected Error
java.lang.OutOfMemoryError: unable to create new native thread
 at java.lang.Thread.start0(Native Method)
 at java.lang.Thread.start(Thread.java:574)
 at sun.rmi.transport.DGCClient$EndpointEntry.<init>(DGCClient.java:223)
 at sun.rmi.transport.DGCClient$EndpointEntry.lookup(DGCClient.java:193)
 at sun.rmi.transport.DGCClient.registerRefs(DGCClient.java:111)
 at sun.rmi.transport.LiveRef.read(LiveRef.java:277)
 at sun.rmi.server.UnicastRef2.readExternal(UnicastRef2.java:54)
 at java.rmi.server.RemoteObject.readObject(RemoteObject.java:438)
 at sun.reflect.GeneratedMethodAccessor177.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:946)
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1818)
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1718)
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1304)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:349)
 at java.rmi.MarshalledObject.get(MarshalledObject.java:135)
 at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:257)
 at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1533)
 at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:634)
 at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627)
 at javax.naming.InitialContext.lookup(InitialContext.java:351)
 at uk.co.firstchoice.cyrus.server.jmx.locator.RemoteJBossServerLocator.locate(Unknown Source)
 at uk.co.firstchoice.cyrus.server.seasoncontroller.SeasonControlFacade.getRow(SeasonControlFacade.java:148)
 at uk.co.firstchoice.cyrus.server.seasoncontroller.SeasonControlFacade.getOnSaleDate(SeasonControlFacade.java:396)
 at uk.co.firstchoice.cyrus.server.consensus.InventoryItemFacade.getOnSaleDay(InventoryItemFacade.java:76)
 at uk.co.firstchoice.cyrus.server.consensus.InventoryItemFacade.getOnSalePeriod(InventoryItemFacade.java:54)
 at uk.co.firstchoice.cyrus.server.consensus.ConsensusBean.createConsensus(ConsensusBean.java:298)
 at uk.co.firstchoice.cyrus.server.consensusengine.ConsensusEngineConsumerBean.process(ConsensusEngineConsumerBean.java:204)
 at uk.co.firstchoice.cyrus.framework.engine.AbstractJmsEngine.onMessage(AbstractJmsEngine.java:251)
 at sun.reflect.GeneratedMethodAccessor297.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
 at org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:495)
 at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
 at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:116)
 at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
 at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
 at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
 at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
 at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:109)
 at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
 at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
 at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:402)
 at org.jboss.ejb.Container.invoke(Container.java:960)
 at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:1092)
 at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onMessage(JMSContainerInvoker.java:1392)
 at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:266)
 at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:906)
 at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:170)
 at org.jboss.mq.SpySession.run(SpySession.java:323)
 at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194)
 at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761)
 at java.lang.Thread.run(Thread.java:595)

 at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:262)
 at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
 at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
 at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:109)
 at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
 at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
 at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:402)
 at org.jboss.ejb.Container.invoke(Container.java:960)
 at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:1092)
 at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onMessage(JMSContainerInvoker.java:1392)
 at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:266)
 at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:906)
 at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:170)
 at org.jboss.mq.SpySession.run(SpySession.java:323)
 at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194)
 at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761)
 at java.lang.Thread.run(Thread.java:595)
Caused by: javax.ejb.EJBException: Unexpected Error
java.lang.OutOfMemoryError: unable to create new native thread
 at java.lang.Thread.start0(Native Method)
 at java.lang.Thread.start(Thread.java:574)
 at sun.rmi.transport.DGCClient$EndpointEntry.<init>(DGCClient.java:223)
 at sun.rmi.transport.DGCClient$EndpointEntry.lookup(DGCClient.java:193)
 at sun.rmi.transport.DGCClient.registerRefs(DGCClient.java:111)
 at sun.rmi.transport.LiveRef.read(LiveRef.java:277)
 at sun.rmi.server.UnicastRef2.readExternal(UnicastRef2.java:54)
 at java.rmi.server.RemoteObject.readObject(RemoteObject.java:438)
 at sun.reflect.GeneratedMethodAccessor177.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:946)
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1818)
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1718)
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1304)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:349)
 at java.rmi.MarshalledObject.get(MarshalledObject.java:135)
 at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:257)
 at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1533)
 at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:634)
 at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627)
 at javax.naming.InitialContext.lookup(InitialContext.java:351)
 at uk.co.firstchoice.cyrus.server.jmx.locator.RemoteJBossServerLocator.locate(Unknown Source)
 at uk.co.firstchoice.cyrus.server.seasoncontroller.SeasonControlFacade.getRow(SeasonControlFacade.java:148)
 at uk.co.firstchoice.cyrus.server.seasoncontroller.SeasonControlFacade.getOnSaleDate(SeasonControlFacade.java:396)
 at uk.co.firstchoice.cyrus.server.consensus.InventoryItemFacade.getOnSaleDay(InventoryItemFacade.java:76)
 at uk.co.firstchoice.cyrus.server.consensus.InventoryItemFacade.getOnSalePeriod(InventoryItemFacade.java:54)
 at uk.co.firstchoice.cyrus.server.consensus.ConsensusBean.createConsensus(ConsensusBean.java:298)
 at uk.co.firstchoice.cyrus.server.consensusengine.ConsensusEngineConsumerBean.process(ConsensusEngineConsumerBean.java:204)
 at uk.co.firstchoice.cyrus.framework.engine.AbstractJmsEngine.onMessage(AbstractJmsEngine.java:251)
 at sun.reflect.GeneratedMethodAccessor297.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
 at org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:495)
 at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
 at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:116)
 at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
 at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
 at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
 at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
 at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:109)
 at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
 at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
 at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:402)
 at org.jboss.ejb.Container.invoke(Container.java:960)
 at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:1092)
 at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onMessage(JMSContainerInvoker.java:1392)
 at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:266)
 at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:906)
 at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:170)
 at org.jboss.mq.SpySession.run(SpySession.java:323)
 at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194)
 at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761)
 at java.lang.Thread.run(Thread.java:595)

 at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:237)
 ... 16 more



      Also sometimes we get the below socket time out exception: 

      
2009-10-16 12:57:08,365 ERROR [uk.co.firstchoice.cyrus.server.consensus.ConsensusBean] Couldn't look up consensus weights.
java.lang.reflect.UndeclaredThrowableException
 at uk.co.firstchoice.cyrus.server.jmx.locator.RemoteJBossServerLocator.locate(Unknown Source)
 at uk.co.firstchoice.cyrus.server.seasoncontroller.SeasonControlFacade.getRow(SeasonControlFacade.java:148)
 at uk.co.firstchoice.cyrus.server.seasoncontroller.SeasonControlFacade.getOnSaleDate(SeasonControlFacade.java:396)
 at uk.co.firstchoice.cyrus.server.consensus.InventoryItemFacade.getOnSaleDay(InventoryItemFacade.java:76)
 at uk.co.firstchoice.cyrus.server.consensus.InventoryItemFacade.getOnSalePeriod(InventoryItemFacade.java:54)
 at uk.co.firstchoice.cyrus.server.consensus.ConsensusBean.createConsensus(ConsensusBean.java:298)
 at uk.co.firstchoice.cyrus.server.consensusengine.ConsensusEngineConsumerBean.process(ConsensusEngineConsumerBean.java:204)
 at uk.co.firstchoice.cyrus.framework.engine.AbstractJmsEngine.onMessage(AbstractJmsEngine.java:251)
 at sun.reflect.GeneratedMethodAccessor297.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
 at org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:495)
 at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
 at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:116)
 at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
 at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
 at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
 at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
 at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:109)
 at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
 at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
 at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:402)
 at org.jboss.ejb.Container.invoke(Container.java:960)
 at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:1092)
 at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onMessage(JMSContainerInvoker.java:1392)
 at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:266)
 at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:906)
 at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:170)
 at org.jboss.mq.SpySession.run(SpySession.java:323)
 at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194)
 at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:761)
 at java.lang.Thread.run(Thread.java:595)
Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
 java.net.SocketTimeoutException: Read timed out]
 at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:780)
 at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627)
 at javax.naming.InitialContext.lookup(InitialContext.java:351)
 ... 33 more
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
 java.net.SocketTimeoutException: Read timed out
 at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:273)
 at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:171)
 at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:94)
 at org.jnp.server.NamingServer_Stub.lookup(Unknown Source)
 at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:667)
 ... 35 more
Caused by: java.net.SocketTimeoutException: Read timed out
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
 at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
 at java.io.DataInputStream.readByte(DataInputStream.java:241)
 at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:215)
 ... 39 more




      Please let me know what may the reason for this failure.

      I have also taken trace details when the server was hangling. I have used pstack and jstack commands.

      Based on the review of this I will attach those files in the next post.
      How to attach a file in this forum?

      Thanks,
      Regards,
      Prabhu

      • 2139 Views
      • Tags: