JBM server shutdown curiously when jboss cluster is running.
We have three nodes in our jboss cluster. We deployed our JBM server on one of three nodes. But in recent days we found our JBM server shutdown curiously when the jboss cluster is running. And all topic was closed.
We checked the server.log in the /server/default/log and found the following information:
2008-09-03 11:29:42,628 INFO [DRM Async Publisher#0] destination.QueueService ( QueueService.java:189) - Queue[/queue/DLQ] stopped 2008-09-03 11:29:42,630 INFO [DRM Async Publisher#0] destination.QueueService ( QueueService.java:189) - Queue[/queue/ExpiryQueue] stopped 2008-09-03 11:29:42,631 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/testTopic] stopped 2008-09-03 11:29:42,633 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/securedTopic] stopped 2008-09-03 11:29:42,634 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/testDurableTopic] stopped 2008-09-03 11:29:42,635 INFO [DRM Async Publisher#0] destination.QueueService ( QueueService.java:189) - Queue[/queue/testQueue] stopped 2008-09-03 11:29:42,636 INFO [DRM Async Publisher#0] destination.QueueService ( QueueService.java:189) - Queue[/queue/A] stopped 2008-09-03 11:29:42,637 INFO [DRM Async Publisher#0] destination.QueueService ( QueueService.java:189) - Queue[/queue/B] stopped 2008-09-03 11:29:42,638 INFO [DRM Async Publisher#0] destination.QueueService ( QueueService.java:189) - Queue[/queue/C] stopped 2008-09-03 11:29:42,640 INFO [DRM Async Publisher#0] destination.QueueService ( QueueService.java:189) - Queue[/queue/D] stopped 2008-09-03 11:29:42,641 INFO [DRM Async Publisher#0] destination.QueueService ( QueueService.java:189) - Queue[/queue/ex] stopped 2008-09-03 11:29:42,642 INFO [DRM Async Publisher#0] destination.QueueService ( QueueService.java:189) - Queue[/queue/PrivateDLQ] stopped 2008-09-03 11:29:42,644 INFO [DRM Async Publisher#0] destination.QueueService ( QueueService.java:189) - Queue[/queue/PrivateExpiryQueue] stopped 2008-09-03 11:29:42,645 INFO [DRM Async Publisher#0] destination.QueueService ( QueueService.java:189) - Queue[/queue/QueueWithOwnDLQAndExpiryQueue] stopped 2008-09-03 11:29:42,646 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/TopicWithOwnDLQAndExpiryQueue] stopped 2008-09-03 11:29:42,648 INFO [DRM Async Publisher#0] destination.QueueService ( QueueService.java:189) - Queue[/queue/QueueWithOwnRedeliveryDelay] stopped 2008-09-03 11:29:42,649 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/TopicWithOwnRedeliveryDelay] stopped 2008-09-03 11:29:42,650 INFO [DRM Async Publisher#0] destination.QueueService ( QueueService.java:189) - Queue[/queue/testDistributedQueue] stopped 2008-09-03 11:29:42,651 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/testDistributedTopic] stopped 2008-09-03 11:29:42,653 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/cmtopic] stopped 2008-09-03 11:29:42,709 ERROR [WorkerThread#110[192.166.32.132:3212]] socket.ServerThread ( ServerThread.java:377) - Worker thread initialization failure java.io.EOFException at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:529) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:373) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166) 2008-09-03 11:29:42,754 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/cmomttopic] stopped 2008-09-03 11:29:42,756 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/Fm4NeaTopic] stopped 2008-09-03 11:29:42,758 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/FmTopic] stopped 2008-09-03 11:29:42,760 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/FmOmtTopic] stopped 2008-09-03 11:29:42,864 INFO [JMS SessionPool Worker-2] server.AbstractCommonMDB ( AbstractCommonMDB.java:48) - [RouterMDB]receive msg id:50 sequenceId=79742 2008-09-03 11:29:42,866 INFO [JMS SessionPool Worker-2] server.AbstractCommonMDB ( AbstractCommonMDB.java:48) - [RouterMDB]receive msg id:50 sequenceId=79743 2008-09-03 11:29:49,837 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/FmNbiTopic] stopped 2008-09-03 11:29:49,840 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/LMTopic] stopped 2008-09-03 11:29:49,842 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/LMOmtTopic] stopped 2008-09-03 11:29:49,844 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/sectopic] stopped 2008-09-03 11:29:49,846 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/RouteTopic] stopped 2008-09-03 11:29:49,847 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/smtopic] stopped 2008-09-03 11:29:49,849 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/topotopic] stopped 2008-09-03 11:29:49,850 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/Fm2TopoTopic] stopped 2008-09-03 11:29:49,853 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/PmTopic] stopped 2008-09-03 11:29:49,854 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/Pm4NeaTopic] stopped 2008-09-03 11:29:49,855 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/Pm4ServerTopic] stopped 2008-09-03 11:29:49,856 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/TDMTopic] stopped 2008-09-03 11:29:49,860 INFO [DRM Async Publisher#0] destination.TopicService ( TopicService.java:179) - Topic[/topic/TDMOmtTopic] stopped 2008-09-03 11:29:49,867 ERROR [WorkerThread#29[192.166.32.8:59979]] remoting.ServerInvoker ( ServerInvoker.java:1820) - Error executing server oneway invocation request: InvocationRequest[12e727b, JMS, org.jboss.jms.wireformat.SessionSendRequest@5deceb] org.jboss.jms.exception.MessagingShutdownException: Cannot handle invocation since messaging server is not active (it is either starting up or shutting down) at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java: 133) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809) at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815) at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63) at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166) 2008-09-03 11:29:49,872 ERROR [WorkerThread#29[192.166.32.8:59979]] remoting.ServerInvoker ( ServerInvoker.java:1820) - Error executing server oneway invocation request: InvocationRequest[6d7710, JMS, org.jboss.jms.wireformat.SessionSendRequest@a3d239] org.jboss.jms.exception.MessagingShutdownException: Cannot handle invocation since messaging server is not active (it is either starting up or shutting down) at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java: 133) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809) at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815) at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63) at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:572) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:387) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:166)
it was already confirmed that nobody executed shutdown() or stop() operation using jboss web-console. It's just like JBM server shutdown itself, and we don't know what is the cause.
any help will be greatly appreciated here.
thanks.