3 Replies Latest reply on Jan 22, 2009 10:39 AM by brian.stansberry

JBoss Clustering hang because of log4j

chicken Jan 16, 2009 3:47 AM

Dear all.
I run my application on JBoss Application Clustering Ver 4.2.3
JDK: 1.5.0_16
Jboss hangs on in every morning and I find out these error in the server.log file:

2009-01-16 02:39:53,793 DEBUG [org.jboss.web.tomcat.service.session.JBossCacheManager] Looking for sessions that have expired ...
2009-01-16 02:40:03,804 DEBUG [org.jboss.web.tomcat.service.session.JBossCacheManager] Looking for sessions that have expired ...
2009-01-16 02:40:09,640 ERROR [org.jboss.remoting.transport.socket.ServerThread] Worker thread initialization failure
java.lang.reflect.InvocationTargetException
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
 at java.lang.reflect.Constructor.newInstance(Unknown Source)
 at org.jboss.remoting.transport.socket.ServerThread.createServerSocketWrapper(ServerThread.java:755)
 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:375)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173)
Caused by: java.io.EOFException
 at java.io.ObjectInputStream$PeekInputStream.readFully(Unknown Source)
 at java.io.ObjectInputStream$BlockDataInputStream.readShort(Unknown Source)
 at java.io.ObjectInputStream.readStreamHeader(Unknown Source)
 at java.io.ObjectInputStream.<init>(Unknown Source)
 at org.jboss.remoting.loading.ObjectInputStreamWithClassLoader.<init>(ObjectInputStreamWithClassLoader.java:95)
 at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.createInput(JavaSerializationManager.java:54)
 at org.jboss.remoting.marshal.serializable.SerializableUnMarshaller.getMarshallingStream(SerializableUnMarshaller.java:72)
 at org.jboss.remoting.marshal.serializable.SerializableUnMarshaller.getMarshallingStream(SerializableUnMarshaller.java:55)
 at org.jboss.remoting.transport.socket.ClientSocketWrapper.createInputStream(ClientSocketWrapper.java:179)
 at org.jboss.remoting.transport.socket.ClientSocketWrapper.createStreams(ClientSocketWrapper.java:162)
 at org.jboss.remoting.transport.socket.ClientSocketWrapper.<init>(ClientSocketWrapper.java:66)
 at org.jboss.remoting.transport.socket.ServerSocketWrapper.<init>(ServerSocketWrapper.java:46)
 ... 7 more
2009-01-16 02:40:10,671 DEBUG [org.jboss.web.tomcat.service.session.JBossCacheManager] Creating an empty ClusteredSession
2009-01-16 02:40:10,672 DEBUG [org.jboss.web.tomcat.service.session.SessionIDGenerator] getSessionId called: OoRDZiWqntrb2pk9zRusNw**
2009-01-16 02:40:10,672 DEBUG [org.jboss.web.tomcat.service.session.JBossCacheManager] Session with id=OoRDZiWqntrb2pk9zRusNw** added. Current active sessions 1
2009-01-16 02:40:10,677 DEBUG [org.jboss.web.tomcat.service.session.JBossCacheManager] Created a ClusteredSession with id: OoRDZiWqntrb2pk9zRusNw**
2009-01-16 02:40:10,680 ERROR [STDERR] log4j:ERROR Could not close org.apache.log4j.helpers.QuietWriter@1b972d9
2009-01-16 02:40:10,680 ERROR [STDERR] java.io.IOException: Ã‚Â¼ÃƒÂ•Ã‚Â»ÃƒÂ³Ã‚ÂµÃƒÂˆ NFS ÃƒÂ†ÃƒÂ„ÃƒÂ€ÃƒÂ� ÃƒÂ‡ÃƒÂšÃ‚ÂµÃƒÂ©
2009-01-16 02:40:10,681 ERROR [STDERR] at java.io.FileOutputStream.close0(Native Method)
2009-01-16 02:40:10,681 ERROR [STDERR] at java.io.FileOutputStream.close(Unknown Source)
2009-01-16 02:40:10,682 ERROR [STDERR] at sun.nio.cs.StreamEncoder$CharsetSE.implClose(Unknown Source)
2009-01-16 02:40:10,682 ERROR [STDERR] at sun.nio.cs.StreamEncoder.close(Unknown Source)
2009-01-16 02:40:10,682 ERROR [STDERR] at java.io.OutputStreamWriter.close(Unknown Source)
2009-01-16 02:40:10,682 ERROR [STDERR] at java.io.FilterWriter.close(Unknown Source)
2009-01-16 02:40:10,682 ERROR [STDERR] at org.apache.log4j.FileAppender.closeFile(FileAppender.java:184)
2009-01-16 02:40:10,682 ERROR [STDERR] at org.apache.log4j.DailyRollingFileAppender.rollOver(DailyRollingFileAppender.java:313)
2009-01-16 02:40:10,683 ERROR [STDERR] at org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:353)
2009-01-16 02:40:10,683 ERROR [STDERR] at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
2009-01-16 02:40:10,683 ERROR [STDERR] at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
2009-01-16 02:40:10,684 ERROR [STDERR] at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
2009-01-16 02:40:10,684 ERROR [STDERR] at org.apache.log4j.Category.callAppenders(Category.java:206)
2009-01-16 02:40:10,684 ERROR [STDERR] at org.apache.log4j.Category.forcedLog(Category.java:391)
2009-01-16 02:40:10,684 ERROR [STDERR] at org.apache.log4j.Category.info(Category.java:666)
2009-01-16 02:40:10,684 ERROR [STDERR] at security.ACL.verifyACL(Unknown Source)
2009-01-16 02:40:10,684 ERROR [STDERR] at org.apache.jsp.index_jsp._jspService(index_jsp.java:56)
2009-01-16 02:40:10,685 ERROR [STDERR] at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
2009-01-16 02:40:10,685 ERROR [STDERR] at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
2009-01-16 02:40:10,685 ERROR [STDERR] at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373)
2009-01-16 02:40:10,685 ERROR [STDERR] at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:336)
2009-01-16 02:40:10,685 ERROR [STDERR] at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265)
2009-01-16 02:40:10,685 ERROR [STDERR] at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
2009-01-16 02:40:10,685 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
2009-01-16 02:40:10,686 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
2009-01-16 02:40:10,686 ERROR [STDERR] at net.util.PGFilterCharset.doFilter(Unknown Source)
2009-01-16 02:40:10,686 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
2009-01-16 02:40:10,686 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
2009-01-16 02:40:10,686 ERROR [STDERR] at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
2009-01-16 02:40:10,686 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
2009-01-16 02:40:10,687 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
2009-01-16 02:40:10,687 ERROR [STDERR] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
2009-01-16 02:40:10,687 ERROR [STDERR] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
2009-01-16 02:40:10,687 ERROR [STDERR] at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
2009-01-16 02:40:10,687 ERROR [STDERR] at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:97)
2009-01-16 02:40:10,687 ERROR [STDERR] at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
2009-01-16 02:40:10,687 ERROR [STDERR] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
2009-01-16 02:40:10,688 ERROR [STDERR] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
2009-01-16 02:40:10,688 ERROR [STDERR] at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
2009-01-16 02:40:10,688 ERROR [STDERR] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
2009-01-16 02:40:10,688 ERROR [STDERR] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
2009-01-16 02:40:10,688 ERROR [STDERR] at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
2009-01-16 02:40:10,688 ERROR [STDERR] at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
2009-01-16 02:40:10,689 ERROR [STDERR] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
2009-01-16 02:40:10,689 ERROR [STDERR] at java.lang.Thread.run(Unknown Source)
2009-01-16 02:40:10,859 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/].[jsp]] Servlet.service() for servlet jsp threw exception
java.lang.IllegalStateException: Current state = FLUSHED, new state = CODING_END
 at java.nio.charset.CharsetEncoder.throwIllegalStateException(Unknown Source)
 at java.nio.charset.CharsetEncoder.encode(Unknown Source)
 at sun.nio.cs.StreamEncoder$CharsetSE.flushLeftoverChar(Unknown Source)
 at sun.nio.cs.StreamEncoder$CharsetSE.implClose(Unknown Source)
 at sun.nio.cs.StreamEncoder.close(Unknown Source)
 at java.io.OutputStreamWriter.close(Unknown Source)
 at java.io.FilterWriter.close(Unknown Source)
 at org.apache.log4j.FileAppender.closeFile(FileAppender.java:184)
 at org.apache.log4j.FileAppender.reset(FileAppender.java:339)
 at org.apache.log4j.FileAppender.setFile(FileAppender.java:284)
 at org.apache.log4j.DailyRollingFileAppender.rollOver(DailyRollingFileAppender.java:331)
 at org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:353)
 at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
 at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
 at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
 at org.apache.log4j.Category.callAppenders(Category.java:206)
 at org.apache.log4j.Category.forcedLog(Category.java:391)
 at org.apache.log4j.Category.info(Category.java:666)
 at security.ACL.verifyACL(Unknown Source)
 at org.apache.jsp.index_jsp._jspService(index_jsp.java:56)
 at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
 at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373)
 at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:336)
 at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at net.util.PGFilterCharset.doFilter(Unknown Source)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
 at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
 at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:97)
 at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
 at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
 at java.lang.Thread.run(Unknown Source)
2009-01-16 02:40:12,657 DEBUG [org.jboss.invocation.pooled.server.ServerThread] beginning dorun
2009-01-16 02:40:13,639 ERROR [org.jboss.invocation.pooled.server.ServerThread] Failed to initialize
java.io.EOFException
 at java.io.ObjectInputStream$PeekInputStream.readFully(Unknown Source)
 at java.io.ObjectInputStream$BlockDataInputStream.readShort(Unknown Source)
 at java.io.ObjectInputStream.readStreamHeader(Unknown Source)
 at java.io.ObjectInputStream.<init>(Unknown Source)
 at org.jboss.invocation.pooled.interfaces.OptimizedObjectInputStream.<init>(OptimizedObjectInputStream.java:147)
 at org.jboss.invocation.pooled.server.ServerThread.dorun(ServerThread.java:265)
 at org.jboss.invocation.pooled.server.ServerThread.run(ServerThread.java:156)
2009-01-16 02:40:13,672 DEBUG [org.jboss.invocation.pooled.server.ServerThread] beginning dorun
2009-01-16 02:40:13,813 DEBUG [org.jboss.web.tomcat.service.session.JBossCacheManager] Looking for sessions that have expired ...
2009-01-16 02:40:14,639 ERROR [org.jboss.invocation.pooled.server.ServerThread] Failed to initialize
java.io.EOFException
 at java.io.ObjectInputStream$PeekInputStream.readFully(Unknown Source)
 at java.io.ObjectInputStream$BlockDataInputStream.readShort(Unknown Source)
 at java.io.ObjectInputStream.readStreamHeader(Unknown Source)
 at java.io.ObjectInputStream.<init>(Unknown Source)
 at org.jboss.invocation.pooled.interfaces.OptimizedObjectInputStream.<init>(OptimizedObjectInputStream.java:147)
 at org.jboss.invocation.pooled.server.ServerThread.dorun(ServerThread.java:265)
 at org.jboss.invocation.pooled.server.ServerThread.run(ServerThread.java:156)
2009-01-16 02:40:15,640 ERROR [jacorb.giop.conn] Failed to read GIOP message in ServerGIOPConnection to 192.168.10.30:4133 (65f1ba), incorrect magic number --> connection closed
2009-01-16 02:40:23,823 DEBUG [org.jboss.web.tomcat.service.session.JBossCacheManager] Looking for sessions that have expired ...
2009-01-16 02:40:28,831 DEBUG [org.jboss.web.tomcat.service.session.JBossCacheManager] Creating an empty ClusteredSession
2009-01-16 02:40:28,832 DEBUG [org.jboss.web.tomcat.service.session.SessionIDGenerator] getSessionId called: PzwhNRPFUSTycVtQNRJRqg**
2009-01-16 02:40:28,832 DEBUG [org.jboss.web.tomcat.service.session.JBossCacheManager] Session with id=PzwhNRPFUSTycVtQNRJRqg** added. Current active sessions 2
2009-01-16 02:40:28,832 DEBUG [org.jboss.web.tomcat.service.session.JBossCacheManager] Created a ClusteredSession with id: PzwhNRPFUSTycVtQNRJRqg**
2009-01-16 02:40:28,835 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/].[jsp]] Servlet.service() for servlet jsp threw exception
java.lang.IllegalStateException: Current state = FLUSHED, new state = CODING

could you help me to solve this problem. It takes me long time but I was not successful.

1. Re: JBoss Clustering hang because of log4j

brian.stansberry Jan 19, 2009 12:11 PM (in response to chicken)

2009-01-16 02:40:10,680 ERROR [STDERR] java.io.IOException: ÃƒÂ‚Ã‚Â¼ÃƒÂƒÃ‚Â•ÃƒÂ‚Ã‚Â»ÃƒÂƒÃ‚Â³ÃƒÂ‚Ã‚ÂµÃƒÂƒÃ‚Âˆ NFS ÃƒÂƒÃ‚Â†ÃƒÂƒÃ‚Â„ÃƒÂƒÃ‚Â€ÃƒÂƒÃ‚Â� ÃƒÂƒÃ‚Â‡ÃƒÂƒÃ‚ÂšÃƒÂ‚Ã‚ÂµÃƒÂƒÃ‚Â©

Can you tell me what all those special characters say?

Looks like it's a failure to close the server.log file for one day so it can roll over to the next day's version. But the other exceptions seem unrelated other than that they are IO related and occur around the same time. Do those non-log4j exceptions only occur at this time?
Actions
2. Re: JBoss Clustering hang because of log4j

chicken Jan 21, 2009 2:20 AM (in response to chicken)

Thank you for your help
"bstansberry@jboss.com" wrote:

Can you tell me what all those special characters say?

I think this is broken Korean Message. I don't know how to config JBoss write to the log file in Korean correctly. please tell me if you know.

"bstansberry@jboss.com" wrote:

Looks like it's a failure to close the server.log file for one day so it can roll over to the next day's version. But the other exceptions seem unrelated other than that they are IO related and occur around the same time. Do those non-log4j exceptions only occur at this time?

yes, all those exceptions don't appear until the log4j exception occurs.
After a failure to close the server.log, I get many EOF exceptions
Actions
3. Re: JBoss Clustering hang because of log4j

brian.stansberry Jan 22, 2009 10:39 AM (in response to chicken)

Sorry, I don't have a good answer for you on this one; looks like a log4j issue. If anyone else on this forum has any idea, please help out.
Actions

Go to original post