3 Replies Latest reply on Jan 22, 2009 10:39 AM by Brian Stansberry

    JBoss Clustering hang because of log4j

    chicken chicken Newbie

      Dear all.
      I run my application on JBoss Application Clustering Ver 4.2.3
      JDK: 1.5.0_16
      Jboss hangs on in every morning and I find out these error in the server.log file:

      2009-01-16 02:39:53,793 DEBUG [org.jboss.web.tomcat.service.session.JBossCacheManager] Looking for sessions that have expired ...
      2009-01-16 02:40:03,804 DEBUG [org.jboss.web.tomcat.service.session.JBossCacheManager] Looking for sessions that have expired ...
      2009-01-16 02:40:09,640 ERROR [org.jboss.remoting.transport.socket.ServerThread] Worker thread initialization failure
      java.lang.reflect.InvocationTargetException
       at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
       at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
       at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
       at java.lang.reflect.Constructor.newInstance(Unknown Source)
       at org.jboss.remoting.transport.socket.ServerThread.createServerSocketWrapper(ServerThread.java:755)
       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:375)
       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173)
      Caused by: java.io.EOFException
       at java.io.ObjectInputStream$PeekInputStream.readFully(Unknown Source)
       at java.io.ObjectInputStream$BlockDataInputStream.readShort(Unknown Source)
       at java.io.ObjectInputStream.readStreamHeader(Unknown Source)
       at java.io.ObjectInputStream.<init>(Unknown Source)
       at org.jboss.remoting.loading.ObjectInputStreamWithClassLoader.<init>(ObjectInputStreamWithClassLoader.java:95)
       at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.createInput(JavaSerializationManager.java:54)
       at org.jboss.remoting.marshal.serializable.SerializableUnMarshaller.getMarshallingStream(SerializableUnMarshaller.java:72)
       at org.jboss.remoting.marshal.serializable.SerializableUnMarshaller.getMarshallingStream(SerializableUnMarshaller.java:55)
       at org.jboss.remoting.transport.socket.ClientSocketWrapper.createInputStream(ClientSocketWrapper.java:179)
       at org.jboss.remoting.transport.socket.ClientSocketWrapper.createStreams(ClientSocketWrapper.java:162)
       at org.jboss.remoting.transport.socket.ClientSocketWrapper.<init>(ClientSocketWrapper.java:66)
       at org.jboss.remoting.transport.socket.ServerSocketWrapper.<init>(ServerSocketWrapper.java:46)
       ... 7 more
      2009-01-16 02:40:10,671 DEBUG [org.jboss.web.tomcat.service.session.JBossCacheManager] Creating an empty ClusteredSession
      2009-01-16 02:40:10,672 DEBUG [org.jboss.web.tomcat.service.session.SessionIDGenerator] getSessionId called: OoRDZiWqntrb2pk9zRusNw**
      2009-01-16 02:40:10,672 DEBUG [org.jboss.web.tomcat.service.session.JBossCacheManager] Session with id=OoRDZiWqntrb2pk9zRusNw** added. Current active sessions 1
      2009-01-16 02:40:10,677 DEBUG [org.jboss.web.tomcat.service.session.JBossCacheManager] Created a ClusteredSession with id: OoRDZiWqntrb2pk9zRusNw**
      2009-01-16 02:40:10,680 ERROR [STDERR] log4j:ERROR Could not close org.apache.log4j.helpers.QuietWriter@1b972d9
      2009-01-16 02:40:10,680 ERROR [STDERR] java.io.IOException: ¼Õ»óµÈ NFS ÆÄÀÃ� ÇÚµé
      2009-01-16 02:40:10,681 ERROR [STDERR] at java.io.FileOutputStream.close0(Native Method)
      2009-01-16 02:40:10,681 ERROR [STDERR] at java.io.FileOutputStream.close(Unknown Source)
      2009-01-16 02:40:10,682 ERROR [STDERR] at sun.nio.cs.StreamEncoder$CharsetSE.implClose(Unknown Source)
      2009-01-16 02:40:10,682 ERROR [STDERR] at sun.nio.cs.StreamEncoder.close(Unknown Source)
      2009-01-16 02:40:10,682 ERROR [STDERR] at java.io.OutputStreamWriter.close(Unknown Source)
      2009-01-16 02:40:10,682 ERROR [STDERR] at java.io.FilterWriter.close(Unknown Source)
      2009-01-16 02:40:10,682 ERROR [STDERR] at org.apache.log4j.FileAppender.closeFile(FileAppender.java:184)
      2009-01-16 02:40:10,682 ERROR [STDERR] at org.apache.log4j.DailyRollingFileAppender.rollOver(DailyRollingFileAppender.java:313)
      2009-01-16 02:40:10,683 ERROR [STDERR] at org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:353)
      2009-01-16 02:40:10,683 ERROR [STDERR] at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
      2009-01-16 02:40:10,683 ERROR [STDERR] at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
      2009-01-16 02:40:10,684 ERROR [STDERR] at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
      2009-01-16 02:40:10,684 ERROR [STDERR] at org.apache.log4j.Category.callAppenders(Category.java:206)
      2009-01-16 02:40:10,684 ERROR [STDERR] at org.apache.log4j.Category.forcedLog(Category.java:391)
      2009-01-16 02:40:10,684 ERROR [STDERR] at org.apache.log4j.Category.info(Category.java:666)
      2009-01-16 02:40:10,684 ERROR [STDERR] at security.ACL.verifyACL(Unknown Source)
      2009-01-16 02:40:10,684 ERROR [STDERR] at org.apache.jsp.index_jsp._jspService(index_jsp.java:56)
      2009-01-16 02:40:10,685 ERROR [STDERR] at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
      2009-01-16 02:40:10,685 ERROR [STDERR] at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
      2009-01-16 02:40:10,685 ERROR [STDERR] at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373)
      2009-01-16 02:40:10,685 ERROR [STDERR] at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:336)
      2009-01-16 02:40:10,685 ERROR [STDERR] at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265)
      2009-01-16 02:40:10,685 ERROR [STDERR] at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
      2009-01-16 02:40:10,685 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
      2009-01-16 02:40:10,686 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      2009-01-16 02:40:10,686 ERROR [STDERR] at net.util.PGFilterCharset.doFilter(Unknown Source)
      2009-01-16 02:40:10,686 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      2009-01-16 02:40:10,686 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      2009-01-16 02:40:10,686 ERROR [STDERR] at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
      2009-01-16 02:40:10,686 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      2009-01-16 02:40:10,687 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      2009-01-16 02:40:10,687 ERROR [STDERR] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
      2009-01-16 02:40:10,687 ERROR [STDERR] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
      2009-01-16 02:40:10,687 ERROR [STDERR] at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
      2009-01-16 02:40:10,687 ERROR [STDERR] at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:97)
      2009-01-16 02:40:10,687 ERROR [STDERR] at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
      2009-01-16 02:40:10,687 ERROR [STDERR] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
      2009-01-16 02:40:10,688 ERROR [STDERR] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
      2009-01-16 02:40:10,688 ERROR [STDERR] at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
      2009-01-16 02:40:10,688 ERROR [STDERR] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
      2009-01-16 02:40:10,688 ERROR [STDERR] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
      2009-01-16 02:40:10,688 ERROR [STDERR] at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
      2009-01-16 02:40:10,688 ERROR [STDERR] at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
      2009-01-16 02:40:10,689 ERROR [STDERR] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
      2009-01-16 02:40:10,689 ERROR [STDERR] at java.lang.Thread.run(Unknown Source)
      2009-01-16 02:40:10,859 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/].[jsp]] Servlet.service() for servlet jsp threw exception
      java.lang.IllegalStateException: Current state = FLUSHED, new state = CODING_END
       at java.nio.charset.CharsetEncoder.throwIllegalStateException(Unknown Source)
       at java.nio.charset.CharsetEncoder.encode(Unknown Source)
       at sun.nio.cs.StreamEncoder$CharsetSE.flushLeftoverChar(Unknown Source)
       at sun.nio.cs.StreamEncoder$CharsetSE.implClose(Unknown Source)
       at sun.nio.cs.StreamEncoder.close(Unknown Source)
       at java.io.OutputStreamWriter.close(Unknown Source)
       at java.io.FilterWriter.close(Unknown Source)
       at org.apache.log4j.FileAppender.closeFile(FileAppender.java:184)
       at org.apache.log4j.FileAppender.reset(FileAppender.java:339)
       at org.apache.log4j.FileAppender.setFile(FileAppender.java:284)
       at org.apache.log4j.DailyRollingFileAppender.rollOver(DailyRollingFileAppender.java:331)
       at org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:353)
       at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
       at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
       at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
       at org.apache.log4j.Category.callAppenders(Category.java:206)
       at org.apache.log4j.Category.forcedLog(Category.java:391)
       at org.apache.log4j.Category.info(Category.java:666)
       at security.ACL.verifyACL(Unknown Source)
       at org.apache.jsp.index_jsp._jspService(index_jsp.java:56)
       at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
       at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373)
       at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:336)
       at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
       at net.util.PGFilterCharset.doFilter(Unknown Source)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
       at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
       at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
       at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:97)
       at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
       at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
       at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
       at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
       at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
       at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
       at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
       at java.lang.Thread.run(Unknown Source)
      2009-01-16 02:40:12,657 DEBUG [org.jboss.invocation.pooled.server.ServerThread] beginning dorun
      2009-01-16 02:40:13,639 ERROR [org.jboss.invocation.pooled.server.ServerThread] Failed to initialize
      java.io.EOFException
       at java.io.ObjectInputStream$PeekInputStream.readFully(Unknown Source)
       at java.io.ObjectInputStream$BlockDataInputStream.readShort(Unknown Source)
       at java.io.ObjectInputStream.readStreamHeader(Unknown Source)
       at java.io.ObjectInputStream.<init>(Unknown Source)
       at org.jboss.invocation.pooled.interfaces.OptimizedObjectInputStream.<init>(OptimizedObjectInputStream.java:147)
       at org.jboss.invocation.pooled.server.ServerThread.dorun(ServerThread.java:265)
       at org.jboss.invocation.pooled.server.ServerThread.run(ServerThread.java:156)
      2009-01-16 02:40:13,672 DEBUG [org.jboss.invocation.pooled.server.ServerThread] beginning dorun
      2009-01-16 02:40:13,813 DEBUG [org.jboss.web.tomcat.service.session.JBossCacheManager] Looking for sessions that have expired ...
      2009-01-16 02:40:14,639 ERROR [org.jboss.invocation.pooled.server.ServerThread] Failed to initialize
      java.io.EOFException
       at java.io.ObjectInputStream$PeekInputStream.readFully(Unknown Source)
       at java.io.ObjectInputStream$BlockDataInputStream.readShort(Unknown Source)
       at java.io.ObjectInputStream.readStreamHeader(Unknown Source)
       at java.io.ObjectInputStream.<init>(Unknown Source)
       at org.jboss.invocation.pooled.interfaces.OptimizedObjectInputStream.<init>(OptimizedObjectInputStream.java:147)
       at org.jboss.invocation.pooled.server.ServerThread.dorun(ServerThread.java:265)
       at org.jboss.invocation.pooled.server.ServerThread.run(ServerThread.java:156)
      2009-01-16 02:40:15,640 ERROR [jacorb.giop.conn] Failed to read GIOP message in ServerGIOPConnection to 192.168.10.30:4133 (65f1ba), incorrect magic number --> connection closed
      2009-01-16 02:40:23,823 DEBUG [org.jboss.web.tomcat.service.session.JBossCacheManager] Looking for sessions that have expired ...
      2009-01-16 02:40:28,831 DEBUG [org.jboss.web.tomcat.service.session.JBossCacheManager] Creating an empty ClusteredSession
      2009-01-16 02:40:28,832 DEBUG [org.jboss.web.tomcat.service.session.SessionIDGenerator] getSessionId called: PzwhNRPFUSTycVtQNRJRqg**
      2009-01-16 02:40:28,832 DEBUG [org.jboss.web.tomcat.service.session.JBossCacheManager] Session with id=PzwhNRPFUSTycVtQNRJRqg** added. Current active sessions 2
      2009-01-16 02:40:28,832 DEBUG [org.jboss.web.tomcat.service.session.JBossCacheManager] Created a ClusteredSession with id: PzwhNRPFUSTycVtQNRJRqg**
      2009-01-16 02:40:28,835 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/].[jsp]] Servlet.service() for servlet jsp threw exception
      java.lang.IllegalStateException: Current state = FLUSHED, new state = CODING

      could you help me to solve this problem. It takes me long time but I was not successful.

        • 1. Re: JBoss Clustering hang because of log4j
          Brian Stansberry Master

           

          2009-01-16 02:40:10,680 ERROR [STDERR] java.io.IOException: ¼Õ»óµÈ NFS ÆÄÀÃÂ� ÇÚµé


          Can you tell me what all those special characters say?

          Looks like it's a failure to close the server.log file for one day so it can roll over to the next day's version. But the other exceptions seem unrelated other than that they are IO related and occur around the same time. Do those non-log4j exceptions only occur at this time?

          • 2. Re: JBoss Clustering hang because of log4j
            chicken chicken Newbie

            Thank you for your help

            "bstansberry@jboss.com" wrote:


            Can you tell me what all those special characters say?

            I think this is broken Korean Message. I don't know how to config JBoss write to the log file in Korean correctly. please tell me if you know.

            "bstansberry@jboss.com" wrote:

            Looks like it's a failure to close the server.log file for one day so it can roll over to the next day's version. But the other exceptions seem unrelated other than that they are IO related and occur around the same time. Do those non-log4j exceptions only occur at this time?

            yes, all those exceptions don't appear until the log4j exception occurs.
            After a failure to close the server.log, I get many EOF exceptions


            • 3. Re: JBoss Clustering hang because of log4j
              Brian Stansberry Master

              Sorry, I don't have a good answer for you on this one; looks like a log4j issue. If anyone else on this forum has any idea, please help out.