9 Replies Latest reply on Mar 4, 2010 9:19 AM by jaikiran pai

    Some junk characters displaying at start of jboss gc log file

    Ajeet Raina Newbie

      Hello Guys,

       

      I am new to this forum and expect a help from your side.

      the GC logs in jboss is provisioining with junk characters like:

       

      ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@

      at the first line while the line after that seems to be fine.

       

      I checked with the file format and surprisingly find it :

       

      gc_log : data

       

      which seems that its binary file.

      I am new to Jboss GC concept.

       

      My run.sh says:

       

      #!/bin/sh
      ### ====================================================================== ###
      ##                                                                          ##
      ##  JBoss Bootstrap Script                                                  ##
      ##                                                                          ##
      ### ====================================================================== ###

       

      ### $Id: run.sh 73862 2008-05-30 18:27:03Z mmoyses $ ###

       

      DIRNAME=`dirname $0`
      PROGNAME=`basename $0`
      GREP="grep"

       

      # Set conf if specified, else set to production
      JBOSSCONF="production"
      CONF_SPECIFIED=false

       

      arg_count=1
      eval SWITCH=\${$arg_count}
      while [ ! -z "$SWITCH" ]
      do

       

              if [ "$SWITCH" = "-c" ]; then
                  eval JBOSSCONF=\$`expr $arg_count + 1`
                  CONF_SPECIFIED=true
                  break
              fi

       

              arg_count=`expr $arg_count + 1`
              eval SWITCH=\${$arg_count}
      done

       

      if [ x${CONF_SPECIFIED} = "xfalse" ]
      then
         set -- "-c" ${JBOSSCONF} $@
      fi

       

      # Use the maximum available, or set MAX_FD != -1 to use that
      MAX_FD="maximum"

       

      #
      # Helper to complain.
      #
      #
      warn() {
          echo "${PROGNAME}: $*"
      }

       

      #
      # Helper to puke.
      #
      die() {
          warn $*
          exit 1
      }

       

      # OS specific support (must be 'true' or 'false').
      cygwin=false;
      darwin=false;
      linux=false;
      case "`uname`" in
          CYGWIN*)
              cygwin=true
              ;;

       

          Darwin*)
              darwin=true
              ;;

       

          Linux)
              linux=true
              ;;
      esac

       

      # Read an optional running configuration file
      if [ "x$RUN_CONF" = "x" ]; then

       

          if [ ! -z "$JBOSSCONF" ] && [ -f "$DIRNAME/../server/$JBOSSCONF/run.conf" ]; then
              RUN_CONF="$DIRNAME/../server/$JBOSSCONF/run.conf"
          else
              RUN_CONF="$DIRNAME/run.conf"
          fi
      fi
      if [ -r "$RUN_CONF" ]; then
          . "$RUN_CONF"

       

      fi

       

      # Force IPv4 on Linux systems since IPv6 doesn't work correctly with jdk5 and lower
      if [ "$linux" = "true" ]; then
         JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
      fi

       

      # For Cygwin, ensure paths are in UNIX format before anything is touched
      if $cygwin ; then
          [ -n "$JBOSS_HOME" ] &&
              JBOSS_HOME=`cygpath --unix "$JBOSS_HOME"`
          [ -n "$JAVA_HOME" ] &&
              JAVA_HOME=`cygpath --unix "$JAVA_HOME"`
          [ -n "$JAVAC_JAR" ] &&
              JAVAC_JAR=`cygpath --unix "$JAVAC_JAR"`
      fi

       

      # Setup JBOSS_HOME
      if [ "x$JBOSS_HOME" = "x" ]; then
          # get the full path (without any relative bits)
          JBOSS_HOME=`cd $DIRNAME/..; pwd`
      fi
      export JBOSS_HOME

       

      # Increase the maximum file descriptors if we can
      if [ "$cygwin" = "false" ]; then
          MAX_FD_LIMIT=`ulimit -H -n`
          if [ "$?" -eq 0 ]; then
              if [ "$MAX_FD" = "maximum" -o "$MAX_FD" = "max" ]; then
                  # use the system max
                  MAX_FD="$MAX_FD_LIMIT"
              fi

       

              ulimit -n $MAX_FD
              if [ "$?" -ne 0 ]; then
                  warn "Could not set maximum file descriptor limit: $MAX_FD"
              fi
          else
              warn "Could not query system maximum file descriptor limit: $MAX_FD_LIMIT"
          fi
      fi
      # Setup the JVM
      if [ "x$JAVA" = "x" ]; then
          if [ "x$JAVA_HOME" != "x" ]; then
              JAVA="$JAVA_HOME/bin/java"
          else
              JAVA="java"
          fi
      fi

       

      # Setup the classpath
      runjar="$JBOSS_HOME/bin/run.jar"
      if [ ! -f "$runjar" ]; then
          die "Missing required file: $runjar"
      fi
      JBOSS_BOOT_CLASSPATH="$runjar"

       

      # Tomcat uses the JDT Compiler
      # Only include tools.jar if someone wants to use the JDK instead.
      # compatible distribution which JAVA_HOME points to
      if [ "x$JAVAC_JAR" = "x" ]; then
          JAVAC_JAR_FILE="$JAVA_HOME/lib/tools.jar"
      else
          JAVAC_JAR_FILE="$JAVAC_JAR"
      fi
      if [ ! -f "$JAVAC_JAR_FILE" ]; then
         # MacOSX does not have a seperate tools.jar
         if [ "$darwin" != "true" -a "x$JAVAC_JAR" != "x" ]; then
            warn "Missing file: JAVAC_JAR=$JAVAC_JAR"
            warn "Unexpected results may occur."
         fi
         JAVAC_JAR_FILE=
      fi

       

      if [ "x$JBOSS_CLASSPATH" = "x" ]; then
          JBOSS_CLASSPATH="$JBOSS_BOOT_CLASSPATH"
      else
          JBOSS_CLASSPATH="$JBOSS_CLASSPATH:$JBOSS_BOOT_CLASSPATH"
      fi
      if [ "x$JAVAC_JAR_FILE" != "x" ]; then
          JBOSS_CLASSPATH="$JBOSS_CLASSPATH:$JAVAC_JAR_FILE"
      fi

       

      # If -server not set in JAVA_OPTS, set it, if supported
      SERVER_SET=`echo $JAVA_OPTS | $GREP "\-server"`
      if [ "x$SERVER_SET" = "x" ]; then

       

          # Check for SUN(tm) JVM w/ HotSpot support
          if [ "x$HAS_HOTSPOT" = "x" ]; then
              HAS_HOTSPOT=`"$JAVA" -version 2>&1 | $GREP -i HotSpot`
          fi

       

          # Enable -server if we have Hotspot, unless we can't
          if [ "x$HAS_HOTSPOT" != "x" ]; then
              # MacOS does not support -server flag
              if [ "$darwin" != "true" ]; then
                  JAVA_OPTS="-server $JAVA_OPTS"
              fi
          fi
      fi

       

      # Setup JBosst Native library path
      JBOSS_NATIVE_DIR="$JBOSS_HOME/bin/native"
      if [ -d "$JBOSS_NATIVE_DIR" ]; then
          if $cygwin ; then
              export PATH="$JBOSS_NATIVE_DIR:$PATH"
              JBOSS_NATIVE_DIR=`cygpath --dos "$JBOSS_NATIVE_DIR"`
          fi
          if [ "x$LD_LIBRARY_PATH" = "x" ]; then
              LD_LIBRARY_PATH="$JBOSS_NATIVE_DIR"
          else
              LD_LIBRARY_PATH="$JBOSS_NATIVE_DIR:$LD_LIBRARY_PATH"
          fi
          export LD_LIBRARY_PATH
          if [ "x$JAVA_OPTS" = "x" ]; then
        if [ "x$JAVA_OPTS" = "x" ]; then
              JAVA_OPTS="-Djava.library.path=$JBOSS_NATIVE_DIR"
          else
              JAVA_OPTS="$JAVA_OPTS -Djava.library.path=$JBOSS_NATIVE_DIR"
          fi
      fi

       

      # Setup JBoss sepecific properties
      JAVA_OPTS="-Dprogram.name=$PROGNAME $JAVA_OPTS"

       

      # Setup the java endorsed dirs
      JBOSS_ENDORSED_DIRS="$JBOSS_HOME/lib/endorsed"

       

      # For Cygwin, switch paths to Windows format before running java
      if $cygwin; then
          JBOSS_HOME=`cygpath --path --windows "$JBOSS_HOME"`
          JAVA_HOME=`cygpath --path --windows "$JAVA_HOME"`
          JBOSS_CLASSPATH=`cygpath --path --windows "$JBOSS_CLASSPATH"`
          JBOSS_ENDORSED_DIRS=`cygpath --path --windows "$JBOSS_ENDORSED_DIRS"`
      fi

       

      # Display our environment
      echo "========================================================================="
      echo ""
      echo "  JBoss Bootstrap Environment"
      echo ""
      echo "  JBOSS_HOME: $JBOSS_HOME"
      echo ""
      echo "  JAVA: $JAVA"
      echo ""
      echo "  JAVA_OPTS: $JAVA_OPTS"
      echo ""
      echo "  CLASSPATH: $JBOSS_CLASSPATH"
      echo ""
      echo "========================================================================="
      echo ""

       

      while true; do
         if [ "x$LAUNCH_JBOSS_IN_BACKGROUND" = "x" ]; then

       

        if [ "x$LAUNCH_JBOSS_IN_BACKGROUND" = "x" ]; then
            # Execute the JVM in the foreground
            "$JAVA" $JAVA_OPTS \
               -Djava.endorsed.dirs="$JBOSS_ENDORSED_DIRS" \
               -classpath "$JBOSS_CLASSPATH" \
               org.jboss.Main "$@"
            JBOSS_STATUS=$?
         else
            # Execute the JVM in the background
            "$JAVA" $JAVA_OPTS \
               -Djava.endorsed.dirs="$JBOSS_ENDORSED_DIRS" \
               -classpath "$JBOSS_CLASSPATH" \
               org.jboss.Main "$@" &
            JBOSS_PID=$!
            # Trap common signals and relay them to the jboss process
            trap "kill -HUP  $JBOSS_PID" HUP
            trap "kill -TERM $JBOSS_PID" INT
            trap "kill -QUIT $JBOSS_PID" QUIT
            trap "kill -PIPE $JBOSS_PID" PIPE
            trap "kill -TERM $JBOSS_PID" TERM
            # Wait until the background process exits
            WAIT_STATUS=0
            while [ "$WAIT_STATUS" -ne 127 ]; do
               JBOSS_STATUS=$WAIT_STATUS
               wait $JBOSS_PID 2>/dev/null
               WAIT_STATUS=$?
            done
         fi
         # If restart doesn't work, check you are running JBossAS 4.0.4+
         #    http://jira.jboss.com/jira/browse/JBAS-2483
         # or the following if you're running Red Hat 7.0
         #    http://developer.java.sun.com/developer/bugParade/bugs/4465334.html
         if [ "$JBOSS_STATUS" -eq 10 ]; then
            echo "Restarting JBoss..."
         else
            exit $JBOSS_STATUS
         fi

       

       

       

      My run.conf says:

       

      CustomLog "| /usr/sbin/logrotate /mc.com/gm.log 60" common

       

      JAVA_OPTS="-server -XX:PermSize=128m -XX:MaxPermSize=256m -XX:-PrintTenuringDistribution

       

      No idea why its showing data file.

       

      What I really facing issue is those first line size which is about 400-500MB.

      I tried to remove the first line through script putting sed utility but its not working breaking log pipe.

       

      i did posted it in linux forum but dint find the correct resolution http://www.linuxquestions.org/questions/linux-newbie-8/some-character-adding-up-starting-of-log-file-792511/

       

      pls help

        • 1. Re: Some junk characters displaying at start of jboss gc log file
          Peter Johnson Master

          Ajeet, welcome to the JBoss forum!

           

          I do not see a reference to gc_log anywhere within your script. And what is gc_log? It is no a standard JBoss AS feature. Do you perhaps mean the log file generated by the -Xloggc JVM argument?

          • 2. Re: Some junk characters displaying at start of jboss gc log file
            Ajeet Raina Newbie

            Thanks for the quick response.

             

            Yeah !! The same Xloggc JVM  argument.

             


            JAVA_OPTS="-server -XX:PermSize=128m -XX:MaxPermSize=256m -XX:-PrintTenuringDistribution -XX:NewSize=256m -XX:MaxNewSize=384m -XX:+UseParallelGC -XX:SurvivorRatio=12 -Xmx1280m -Xms1280m -XX:ThreadStackSize=256 -XX:+DisableExplicitGC -verbose:gc -XX:+PrintGCDetails  -XX:+PrintTenuringDistribution -XX:ParallelGCThreads=2 -Xloggc:/project/gm.log -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dsun.lang.ClassLoader.allowArraySyntax=true"Quote:
            -Xloggc:file

             

            I wonder if that is causing the issue.

             

            Surprisingly , gc_log is something I too find developer playing with and am new to that.

            The file format is data(binary) characters and it starts with ^@ surprisingly.

            Whenever log rotate it again adds up ^@ at the start....Confused !!!

            Any idea what makes it be data file rather than simple text log.

            • 3. Re: Some junk characters displaying at start of jboss gc log file
              Peter Johnson Master

              Sounds like the JVM is creating the GC log using Unicode with perhaps a byte-order marker. You might need a text editor that understands Unicode the read the file. Or it could be creating a file based on your localization settings, and you need an editor that understands that to read the file. Of course it is hard to tell since what you posted is getting mangled - posting the hex values would have been more helpful.

               

              By the way, Linux is probably telling you that the file is a data file simply because it is not ASCII (or UTF8). But that is just a guess.

              • 4. Re: Some junk characters displaying at start of jboss gc log file
                Ajeet Raina Newbie

                GC with Unicode..Could be the Reason..Not Sure about that.

                Let me tell you further.

                The First line is the long line provisioned with ^@^@^@ characters and the line after that is simple text file kind-a-stuff.

                I can read those lines easily.

                 

                Seems like these entries:(not exactly the same words but the format is same)

                 

                976.578: [GC 976.578: [Tenured: 18115K->7153K(18116K), 0.3309426 secs] 19523K->7153K(19524K), 0.3459331 secs]
                980.544: [GC 980.544: [Tenured: 18115K->8080K(18116K), 0.1734528 secs] 19523K->8080K(19524K), 0.1949907 secs]
                982.783: [GC 982.783: [Tenured: 18115K->12388K(18116K), 0.1951404 secs] 19523K->12388K(19524K), 0.2119867 secs]
                986.579: [GC 986.579: [Tenured: 20647K->8289K(20648K), 0.1755991 secs] 22311K->8289K(22312K), 0.1924510 secs]

                After I delete the first line through dd on command-mode in linux it reduces lots of size too.

                • 5. Re: Some junk characters displaying at start of jboss gc log file
                  Peter Johnson Master

                  "The First line is the long line provisioned with ^@^@^@ characters"

                  I don't know what you are seeing in your forum posts, but what I see is "provisioned with the ^@^@^@ characters", and I have no idea what ^@^@^@ is. Like I said, knowing the hex values for ^@^@^@ would help. For example, if you are seeing 0xFEFF or 0xFFFE, then these are byte-ordering characters in Unicode text files.

                   

                  If dd reduces the file size, then yes the file is probably Unicode and dd is translating it to ASCII.

                  • 6. Re: Some junk characters displaying at start of jboss gc log file
                    Ajeet Raina Newbie

                    Yes..Whatever character you can see is what I mean.

                    What I had in my mind is deleting this Unicode everytime log is rotated.

                    That could be the temporary solution for me as of now.

                    I tried putting script in postrotate but that dint work.

                    Then I tried writing teh script as:

                     

                    #!/binbash

                    > gc_log  <--- Main Log file

                    cp gc_log tempfile  -<-- copied it to tempfile for deleting the first line

                    sed -i 1D tempfile

                    cat tempfile > gc_log

                     

                    But that dint work.

                     

                    ANy Suggestion.

                    • 7. Re: Some junk characters displaying at start of jboss gc log file
                      Ajeet Raina Newbie

                      Also, few Freakers did sugest me with this too.

                       

                       

                      ngDistribution -XX:NewSize=256m -XX:MaxNewSize=384m -XX:+UseParallelGC -XX:SurvivorRatio=12 -Xmx1280m -Xms1280m -XX:ThreadStackSize=256 -XX:+DisableExplicitGC -verbose:gc -XX:+PrintGCDetails  -XX:+PrintTenuringDistribution -XX:ParallelGCThreads=2 -Xloggc:/project/gm.log -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dsun.lang.ClassLoader.allowArraySyntax=true"
                      Quote:
                      -Xloggc:file
                      Report on each garbage collection event, as with -verbose:gc, but log this data to file. In addition to the information -verbose:gc gives, each reported event will be preceeded by the time (in seconds) since the first garbage-collection event.
                      Always use a local file system for storage of this file to avoid stalling the JVM due to network latency. The file may be truncated in the case of a full file system and logging will continue on the truncated file. This option overrides -verbose:gc if both are given on the command line.

                      Please check you are using both the options. This probably is not the cause for the binary data in the file. But it is there so I highlighted it.

                       

                      Secondly, I am still not sure how can you be so sure that these junk character will be there on the first line. For me these are gc logs and can be at any place.

                       

                      Do you find any exception just above or below these junk character in the log file?

                       

                      You can try invoking this java command on the command line to see what could be the probable exeception that is dumping these chars in the gc log file.

                      • 8. Re: Some junk characters displaying at start of jboss gc log file
                        Ajeet Raina Newbie

                        Hi Guys !!

                         

                        Is there any way we can add Appender for GC logs in log4j?

                        That could solve our issue?

                        • 9. Re: Some junk characters displaying at start of jboss gc log file
                          jaikiran pai Master

                          your_shadow03 wrote:

                           

                          Hi Guys !!

                           

                          Is there any way we can add Appender for GC logs in log4j?

                          That could solve our issue?

                          This has already been answered in your other post here http://community.jboss.org/message/529671#529671

                           

                          In short, no it cannot be done. Because JVM does not use log4j.