9 Replies Latest reply on Nov 15, 2011 9:00 AM by chengwen

    I cann't kill jboss.exe in any way

    chengwen

      I am using jboss 6.0.0.final as a server to provide some service

      os : windows server 2003.

      jdk: jdk1.6

      memory: 4G

      two machines : one jboss with 1000 client , another with about 4000.

       

      In the machine with 1000 clients, the jboss server is running perfectly well. It is running fast and stable for over 1 months.

       

      But in the machine with 4000 clients, the jboss server crashed everyday once or twice without any error log, suddenly.

      Before It's crashing, the log stopped first. And then the server will not response to any request.

      Then I tried to kill jboss.exe in the windows task manager, but it cann't be killed.

      I used the command of "ntsd -c q  -p" , but it doesn't work.

      I used the tool "procexp.exe", but nothing changed.

       

      Then, in order to solve the problem, the only way left is to restart the whole computer.

       

      I wonder why jboss.exe cann't be killed after it is crashed , and how to restart jboss without restarting the computer.

       

      Thank in advance very, very much!

        • 1. Re: I cann't kill jboss.exe in any way
          chengwen

          I am trying hard to find the reason of the hang, But without log, it is very, very difficult.

           

          The memory, the cpu usage looks like as normal.

          And I cann't find any error in DB.

          No error logs in server.log of jboss.

           

          So, what happend?

           

          The only difference of the two servers is the number of clients.

          I used jmx to record the request/second , the 4000-clients server got 300 requests/second on average when it is running.

          • 2. Re: I cann't kill jboss.exe in any way
            jaikiran

            How do you start the server? And what is jboss.exe?

            • 3. Re: I cann't kill jboss.exe in any way
              chengwen

              Thanks jaikiran.

               

              I register JBoss as a windows service using JavaService.

              JavaService allows server-type Java programs to be run in the background as a Windows NT/2000/XP system service (aka Unix daemon process).

               

               

              JavaService will providers a file named "javaService.exe" to help you to register jboss as a windows service.

              I renamed that file to "jboss.exe".

              Then when the service starts, there is a process named "jboss.exe" in windows task manager.

               

              I tried to use jbosssvc.exe which is already provided in jboss 6. But some parameters cannot be set by that. So I changed to JavaService.

              • 4. Re: I cann't kill jboss.exe in any way
                chengwen

                When Jboss is running normally, you can do anything, stop jboss service, restart it , login the web, etc.

                But after it's "hung" suddenly, you cannot stop it any longer. The jboss service is running like it is still ok, the usage of the cpu and memory has no change.

                The only two differences are that all HTTP request got no response. And the jboss processing log stopped exporting to my_running_log.log.

                 

                The two servers provides the same applications. The only difference is the client's number of them.

                 

                I wonder there must be something wrong with my code. Or the usage of the memory? The jvm parameters ?

                I used -Xms256M -Xmx512M

                • 5. Re: I cann't kill jboss.exe in any way
                  jaikiran

                  Have you checked locations like JBOSS_HOME/bin to see if any JVM dump files are created (if it crashed)? What memory usage does the server show when it's "hung"?

                  • 6. Re: I cann't kill jboss.exe in any way
                    chengwen

                    "Hung" is better than "crash" to describe the situation.

                     

                    Yes I do checked. There is no JVM dump files named "hs_err_pid***.log" in JBOSS_HOME/bin.

                     

                    When the service is "hung", the memory usage is about 850M.

                    When the service is "ok", the memory usage is also about 850M.

                    • 7. Re: I cann't kill jboss.exe in any way
                      chengwen

                      And last Friday the 'hung' last for about 4 hours before I noticed.

                      It looks like that the "hung" cannot be recovered by itself, at least it can't do that in 4 hours

                       

                      I have to find the reason of the "hung".

                      Before solve the problem, I have to find the way to restart jboss quickly, without restarting the computer.

                      (It is cost 17 seconds to restart jboss when it can be restart, but it cost over 25 minutes to restart the computer)

                      • 8. Re: I cann't kill jboss.exe in any way
                        peterj

                        JavaService is not recommended for use with JBoss AS, see (about 1/2 way down the page):

                        http://community.jboss.org/wiki/RunJBossAsAServiceOnWindows

                         

                        >>I tried to use jbosssvc.exe which is already provided in jboss 6. But some parameters cannot be set by that.

                        What parameters? You should be able to set anything using service.bat and jbosssvc.exe.

                        • 9. Re: I cann't kill jboss.exe in any way
                          chengwen

                          Thanks Peter.

                           

                          These days, in order to solve the "hung" problem,  I tried the following methods:

                           

                          1) Because of the bug of JavaService, I started jboss with run.bat instead of starting it as a windows service.

                          2) I moved the jboss server to the most powerful computer I have, which has 16G memory and very good CPUs.

                          3) In that best machine, I change the JVM parameters to -Xmx2560M.

                           

                          With the above changes, the server still hung.

                           

                          What's more, I run JConsole to watch the status of jboss.

                          I found that when the server hung, all threads in the tomcat thread pool disappeared.

                          In another word, when the server is working, the threads number of the process I found in the windows task manager is about 940. And when the server hungs, the threads number reduced to about 140. I use JStack to see the threads detail, and I found that the disappreaded 800 threads are all the threads in tomcat thread pool.

                           

                          Part of my server.xml is as follows:

                           

                          <Executor name="tomcatThreadPool" className="org.apache.catalina.core.StandardThreadExecutor" namePrefix="catalina-exec-" maxThreads="800" minSpareThreads="50" maxIdleTime="60000"/>

                           

                          <Connector protocol="HTTP/1.1" port="${jboss.web.http.port}" address="${jboss.bind.address}"

                                     connectionTimeout="180000" enableLookups="false"  maxThreads="1024"

                                   strategy="ms" maxHttpHeaderSize="8192" disableUploadTimeout="true"

                                   redirectPort="${jboss.web.https.port}" acceptCount="1528"

                                   executor="tomcatThreadPool"/>

                           

                           

                          Form server.xml, you can see that I used tomcatThreadPool. The name of the thread in that pool begins with "catalina-exec-". And from the file export by JStack, I found 800 "catalina-exe-" threads before and 0 after the server hung.

                           

                          Dose anyone know why the threads in tomcatThreadPool disappears? What happened?

                           

                          Here is the JConsole SnapShot. See if it helps.

                          JConsole_1102.png