-
1. Re: I cann't kill jboss.exe in any way
chengwen Nov 3, 2011 9:59 AM (in response to chengwen)I am trying hard to find the reason of the hang, But without log, it is very, very difficult.
The memory, the cpu usage looks like as normal.
And I cann't find any error in DB.
No error logs in server.log of jboss.
So, what happend?
The only difference of the two servers is the number of clients.
I used jmx to record the request/second , the 4000-clients server got 300 requests/second on average when it is running.
-
2. Re: I cann't kill jboss.exe in any way
jaikiran Nov 1, 2011 2:30 AM (in response to chengwen)How do you start the server? And what is jboss.exe?
-
3. Re: I cann't kill jboss.exe in any way
chengwen Nov 1, 2011 3:21 AM (in response to jaikiran)Thanks jaikiran.
I register JBoss as a windows service using JavaService.
JavaService allows server-type Java programs to be run in the background as a Windows NT/2000/XP system service (aka Unix daemon process).
JavaService will providers a file named "javaService.exe" to help you to register jboss as a windows service.
I renamed that file to "jboss.exe".
Then when the service starts, there is a process named "jboss.exe" in windows task manager.
I tried to use jbosssvc.exe which is already provided in jboss 6. But some parameters cannot be set by that. So I changed to JavaService.
-
4. Re: I cann't kill jboss.exe in any way
chengwen Nov 3, 2011 10:01 AM (in response to jaikiran)When Jboss is running normally, you can do anything, stop jboss service, restart it , login the web, etc.
But after it's "hung" suddenly, you cannot stop it any longer. The jboss service is running like it is still ok, the usage of the cpu and memory has no change.
The only two differences are that all HTTP request got no response. And the jboss processing log stopped exporting to my_running_log.log.
The two servers provides the same applications. The only difference is the client's number of them.
I wonder there must be something wrong with my code. Or the usage of the memory? The jvm parameters ?
I used -Xms256M -Xmx512M
-
5. Re: I cann't kill jboss.exe in any way
jaikiran Nov 1, 2011 3:20 AM (in response to chengwen)Have you checked locations like JBOSS_HOME/bin to see if any JVM dump files are created (if it crashed)? What memory usage does the server show when it's "hung"?
-
6. Re: I cann't kill jboss.exe in any way
chengwen Nov 1, 2011 3:31 AM (in response to jaikiran)"Hung" is better than "crash" to describe the situation.
Yes I do checked. There is no JVM dump files named "hs_err_pid***.log" in JBOSS_HOME/bin.
When the service is "hung", the memory usage is about 850M.
When the service is "ok", the memory usage is also about 850M.
-
7. Re: I cann't kill jboss.exe in any way
chengwen Nov 1, 2011 4:04 AM (in response to jaikiran)And last Friday the 'hung' last for about 4 hours before I noticed.
It looks like that the "hung" cannot be recovered by itself, at least it can't do that in 4 hours
I have to find the reason of the "hung".
Before solve the problem, I have to find the way to restart jboss quickly, without restarting the computer.
(It is cost 17 seconds to restart jboss when it can be restart, but it cost over 25 minutes to restart the computer)
-
8. Re: I cann't kill jboss.exe in any way
peterj Nov 2, 2011 10:59 AM (in response to chengwen)JavaService is not recommended for use with JBoss AS, see (about 1/2 way down the page):
http://community.jboss.org/wiki/RunJBossAsAServiceOnWindows
>>I tried to use jbosssvc.exe which is already provided in jboss 6. But some parameters cannot be set by that.
What parameters? You should be able to set anything using service.bat and jbosssvc.exe.
-
9. Re: I cann't kill jboss.exe in any way
chengwen Nov 15, 2011 9:00 AM (in response to peterj)Thanks Peter.
These days, in order to solve the "hung" problem, I tried the following methods:
1) Because of the bug of JavaService, I started jboss with run.bat instead of starting it as a windows service.
2) I moved the jboss server to the most powerful computer I have, which has 16G memory and very good CPUs.
3) In that best machine, I change the JVM parameters to -Xmx2560M.
With the above changes, the server still hung.
What's more, I run JConsole to watch the status of jboss.
I found that when the server hung, all threads in the tomcat thread pool disappeared.
In another word, when the server is working, the threads number of the process I found in the windows task manager is about 940. And when the server hungs, the threads number reduced to about 140. I use JStack to see the threads detail, and I found that the disappreaded 800 threads are all the threads in tomcat thread pool.
Part of my server.xml is as follows:
<Executor name="tomcatThreadPool" className="org.apache.catalina.core.StandardThreadExecutor" namePrefix="catalina-exec-" maxThreads="800" minSpareThreads="50" maxIdleTime="60000"/>
<Connector protocol="HTTP/1.1" port="${jboss.web.http.port}" address="${jboss.bind.address}"
connectionTimeout="180000" enableLookups="false" maxThreads="1024"
strategy="ms" maxHttpHeaderSize="8192" disableUploadTimeout="true"
redirectPort="${jboss.web.https.port}" acceptCount="1528"
executor="tomcatThreadPool"/>
Form server.xml, you can see that I used tomcatThreadPool. The name of the thread in that pool begins with "catalina-exec-". And from the file export by JStack, I found 800 "catalina-exe-" threads before and 0 after the server hung.
Dose anyone know why the threads in tomcatThreadPool disappears? What happened?
Here is the JConsole SnapShot. See if it helps.