Server is not Responding ( Jboss 4.2.3.GA with JDK1.5) after

[prakashmvc]

Hi,
I am using Jboss 4.2.3.GA with JDK 1.5.
After starting the server, day after server is not responding (dead)..it means if we check the status of the server it shows RUNNING,but we can't open the console
using the address http://ipoftheserver:8080/

when I attached the visualgc to the server.visualgc status shows dead.

server is started in unix environment.If we open the browser it shows opening the page,but it never opens.

how to investigate this issue?what will be the root cause for this problem?

it would be helful if anyone gives the hint.

Regards,
Prakash

[peterj]

One thing to look at - the server/xxx/log/server.log file. The entries at the end of that file should tell you what jbossas did last.

Another thing to try is to get a thread dump: http://www.jboss.org/community/docs/DOC-12395. This will tell you what each thread is doing.

[prakashmvc]

Thanks peter for your reply..

First thing which you have addressed has already checked before posting to forum. I couldn't find any information in the server log.

Second thing which you have addressed, it's worth to look at the thread dump. I will get the thread dump and try to analyse what going wrong..

I will post the updates later..

Regards,
Prakash

[prakashmvc]

Hi Peter,
Second thing which you had mentioned is didn't work for me.
Following are the updates.

1.Signal the JVM to print out a StackTrace
JbossServer is started in UNIX environment.When I try to run the command kill -3 processid in the UNIX. Process is not killed.
After running this command, if I check the status of the Jboss server it says server is running.

2.Generate a thread dump using the jmx-console
I can't open the jbossConsole using the following url
http://localhost:8080. Page never opens but it says opening the page.
I believe without opening the jbossconsole we can't open the JMX console.
It's not listining for http://localhost:8080.

3.Deploy a thread dump JSP under Java 5.
we can't open the page http://localhost:8080/threaddump/.Server is not responding.

4.Use the jconsole when run under a Sun 1.5+ JVM
I ran the JConsole, under the remote tab I have entered the ip address of the server, portnumber,user and password. I got the message connection failed.

Do we have any alternatives to investigate this issue?

Regards,
Prakash

[peterj]

1) kill -3 does not kill the process - it sends a quit signal to the JVM which then sends the threadump to stdout. You need to ensure that the initialization script used to run jbossas redirects stdout to a file.

2, 3) Based on your earlier statements, I would not expect these to work because they require jbossas to repsond.

4) Did you configure jbossas to allow connections?

[prakashmvc]

Hi Peter,
I got some tool to analyze the thread dump.After analysis what I found is,
In jboss server connection pool is configured to allow 75 connections.
After sometime starting the server,all database connections being blocked,it shows all 75 connections are inactive.
Java code base also, we are closing the connections after performing the operations.
Because of all the database connections are blocked, server couldn't connect to database.Hence it's going to zombie state.

Is there any configuration changes required in jboss to overcome this issue?

Regards,
Prakash

[peterj]

all database connections being blocked,it shows all 75 connections are inactive.

Could you post the stack trace for one of the threads that is blocked on the database?

[prakashmvc]

Hi Peter,
we encountered same problem again but this time no database issues.
I will explain in more detail.

We are using JBoss 4.2.3 in production. its' using sun solaris operating system.

Sometimes server is not responding.we can't open http://ip:8080/
it won't respond for this url.

But if we check the status of the server it says server is running? no error messages been logged in the server.log file.

Since no error messages logged in the log file, is there any way to find why the server is not responding?

Thread dump also, was not useful in this situation as I analyzed.

Regards,
Prakash

[jaikiran]

Sometimes server is not responding.we can't open http://ip:8080/
it won't respond for this url.

What exactly happens? Are there any other system level issues when this happens - like network outage?

[prakashmvc]

When it happened,
1. Network was fine, I could able to ping the server ip address
2. In server log no error messages
3. We are using unix script to start,stop and check the status of the server.
status of the server was showing 'Running'
4. when we open the the browser with http://ip:8080/
statusbar shows connecting....After some time approximately 10 or 15
mins it says page not found

[alberto.silveirajunior]

I'm having a very similar problem with JBOSS 4.0.5GA.

We have a very specific application where each client sends one request every 0.9 secs (Using AJAX for that).

It works fine for approximately 2 days and half and the the server stops to respond requests. After that is not possible access http://ip:8080/ or http://ip:8080/myApplications.

I monitored heap size, memory, threads, and everything looks fine. No errors on log files. It just stop to respond requests...but if I check the server status it says it is running with no problems.

Please...if you have any suggestion about what I can do to solve this problem....please let me know!

Thanks a lot.

[vpraveen]

Any solution to it ??? Me to face the same problem...

Thanks in advance,

Praveen V

[alberto.silveirajunior]

I'm still stuck on this problem. I can't even reproduce the problem in QA, which is a mirror from production.

I've been monitoring JVM, JBOSS, memory, heap memory, threads, network, etc, ...in production and I could not find anything so far.

I notice the following log message when I'm shooting the JBOSS down every night:

"[org.apache.catalina.core.StandardWrapper] Waiting for 854 instance(s) to be deallocated"

Could this message be related to this symptoms?

Please any comments/help are welcome!!!

Thanks....

[jaikiran]

"alberto.silveirajunior" wrote:

I notice the following log message when I'm shooting the JBOSS down every night:

"[org.apache.catalina.core.StandardWrapper] Waiting for 854 instance(s) to be deallocated"

Could this message be related to this symptoms?

Probably yes. Looking at the code of the StandardWrapper (which is a Tomcat thing) http://svn.apache.org/repos/asf/tomcat/container/tc5.5.x/catalina/src/share/org/apache/catalina/core/StandardWrapper.java, this is the number of servlet instances that have been created for the servlet. And 854 is really a very high number. From what i see the maxInstances by default is 20 so i don't know how 854 instances were created in first place. Are any configurations changed on your setup?

[jaikiran]

"alberto.silveirajunior" wrote:


"[org.apache.catalina.core.StandardWrapper] Waiting for 854 instance(s) to be deallocated"

By the way, a quick google search tells me the reason why the instances are not unloaded :

Usually servlets that haven't finished servicing requests.