-
1. Re: Updating the server task to detect OutOfMemoryErrors
ryan.campbell Mar 9, 2007 10:52 AM (in response to starksm64)This is going to require integration between the server tasks and a custom test runner.
IE, the test runner would ask the server manager to check for liveness of each running server between each test. Just a JMX call to the ServerInfo bean or whatever. -
2. Re: Updating the server task to detect OutOfMemoryErrors
starksm64 Mar 9, 2007 11:15 AM (in response to starksm64)I moved this to the testsuite forum.
-
3. Re: Updating the server task to detect OutOfMemoryErrors
wolfc Mar 14, 2007 6:03 PM (in response to starksm64)I don't think that is good enough. The OOM is preceeded by 1 hour of almost continious garbage collection and then something dies, but you can't predict what. I don't see how a service can check whether everything is alive. Even trying an uncaught on the ThreadGroup will probably get killed.
The only thing I've found to be working is an external program monitoring the log file for OOM. -
4. Re: Updating the server task to detect OutOfMemoryErrors
ryan.campbell Mar 14, 2007 9:03 PM (in response to starksm64)The JMX call could have a timeout of 10 seconds. We could just spawn a thread and sleep for 10 seconds, right? If you don't get an answer, the server is dead.
I agree you couldn't check whether everything is alive, just whether there is any memory left. I think Dimitris created an MBean to do this?
The problem with the log file approach is that the all log gets huge, and this would really slow down the runs, even when the server is fine. The JMX call should be performant if things are ok, which is the main case. -
5. Re: Updating the server task to detect OutOfMemoryErrors
ryan.campbell Mar 15, 2007 8:55 AM (in response to starksm64)A simple implementation of this would extend the org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner class. The endTest() method could be overriden to call a new method on org.jboss.test.util.ServerController which requests a liveness check for all running servers.
-
6. Re: Updating the server task to detect OutOfMemoryErrors
dimitris Mar 15, 2007 12:53 PM (in response to starksm64)There is a memory monitor that can emit notifications at warning and critical levels. You can combine this with a scripting listener that gets the notification and does something (shutdown the server with a special exit code?) We can do this out of the box with a simple -service.xml descriptor.
-
7. Re: Updating the server task to detect OutOfMemoryErrors
ryan.campbell Mar 17, 2007 6:40 PM (in response to starksm64)Ok thanks Dimtris. I don't think we want to modify the configurations under test, so we'll just make a call to ServerInfo bean between each test. If we get below 5% free memory, we can halt the test run and kill any servers.
IE, call the jboss.system:type=ServerInfo mbean and compare the FreeMemory to the TotalMemory attribute. -
8. Re: Updating the server task to detect OutOfMemoryErrors
ryan.campbell Mar 17, 2007 6:55 PM (in response to starksm64)Assigned to Martin: http://jira.jboss.com/jira/browse/JBAS-4211
-
9. Re: Updating the server task to detect OutOfMemoryErrors
mvecera Mar 19, 2007 6:26 AM (in response to starksm64)So I am supposed to override the JUnitTestRunner class and in the endTask() method call the ServerInfo mbean and check for free memory? Is it right?
-
10. Re: Updating the server task to detect OutOfMemoryErrors
mvecera Mar 19, 2007 10:47 AM (in response to starksm64)As I see, there is only one method to get the thing working. I have to write my own Ant task and run it against the started server after every junit task is called...
-
11. Re: Updating the server task to detect OutOfMemoryErrors
mvecera Mar 19, 2007 10:58 AM (in response to starksm64)As I see, there is only one method to get the thing working. I have to write my own Ant task and run it against the started server after every junit task is called...
-
12. Re: Updating the server task to detect OutOfMemoryErrors
ryan.campbell Mar 19, 2007 3:24 PM (in response to starksm64)Yeah, that's the jist of it. Though as Carlo pointed out, if the server is experiencing an OOME, you won't get a response. So you need some kind of timeout on the call to the server.
-
13. Re: Updating the server task to detect OutOfMemoryErrors
ryan.campbell Mar 19, 2007 3:30 PM (in response to starksm64)(Just saw your second post)
If you mean that you need to extend the JUnitTask, I agree; there doesn't seem to be a way to substitute your own TestRunner implementation.
Studying JUnitTask and JUnitTestRunner in the Ant source distro are good first steps for this task. -
14. Re: Updating the server task to detect OutOfMemoryErrors
mvecera Mar 20, 2007 10:07 AM (in response to starksm64)I have created new ant task server:memcheck (MemCheckServerTask. java), see file attached to JBAS-4211.
It must be put to Branch_4_2 and diff files must be applied.
Then you can run ./testsuite/build.sh jboss-memcheck-tests.
The only problem is that you must have build/output/jboss-xxx/client/jbossall-client.jar in CLASSPATH environment variable for the task to work.
I was not able to solve the classpath problem. Could anybody just have a look on it to help me a bit please?