testsuite's log4j configuration leads to large reports

[jaroslaw.kijanowski]

Testsuite's log4j was configured to print DEBUG messages to system.out.
This leads to a large TESTS-TestSuites.xml file.

There are two targets in build.xml:
tests-report-xml
tests-report-text
The first one generates testsuite/output/reports/text/TESTS-TestSuites.txt
The second one testsuite/output/reports/xml/TESTS-TestSuites.xml
Both files are small summaries (see below).

Both targets uses a XSL Transformation to transform this huge TESTS-TestSuites.xml file. If this file is larger than 100MB, JAVA's embedded XSLT is unable finish this process:
812 tests = 98 MB = 11 minutes 27 seconds
813 tests = 100 MB = 21 minutes 26 seconds
817 tests (the whole testsuite) = 111MB = OOME after over 80 minutes

So we can:
- use another XSLT - STX, but this needs new stylesheets, new libraries
- make the noisiest tests less verbose by commenting log.debug()
- make log4j less verbose (this was already done by changing level from DEBUG to FATAL)
- skip both targets above - do we need these files generated by that targets?

TESTS-TestSuites.txt:
**********************
<?xml version="1.0" encoding="UTF-8"?>
<time-of-test>2007-01-25.10-47</time-of-test><jdk-vendor>Sun Microsystems Inc.</jdk-vendor><jdk-version>1.4.2_09</jdk-version><jvm-vm-spec-version>1.0</jvm-vm-spec-version><jvm-name>Java HotSpot(TM) Client VM</jvm-name><jvm-version>1.4.2_09-b05</jvm-version><jvm-info>mixed mode</jvm-info><java-spec-version>1.4</java-spec-version><java-class-version>48.0</java-class-version><os-name>Linux</os-name><os-arch>i386</os-arch><os-version>2.6.9-42.0.2.ELsmp</os-version>4167416511
**********************

TESTS-TestSuites.txt:
**********************
JBoss daily test results

SUMMARY

Number of tests run: 4167

--------------------------------------------

Successful tests: 4165

Errors: 1

Failures: 1

--------------------------------------------



[time of test: 2007-01-25.10-47 GMT]
[java.version: 1.4.2_09]
[java.vendor: Sun Microsystems Inc.]
[java.vm.version: 1.4.2_09-b05]
[java.vm.name: Java HotSpot(TM) Client VM]
[java.vm.info: mixed mode]
[os.name: Linux]
[os.arch: i386]
[os.version: 2.6.9-42.0.2.ELsmp]

Useful resources:

- http://jboss.sourceforge.net/junit-results/32/2007-01-25.10-47 for
the junit report of this test.


NOTE: If there are any errors shown above - this mail is only highlighting
them - it is NOT indicating that they are being looked at by anyone.

It is assumed that whoever makes change(s) to jboss that
break the test will be fixing the test or jboss, as appropriate!

--------------------------------------------



DETAILS OF ERRORS



Suite: org.jboss.test.pooled.test.SSLSocketsUnitTestCase
Test: testClientCertSSLAccess
Type: error
Exception: java.rmi.AccessException
Message: SecurityException; nested exception is: javax.security.auth.login.FailedLoginException: Supplied Credential did not match existing credential for null
---------------------------------



Suite: org.jboss.test.security.test.SRPLoginModuleUnitTestCase
Test: testSRPLoginWithAuxChallenge
Type: failure
Exception: junit.framework.AssertionFailedError
Message: Unable to complete login: Failed to complete SRP login, msg=Failed to encrypt aux challenge
---------------------------------

**********************

[starksm64]

They are used for the reports summary as well as the email msgs. Perhaps we need a better formatter that produces a more concise summary that does not include all of the console output.

[dimitris]

Where are those summaries used? I mean when you browse CC results.

I very much like the stdout output on every test, so I'd rather drop the 2 summaries.

[jaroslaw.kijanowski]

Which formatter do you mean? JUnit? there are just three - xml, brief and plain... or a custom one.

We need for every test the DEBUG messages - this lead to bigger TEST-testname.xml files - this leads to a huge (unparseable by XSLT) TESTS-Testsuites.xml file.

What do you think about this:
Let's generate TEST-testname.xml files like before, with all the DEBUG messages
Let's parse 817 TEST-testname.xml files (the whole testsuite) to 817 small TEST-testname_short_version.xml files, just for the purpose to create this two summaries.
Now we could generate two TESTS-Testsuites.xml files, the big one for the html report (which does not nead a XSL Transformation) and the small one for these summaries where XSLT is involved.

Pros: We can leave DEBUG messages in the html report

Cons:
- we need a lot of more space ~ 200MB per testsuite
- it could take a while (how long? no idea...) to parse 817 files to 817 smaller files

[starksm64]

I was thinking of a custom xml formatter that produced the current TEST-testname.xml along with a TEST-summary-testname.xml that would not contain any stderr/out output to feed into the summary. It may also be easier to just write a simple summary java program and bypas the use of xsl for the simple xml/txt summaries.

[jaroslaw.kijanowski]

I think it's easier to get xml files without stderr/out by writing a new stylesheet. writing an own formatter or a java program would need to integrate it with the testsuite.

Finally these "short-version" files could be removed so additional space is not needed.

[starksm64]

If that works, great. I was not clear on whether a new stylesheet avoids the current memory problem.

[clebert.suconic]

If that works, great. I was not clear on whether a new stylesheet avoids the current memory problem.

I agree with Scott.. I don't think just replacing the XSL would fix the memory problem.

The stdout still on the XML for each test. When JUnitReport aggregates all the tests in a single XML, you will still have a problem to read that huge XML entry. (The System.out/ System.err). And that's what I think is happening.

[jaroslaw.kijanowski]

A huge TESTS-Testsuites.xml file is ok as long it is not parsed by XSLT. To generate a html report we don't use XSLT. We use XSLT just for generating these two summaries.

So my idea is to use XSLT not for the huge TESTS-Testsuites.xml, but to generate smaller TEST-testname_short_version.xml files. Then, JUnitReport will generate a second TESTS-Testsuites_short_version.xml by aggregating these "short_version" xml files and not the bigger ones.

Finally TESTS-Testsuites_short_version.xml would be used for generating summaries with the old stylesheets.