test target

[tom.elrod]

I am working on converting my test.functional target from the old JBossRemoting build to the new jboss build. The old one looks like:

<target name="tests.functional" depends="tests.jars">
 <mkdir dir="${output.tests.results}"/>
 <junit printsummary="true" fork="yes" includeantruntime="true">
 <classpath>
 <path refid="tests.classpath"/>
 <pathelement location="${output.lib.dir}/jboss-remoting-tests.jar"/>
 </classpath>
 <!-- this is needed for the remoting.marshall.dynamic.remote.MarshallerLoadingServer -->
 <jvmarg value="-Dloader.path=${output.lib.dir}/jboss-remoting-loading-test.jar"/>
 <formatter type="xml"/>
 <batchtest fork="yes" todir="${output.tests.results}"
 haltonfailure="no">
 <formatter type="xml"/>
 <fileset dir="${tests.compile.dir}">
 <include name="**/remoting/**/*TestCase.class"/>
 <exclude name="**/remoting/**/performance/**"/>
 </fileset>
 </batchtest>
 </junit>
 </target>

There are few specializations I need. First, is want to specifiy exactly what will be included on the classpath for the test run (so jboss-remoting-loading-test.jar is not included). Then want to pass a jvmarg indicating where the jboss-remoting-loading-test.jar is located (this is a for a test of remote classloading).

My jbossbuild.xml currently looks like:

<source id="tests" test="org/jboss/test/remoting/**/*TestCase.java">
 <include input="main"/>
 <include component="apache-commons"/>
 <include component="oswego-concurrent"/>
 <include component="apache-log4j"/>
 <include component="junit-junit"/>
 <include component="sun-jmx"/>
 <include component="sun-jaxp"/>

 <include component="javagroups-javagroups"/>

 <include component="common"/>
 <include component="naming"/>
 <include component="j2se"/>
 <include component="system"/>
 <include component="aop"/>
 </source>

I am still not totally clear on the source tags and what exactly they are defining (can these be any free form strings, or do they have to be 'main', 'test', etc.?). I found where specifing a 'test' attribute with the string expresession for the test you want run will be used within the 'runtest' targetdef from task.xml (which will indeed run my tests when run the 'test' target for the build. Am not sure how to specify a string expression for the files I want excluded.

So now just need to be able to specify exactly the jars included on the classpath for the test run and how to pass the jvmarg parameter. I see that runtest uses pathElements for the classpath, but don't know where the values come from. I guess the jvmarg can be added to the runtest taskdef as well, but am not sure how to make it conditional depending of if there is a variable defined for it within the jbossbuild.xml.

Thanks.

-Tom

[ryan.campbell]

I'll start with the easiest questions first :-)

I am still not totally clear on the source tags and what exactly they are defining (can these be any free form strings, or do they have to be 'main', 'test', etc.?).

These are just the names of the directories under your src directory. They have no special meaning to the build system. So if you had a src/foo directory, you could generate compile targets for it like this:

<source id="foo"/>

which would result in a build.foo target.

Am not sure how to specify a string expression for the files I want excluded.

Yes, that's certainly a limitation. Is this an actual need at this point? If so, we could change that attribute to be an Ant patternset refid instead of the actual pattern.

So now just need to be able to specify exactly the jars included on the classpath for the test run and how to pass the jvmarg parameter. I see that runtest uses pathElements for the classpath, but don't know where the values come from.

The pathElements tag is a special tag which resolves to the inputs of your source - and the jars in the Classpath of your inputs manfest. So if you have:

<source id="tests">
 <input="jboss-common.jar"/>
</source>

The will include jboss-common.jar in your test's classpath, as well as all of jboss-common.jar's dependencies. You can see this working by using the special show target:

ant -f jbossbuild.xml show -Dshow=runtests.test

This will let you will see the target that is generated, and what the actually resolves to. Also, see the bottom of this page:
http://www.jboss.org/index.html?module=bb&op=viewtopic&t=58993&postdays=0&postorder=asc&start=10

[ryan.campbell]

I guess the jvmarg can be added to the runtest taskdef as well, but am not sure how to make it conditional depending of if there is a variable defined for it within the jbossbuild.xml.

Usually, you can add an attribute to an element like this, and then do conditional processing based on the presence of that attribute. However, in this case we have an optional parameter (your jvm arguments), which the template processing can't handle. I think we will need to extend the template language to allow you to do something like this:

<source id="test" tests="*TestCase.java">
 <input id="main"/>
 <jvmarg value="-Dloader.path=${output.lib.dir}/jboss-remoting-loading-test.jar"/>
 </source>

and then in our tasks.xml where we make the JUnit call we could have:

<junit ...
 <pathelements>
 <jvmargs/>
 </junit>

And the above would expand to include all the jmvarg elements you defined. This would be a generic mechanism for any kind of element.

Adrian? Do you see any problem with having dynamic elements like this? I haven't investigated deeply, but ant's DynamicElement seems like it might fit the bill.

We need to focus on getting the build complete, but I will add this as an enhancement in JIRA if Adrian agrees.

[adrian.brock]

I've never liked the idea of passing arguments to tests this way.

For a start, it makes it difficult to run the those tests in IDEs.

But more importantly, it also doesn't scale when more than one test
in the same suite wants to do a similar but incompatible thing.

I prefer the mechanism where the test bootstraps its own environment, though
I've never found a good generic solution to this problem.
i.e. being able to run the tests as unit tests inside the home project, and also
bundling the tests for binary distribution into some larger testsuite or QA
environment.

I guess this is one for the custom TestRunner we want to write for other reasons
like being able to simulate a JBoss crash or reboot?

[adrian.brock]

Objections and ideals over.

To reproduce what already exists, like you suggested,
you could add specialised parameters to the source definition that can affect the macro generating the junit task.

But this would require some coding.

Probably the most general solution is to allow any element to be added
to a macro by making each type implement DynamicElement not just DynamicAttribute?

<source id="tests" test="whatever">
 <junit-elements> <!-- This is just held inside the definition as a DynamicElement -->
...
 </junit-elements>
</source>

Then insert the junit-elements directly into the junit task (which might be empty).

 <targetdef target="runtest">
 <component/>
 <source when="@{test}">
 <mkdir dir="@{testDir}"/>
 <junit fork="true"
 printSummary="true">
 <formatter type="plain"/>
 <classpath>
 <pathElements/>
 </classpath>
 <batchtest todir="@{testDir}">
 <fileset dir="@{sourceDir}" includes="@{test}"/>
 </batchtest>
 <junit-elements/> <!-- HERE source/junit-elements -->
 </junit>
 </source>
 </targetdef>

The downside is the loss of validation if you mistype and sub-element, e.g.

<source id="tests">
 <incude id="main"/> // OOPS! this is now a DynamicElement when it should be a parser exception
</source>

[adrian.brock]

To make this easy to parse you could add an "elements" attribute
identify what you want to replace.

 <source when="@{test}" elements="junit-elements">
 <mkdir dir="@{testDir}"/>
 <junit fork="true"
 printSummary="true">
 <formatter type="plain"/>
 <classpath>
 <pathElements/>
 </classpath>
 <batchtest todir="@{testDir}">
 <fileset dir="@{sourceDir}" includes="@{test}"/>
 </batchtest>
 <junit-elements/> <!-- HERE source/junit-elements -->
 </junit>
 </source>

[tom.elrod]

Yes, that's certainly a limitation. Is this an actual need at this point? If so, we could change that attribute to be an Ant patternset refid instead of the actual pattern.

I really only need this to segment out the performance tests, which use the same XXXTestCase naming convention and is under the same org.jboss.test.remoting root package as the rest of the other functional remoting tests. This is because I don't always want to run the performance tests when running functional tests, as they takes a long time.

I would not be opposed to just changing the package structure to be something like org.jboss.test.remoting.functional and org.jboss.test.remoting.performance so I can segment out the different tests. The only problem I see with this, is there is no way that I know of to create another source element for this (because both would live under src/tests/ directory, both would have to be named 'tests'). I could create a src/performance/ directory and put the tests there, but this does not seem appropriate.

The pathElements tag is a special tag which resolves to the inputs of your source - and the jars in the Classpath of your inputs manfest. So if you have:

Using the show target for the runtests.test, didn't work for me:

C:\Projects\JBoss\JBossRemoting\remoting>ant -f jbossbuild.xml show -Dshow=runtests.test
Buildfile: jbossbuild.xml

BUILD FAILED
Target `show' does not exist in this project.

Total time: 0 seconds
C:\Projects\JBoss\JBossRemoting\remoting>ant -f release.xml show -Dshow=runtests.test
Buildfile: release.xml

show:

BUILD SUCCESSFUL
Total time: 0 seconds

However, using ant -v shows that '..\remoting\output\classes\main' (which I would expect due to having ) and '..\remoting\output\classes\tests' is include in the JUnit classpath when running the tests. So trying to exclude certain test jars won't matter anyways. Am not sure how to find a solution to my particular problem of scoping the classes I want to include in test runs, but maybe a solution to the previous problem (performance vs functional tests) will present a solution to this as well.

I've never liked the idea of passing arguments to tests this way.

So given that my test example may not be a good example of why jvmarg is needed, I have another, more valid one. I have been working on SSL transport for remoting. This means that I'll need to be able to declare where the key store is located via a system property (i.e. -Djavax.net.ssl.keyStore=.keystore). This one is outside my control :)

[adrian.brock]

I've never liked the idea of passing arguments to tests this way.

So given that my test example may not be a good example of why jvmarg is needed, I have another, more valid one. I have been working on SSL transport for remoting. This means that I'll need to be able to declare where the key store is located via a system property (i.e. -Djavax.net.ssl.keyStore=.keystore). This one is outside my control :)

TestCase.setup() and System.setProperty()
or
docs/guide/security/jsse/JSSERefGuide.html#CustomizingStores
For mocking something up that can be used in a testcase while still maintaining the
ability for the test to be run with other tests that also want to do SSL with different trust
configuration and/or checking failure conditions like expriing the certificate
or removing one from the trusted chain.

e.g.
http://cvs.sourceforge.net/viewcvs.py/jboss/jboss/src/main/org/jboss/invocation/http/interfaces/AnyhostVerifier.java?rev=1.6&view=markup

[ryan.campbell]

Probably the most general solution is to allow any element to be added to a macro by making each type implement DynamicElement not just DynamicAttribute?

Yes, that's what I was thinking.

I like the junit-elements idea.

The downside is the loss of validation if you mistype and sub-element.

Yes, I think is a serious flaw. I could add a in tasks.xml which would list the allowed dynamic elements to prevent silent errors such as the one above.

 <dynamicElements>
 <dynamicElement name="junit-element"/>
 </dynamicElements>

I don't think I should add this at the targetdef/template level because a given targetdef won't necessarily know/care about all dynamic elements.

http://jira.jboss.com/jira/browse/JBBUILD-70

[ryan.campbell]

The only problem I see with this, is there is no way that I know of to create another source element for this (because both would live under src/tests/ directory, both would have to be named 'tests').

For performance tests, you could add a seperate target which calls Junit directly. It would need JBBUILD-71 implemented to be able reference the classspath and other things you'll need. It is something we need in general, but will help with this particular case.

Using the show target for the runtests.test, didn't work for me:

Sorry, that should have been

ant -f jbossbuild.xml show -Dshow=runtest.tests

It is based on the ID of your source. target.id

Am not sure how to find a solution to my particular problem of scoping the classes I want to include in test runs, but maybe a solution to the previous problem (performance vs functional tests) will present a solution to this as well.


Honestly, all of this is making me wonder how much work we should put into making testing a core part of jbossbuild. I've already raised 2 JIRA issues for you to be able to run your tests, so this tells me we may need to rethink how we define tests for a project. I know aop has an extensive testsuite which seems like it would accept no generic solution.

Perhaps projects should define their own test targets, at least until we get the build part of jbossbuild complete.