8 Replies Latest reply on Feb 16, 2007 3:21 PM by jaroslaw.kijanowski

    testcases overwritten

    akostadinov

      See http://jira.jboss.com/jira/browse/JBAS-4103 for descripttion of the problem.

      Any ideas how to fix? Should we rename testcases run by all but tests-standard-unit target?



      Below is what I found about it for 4.0 branch (there could be some more not caught here):
      1. cat tests.log | grep "\[junit\] Running" | sed -e 's/.\+ Running
      \(.\+\)/\1/' | sort | uniq -d > tests.log.dup
      2. sed -e 's/.\+nowrap.\+href="\(.\+\)\.html".\+/\1/p' -e 'd'
      reports/allclasses-frame.html | sed -e 'y/\//./' > tests.log.report
      3. cat tests.log.* | sort | uniq -d

      I get these (45 total):
      org.jboss.test.bmp.test.BmpUnitTestCase
      org.jboss.test.bmp.test.SmallCacheBmpUnitTestCase
      org.jboss.test.cmp2.ageout.test.JDBC2PmAgeOutUnitTestCase
      org.jboss.test.cmp2.audit.test.AuditUnitTestCase
      org.jboss.test.cmp2.batchcascadedelete.test.BatchCascadeDeleteUnitTestCase
      org.jboss.test.cmp2.cacheinvalidation.test.CacheInvalidationUnitTestCase
      org.jboss.test.cmp2.cacheinvalidation.test.JDBC2PmCacheInvalidationUnitTestCase
      org.jboss.test.cmp2.cmrstress.CMRStressTestCase
      org.jboss.test.cmp2.cmr.test.CMRPostCreatesWrittenUnitTestCase
      org.jboss.test.cmp2.cmrtransaction.test.CMRTransactionUnitTestCase
      org.jboss.test.cmp2.cmrtree.test.CascadeDeleteUnitTestCase
      org.jboss.test.cmp2.commerce.CompleteUnitTestCase
      org.jboss.test.cmp2.dbschema.relationship.RelationshipSchemaUnitTestCase
      org.jboss.test.cmp2.distinct.DistinctUnitTestCase
      org.jboss.test.cmp2.ejbselect.EJBSelectUnitTestCase
      org.jboss.test.cmp2.ejbselect.JDBC2PmEJBSelectUnitTestCase
      org.jboss.test.cmp2.enums.test.EnumUnitTestCase
      org.jboss.test.cmp2.enums.test.JDBC2PmEnumUnitTestCase
      org.jboss.test.cmp2.fkmapping.test.FKMappingUnitTestCase
      org.jboss.test.cmp2.fkstackoverflow.test.FKStackOverflowUnitTestCase
      org.jboss.test.cmp2.idxandusersql.test.IdxAndUsersqlUnitTestCase
      org.jboss.test.cmp2.jbas1361.JBAS1361UnitTestCase
      org.jboss.test.cmp2.jbas1665.JBAS1665UnitTestCase
      org.jboss.test.cmp2.jbas3541.JBAS3541UnitTestCase
      org.jboss.test.cmp2.jbas3541.JDBC2PmJBAS3541UnitTestCase
      org.jboss.test.cmp2.jbas3717.JBAS3717UnitTestCase
      org.jboss.test.cmp2.jbas979.JBAS979UnitTestCase
      org.jboss.test.cmp2.keygen.test.KeyGenerationUnitTestCase
      org.jboss.test.cmp2.lob.LOBUnitTestCase
      org.jboss.test.cmp2.optimisticlock.test.OptimisticLockUnitTestCase
      org.jboss.test.cmp2.partitionindex.PartitionIndexUnitTestCase
      org.jboss.test.cmp2.passivation.test.EntityPassivationUnitTestCase
      org.jboss.test.cmp2.perf.test.PerfUnitTestCase
      org.jboss.test.cmp2.readonly.ReadonlyUnitTestCase
      org.jboss.test.cmp2.relationship.RelationshipUnitTestCase
      org.jboss.test.cmp2.simple.PageSizeUnitTestCase
      org.jboss.test.cmp2.simple.SimpleUnitTestCase
      org.jboss.test.jacc.test.external.AuthzDelegationUnitTestCase
      org.jboss.test.jmx.test.RMIAdaptorAuthorizationUnitTestCase
      org.jboss.test.security.test.auth.AppCallbackHandlerUnitTestCase
      org.jboss.test.testbean.test.BeanUnitTestCase
      org.jboss.test.testbyvalue.test.ByValueUnitTestCase
      org.jboss.test.webservice.jbws309.JBWS309TestCase
      org.jboss.test.web.test.FormAuthUnitTestCase
      org.jboss.test.web.test.UserInRoleUnitTestCase

      -----------------------------------------------------------------------------------------

      Below I what Jarek found about 4.0.5.GA_CP status:
      tests.log:
      number of test cases:
      grep "Running" tests.log|wc -l
      730

      number of tests:
      grep "Tests run" tests.log|awk '{print $4}'|awk -F, '{sum+=$1} END {print sum}'
      4182

      html log:
      number of test cases:
      726 (4 test cases lost)

      number of tests:
      4167 (15 tests lost)

      These tests are run twice, but I get only one report:
      org.jboss.test.jmx.test.RMIAdaptorAuthorizationUnitTestCase (2 tests)
      org.jboss.test.security.test.auth.AppCallbackHandlerUnitTestCase (3
      tests)

      This test is run five times, but only three reports:
      org.jboss.test.webservice.jbws309.JBWS309TestCase (5 tests)

        • 1. Re: testcases overwritten
          brian.stansberry

          The run-junit target takes a param "junit.configuration" which if set will be appended to the report name. All the places in the testsuite where this is called should set a value for this parameter. Places where the tests are executed not via run-junit (see tests-clustering-unit target) should include this kind of stuff in the call to junit:

          <!-- definition of the property for keeping results between configuration -->
          <sysproperty key="jboss-junit-configuration" value="${jboss-junit-configuration}"/>
          <formatter classname="org.jboss.ant.taskdefs.XMLJUnitMultipleResultFormatter" usefile="${junit.formatter.usefile}" extension="-${jboss-junit-configuration}.xml" />
          


          Here ${jboss.junit.configuration} is passed in by the target that calls this one. If that kind of indirection isn't used (e.g. in tests-security-manager), the use of ${jboss.junit.configuration} could skipped and the value set directly.

          • 2. Re: testcases overwritten
            akostadinov

            Thanks, Brian.

            I would like to hear if the best approach is renaming or there is some better alternative.

            And if we consider to rename how should that be done. For example should tests run by tests-security-manager target be named "testname(security-manager)"?

            • 3. Re: testcases overwritten
              starksm64

              The tests cannot be renamed in that fashion because testname(security-manager) is not a valid java package name. Tests that require a specific configuration should have a distinct naming convention that excludes them from the less selective runs. Tests that can be run under multiple configurations should use the junit.configuration value to distinguish the run.

              In general it looks like we need to go through the duplicates and determine if its because of poor names or poor exclusion filters.

              • 4. Re: testcases overwritten
                akostadinov

                By renaming I mean to use junit.configuration so tests appear renamed in reports. Thus testname(security-manager) means we use junit.configuration=security-manager.

                So I see two options:
                1. use junit.configuration value to rename all tests except these run by tests-standard-unit target.
                2. resolve duplicates by tunning include/exclude patterns and invent some mechanism to notify if we have duplicate testcases again.

                "1" seems much easier to me.

                Let me know what do you think.

                • 5. Re: testcases overwritten
                  starksm64

                  Ok, yes I would start with 1 and then work to update tests filters/names for tests that do not need to be run multiple times.

                  • 6. Re: testcases overwritten
                    akostadinov

                    Agreed.

                    One more thing to clear is which branches we should do that to?

                    4.0, 4.2 and trunc I think? What about released versions?

                    • 7. Re: testcases overwritten

                      The CP versions can be taken care of by Jarek.

                      • 8. Re: testcases overwritten
                        jaroslaw.kijanowski

                        Sure, I will update 3.2.7, 4.0.2, 4.0.3, 4.0.4 and 4.0.5