Can we start writing jms test clients to check compliance wi

[rajdeep_dua]

Hi
Is the development in a state where jms api client can be written.i thought of start contribution by writing jms 1.1 compliance tests.

saw a JMSClient in org.jboss.messaging.jms.tools, but not sure where is it going to pick up the initial context properties from(there is p2p.properties in resources).any pointers will be helpful

IS there anything else where i can start contributing before diving into the core development.

[rajdeep_dua]

i was able to run a simple jms api test client with this

Properties props = new Properties();
 props.put("java.naming.factory.initial","org.jboss.messaging.util.StandaloneInitialContextFactory");
 ctx = new InitialContext(props);

please let me know if my writing test cases for jms compliance is ok
i am using test package --

org.jboss.test.messaging.jmsapi

[ovidiu.feodorov]

Thank you for your involvement.

Yes, we are at the development stage when a JMS API client can be written, or at least, we're very close to that. Please look at the JMS Facade thread http://jboss.org/index.html?module=bb&op=viewtopic&t=61556, you'll see where we are. However, that client would have a very limited functionality. A lot of JMS API calls just throw NotYetImplementedException, so I am sure that at this stage, the client API won't pass any compliance test.

The client you used, even if it is functional, it is deprecated, so I would suggest not to start testing anything on it, since we'll drop that code anyway.

Having someone who specializes compliance tests, or other kind of testing for that matter, is essential, so your help is more than welcome. I hope we'll have something that will start passing compliance tests in several weeks.

Until then, I would like you to start thinking at a more conprehensive testing and benchmarking harness that will grow toghether with the project.

This is what I have in mind:

We'll have four kinds of tests:
- unit tests: written by the developers, at the same time or before writing the code itself. "Old" XP principle. They are essential, since they'll catch regression bugs and we'll allow us to refactor more freely.
- compliance tests: your stuff. Ideally, I don't want developers involved with them, they should be performed by someone who is not closely involved with the code (or not yet), just because his/her only concern is to see how solid is the implementation from the JMS point of view.
- integration test: test how well our JMS provider integrates with the AS; mainly the QA team's job.
- stress tests and benchmarks: I hope to have a benchmarking framework that will allow us to closely follow key metrics for our implementation. As a very quick example, I am thinking of something like: how does our implementation behave when one hundred producers concurrently send 10KB messages? What are the key metrics to follow? How do these metrics evolve from a release to another? Does our implementation scale?


Are you interested in getting involved with that?
How did you think of doing compliance tests? SpecJ?

[adrian.brock]

I'd recommend reworking what is already in the testsuite
to work over both JBossMQ and JBoss Messaging:
http://cvs.sourceforge.net/viewcvs.py/jboss/jbosstest/src/main/org/objectweb/jtests/
However, that is only jms1.0.2b level tests.

Some of the JBossMQ tests are also pretty generic where they don't use
internal apis for things like creating/clearing message queues, etc.
http://cvs.sourceforge.net/viewcvs.py/jboss/jbosstest/src/main/org/jboss/test/jbossmq/

Since JBoss Inc. is a j2ee licensee we have access to the JMS 1.1. TCK,
but only our employees can run it.

It is still a good idea to have a thorough testsuite for the developers who
are not JBoss Inc employees or don't work for another j2ee licensee.

[rajdeep_dua]

Adrian objectwebs test cases you are talking about - are they same as the joram test suite.
i had intergrated it with pramat's jms server 3 years back.

i think they have upgraded the test suite for 1.1 version also.

we can integrated those test cases.

My first test client failed yesterday

connection.getClientID()

ClientID is always the tricky part

throws OperationNotsupportedException. will post the exact stact trace.

For performance test there is a grinder framework available
Sonic also has a test harness which is downloadable.
It covers many permutations/combinations outlined above.I can start looking into that also.

please let me know where should i start posting testcases?

[rajdeep_dua]

For check on compliance i am following the specification itself.
each api call has various senarios where things can bomb or atleast there is a risk!

[ovidiu.feodorov]

Try to build your compliance tests starting from what Adrian suggested
http://cvs.sourceforge.net/viewcvs.py/jboss/jbosstest/src/main/org/objectweb/jtests/
and
http://cvs.sourceforge.net/viewcvs.py/jboss/jbosstest/src/main/org/jboss/test/jbossmq/

No need to get into JBoss-specific API testing just yet. We'll do that later. If you could come up with a comprehensive JMS 1.1 test suite, that would be great.

[rajdeep_dua]

As a first step towards compliance with 1.1, i have integrated the latest testsuite from Joram which is for 1.1 compliance

This i tested with JBoss 4.0 release code since the latest jms server is still not ready.

There are total of 186 tests in this test suite, 2 failed and there were errors in 2.

I had to modify

org.objectweb.jtests.providers.admin.JBossMQAdmin

Added te following methods

public void createConnectionFactory(String name) {
 createQueueConnectionFactory(name);
 }

 public void deleteConnectionFactory(String name) {
 deleteQueueConnectionFactory(name);
 }

Here is the test summary report
commands, ant tests-objectweb-jms followed by ant tests-report-text in the testsuite module(note : i replaced the org.objectweb.* with the newer version from their CVS)

JBoss daily test results

SUMMARY

Number of tests run: 186

--------------------------------------------

Successful tests: 180

Errors: 2

Failures: 4

--------------------------------------------

[time of test: 2005-03-21.08-16 GMT]
[java.version: 1.4.2]
[java.vendor: Sun Microsystems Inc.]
[java.vm.version: 1.4.2-b28]
[java.vm.name: Java HotSpot(TM) Client VM]
[java.vm.info: mixed mode]
[os.name: Windows XP]
[os.arch: x86]
[os.version: 5.1]

Useful resources:

- http://jboss.sourceforge.net/junit-results/32/2005-03-21.08-16 for
the junit report of this test.


NOTE: If there are any errors shown above - this mail is only highlighting
them - it is NOT indicating that they are being looked at by anyone.

It is assumed that whoever makes change(s) to jboss that
break the test will be fixing the test or jboss, as appropriate!

--------------------------------------------
DETAILS OF ERRORS

Suite: org.objectweb.jtests.jms.conform.message.properties.JMSXPropertyTest
Test: testSupportsJMSXDeliveryCount
Type: failure
Exception: junit.framework.AssertionFailedError
Message: JMSXDeliveryCount property is not supported
---------------------------------
Suite: org.objectweb.jtests.jms.conform.message.properties.JMSXPropertyTest
Test: testJMSXDeliveryCount
Type: failure
Exception: junit.framework.AssertionFailedError
Message: Message property 'JMSXDeliveryCount' not set.
---------------------------------
Suite: org.objectweb.jtests.jms.conform.selector.SelectorSyntaxTest
Test: testInvalidIdentifiersStart
Type: error
Exception: org.jboss.mq.selectors.TokenMgrError
Message: Lexical error at line 1, column 1. Encountered: "%" (37), after : ""
---------------------------------
Suite: org.objectweb.jtests.jms.conform.selector.SelectorTest
Test: unknown
Type: error
Exception: junit.framework.AssertionFailedError
Message: Timeout occurred
---------------------------------
Suite: org.objectweb.jtests.jms.conform.session.QueueSessionTest
Test: testRollbackRececeivedMessage
Type: failure
Exception: junit.framework.ComparisonFailure
Message: expected:<...RollbackRececeivedMessage> but was:<...JMSDeliveryModeInSelector:1>
---------------------------------
Suite: org.objectweb.jtests.jms.conform.session.UnifiedSessionTest
Test: testCreateDurableConnectionConsumerOnQueueConnection
Type: failure
Exception: junit.framework.AssertionFailedError
Message: Should throw a javax.jms.IllegalStateException
---------------------------------

Please let me know how this can be checked in.

I think we might have to modify JBossMQAdmin if the MBean APIs etc are changing for the serverless impl.

Guys let me know your thoughts on this.

[adrian.brock]

"rajdeep_dua" wrote:

I think we might have to modify JBossMQAdmin if the MBean APIs etc are changing for the serverless impl.

Guys let me know your thoughts on this.

The idea would be to make the admin class configurable by a system property.
Then the tests could be run multiple times:
1) over JBossMQ
2) over serverless jms
3) over JBoss Messaging + JMS Facade
4) over JBoss + A.N.Other jms provider

[adrian.brock]

rajdeep, if you send me your sourceforge id/email address we can
see about getting you cvs r/w to integrate this work into the testsuite.

[rajdeep_dua]

Adrian,

My sourceforge id is: rajdeep_dua

My mail id is [e-mail]

[ovidiu.feodorov]

Rajdeep,

Please let me know when you start working on the test suite. I still don't recommend to apply it to the Mess (see http://www.jboss.org/index.html?module=bb&op=viewtopic&t=61556), but it's great if you get it ready to run on JBossMQ and you create the hooks for the Mess and other JMS provider.

[rajdeep_dua]

Hi Ovidiu,
Started working on the test suite last week end.
(Had some issues with compiling the jmx module,hence the delay)

Here is the brief of the test suite structure
Location
${jboss-head}/jms/tests
Package :
org.jboss.test.messaging.jmsapi
org.jboss.test.messaging.jmsapi.connection
org.jboss.test.messaging.jmsapi.flow
org.jboss.test.messaging.jmsapi.message
...

Actors

All TestCases extend org.jboss.test.messaging.MessagingTestCase
Provider specfic Subclasses of Abstract class org.jboss.test.messaging.util.AdminUtility : responsible for getting the initial context, binding /unbinding of the administered objects.

Property file decides which subclass of this AdminUtility gets picked up.Sub class instantiated using reflection to keep the code independent of teh actual impl

e.g
org.jboss.test.messaging.util.provider.JBossMessAdminUtility extends
org.jboss.test.messaging.util.AdminUtility

JBossMessAdminUtility uses ServerWrapper internally(for the time being)

org.jboss.test.messaging.util.provider.JBossMQAdminUtility : for JBossMQ..

Let me know if this is ok,.
{Will come out with the design proposal for the perofrmance test framework in the other thread.}

[ovidiu.feodorov]

Looks fine. Go ahead.

[ovidiu.feodorov]

An observation related to naming: org.jboss.test.messaging.util is intendend to contain tests for Messaging utility classes, not utilities. Please use org.jboss.test.messaging.tools for that.