14 Replies Latest reply on Apr 17, 2004 11:05 AM by acoliver

    TODO: Script Runner

    acoliver

      Client Side:

      This script runner should:

      1. read a script file
      2. Connect to a port
      3. run the script against the server validating replies.
      4. repeat deviences and fail with asserts in junit tests

      It will need to offer some "IF/THEN/ELSE" semantics in order to deal with multiple acceptable states/configurations as well as wildcards (because time/date, etc might vary)

      There is code ready for this, but it needs more work.

      Server Side:

      This script runner should:

      1. read a script file
      2. listen on a port
      3. wait for the client to send the expect output and then return the replies
      4. log everything

      This will serve as a dev tool. I've already needed this several times but haven't gotten to it yet. The reason being that its very difficult to debug why certain clients work and don't without seeing what they send. Yes we can log that but it would be useful to run through scenarios that are much harder to replicate without throw away code.

      Estimated time: 2-4 days (including some scripts)

        • 1. Re: TODO: Script Runner
          mikea-xoba

          may be interested in tackling this.

          just want to make sure that this is solely seen as a development or testing tool, and not something for production.

          if just for development, should be pretty straightforward. if for production, may be more difficult because then we'd have security to worry about too.

          basically, i wouldn't want to create a scripting capability just to see some viruses taking advantage of it!

          • 2. Re: TODO: Script Runner
            kabirkhan

            AFAIK it is just for development/testing. Here's some stuff from Andy a month or so ago:

            We’ll need to create a “injbosstest” target soon. I’m working on a script runner which will let us write branched conditional scripts like below inside of unit tests. I’m also writing a “server side” one which will let us debug things easier.

            S: 220 %ANYTHING%
            C: EHLO noone.nowhere.com
            END
            S: 250-%ANYTHING% Hello noone.nowhere.com (localhost.localdomain [127.0.0.1])
            C: %NOTHING%
            END
            S: 250 AUTH LOGIN PLAIN
            C: AUTH LOGIN
            END
            S: 334 VXNlcm5hbWU6
            C: %encode(test)
            END
            S: 334 UGFzc3dvcmQ6
            C: %encode(testpw)
            END
            S: 235 Authentication Successful
            C: MAIL FROM: <test@badmojo.superlinksoftware.com>
            END
            S: 250 Sender <test@badmojo.superlinksoftware.com> OK
            C: RCPT TO: <test@badmojo.superlinksoftware.com>
            END
            S: 250 Recipient <test@badmojo.superlinksoftware.com> OK
            C: DATA
            END
            S: 354 Ok Send data ending with .
            C: bla
            bla
            .
            END

            • 3. Re: TODO: Script Runner

              How do you see tying JUnit into this? I hacked my own assert thing for some tests a few years ago, where you send in input, send in expected output, and compare to actual output.

              So you would put the expected output in the text file as well, and then you could do asserts off of that.

              Steve

              • 4. Re: TODO: Script Runner
                mikea-xoba

                not so sure yet since i'm still working on the core stuff. i'd certainly like to put on a junit facade since it'd be good to run these script tests along with all the others.

                this stuff's not unit testing per se, but rather more like integration testing, since we'll ultimately also want to test for things like bounce messages getting delivered etc., and that'll involve coordinating multiple subsystems.

                • 5. Re: TODO: Script Runner
                  acoliver

                  It'll work fine with Junit. I've done it before. JAMES does something similar. JBoss even does something similar. What won't work is multi-system testing. For that we'll need yet another ant target which requires you to do your own set up. That will be more like a Test kit than a set of unit tests. This is more for testing more basic stuff that we can't test on a component basis.

                  • 6. Re: TODO: Script Runner
                    mikea-xoba

                    still plodding along on this... i think its coming along well, though. hope to commit something in the not too distant future and have an impact on M1.

                    my notion of 'multi-system' was rather 'multi-subsystem' where in addition to connecting as a client to the server on port 9000, it would also put up a 'dummy server' on port 25 in the background to make sure certain test messages got delivered via jboss-mail within a specified period of time (including bounces). doesn't seem too difficult to do.

                    • 7. Re: TODO: Script Runner
                      acoliver

                      Its a great idea. For unit test environments I'd rather not run on standard ports. The reasons should be obvious. Port 9000 is likely used by nothing, port 25 is likely monitored by sendmail or something. If we run the unit tests on port 9000, 9001,9002,9004 we have a lot less pain. I think this can wait till after M1. I'm feeling we're ready for M1 with a few build changes and documentation.

                      • 8. Re: TODO: Script Runner
                        mikea-xoba

                        probably a good idea. in that case the 'test server' would deliver to a nonstandard port in addition to receiving on a nonstandard port (9000, etc).

                        • 9. Re: TODO: Script Runner
                          mikea-xoba

                          committed something which is not precisely ready for prime-time yet, but nevertheless can currently pull its weight for some basic tests on the client side and is already integrated into our junit testing target. i'll be filling in the weak and missing spots over the next few days. will also document it.

                          its basically an extensible pluggable framework for scripts running against a protocol (like smtp or pop or nntp etc etc), where scripts, processing modules, and expected responses are described in a validated xml file. that file is now src/resources/prot/smtp-scripts.xml.

                          the junit target is currently running against my server 'ceau.com' (a miserable ancient underprovisioned pc --- cpu fan's been broken for 4 years now) running our out-of-the box jboss-mail on port 9000, but that can be configured in the xml to anywhere else.

                          mike

                          • 10. Re: TODO: Script Runner
                            acoliver

                            This is good news. I suggest either localhost by default but overridable or getHostName by default as opposed to XML. Also maybe the override could be a simple environment variable that we could use throughout "MS_HOSTNAME=foo.org" for the build... I don't really want to do a manual edit of a file for build/unit where it can be helped.

                            • 11. Re: TODO: Script Runner
                              mikea-xoba

                              right. my choice of 'ceau.com' was mainly motivated by not wanting to get in the way of other tests which put up Server mbeans, for instance, that listen on port 9000 by default --- so a fully running externally set-up localhost server on port 9000 would tend to break the unit tests.

                              what about starting/stopping a full localhost jboss-mail server in the setUp() and tearDown() methods of the test? in a sense its similar to putting up a single mbean like in some of the other tests, but of course more extensive since 400-500 other mbeans would follow along... but if we did that, it may still wind up saving some time since the environment would be preset and the penalty would be only 1-2 minutes for starting/stopping a full jboss deployment. but then if we started a full jboss server in the background for testing, we'd probably try to get efficiencies with the other tests too by not putting up single mbeans in them.

                              any comments/thoughts/criticisms welcome --- i'm not really sure what the best way to do this is.

                              • 12. Re: TODO: Script Runner
                                acoliver

                                I'm planning to change the default port to 25/110. 9000 and 9001 was just for while I didn't want anyone to ACCIDENTLY set up an open relay while we worked out the kinks. I'm confident that our defaults won't be an open relay at least by M1. I'm planning to cut M1RC1 this weekend.

                                • 13. Re: TODO: Script Runner
                                  mikea-xoba

                                  great news for M1RC1. i'm minimally tidying up/filling out some of the code (mainly script runner) and plan to commit shortly. but i noticed in another forum thread you mentioned not to commit till after RC1, and i wasn't sure that would apply here. (just let me know if so and i'll postpone).

                                  with regard to ports and external jboss-mail servers etc, the junit tests which run these scripts fail with explanatory failure messages, so if someone runs the tests without running against a full mail server the failure messages will indicate the scripts couldn't connect. i think this behaviour should hold us over till we figure out the best way to set the script runner up.

                                  mike

                                  • 14. Re: TODO: Script Runner
                                    acoliver

                                    go ahead. See the TestServer and all for how I think we should set up. Just start the pieces we need inside of the unit tests. Later we'll have multinode tests. Right now we need basic tests like "if there is user XYZ in the respository and I send him a mail from himself, does it get delivered" and the such. End to end testing of the SMTP/POP stuff.