1 2 Previous Next 23 Replies Latest reply on May 8, 2008 9:58 AM by clebert.suconic

    Build broken with new AIO tests

    timfox

      The test run is currently broken since the new AIO tests require configure to be manually run from the native directory before running the test suite.

      A user should be able to just run ant unit-tests without any extra stages.

      Clebert - can you automate this please?

        • 1. Re: Build broken with new AIO tests
          timfox

          Even after running ./native/configure all the new aio tests fail for me:

          AIOSequentialFileFactoryTest and RealAIOJournalImplTest:

          java.lang.UnsatisfiedLinkError: init
           at org.jboss.messaging.core.asyncio.impl.AsynchronousFileImpl.init(Native Method)
           at org.jboss.messaging.core.asyncio.impl.AsynchronousFileImpl.open(AsynchronousFileImpl.java:106)
           at org.jboss.messaging.core.journal.impl.AIOSequentialFile.open(AIOSequentialFile.java:123)
           at org.jboss.messaging.core.journal.impl.JournalImpl.createFile(JournalImpl.java:1255)
           at org.jboss.messaging.core.journal.impl.JournalImpl.load(JournalImpl.java:975)
           at org.jboss.messaging.tests.unit.core.journal.impl.JournalImplTestBase.load(JournalImplTestBase.java:179)
           at org.jboss.messaging.tests.unit.core.journal.impl.JournalImplTestUnit.testState(JournalImplTestUnit.java:79)
          


          • 2. Re: Build broken with new AIO tests
            clebert.suconic

            I have added the .SO under /native/bin. At the moment you have the 64bit version there.

            If you have a 32bits OS, you have to run ./bootstrap under native and copy the library there (The Makefile will place it under project/native/src/.libs). The README under native should tell what are the install requirements. (libaio-dev, automake, autoconf, libtool, GCC, G++, lib6-dev).

            If you are running those tests inside Eclipse, you could either define java.library.path through the IDE:

            * you need to define -Djava.library.path=${project-root}/native/src/.libs when calling the JVM
             * If you are running this test in eclipse you should do:
             * I - Run->Open Run Dialog
             * II - Find the class on the list (you will find it if you already tried running this testcase before)
             * III - Add -Djava.library.path=<your project place>/native/src/.libs
            



            Or on the script that starts your IDE you should:

            export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/pathToYourProject/native/bin.

            The only automation that could be made is to place the libraries under the SVN. We could try adding a 32bit version and 64bit for a recent kernel, but If that doesn't work the library has to be compiled again.. unfortunately. (Unfortunately Linux binary redistribution sucks... but I will see what could be improved).


            • 3. Re: Build broken with new AIO tests
              timfox

              This needs to be transparent to the user.

              The build should detect
              a) Whether Linux is available
              b) Whether 32bit or 64 bit

              If tests aren't running on Linux then the AIO tests should automatically be excluded.

              If 32 bit or 64 bit then the correct libraries should be automatically built and installed.

              Bottom line is there should be no manual steps. It's ugly.

              • 4. Re: Build broken with new AIO tests
                timfox

                I don't understand why the binary is in SVN.

                Can't you just change the build script to automatically build the binary at test run time and copy it to bin? Then it would be in the correct 32bit or 64bit format.

                I don't like this idea of platform specific binaries in SVN

                • 5. Re: Build broken with new AIO tests
                  clebert.suconic

                   

                  Bottom line is there should be no manual steps. It's ugly.



                  I"m trying to automate this as much as I can.

                  The loadlibrary function on the code is testing for both 32 and 64 bits versions.

                  We can do an exclude if not Linux.. but Native code will aways require an initial setup. We can automate the setup somehow. (Maybe RPMs, .DEB.. I don't know). But that's something I'm still working on.

                  • 6. Re: Build broken with new AIO tests
                    clebert.suconic

                     

                    "timfox" wrote:
                    I don't understand why the binary is in SVN.

                    Can't you just change the build script to automatically build the binary at test run time and copy it to bin? Then it would be in the correct 32bit or 64bit format.

                    I don't like this idea of platform specific binaries in SVN



                    It's impossible to force a build without asking the user to first install these requirements:

                    - libtoos
                    - automake
                    - autoconf
                    - autoreconf
                    - libaio-dev
                    - lib6-dev
                    - gcc
                    - g++
                    - JAVA_HOME defined to a JDK path
                    (We can change the variable name to JDK_HOME if you like)

                    If we have the native library on SVN, you would require only libaio-dev.


                    We will need to think about RPMs, .Deb.. or any sort of binary packaging.

                    • 7. Re: Build broken with new AIO tests
                      clebert.suconic

                       

                      If we have the native library on SVN, you would require only libaio-dev.



                      Actually.. just libaio. (no -dev)

                      • 8. Re: Build broken with new AIO tests
                        clebert.suconic

                        I have changed the native library load a bit to make sure the conditional test for 32 and 64 bits are working.

                        To make your life easier during development, you could change the script that starts your IDE to something like this:

                        export LD_LIBRARY_PATH=/extra/work-clebert/workspaces/jboss-messaging-trunk/trunk/native/bin:$LD_LIBRARY_PATH
                        /usr/share/eclipse/eclipse
                        



                        This way you can run any testcase directly from your IDE.

                        If you don't want to define that.. you can add the argument -Djava.library.path=<path to the same place> on the JVM arguments before starting your test.. but this is a little bit annoying having to do it every time. If you choose this path, on Eclipse you can do:

                        I - Run->Open Run Dialog
                        II - Under JUnit, find the Item specific to the test you want to run
                        III - on the Arguments folder, change the VM ARguments:
                        -Djava.library.path=<path to the place of the .SO>


                        But you aways need to make sure you have a kernel 2.6+ and libaio available. (yum libaio, aptitude install libaio, or the equivalent deb command).

                        • 9. Re: Build broken with new AIO tests
                          clebert.suconic

                          There are few leaks on the testcases where the Journals and files are not closed, what will cause few messages like:

                          "Can't Initialize AIO"

                          I have fixed that on the testsuite.. and I should commit this tomorrow by the end of the day. (I have to make some tests with some other commits I'm doing, so that's why I'm not doing it now).

                          For now please just ignore those messages.

                          • 10. Re: Build broken with new AIO tests
                            timfox

                            When I run the test suite on my 32bit linux laptop, it reports:

                            07:26:58,410 WARN @main [JournalStorageManager] AIO wasn't located on this platform, using just standard Java NIO. If you are on Linux, install LibAIO and the required wrapper and you will get a lot of performance benefit

                            Shouldn't this be picked up automatically? What am I missing here?

                            • 11. Re: Build broken with new AIO tests
                              ataylor

                               

                              Shouldn't this be picked up automatically? What am I missing here?


                              thats what i assumed, am i right in thinking that the only libs that we need are either libJBMLibAIO32.so or libJBMLibAIO64.so or is there something else that needs installing?

                              Ive updated the standalone distro to ship with these libs so should get used out of the box.

                              When running the perf examples i get no difference in performance between nio and asyncio, in fact when i set journal-sync to false nio is faster. What should i expect?

                              • 12. Re: Build broken with new AIO tests
                                timfox

                                 

                                "ataylor" wrote:

                                When running the perf examples i get no difference in performance between nio and asyncio, in fact when i set journal-sync to false nio is faster.


                                That's to be expected.

                                Setting journal-sync to false means transactions aren't synched at commit/rollback - they're just written to the OS buffers and the OS lazily writes them.

                                So you get performance at the expense of reliability.

                                With AIO you get performance with no reliability hit (that's the theory anyway)

                                • 13. Re: Build broken with new AIO tests
                                  clebert.suconic

                                   

                                  "timfox" wrote:
                                  When I run the test suite on my 32bit linux laptop, it reports:

                                  07:26:58,410 WARN @main [JournalStorageManager] AIO wasn't located on this platform, using just standard Java NIO. If you are on Linux, install LibAIO and the required wrapper and you will get a lot of performance benefit

                                  Shouldn't this be picked up automatically? What am I missing here?



                                  You also need libaio:

                                  on the U word OS
                                  aptitude install libaio
                                  or

                                  On Fedora
                                  yum libaio


                                  Please let me know if you cant still load after making sure libaio is installed.

                                  • 14. Re: Build broken with new AIO tests
                                    clebert.suconic

                                     

                                    "ataylor" wrote:
                                    Shouldn't this be picked up automatically? What am I missing here?


                                    thats what i assumed, am i right in thinking that the only libs that we need are either libJBMLibAIO32.so or libJBMLibAIO64.so or is there something else that needs installing?

                                    Ive updated the standalone distro to ship with these libs so should get used out of the box.

                                    When running the perf examples i get no difference in performance between nio and asyncio, in fact when i set journal-sync to false nio is faster. What should i expect?


                                    On the tests I have done... AIO would be a little bit faster than non Sync (when calling the Storage classes directly), but the same tests also shows the transport is maxing the numbers. (so.. at this point it should be the same)

                                    1 2 Previous Next