13 Replies Latest reply on Aug 6, 2010 3:00 AM by Istvan Nagy

    War Stories: Apache Native Runtime on JBoss 5

    Richard Kennard Novice

      6 months after the release of JBoss 5, it's probably time we updated this wiki page...

      http://www.jboss.org/community/wiki/HowToAddAprToJBoss

      ...I'm happy to do it if someone can help me get APR working. Before following the wiki (which is for JBoss 4.2), this is what JBoss 5 is saying...

      08:32:06,688 main INFO [AprLifecycleListener] The Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/local/java/jdk1.6.0_14/jre/lib/i386/server:/usr/local/java/jdk1.6.0_14/jre/lib/i386:/usr/local/java/jdk1.6.0_14/jre/../lib/i386:/usr/java/packages/lib/i386:/lib:/usr/lib


      But after following the wiki, it says:

      01:17:59,016 main INFO [AprLifecycleListener] The Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /jboss/prod/jboss-5.1.0.GA/bin/META-INF/lib/linux2/x86


      The message is slightly different but still a failure. The .so files are all there, and correctly symlinked:

      username@haruko:/jboss/prod/jboss-5.1.0.GA/bin/META-INF/lib/linux2/x86$ ls -l
      total 3388
      drwxrwxr-x 2 username username 4096 2008-10-30 12:08 engines
      lrwxrwxrwx 1 username username 17 2009-07-06 01:14 libapr-1.so -> libapr-1.so.0.2.8
      lrwxrwxrwx 1 username username 17 2009-07-06 01:14 libapr-1.so.0 -> libapr-1.so.0.2.8
      -rwxr-xr-x 1 username username 515542 2008-10-30 12:09 libapr-1.so.0.2.8
      lrwxrwxrwx 1 username username 21 2009-07-06 01:14 libaprutil-1.so -> libaprutil-1.so.0.2.8
      lrwxrwxrwx 1 username username 21 2009-07-06 01:14 libaprutil-1.so.0 -> libaprutil-1.so.0.2.8
      -rwxr-xr-x 1 username username 288578 2008-10-30 12:10 libaprutil-1.so.0.2.8
      lrwxrwxrwx 1 username username 18 2009-07-06 01:14 libcrypto.so -> libcrypto.so.0.9.8
      lrwxrwxrwx 1 username username 18 2009-07-06 01:14 libcrypto.so.0 -> libcrypto.so.0.9.8
      -r-xr-xr-x 1 username username 1204967 2008-10-30 12:08 libcrypto.so.0.9.8
      lrwxrwxrwx 1 username username 17 2009-07-06 01:14 libexpat.so -> libexpat.so.0.1.0
      lrwxrwxrwx 1 username username 17 2009-07-06 01:14 libexpat.so.0 -> libexpat.so.0.1.0
      -rwxr-xr-x 1 username username 303414 2008-10-30 12:10 libexpat.so.0.1.0
      lrwxrwxrwx 1 username username 15 2009-07-06 01:14 libssl.so -> libssl.so.0.9.8
      lrwxrwxrwx 1 username username 15 2009-07-06 01:14 libssl.so.0 -> libssl.so.0.9.8
      -r-xr-xr-x 1 username username 256316 2008-10-30 12:08 libssl.so.0.9.8
      lrwxrwxrwx 1 username username 23 2009-07-06 01:14 libtcnative-1.so -> libtcnative-1.so.0.1.16
      lrwxrwxrwx 1 username username 23 2009-07-06 01:14 libtcnative-1.so.0 -> libtcnative-1.so.0.1.16
      -rwxr-xr-x 1 username username 776610 2008-10-30 12:10 libtcnative-1.so.0.1.16
      lrwxrwxrwx 1 username username 13 2009-07-06 01:14 libz.so -> libz.so.1.2.3
      lrwxrwxrwx 1 username username 13 2009-07-06 01:14 libz.so.1 -> libz.so.1.2.3
      -rwxr-xr-x 1 username username 75778 2008-10-30 12:05 libz.so.1.2.3


      We have this working in Tomcat 5 and 6, in multiple installations, but for some reason no such luck with JBoss 5.1 GA. Steps we took:

      1. Download from http://www.jboss.org/jbossweb/downloads/jboss-native/
      2. Unzip/tar wget -O native.tar.gz --no-check-certificate http://labs.jboss.com/file-access/default/members/jbossweb/freezone/dist/2.0.6.GA/jboss-native-2.0.6-linux2-x86-ssl.tar.gz tar -xvvf native.tar.gz
      3. Move files into correct locations: export JBOSS_HOME=/jboss/prod/jboss-5.1.0.GA mv bin/META-INF $JBOSS_HOME/bin/META-INF
      4. Set permissions (same as other files in JBOSS_HOME/bin)
      5. Start JBoss
      6. It recognises that the folder /jboss/prod/jboss-5.1.0.GA/bin/META-INF/lib/linux2/x86 is now present, and changes the warning message as described above, but says the files are not present even though they are.


      Can anybody please help getting APR working with JBoss 5? If we're successful I'll update the wiki.

      Regards,

      Richard.




        • 2. Re: War Stories: Apache Native Runtime on JBoss 5
          Peter Johnson Master

          Please note that the run script screws up the ld_library_path - it replaces the existing path with only bin/META-INF/etc-etc-etc, which is why the SO won't load - the directories containing the required system library are no longer in the path.

          • 3. Re: War Stories: Apache Native Runtime on JBoss 5
            Richard Kennard Novice

            Hi Peter,

            Thanks for your prompt help, and your advice in the other thread as well.

            We've fixed run.sh so that it correctly sets LD_LIBRARY_PATH and also the -D java arg (seemingly) correctly, but still no luck.

            Echo of $LD_LIBRARY_PATH shows:

            LD_LIBRARY_PATH=/jboss/prod/jboss-5.1.0.GA/bin/META-INF/lib/linux2/x86:/usr/local/java/jdk1.6.0_14/jre/lib/i386/server:/usr/local/java/jdk1.6.0_14/jre/lib/i386:/usr/local/java/jdk1.6.0_14/jre/../lib/i386


            (seems correct)

            This appears in the logs, which seems to show all the correct paths (our APR one, plus all the previous ones which showed before that path was present)

            10:26:29,648 main INFO [AprLifecycleListener] The Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /jboss/prod/jboss-5.1.0.GA/bin/META-INF/lib/linux2/x86:/usr/local/java/jdk1.6.0_14/jre/lib/i386/server:/usr/local/java/jdk1.6.0_14/jre/lib/i386:/usr/local/java/jdk1.6.0_14/jre/../lib/i386


            This is with 2.0.6 on Linux32 / JBoss 5.1 GA.

            Any ideas of what we can try next?

            Regards,

            Richard.

            • 4. Re: War Stories: Apache Native Runtime on JBoss 5
              Richard Kennard Novice

              (we also tried 2.0.5 and 2.0.4. with the same results)

              • 6. Re: War Stories: Apache Native Runtime on JBoss 5
                Peter Johnson Master

                Hmm, I didn't write down which Linux distro I used, but I imagine I did this at home with Kubuntu 8.04 (32-bit in a VM) and Ubuntu 8.10 (64-bit). I probably still have the 32-bit setup, I could try it again tonight or tomorrow and let you know. Or if I have time I could try my Fedora 11 (32-bit) system at work.

                • 7. Re: War Stories: Apache Native Runtime on JBoss 5
                  Richard Kennard Novice

                  Hi guys,

                  Thanks for all your help!

                  We finally found the problem using DEBUG on org.apache.catalina.core as suggested.

                  We were getting an "Operation not permitted" error back from the OS on the native library load, because we had the partition flags set to noexec (Linux) for security reasons. JAI worked because it was loading from the java folders instead of the jboss folders.

                  With noexec on the partition, these are the logs we got back from DEBUG on org.apache.catalina.core: (note this error message is great, but still a bit misleading because it complains the file can't load from JBOSS_HOME/server/all/bin instead of just JBOSS_HOME/bin - we tried placing them in server/all/bin at first too ;-)

                  Caused by: java.lang.UnsatisfiedLinkError: (Error on: /jboss/prod/jboss-5.1.0.GA/bin/META-INF/lib/linux2/x86/libtcnative-1.so)/jboss/prod/jboss-5.1.0.GA/bin/META-INF/lib/linux2/x86/libtcnative-1.so.0.1.16: /jboss/prod/jboss-5.1.0.GA/bin/META-INF/lib/linux2/x86/libtcnative-1.so.0.1.16: failed to map segment from shared object: Operation not permitted, no libtcnative-1 in java.library.path Can't find: /jboss/prod/jboss-5.1.0.GA/server/all/bin/META-INF/lib/linux2/x86/libapr-1.so (/jboss/prod/jboss-5.1.0.GA/bin/META-INF/lib/linux2/x86)


                  So we enabled exec on the partition and now we get:

                  16:48:19,721 main DEBUG [AprLifecycleListener] Loaded Apache Tomcat Native library 1.1.16.
                  16:48:19,721 main INFO [AprLifecycleListener] APR capabilities: IPv6 [true], sendfile [true], random [true].


                  Yay!

                  Next problem - after the two lines above are output, JBoss freezes up. There are no more logs output, and the HTTP ports don't open.

                  I've seen that happen on Tomcat with APR before as well, and the problem seemed related to the OpenSSL initialisation. We fixed it by disabling SSL (SSLEngine=off in server.xml) as follows:

                  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="off" />


                  Anyway, now the Apr HTTP connectors initialise and we get Http11AprProtocol showing as below, so APR is working fine:

                  17:09:32,370 main INFO [Http11AprProtocol] Starting Coyote HTTP/1.1 on http-192.168.1.3-8080
                  17:09:32,583 main INFO [Http11AprProtocol] Starting Coyote HTTP/1.1 on http-192.168.1.6-8080
                  17:09:32,597 main INFO [Http11AprProtocol] Starting Coyote HTTP/1.1 on http-192.168.1.9-8080
                  17:09:32,762 main INFO [Http11AprProtocol] Starting Coyote HTTP/1.1 on http-192.168.1.12-8080
                  17:09:32,815 main INFO [Http11AprProtocol] Starting Coyote HTTP/1.1 on http-192.168.1.15-8080
                  17:09:32,878 main INFO [Http11AprProtocol] Starting Coyote HTTP/1.1 on http-192.168.1.18-8080


                  So for now we have APR working, but SSL is now unusable.

                  Any idea why enabling SSLEngine locks up the init process? Here is where it's locking up:

                  "main" prio=10 tid=0x09508000 nid=0x6a11 runnable [0x73609000]
                   java.lang.Thread.State: RUNNABLE
                   at org.apache.tomcat.jni.SSL.initialize(Native Method)
                   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                   at java.lang.reflect.Method.invoke(Method.java:597)
                   at org.apache.catalina.core.AprLifecycleListener.initializeSSL(AprLifecycleListener.java:220)
                   - locked <0x76aedff0> (a java.lang.Class for org.apache.catalina.core.AprLifecycleListener)
                   at org.apache.catalina.core.AprLifecycleListener.lifecycleEvent(AprLifecycleListener.java:84)
                   at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
                   at org.apache.catalina.core.StandardServer.initialize(StandardServer.java:796)
                   at org.jboss.web.tomcat.service.deployers.TomcatService.startService(TomcatService.java:313)


                  It's a native method too of course...

                  /**
                   * Initialize OpenSSL support.
                   * This function needs to be called once for the
                   * lifetime of JVM. Library.init() has to be called before.
                   * @param engine Support for external a Crypto Device ("engine"),
                   * usually
                   * a hardware accellerator card for crypto operations.
                   * @return APR status code
                   */
                   public static native int initialize(String engine);


                  We get it locking up at this point when SSLEngine=on. When it's off, JBoss will start correctly and use APR, but we can't add any HTTPS connectors.

                  Best Regards,

                  Richard



                  • 8. Re: War Stories: Apache Native Runtime on JBoss 5
                    Richard Kennard Novice

                    Hi Guys,

                    I found the problem with the SSL initialization freeze-up.

                    It seems OpenSSL takes 14 minutes to initialise on our server due to lack of entropy.

                    Ref: TomcatUsers mailing list post (http://readlist.com/lists/tomcat.apache.org/users/9/46003.html)

                    Subject: Re: APR with tomcat - long boot
                    Group: Tomcat-users
                    From: Markus Schonhaber
                    Date: 17 Jun 2007
                    
                    This may be normal if you have configured a HTTPS connector and your APR
                    uses /dev/random as random source. /dev/random will block if there's not
                    enough entropy available.
                    If you want to speed things up, you may move the mouse like crazy while
                    Tomcat is starting, hack on the keyboard like even crazier or compile
                    APR with --with-devrandom=/dev/urandom
                    as an option to configure.
                    
                    Regards
                    mks


                    We will try a recompile of APR as suggested above, and get back to the list with further results when we have them.

                    Is it possible for future APR versions to use /dev/urandom as described above for OpenSSL? Or is that unsafe?

                    I'm not too sure how /dev/random works, but if it really relies on things like moving a mouse or keypresses to get randomness, it seems counter-productive to rely on it in a server environment like JBoss would usually run on?

                    • 9. Re: War Stories: Apache Native Runtime on JBoss 5
                      Jean-Frederic Clere Master

                      I think the problem is the entropy collection for SSL.
                      try SSLRandomSeed="/dev/urandom" or wait a little more.

                      • 10. Re: War Stories: Apache Native Runtime on JBoss 5
                        Richard Kennard Novice

                        Thanks, that was definitely the problem. 14 minutes startup is now under 0.2s :-)

                        I didn't know we could specify SSLRandomSeed as an attribute either, so thank you for that. Now we don't need to recompile APR.

                        Our HTTPS connectors are all failing now of course, because the certificates are not in the correct format for OpenSSL, so we'll work on those next.

                        Thanks to both of you again for your help. I will update the FAQ with our findings.

                        Richard.

                        • 11. Re: War Stories: Apache Native Runtime on JBoss 5
                          Jean-Frederic Clere Master

                          For the certificates use keytool to export the certificates and keys to the PKCS12 format that both java and openssl are supporting.

                          • 12. Re: War Stories: Apache Native Runtime on JBoss 5
                            Richard Kennard Novice

                            Brilliant. Thanks everyone. I have updated the FAQ.

                            • 13. Re: War Stories: Apache Native Runtime on JBoss 5
                              Istvan Nagy Newbie

                              Hi,

                               

                              please check for my opensolaris issue fixed with a downgrade of the native APR-libs.

                              @link: http://community.jboss.org/wiki/HowToAddAprToJBoss#cf

                               

                              cheers,

                              Istvan