Hi Peter,

Thanks for your prompt help, and your advice in the other thread as well.

We've fixed run.sh so that it correctly sets LD_LIBRARY_PATH and also the -D java arg (seemingly) correctly, but still no luck.

Echo of $LD_LIBRARY_PATH shows:

LD_LIBRARY_PATH=/jboss/prod/jboss-5.1.0.GA/bin/META-INF/lib/linux2/x86:/usr/local/java/jdk1.6.0_14/jre/lib/i386/server:/usr/local/java/jdk1.6.0_14/jre/lib/i386:/usr/local/java/jdk1.6.0_14/jre/../lib/i386

10:26:29,648 main INFO [AprLifecycleListener] The Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /jboss/prod/jboss-5.1.0.GA/bin/META-INF/lib/linux2/x86:/usr/local/java/jdk1.6.0_14/jre/lib/i386/server:/usr/local/java/jdk1.6.0_14/jre/lib/i386:/usr/local/java/jdk1.6.0_14/jre/../lib/i386

(we also tried 2.0.5 and 2.0.4. with the same results)

Hi guys,

Thanks for all your help!

We finally found the problem using DEBUG on org.apache.catalina.core as suggested.

We were getting an "Operation not permitted" error back from the OS on the native library load, because we had the partition flags set to noexec (Linux) for security reasons. JAI worked because it was loading from the java folders instead of the jboss folders.

With noexec on the partition, these are the logs we got back from DEBUG on org.apache.catalina.core: (note this error message is great, but still a bit misleading because it complains the file can't load from JBOSS_HOME/server/all/bin instead of just JBOSS_HOME/bin - we tried placing them in server/all/bin at first too ;-)

Caused by: java.lang.UnsatisfiedLinkError: (Error on: /jboss/prod/jboss-5.1.0.GA/bin/META-INF/lib/linux2/x86/libtcnative-1.so)/jboss/prod/jboss-5.1.0.GA/bin/META-INF/lib/linux2/x86/libtcnative-1.so.0.1.16: /jboss/prod/jboss-5.1.0.GA/bin/META-INF/lib/linux2/x86/libtcnative-1.so.0.1.16: failed to map segment from shared object: Operation not permitted, no libtcnative-1 in java.library.path Can't find: /jboss/prod/jboss-5.1.0.GA/server/all/bin/META-INF/lib/linux2/x86/libapr-1.so (/jboss/prod/jboss-5.1.0.GA/bin/META-INF/lib/linux2/x86)

16:48:19,721 main DEBUG [AprLifecycleListener] Loaded Apache Tomcat Native library 1.1.16.
16:48:19,721 main INFO [AprLifecycleListener] APR capabilities: IPv6 [true], sendfile [true], random [true].

<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="off" />

17:09:32,370 main INFO [Http11AprProtocol] Starting Coyote HTTP/1.1 on http-192.168.1.3-8080
17:09:32,583 main INFO [Http11AprProtocol] Starting Coyote HTTP/1.1 on http-192.168.1.6-8080
17:09:32,597 main INFO [Http11AprProtocol] Starting Coyote HTTP/1.1 on http-192.168.1.9-8080
17:09:32,762 main INFO [Http11AprProtocol] Starting Coyote HTTP/1.1 on http-192.168.1.12-8080
17:09:32,815 main INFO [Http11AprProtocol] Starting Coyote HTTP/1.1 on http-192.168.1.15-8080
17:09:32,878 main INFO [Http11AprProtocol] Starting Coyote HTTP/1.1 on http-192.168.1.18-8080

"main" prio=10 tid=0x09508000 nid=0x6a11 runnable [0x73609000]
 java.lang.Thread.State: RUNNABLE
 at org.apache.tomcat.jni.SSL.initialize(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.catalina.core.AprLifecycleListener.initializeSSL(AprLifecycleListener.java:220)
 - locked <0x76aedff0> (a java.lang.Class for org.apache.catalina.core.AprLifecycleListener)
 at org.apache.catalina.core.AprLifecycleListener.lifecycleEvent(AprLifecycleListener.java:84)
 at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
 at org.apache.catalina.core.StandardServer.initialize(StandardServer.java:796)
 at org.jboss.web.tomcat.service.deployers.TomcatService.startService(TomcatService.java:313)

/**
 * Initialize OpenSSL support.
 * This function needs to be called once for the
 * lifetime of JVM. Library.init() has to be called before.
 * @param engine Support for external a Crypto Device ("engine"),
 * usually
 * a hardware accellerator card for crypto operations.
 * @return APR status code
 */
 public static native int initialize(String engine);

Hi Guys,

I found the problem with the SSL initialization freeze-up.

It seems OpenSSL takes 14 minutes to initialise on our server due to lack of entropy.

Ref: TomcatUsers mailing list post (http://readlist.com/lists/tomcat.apache.org/users/9/46003.html)

Subject: Re: APR with tomcat - long boot
Group: Tomcat-users
From: Markus Schonhaber
Date: 17 Jun 2007

This may be normal if you have configured a HTTPS connector and your APR
uses /dev/random as random source. /dev/random will block if there's not
enough entropy available.
If you want to speed things up, you may move the mouse like crazy while
Tomcat is starting, hack on the keyboard like even crazier or compile
APR with --with-devrandom=/dev/urandom
as an option to configure.

Regards
mks

Thanks, that was definitely the problem. 14 minutes startup is now under 0.2s :-)

I didn't know we could specify SSLRandomSeed as an attribute either, so thank you for that. Now we don't need to recompile APR.

Our HTTPS connectors are all failing now of course, because the certificates are not in the correct format for OpenSSL, so we'll work on those next.

Thanks to both of you again for your help. I will update the FAQ with our findings.

Richard.

Brilliant. Thanks everyone. I have updated the FAQ.