Strange Problem with HTTP Invoker Using Tomcat Coyote-HTTPS

[ahahamyan]

We are using JBoss 3.2.2/Tomcat 4.1.27 on RedHat 9. The client application (Swing) communicates with the app server via the HTTP invoker (HTTPS) on port 443. For testing we are using self-signed certs. We also use a custom JAAS login module for client authentication. The application deployed is an ear made up on stateless session beans using CMT against an Oracle datasource. We observed this problem only on Linux, it does not happen on Windows.

The problem has to do with something in the http invoker trying to make calls out to java.sun.com and www.jboss.org to lookup dtds which causes the client invocation to hang. In our environment, outside access is not allowed from app server boxes. The server comes up fine without any problems, however the first time a client call is made, four dtds are requested (www.jboss.org and java.sun.com) , they are :

/j2ee/dtd/jboss_3_2.dtd
/j2ee/dtd/security_config.dtd
/j2ee/dtd/jbosscmp-jdbc_3_2.dtd
/j2ee/dtd/jbosscmp-jdbc_3_0.dtd
/dtd/web-app_2_3.dtd

If outside access is not allowed the client invocation hangs and netstat shows a SYN_SENT entry going out to the external site. About 15 minutes later the invocation seems to timeout, the client receives the expected response and things proceed fine thereafter, further client invocations do not hang. For now we modified etc/hosts to point www.jboss.org and java.sun.com to 127.0.0.1. This causes the invocation to succeed, however we observed that something on the app server tries to lookup the dtd by going to the external sites exactly every 5 minutes. I copied the dtds to an internal box running apache and pointed the www.jboss.org and java.sun.com to that machine and observed that even though the http response was a 200, it still tries to validate the dtds every five minutes. The same exact test on Windows worked with no problems, no http requests were observed from the app server to lookup any dtds.

Basically, the first client invocation acts as a trigger to start the lookup of the five dtds every five minutes, regardless of whether they are found or not. Turning on debugging reveals the following exceptions which occur irregardless of client invocation hanging or running on windows/linux. The exceptions show up as DEBUG.

Any help is greatly appreciated.

Thanks,

Arnold Hahamyan

2004-06-04 11:03:14,484 DEBUG [org.apache.tomcat.util.threads.ThreadPool] Getting new thread data
2004-06-04 11:03:14,781 DEBUG [org.apache.tomcat.util.net.jsse.JSSE14Support] Error getting client certs
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(DashoA6275)
at org.apache.tomcat.util.net.jsse.JSSE14Support.getX509Certificates(JSSE14Support.java:151)
at org.apache.tomcat.util.net.jsse.JSSESupport.getPeerCertificateChain(JSSESupport.java:161)
at org.apache.coyote.http11.Http11Processor.action(Http11Processor.java:748)
at org.apache.coyote.Response.action(Response.java:222)
at org.apache.coyote.tomcat4.CoyoteAdapter.postParseRequest(CoyoteAdapter.java:321)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:221)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:601)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:392)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:565)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:619)
at java.lang.Thread.run(Thread.java:534)
2004-06-04 11:03:14,812 INFO [org.jboss.security.plugins.JaasSecurityManagerService] Created securityMgr=org.jboss.security.plugins.JaasSecurityManager@1d1cdf7
2004-06-04 11:03:14,812 DEBUG [org.jboss.security.plugins.JaasSecurityManager.http-invoker] CachePolicy set to: org.jboss.util.TimedCachePolicy@1292ba7
2004-06-04 11:03:14,812 INFO [org.jboss.security.plugins.JaasSecurityManagerService] setCachePolicy, c=org.jboss.util.TimedCachePolicy@1292ba7
2004-06-04 11:03:14,812 INFO [org.jboss.security.plugins.JaasSecurityManagerService] Added http-invoker, org.jboss.security.plugins.SecurityDomainContext@1f11e0f to map
2004-06-04 11:03:15,156 DEBUG [org.apache.tomcat.util.net.jsse.JSSE14Support] Error getting client certs
javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at com.sun.net.ssl.internal.ssl.SSLSessionImpl.getPeerCertificates(DashoA6275)
at org.apache.tomcat.util.net.jsse.JSSE14Support.getX509Certificates(JSSE14Support.java:151)
at org.apache.tomcat.util.net.jsse.JSSESupport.getPeerCertificateChain(JSSESupport.java:161)
at org.apache.coyote.http11.Http11Processor.action(Http11Processor.java:748)
at org.apache.coyote.Response.action(Response.java:222)
at org.apache.coyote.tomcat4.CoyoteAdapter.postParseRequest(CoyoteAdapter.java:321)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:221)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:601)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:392)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:565)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:619)
at java.lang.Thread.run(Thread.java:534)
2004-06-04 11:03:15,203 DEBUG [org.apache.coyote.http11.Http11Protocol] IOException reading request
javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Software caused connection abort: socket write error
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.d(DashoA6275)
at com.sun.net.ssl.internal.ssl.AppInputStream.available(DashoA6275)
at org.apache.tomcat.util.net.TcpConnection.shutdownInput(TcpConnection.java:138)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:406)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:565)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:619)
at java.lang.Thread.run(Thread.java:534)
Caused by: javax.net.ssl.SSLException: java.net.SocketException: Software caused connection abort: socket write error
at com.sun.net.ssl.internal.ssl.BaseSSLSocketImpl.a(DashoA6275)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(DashoA6275)
at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:668)
at org.apache.coyote.http11.filters.ChunkedOutputFilter.end(ChunkedOutputFilter.java:206)
at org.apache.coyote.http11.InternalOutputBuffer.endRequest(InternalOutputBuffer.java:379)
at org.apache.coyote.http11.Http11Processor.action(Http11Processor.java:714)
at org.apache.coyote.Response.action(Response.java:222)
at org.apache.coyote.Response.finish(Response.java:343)
at org.apache.coyote.tomcat4.OutputBuffer.close(OutputBuffer.java:326)
at org.apache.coyote.tomcat4.CoyoteOutputStream.close(CoyoteOutputStream.java:125)
at java.io.ObjectOutputStream$BlockDataOutputStream.close(ObjectOutputStream.java:1590)
at java.io.ObjectOutputStream.close(ObjectOutputStream.java:662)
at org.jboss.invocation.http.servlet.InvokerServlet.processRequest(InvokerServlet.java:140)
at org.jboss.invocation.http.servlet.InvokerServlet.doPost(InvokerServlet.java:195)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.jboss.web.tomcat.security.JBossSecurityMgrRealm.invoke(JBossSecurityMgrRealm.java:220)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:494)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.CertificatesValve.invoke(CertificatesValve.java:246)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.jboss.web.tomcat.tc4.statistics.ContainerStatsValve.invoke(ContainerStatsValve.java:76)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2416)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:65)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:577)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:223)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:601)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:392)
... 3 more
Caused by: java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at com.sun.net.ssl.internal.ssl.OutputRecord.a(DashoA6275)

[hbaxmann]

The reason for this behavior is described in:

http://www-106.ibm.com/developerworks/library/x-tipent.html

HTH a bit

bax

[ahahamyan]

Well, Jboss already has a local entity resolver implementation (JbossEntityResolver.java) that it uses to lookup dtds from jboss.jar. Several of the dtds being requested (not all of them suprisingly) are already registered in the JBossEntityResolver.java to be looked up from jboss.jar (org.jboss.metadata package). The XMLFileLoader registers this implementation class as the resolver. Therefore I would have expected at least the ones registered there not to be looked up. Besides, that still does not explain why on Windows no lookups are requested.

Still a mystery.

[starksm64]

What does the given exception have to do with this? The http invoker knows nothing about the indicated dtds so I don't see how that is involved. Take a thread dump of the server when its sitting in a hung outbound connection to see what is triggering the dtd lookup.

[ahahamyan]

Thanks for the thread dump suggestion, that pointed out a piece of our code that was causing the problem. This was not a JBoss issue at all. Thanks again for your help.

[praneel]

Having same problem..

Could not catch what exactly thread dump of the server means and looking around for some more great idea to Fix this .

Help appri..