WSDLGetInterceptor block WebService call

[tsukasa]

Hello,

I'm continously getting locked http-threads in

WSDLGetInterceptor.java:127 when having "some" load. See attached thread dump.

JBoss AS 7.1.1

CXF 4.1.0.FINAL (ant script used for replacement expecting this could solve this problem)

Scenario:

I'm using Eclipse BIRT webviewer 4.2.1 to generate reports. This ReportEngine uses SOAP 1.1 WebServices to fetch data for configured reports.

These Webservices are annotated with

@WebService

@SOAPBinding(style = Style.RPC)

I also tried providing the wsdl using "wsdlLocation" attribute of @Webservice annotation whcih didn't change the behaviour.

After a while of requesting data from those Webservices (by running the rendering the reports) one http-*-thread blocks the other during on JaxWsEndpointImpl

"http-hostname.example.com-10.238.7.109-9080-1" - Thread t@190

      java.lang.Thread.State: BLOCKED on org.apache.cxf.jaxws.support.JaxWsEndpointImpl@17fdb90 owned by: http-hostname.example.com-10.238.7.109-9080-5

The blocking thread http-hostname.example.com-10.238.7.109-9080-5 will always be locked during writing to the socket of WSDLGetInterceptor

"http-hostname.example.com-10.238.7.109-9080-5" - Thread t@3376

   java.lang.Thread.State: RUNNABLE

    at java.net.SocketOutputStream.socketWrite0(Native Method)

    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)

    at java.net.SocketOutputStream.write(SocketOutputStream.java:136)

    at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:724)

    at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:449)

    at org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:299)

    at org.apache.coyote.http11.Http11Processor.action(Http11Processor.java:1027)

    at org.apache.coyote.Response.action(Response.java:190)

    at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:347)

    at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:316)

    at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:98)

    at org.apache.cxf.io.AbstractWrappedOutputStream.flush(AbstractWrappedOutputStream.java:84)

    at com.ctc.wstx.io.UTF8Writer.flush(UTF8Writer.java:103)

    at com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:225)

    at com.ctc.wstx.sw.BufferingXmlWriter.close(BufferingXmlWriter.java:198)

    at com.ctc.wstx.sw.BaseStreamWriter._finishDocument(BaseStreamWriter.java:1400)

    at com.ctc.wstx.sw.BaseStreamWriter.writeEndDocument(BaseStreamWriter.java:532)

    at org.apache.cxf.staxutils.StaxUtils.writeDocument(StaxUtils.java:731)

    at org.apache.cxf.staxutils.StaxUtils.writeDocument(StaxUtils.java:713)

    at org.apache.cxf.staxutils.StaxUtils.writeNode(StaxUtils.java:903)

    at org.apache.cxf.frontend.WSDLGetInterceptor.handleMessage(WSDLGetInterceptor.java:127)

    - locked org.apache.cxf.jaxws.support.JaxWsEndpointImpl@17fdb90

    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262)

    - locked org.apache.cxf.phase.PhaseInterceptorChain@8e3097

It is not possible to predict on which webservice the lock happens but the chance of locking increases if there are plenty of parallel calls.

I'm struggeling with this for almost a week but didn't find any working solution.

Does anyone have an idea whether this is a client side or server side related problem?

Do you need any more information?

[asoldano]

I don't have a real answer here; what I can say is that we have a thread which is RUNNABLE and trying to write the requested wsdl document on a socket. Subsequent requests for the same wsdl (or related schemas) for the same endpoint are blocked waiting for that, because there's a synchronized block in WSDLGetInterceptor for processing a single wsdl request per endpoint at the same time. So far so good. The actual question here is why the socket cannot be written. That is usually due to the reader of the socket not being able to keep pace or to some network issues, or something similar.

In any case this does not look something cxf or jbws specific. Just out of curiosity, did you try different version of the JVM just to exclude any possible bug in it (but I don't really think the problem is there tbh) ?

[tsukasa]

I have tried this with Sun(Oracle) JDK 1.6.24 and 1.6.39 on Win 7 (64) and Solaris.

Reportengine and Webservices are currently running on the same JBoss instance (standalone mode) during demo deployment phase, so there shouldn't be any networking issue. Not keeping pace of the reader could be explained by short locking phases. but the locks I'm talking about seem to be permanent. (in Jboss 5 report generation took about 15 seconds per report and I'm currently talking about locks that stay alive until the jboss gets killed. AFAIK in case of network disconnect some IOexception should occur leading to closing the socket.)

[tsukasa]

I resolved the problem be refactoring the report data service architecture. I replaced the SOAP stack and used remote bean invocation for report generation instead.