5 Replies Latest reply on Sep 28, 2009 11:11 AM by Alessio Soldano

    Intermittent failures in multithreaded test

    Andrew Dinn Master

      I am getting intermittent and varying errors out of the Netty/SOAP layer when running the JBossTS WSTX tests on JBossAS 6.0.0 trunk i.e. using JBossWS 3.2.0 Native. The error does not vary from run to run but different errors manifest if I modify the test or run it under different circumstances. Here is one sample stack trace:

      13:54:39,868 ERROR [org.jboss.ws.core.CommonClient] Exception caught while (preparing for) performing the invocation: : org.jboss.ws.WSException: Cannot obtain deserializer factory for: [xmlType={http://docs.oasis-open.org/ws-tx/wscoor/2006/06}RegisterResponseType,javaType=class org.oasis_open.docs.ws_tx.wscoor._2006._06.RegisterResponseType]
       at org.jboss.ws.core.soap.XMLContent.getDeserializerFactory(XMLContent.java:285)
       at org.jboss.ws.core.soap.XMLContent.unmarshallObjectContents(XMLContent.java:176)
       at org.jboss.ws.core.soap.XMLContent.transitionTo(XMLContent.java:96)
       at org.jboss.ws.core.soap.SOAPContentElement.transitionTo(SOAPContentElement.java:140)
       at org.jboss.ws.core.soap.SOAPBodyElementDoc.transitionTo(SOAPBodyElementDoc.java:85)
       at org.jboss.ws.core.soap.SOAPContentElement.getObjectValue(SOAPContentElement.java:172)
       at org.jboss.ws.core.EndpointInvocation.transformPayloadValue(EndpointInvocation.java:273)
       at org.jboss.ws.core.EndpointInvocation.getReturnValue(EndpointInvocation.java:197)
       at org.jboss.ws.core.CommonClient.syncOutputParams(CommonClient.java:512)
       at org.jboss.ws.core.CommonClient.invoke(CommonClient.java:398)
       at org.jboss.ws.core.jaxws.client.ClientImpl.invoke(ClientImpl.java:290)
       at org.jboss.ws.core.jaxws.client.ClientProxy.invoke(ClientProxy.java:170)
       at org.jboss.ws.core.jaxws.client.ClientProxy.invoke(ClientProxy.java:150)
       at $Proxy177.registerOperation(Unknown Source)
       at com.arjuna.wsc11.RegistrationCoordinator.register(RegistrationCoordinator.java:56)
       at com.arjuna.mwlabs.wst11.at.remote.TransactionManagerImple.registerParticipant(TransactionManagerImple.java:191)
       at com.arjuna.mwlabs.wst11.at.remote.TransactionManagerImple.enlistForCompletion(TransactionManagerImple.java:136)
       at com.arjuna.mwlabs.wst11.at.remote.UserTransactionImple.enlistCompletionParticipants(UserTransactionImple.java:224)
       at com.arjuna.mwlabs.wst11.at.remote.UserTransactionImple.begin(UserTransactionImple.java:89)
       at com.arjuna.mwlabs.wst11.at.remote.UserTransactionImple.begin(UserTransactionImple.java:75)
       at com.arjuna.wst11.tests.junit.basic.ThreadedObject.run(ThreadedTransaction.java:53)
      


      and here is another

      13:54:41,828 ERROR [org.jboss.ws.core.CommonClient] Exception caught while (preparing for) performing the invocation: : java.lang.IllegalStateException: Cannot find property: javax.xml.ws.handler.message.outbound
       at org.jboss.ws.core.jaxws.handler.MessageContextJAXWS.processPivot(MessageContextJAXWS.java:110)
       at org.jboss.ws.core.jaxws.client.ClientImpl.processPivot(ClientImpl.java:355)
       at org.jboss.ws.core.CommonClient.processPivotInternal(CommonClient.java:480)
       at org.jboss.ws.core.CommonClient.invoke(CommonClient.java:347)
       at org.jboss.ws.core.jaxws.client.ClientImpl.invoke(ClientImpl.java:290)
       at org.jboss.ws.core.jaxws.client.ClientProxy.invoke(ClientProxy.java:170)
       at org.jboss.ws.core.jaxws.client.ClientProxy.invoke(ClientProxy.java:150)
       at $Proxy178.commitOperation(Unknown Source)
       at com.arjuna.webservices11.wsat.client.CompletionCoordinatorClient.sendCommit(CompletionCoordinatorClient.java:87)
       at com.arjuna.wst11.stub.CompletionStub.commit(CompletionStub.java:44)
       at com.arjuna.mwlabs.wst11.at.remote.UserTransactionImple.commitWithoutAck(UserTransactionImple.java:333)
       at com.arjuna.mwlabs.wst11.at.remote.UserTransactionImple.commit(UserTransactionImple.java:116)
       at com.arjuna.wst11.tests.junit.basic.ThreadedObject.run(ThreadedTransaction.java:61)
      


      As you no doubt have already deciphered from the stack trace one of these exceptions happens under the TX begin operation and the other under the TX commit.

      After seeing these two errors I then get repeated errors when I rerun the test
      14:50:23,718 ERROR [org.jboss.ws.core.soap.EnvelopeBuilderDOM] Exception while building envelope: ClientAbortException: java.io.IOException: Invalid chunk header
       at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:342)
       at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:372)
       at org.apache.catalina.connector.InputBuffer.readByte(InputBuffer.java:354)
       at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:105)
       at org.apache.xerces.impl.XMLEntityManager$RewindableInputStream.read(Unknown Source)
       at org.apache.xerces.impl.XMLEntityManager.setupCurrentEntity(Unknown Source)
       at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source)
       at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
       at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
       at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
       at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
       at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
       at org.jboss.wsf.common.DOMUtils.parse(DOMUtils.java:201)
       at org.jboss.ws.core.soap.EnvelopeBuilderDOM.build(EnvelopeBuilderDOM.java:84)
       at org.jboss.ws.core.soap.MessageFactoryImpl.createMessage(MessageFactoryImpl.java:294)
       at org.jboss.ws.core.soap.MessageFactoryImpl.createMessage(MessageFactoryImpl.java:193)
       at org.jboss.wsf.stack.jbws.RequestHandlerImpl.processRequest(RequestHandlerImpl.java:453)
       at org.jboss.wsf.stack.jbws.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:294)
       at org.jboss.wsf.stack.jbws.RequestHandlerImpl.doPost(RequestHandlerImpl.java:204)
       at org.jboss.wsf.stack.jbws.RequestHandlerImpl.handleHttpRequest(RequestHandlerImpl.java:130)
       at org.jboss.wsf.common.servlet.AbstractEndpointServlet.service(AbstractEndpointServlet.java:87)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
       at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:273)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:277)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
       at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)
       at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
       at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
       at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
       at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
       at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
       at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:358)
       at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:872)
       at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653)
       at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:951)
       at java.lang.Thread.run(Thread.java:619)
      Caused by: java.io.IOException: Invalid chunk header
       at org.apache.coyote.http11.filters.ChunkedInputFilter.parseChunkHeader(ChunkedInputFilter.java:278)
       at org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:132)
       at org.apache.coyote.http11.InternalInputBuffer.doRead(InternalInputBuffer.java:706)
       at org.apache.coyote.Request.doRead(Request.java:428)
       at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:332)
       ... 41 more
      


      14:50:23,736 ERROR [org.jboss.ws.core.jaxws.SOAPFaultHelperJAXWS] SOAP request exception: org.jboss.ws.core.CommonSOAPFaultException
       at org.jboss.ws.core.soap.EnvelopeBuilderDOM.build(EnvelopeBuilderDOM.java:94)
       at org.jboss.ws.core.soap.MessageFactoryImpl.createMessage(MessageFactoryImpl.java:294)
       at org.jboss.ws.core.soap.MessageFactoryImpl.createMessage(MessageFactoryImpl.java:193)
       at org.jboss.wsf.stack.jbws.RequestHandlerImpl.processRequest(RequestHandlerImpl.java:453)
       at org.jboss.wsf.stack.jbws.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:294)
       at org.jboss.wsf.stack.jbws.RequestHandlerImpl.doPost(RequestHandlerImpl.java:204)
       at org.jboss.wsf.stack.jbws.RequestHandlerImpl.handleHttpRequest(RequestHandlerImpl.java:130)
       at org.jboss.wsf.common.servlet.AbstractEndpointServlet.service(AbstractEndpointServlet.java:87)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
       at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:273)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:277)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
       at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)
       at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
       at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
       at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
       at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
       at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
       at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:358)
       at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:872)
       at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653)
       at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:951)
       at java.lang.Thread.run(Thread.java:619)
      


      and also

      14:50:23,736 ERROR [org.jboss.ws.core.jaxrpc.SOAPFaultHelperJAXRPC] SOAP request exception: javax.xml.rpc.soap.SOAPFaultException
       at org.jboss.ws.core.jaxrpc.SOAPFaultHelperJAXRPC.exceptionToFaultMessage(SOAPFaultHelperJAXRPC.java:189)
       at org.jboss.ws.core.jaxws.SOAPFaultHelperJAXWS.exceptionToFaultMessage(SOAPFaultHelperJAXWS.java:183)
       at org.jboss.ws.core.jaxws.binding.SOAP11BindingJAXWS.createFaultMessageFromException(SOAP11BindingJAXWS.java:102)
       at org.jboss.ws.core.CommonSOAPBinding.bindFaultMessage(CommonSOAPBinding.java:671)
       at org.jboss.wsf.stack.jbws.RequestHandlerImpl.processRequest(RequestHandlerImpl.java:490)
       at org.jboss.wsf.stack.jbws.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:294)
       at org.jboss.wsf.stack.jbws.RequestHandlerImpl.doPost(RequestHandlerImpl.java:204)
       at org.jboss.wsf.stack.jbws.RequestHandlerImpl.handleHttpRequest(RequestHandlerImpl.java:130)
       at org.jboss.wsf.common.servlet.AbstractEndpointServlet.service(AbstractEndpointServlet.java:87)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
       at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:273)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:277)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
       at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)
       at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
       at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
       at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
       at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
       at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
       at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:358)
       at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:872)
       at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653)
       at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:951)
       at java.lang.Thread.run(Thread.java:619)
      


      My test runs 10 web service transactions in parallel, each in its own thread. The TX begin operation employs a JaxWS request to activate the TX and a second JAxWS request to register the client as a termination client. The latter is where the first error manifests. The commit operation employs JaxWS request to actually terminate the TX and this is where the second error manifests.

      Which of the initial two errors actually happens and in how many of the threads depends upon what I run. When I run the full WSTX 1.1. test suite in a newly booted AS I get multiple instances of each of the initial two exceptions in various of the threads yetsome threads also run without exceptions. If I cut out all other tests in the test suite the multi-threaded test runs without error. In other circumstances I have just seen multiple occurences of the second error.

      The same test code ran without error on 5.1.0.GA.