Intermittent failures in multithreaded test
I am getting intermittent and varying errors out of the Netty/SOAP layer when running the JBossTS WSTX tests on JBossAS 6.0.0 trunk i.e. using JBossWS 3.2.0 Native. The error does not vary from run to run but different errors manifest if I modify the test or run it under different circumstances. Here is one sample stack trace:
13:54:39,868 ERROR [org.jboss.ws.core.CommonClient] Exception caught while (preparing for) performing the invocation: : org.jboss.ws.WSException: Cannot obtain deserializer factory for: [xmlType={http://docs.oasis-open.org/ws-tx/wscoor/2006/06}RegisterResponseType,javaType=class org.oasis_open.docs.ws_tx.wscoor._2006._06.RegisterResponseType]
at org.jboss.ws.core.soap.XMLContent.getDeserializerFactory(XMLContent.java:285)
at org.jboss.ws.core.soap.XMLContent.unmarshallObjectContents(XMLContent.java:176)
at org.jboss.ws.core.soap.XMLContent.transitionTo(XMLContent.java:96)
at org.jboss.ws.core.soap.SOAPContentElement.transitionTo(SOAPContentElement.java:140)
at org.jboss.ws.core.soap.SOAPBodyElementDoc.transitionTo(SOAPBodyElementDoc.java:85)
at org.jboss.ws.core.soap.SOAPContentElement.getObjectValue(SOAPContentElement.java:172)
at org.jboss.ws.core.EndpointInvocation.transformPayloadValue(EndpointInvocation.java:273)
at org.jboss.ws.core.EndpointInvocation.getReturnValue(EndpointInvocation.java:197)
at org.jboss.ws.core.CommonClient.syncOutputParams(CommonClient.java:512)
at org.jboss.ws.core.CommonClient.invoke(CommonClient.java:398)
at org.jboss.ws.core.jaxws.client.ClientImpl.invoke(ClientImpl.java:290)
at org.jboss.ws.core.jaxws.client.ClientProxy.invoke(ClientProxy.java:170)
at org.jboss.ws.core.jaxws.client.ClientProxy.invoke(ClientProxy.java:150)
at $Proxy177.registerOperation(Unknown Source)
at com.arjuna.wsc11.RegistrationCoordinator.register(RegistrationCoordinator.java:56)
at com.arjuna.mwlabs.wst11.at.remote.TransactionManagerImple.registerParticipant(TransactionManagerImple.java:191)
at com.arjuna.mwlabs.wst11.at.remote.TransactionManagerImple.enlistForCompletion(TransactionManagerImple.java:136)
at com.arjuna.mwlabs.wst11.at.remote.UserTransactionImple.enlistCompletionParticipants(UserTransactionImple.java:224)
at com.arjuna.mwlabs.wst11.at.remote.UserTransactionImple.begin(UserTransactionImple.java:89)
at com.arjuna.mwlabs.wst11.at.remote.UserTransactionImple.begin(UserTransactionImple.java:75)
at com.arjuna.wst11.tests.junit.basic.ThreadedObject.run(ThreadedTransaction.java:53)
and here is another
13:54:41,828 ERROR [org.jboss.ws.core.CommonClient] Exception caught while (preparing for) performing the invocation: : java.lang.IllegalStateException: Cannot find property: javax.xml.ws.handler.message.outbound at org.jboss.ws.core.jaxws.handler.MessageContextJAXWS.processPivot(MessageContextJAXWS.java:110) at org.jboss.ws.core.jaxws.client.ClientImpl.processPivot(ClientImpl.java:355) at org.jboss.ws.core.CommonClient.processPivotInternal(CommonClient.java:480) at org.jboss.ws.core.CommonClient.invoke(CommonClient.java:347) at org.jboss.ws.core.jaxws.client.ClientImpl.invoke(ClientImpl.java:290) at org.jboss.ws.core.jaxws.client.ClientProxy.invoke(ClientProxy.java:170) at org.jboss.ws.core.jaxws.client.ClientProxy.invoke(ClientProxy.java:150) at $Proxy178.commitOperation(Unknown Source) at com.arjuna.webservices11.wsat.client.CompletionCoordinatorClient.sendCommit(CompletionCoordinatorClient.java:87) at com.arjuna.wst11.stub.CompletionStub.commit(CompletionStub.java:44) at com.arjuna.mwlabs.wst11.at.remote.UserTransactionImple.commitWithoutAck(UserTransactionImple.java:333) at com.arjuna.mwlabs.wst11.at.remote.UserTransactionImple.commit(UserTransactionImple.java:116) at com.arjuna.wst11.tests.junit.basic.ThreadedObject.run(ThreadedTransaction.java:61)
As you no doubt have already deciphered from the stack trace one of these exceptions happens under the TX begin operation and the other under the TX commit.
After seeing these two errors I then get repeated errors when I rerun the test
14:50:23,718 ERROR [org.jboss.ws.core.soap.EnvelopeBuilderDOM] Exception while building envelope: ClientAbortException: java.io.IOException: Invalid chunk header at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:342) at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:372) at org.apache.catalina.connector.InputBuffer.readByte(InputBuffer.java:354) at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:105) at org.apache.xerces.impl.XMLEntityManager$RewindableInputStream.read(Unknown Source) at org.apache.xerces.impl.XMLEntityManager.setupCurrentEntity(Unknown Source) at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.DOMParser.parse(Unknown Source) at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source) at org.jboss.wsf.common.DOMUtils.parse(DOMUtils.java:201) at org.jboss.ws.core.soap.EnvelopeBuilderDOM.build(EnvelopeBuilderDOM.java:84) at org.jboss.ws.core.soap.MessageFactoryImpl.createMessage(MessageFactoryImpl.java:294) at org.jboss.ws.core.soap.MessageFactoryImpl.createMessage(MessageFactoryImpl.java:193) at org.jboss.wsf.stack.jbws.RequestHandlerImpl.processRequest(RequestHandlerImpl.java:453) at org.jboss.wsf.stack.jbws.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:294) at org.jboss.wsf.stack.jbws.RequestHandlerImpl.doPost(RequestHandlerImpl.java:204) at org.jboss.wsf.stack.jbws.RequestHandlerImpl.handleHttpRequest(RequestHandlerImpl.java:130) at org.jboss.wsf.common.servlet.AbstractEndpointServlet.service(AbstractEndpointServlet.java:87) at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:273) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:277) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92) at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126) at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:358) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:872) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:951) at java.lang.Thread.run(Thread.java:619) Caused by: java.io.IOException: Invalid chunk header at org.apache.coyote.http11.filters.ChunkedInputFilter.parseChunkHeader(ChunkedInputFilter.java:278) at org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:132) at org.apache.coyote.http11.InternalInputBuffer.doRead(InternalInputBuffer.java:706) at org.apache.coyote.Request.doRead(Request.java:428) at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:332) ... 41 more
14:50:23,736 ERROR [org.jboss.ws.core.jaxws.SOAPFaultHelperJAXWS] SOAP request exception: org.jboss.ws.core.CommonSOAPFaultException at org.jboss.ws.core.soap.EnvelopeBuilderDOM.build(EnvelopeBuilderDOM.java:94) at org.jboss.ws.core.soap.MessageFactoryImpl.createMessage(MessageFactoryImpl.java:294) at org.jboss.ws.core.soap.MessageFactoryImpl.createMessage(MessageFactoryImpl.java:193) at org.jboss.wsf.stack.jbws.RequestHandlerImpl.processRequest(RequestHandlerImpl.java:453) at org.jboss.wsf.stack.jbws.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:294) at org.jboss.wsf.stack.jbws.RequestHandlerImpl.doPost(RequestHandlerImpl.java:204) at org.jboss.wsf.stack.jbws.RequestHandlerImpl.handleHttpRequest(RequestHandlerImpl.java:130) at org.jboss.wsf.common.servlet.AbstractEndpointServlet.service(AbstractEndpointServlet.java:87) at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:273) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:277) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92) at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126) at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:358) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:872) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:951) at java.lang.Thread.run(Thread.java:619)
and also
14:50:23,736 ERROR [org.jboss.ws.core.jaxrpc.SOAPFaultHelperJAXRPC] SOAP request exception: javax.xml.rpc.soap.SOAPFaultException at org.jboss.ws.core.jaxrpc.SOAPFaultHelperJAXRPC.exceptionToFaultMessage(SOAPFaultHelperJAXRPC.java:189) at org.jboss.ws.core.jaxws.SOAPFaultHelperJAXWS.exceptionToFaultMessage(SOAPFaultHelperJAXWS.java:183) at org.jboss.ws.core.jaxws.binding.SOAP11BindingJAXWS.createFaultMessageFromException(SOAP11BindingJAXWS.java:102) at org.jboss.ws.core.CommonSOAPBinding.bindFaultMessage(CommonSOAPBinding.java:671) at org.jboss.wsf.stack.jbws.RequestHandlerImpl.processRequest(RequestHandlerImpl.java:490) at org.jboss.wsf.stack.jbws.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:294) at org.jboss.wsf.stack.jbws.RequestHandlerImpl.doPost(RequestHandlerImpl.java:204) at org.jboss.wsf.stack.jbws.RequestHandlerImpl.handleHttpRequest(RequestHandlerImpl.java:130) at org.jboss.wsf.common.servlet.AbstractEndpointServlet.service(AbstractEndpointServlet.java:87) at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:273) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:277) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92) at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126) at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:358) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:872) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:951) at java.lang.Thread.run(Thread.java:619)
My test runs 10 web service transactions in parallel, each in its own thread. The TX begin operation employs a JaxWS request to activate the TX and a second JAxWS request to register the client as a termination client. The latter is where the first error manifests. The commit operation employs JaxWS request to actually terminate the TX and this is where the second error manifests.
Which of the initial two errors actually happens and in how many of the threads depends upon what I run. When I run the full WSTX 1.1. test suite in a newly booted AS I get multiple instances of each of the initial two exceptions in various of the threads yetsome threads also run without exceptions. If I cut out all other tests in the test suite the multi-threaded test runs without error. In other circumstances I have just seen multiple occurences of the second error.
The same test code ran without error on 5.1.0.GA.
