-
1. Re: WS-BA Coordinator Crash Recovery
Hi Masao,
I think that this means that BA recovery module is able to recover a transaction even if a coordinator crashes before it writes the transaction log.
Yes, that is what it means.
Then I ran XTS demo application(node1:Client, node2:WS) and created a BA transaction and I killed node1 at the time just before client calls UserBusinessActivity.close(). In the code, at BasicClient line 517.
I expected that the participants would begin compensate or other error handling proccess from sometime, but they look like just repeating sending Completed messages. (The timer thread calls ParticipantCompletionParticipantEngine.commsTimeout() and it does not stops)
Is this expected behavior?
Or do we need something to do to work crash recovery correctly?
Yes, this is the expected behaviour and yes there is something you need to do
Is your coordinator running on node 1 or node 2? This affects what happens but in either case the transactions will eventually be compensated.
First let us assume that the coordinator is running on node 1 so it has also died when the client died.
The web services running on node 2 do not know what has happened to the transaction so they cannot decide what to do. Think about what might happen if, say, the theatre service decided to close or compensate without waiting to be told by the coordinator. If the crash had happened earlier then the restaurant serice might not have completed so a close would cause an inconsistent outcome. If it had happened a little later then the restaurant service might already have completed and closed so a compensate would also be inconsistent. The theatre service does nto know enough to be able to make a safe decision.
So, when the coordinatro dies the services keep sending completed requests just to remind it that they are still there and to make sure it knows that they need to be told either to close or compensate. Of course, nothing will happen until you restart the AS on node 1 because there is nothing to receive these messages. When it has restared the coordinator gets a completed message from each web service which contains the id of the business activity. It does not respond until it has checked its log records. If it finds that there is no record of the activity it knows that none of the participants could have closed so it must be safe to tell them all to compensate. It does not know which participants were involved in the transaction and it certainly does not know whethert they have completed or not but it does not actally need to know this. All it needs to do is send an UnknownTransaction fault in response to the completed message. The web service handles this fault by calling compensate. That's why the participants keep resending completed. It makes sure that they eventually get told to compensate.
The second possibility is that the coordinator is running on node 2 (or on another node, call it node 3).
In this case when the client dies the coordinator and web services cannot know that something has gone wrong. If you want the coordinator to deal with a client crash then you need to specify a timeout when you create the business activity. Imagine you set up the same sitation but with a timeout of, say, 5 minutes. So, 5 minutes after the acitvity was started the coordinator will decide to cancel the business activity. Since it has seen a completed request from each web service it will send a compensate message to them and wait for a compensated reply.
If the client was just taking a long time and tried to close or cancel the activity after the 5 minute timeout is up the coordinator will fail to find the activity and will return an UnknownTransaction fault to the client. The close/cancel method throws an UnknownTransaction exception.
So, here is what else you need to do.
- If your coordinator is on node 1 you need to restart the AS on node 1.
- If your coordiator is on node 2 (or node 3) you need to change your applicattion so that it provides a timeout when it creates the activity.
I hope that is all clear now.
-
2. Re: WS-BA Coordinator Crash Recovery
Hi Andrew,
Thank you for your kind reply.
I am clear about the behaviour now. But still I have a problem.
I restart the AS on node 1(my coordinator node) after it died. Then, I watched the behaviour for a while, but it does not work and an error occurs at node1 when the coordinator responds fault. Here is the error message:10:20:14,035 ERROR [STDERR] javax.xml.ws.WebServiceException: java.lang.IllegalArgumentException: Cannot resolve imported resource: http://schemas.xmlsoap.org/soap/envelope 10:20:14,036 ERROR [STDERR] at org.jboss.ws.core.jaxws.spi.ProviderImpl.createServiceDelegate(ProviderImpl.java:71) 10:20:14,036 ERROR [STDERR] at javax.xml.ws.Service.<init>(Service.java:79) 10:20:14,036 ERROR [STDERR] at javax.xml.ws.Service21.<init>(Service21.java:37) 10:20:14,036 ERROR [STDERR] at org.jboss.jbossts.xts.soapfault.SoapFaultService.<init>(SoapFaultService.java:46) 10:20:14,036 ERROR [STDERR] at com.arjuna.webservices11.wsaddr.client.SoapFaultClient.getSoapFaultService(SoapFaultClient.java:106) 10:20:14,037 ERROR [STDERR] at com.arjuna.webservices11.wsaddr.client.SoapFaultClient.getSoapFaultPort(SoapFaultClient.java:114) 10:20:14,037 ERROR [STDERR] at com.arjuna.webservices11.wsaddr.client.SoapFaultClient.sendSoapFault(SoapFaultClient.java:69) 10:20:14,037 ERROR [STDERR] at com.arjuna.wst11.messaging.ParticipantCompletionCoordinatorProcessorImpl.getStatus(ParticipantCompletionCoordinatorProcessorImpl.java:407) 10:20:14,037 ERROR [STDERR] at com.arjuna.webservices11.wsba.sei.BusinessAgreementWithParticipantCompletionCoordinatorPortTypeImpl$8.executeTask(BusinessAgreementWithParticipantCompletionCoordinatorPortTypeImpl.java:223) 10:20:14,037 ERROR [STDERR] at com.arjuna.services.framework.task.TaskWorker.run(TaskWorker.java:65) 10:20:14,037 ERROR [STDERR] at java.lang.Thread.run(Thread.java:636) 10:20:14,038 ERROR [STDERR] Caused by: java.lang.IllegalArgumentException: Cannot resolve imported resource: http://schemas.xmlsoap.org/soap/envelope 10:20:14,038 ERROR [STDERR] at org.jboss.ws.tools.wsdl.WSDLLocatorImpl.getImportInputSource(WSDLLocatorImpl.java:139) 10:20:14,038 ERROR [STDERR] at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.parseSchema(JBossWSDLReaderImpl.java:777) 10:20:14,038 ERROR [STDERR] at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.parseSchema(JBossWSDLReaderImpl.java:657) 10:20:14,038 ERROR [STDERR] at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.parseTypes(JBossWSDLReaderImpl.java:618) 10:20:14,038 ERROR [STDERR] at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.parseDefinitions(JBossWSDLReaderImpl.java:330) 10:20:14,038 ERROR [STDERR] at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.readWSDL(JBossWSDLReaderImpl.java:2292) 10:20:14,038 ERROR [STDERR] at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.readWSDL(JBossWSDLReaderImpl.java:2256) 10:20:14,038 ERROR [STDERR] at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.readWSDL(JBossWSDLReaderImpl.java:2309) 10:20:14,038 ERROR [STDERR] at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.readWSDL(JBossWSDLReaderImpl.java:2330) 10:20:14,038 ERROR [STDERR] at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.readWSDL(JBossWSDLReaderImpl.java:2362) 10:20:14,038 ERROR [STDERR] at org.jboss.ws.tools.wsdl.WSDLDefinitionsFactory.parse(WSDLDefinitionsFactory.java:128) 10:20:14,038 ERROR [STDERR] at org.jboss.ws.metadata.umdm.ServiceMetaData.getWsdlDefinitions(ServiceMetaData.java:293) 10:20:14,038 ERROR [STDERR] at org.jboss.ws.metadata.builder.jaxws.JAXWSClientMetaDataBuilder.buildMetaData(JAXWSClientMetaDataBuilder.java:84) 10:20:14,039 ERROR [STDERR] at org.jboss.ws.core.jaxws.spi.ServiceDelegateImpl.<init>(ServiceDelegateImpl.java:137) 10:20:14,039 ERROR [STDERR] at org.jboss.ws.core.jaxws.spi.ProviderImpl.createServiceDelegate(ProviderImpl.java:65) 10:20:14,039 ERROR [STDERR] ... 10 more 10:20:19,780 ERROR [STDERR] javax.xml.ws.WebServiceException: java.lang.IllegalArgumentException: Cannot resolve imported resource: http://schemas.xmlsoap.org/soap/envelope 10:20:19,781 ERROR [STDERR] at org.jboss.ws.core.jaxws.spi.ProviderImpl.createServiceDelegate(ProviderImpl.java:71) 10:20:19,781 ERROR [STDERR] at javax.xml.ws.Service.<init>(Service.java:79) 10:20:19,781 ERROR [STDERR] at javax.xml.ws.Service21.<init>(Service21.java:37) 10:20:19,781 ERROR [STDERR] at org.jboss.jbossts.xts.soapfault.SoapFaultService.<init>(SoapFaultService.java:46) 10:20:19,781 ERROR [STDERR] at com.arjuna.webservices11.wsaddr.client.SoapFaultClient.getSoapFaultService(SoapFaultClient.java:106) 10:20:19,781 ERROR [STDERR] at com.arjuna.webservices11.wsaddr.client.SoapFaultClient.getSoapFaultPort(SoapFaultClient.java:114) 10:20:19,781 ERROR [STDERR] at com.arjuna.webservices11.wsaddr.client.SoapFaultClient.sendSoapFault(SoapFaultClient.java:69) 10:20:19,781 ERROR [STDERR] at com.arjuna.wst11.messaging.ParticipantCompletionCoordinatorProcessorImpl.getStatus(ParticipantCompletionCoordinatorProcessorImpl.java:407) 10:20:19,781 ERROR [STDERR] at com.arjuna.webservices11.wsba.sei.BusinessAgreementWithParticipantCompletionCoordinatorPortTypeImpl$8.executeTask(BusinessAgreementWithParticipantCompletionCoordinatorPortTypeImpl.java:223) 10:20:19,781 ERROR [STDERR] at com.arjuna.services.framework.task.TaskWorker.run(TaskWorker.java:65) 10:20:19,781 ERROR [STDERR] at java.lang.Thread.run(Thread.java:636) 10:20:19,781 ERROR [STDERR] Caused by: java.lang.IllegalArgumentException: Cannot resolve imported resource: http://schemas.xmlsoap.org/soap/envelope 10:20:19,781 ERROR [STDERR] at org.jboss.ws.tools.wsdl.WSDLLocatorImpl.getImportInputSource(WSDLLocatorImpl.java:139) 10:20:19,781 ERROR [STDERR] at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.parseSchema(JBossWSDLReaderImpl.java:777) 10:20:19,782 ERROR [STDERR] at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.parseSchema(JBossWSDLReaderImpl.java:657) 10:20:19,782 ERROR [STDERR] at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.parseTypes(JBossWSDLReaderImpl.java:618) 10:20:19,782 ERROR [STDERR] at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.parseDefinitions(JBossWSDLReaderImpl.java:330) 10:20:19,782 ERROR [STDERR] at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.readWSDL(JBossWSDLReaderImpl.java:2292) 10:20:19,782 ERROR [STDERR] at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.readWSDL(JBossWSDLReaderImpl.java:2256) 10:20:19,782 ERROR [STDERR] at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.readWSDL(JBossWSDLReaderImpl.java:2309) 10:20:19,782 ERROR [STDERR] at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.readWSDL(JBossWSDLReaderImpl.java:2330) 10:20:19,782 ERROR [STDERR] at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.readWSDL(JBossWSDLReaderImpl.java:2362) 10:20:19,782 ERROR [STDERR] at org.jboss.ws.tools.wsdl.WSDLDefinitionsFactory.parse(WSDLDefinitionsFactory.java:128) 10:20:19,782 ERROR [STDERR] at org.jboss.ws.metadata.umdm.ServiceMetaData.getWsdlDefinitions(ServiceMetaData.java:293) 10:20:19,783 ERROR [STDERR] at org.jboss.ws.metadata.builder.jaxws.JAXWSClientMetaDataBuilder.buildMetaData(JAXWSClientMetaDataBuilder.java:84) 10:20:19,783 ERROR [STDERR] at org.jboss.ws.core.jaxws.spi.ServiceDelegateImpl.<init>(ServiceDelegateImpl.java:137) 10:20:19,783 ERROR [STDERR] at org.jboss.ws.core.jaxws.spi.ProviderImpl.createServiceDelegate(ProviderImpl.java:65) 10:20:19,783 ERROR [STDERR] ... 10 more
Do you have any idea about this?
I will try to figure out more details, anyway.
thanks,
Masao -
3. Re: WS-BA Coordinator Crash Recovery
Hi Masao,
Thanks for posting this error. I am not sure what is going wrong here. It looks like the web services code is failing to find the soap envelope wsdl but I don't know why this is happening -- although I suspect it maypossibly be a packaging problem (the wsdl should be found in the .sar archive). I will attempt to reproduce the error on my machine and see if I can work out why it is failing. I will also notify the web services team and see if they can investigate it.
I will get back to when I find out what is happening. Let me know if in the meantime you find out anything further.
regards,
Andrew Dinn
-
4. Re: WS-BA Coordinator Crash Recovery
Hi Andrew,
This error occurs every time I create this situation on my machine. This is the details:
1) Run AS on node1 and node2 with debug mode.
2) Create a BA transaction from the client on node1.
3) Break the process at BasicClient line 517 and kill AS process on node1.
4) Participants keep sending completed.
5) Restart AS on node1.
6) Coordinator recieves completed and writes the debug log "com.arjuna.wst11.messaging.ParticipantCompletionCoordinatorProcessorImpl.completed_3".
7) Participants begin to send getStatus.
8) Coordinator recieves getStatus and attempts to reply. The error occurs at this time. (Participants keep resending getStatus)
I tracked cause of the error.
It occurs at WSDLLocatorImpl.getImportInputSource() because entityResolver.resolveEntity() returns null. The parameter "wsdlImport" value is "http://schemas.xmlsoap.org/soap/envelope".try { log.trace("Trying to resolve: " + wsdlImport); InputSource inputSource = entityResolver.resolveEntity(wsdlImport, wsdlImport); if (inputSource != null) { latestImportURI = wsdlImport; } else { throw new IllegalArgumentException("Cannot resolve imported resource: " + wsdlImport); } return inputSource; }When I changed the value to "http://schemas.xmlsoap.org/soap/envelope/" over debugger, it returned a InputSource object. This value comes from schemaLocation attribute in WS-C/dev/dd/soapfault/wsdl/soapfault.wsdl.
I would like to make sure that "http://schemas.xmlsoap.org/soap/envelope" is fine for this attribute. Would it be not "http://schemas.xmlsoap.org/soap/envelope/"?However, even if I changed the attribute, another error occurs after that point. So I may have guessed wrong. FYI, here is the another error:
javax.xml.ws.WebServiceException: org.jboss.ws.metadata.wsdl.WSDLException: javax.wsdl.WSDLException: WSDLException (at /definitions/types/s:schema): faultCode=PARSER_ERROR: Problem parsing 'http://schemas.xmlsoap.org/soap/envelope/'.: org.xml.sax.SAXParseException: Content is not allowed in prolog. at org.jboss.ws.core.jaxws.spi.ProviderImpl.createServiceDelegate(ProviderImpl.java:71) at javax.xml.ws.Service.(Service.java:79) at javax.xml.ws.Service21.(Service21.java:37) at org.jboss.jbossts.xts.soapfault.SoapFaultService.(SoapFaultService.java:46) at com.arjuna.webservices11.wsaddr.client.SoapFaultClient.getSoapFaultService(SoapFaultClient.java:106) at com.arjuna.webservices11.wsaddr.client.SoapFaultClient.getSoapFaultPort(SoapFaultClient.java:114) at com.arjuna.webservices11.wsaddr.client.SoapFaultClient.sendSoapFault(SoapFaultClient.java:69) at com.arjuna.wst11.messaging.ParticipantCompletionCoordinatorProcessorImpl.getStatus(ParticipantCompletionCoordinatorProcessorImpl.java:407) at com.arjuna.webservices11.wsba.sei.BusinessAgreementWithParticipantCompletionCoordinatorPortTypeImpl$8.executeTask(BusinessAgreementWithParticipantCompletionCoordinatorPortTypeImpl.java:223) at com.arjuna.services.framework.task.TaskWorker.run(TaskWorker.java:65) at java.lang.Thread.run(Thread.java:636) Caused by: org.jboss.ws.metadata.wsdl.WSDLException: javax.wsdl.WSDLException: WSDLException (at /definitions/types/s:schema): faultCode=PARSER_ERROR: Problem parsing 'http://schemas.xmlsoap.org/soap/envelope/'.: org.xml.sax.SAXParseException: Content is not allowed in prolog. at org.jboss.ws.tools.wsdl.WSDLDefinitionsFactory.parse(WSDLDefinitionsFactory.java:155) at org.jboss.ws.metadata.umdm.ServiceMetaData.getWsdlDefinitions(ServiceMetaData.java:293) at org.jboss.ws.metadata.builder.jaxws.JAXWSClientMetaDataBuilder.buildMetaData(JAXWSClientMetaDataBuilder.java:84) at org.jboss.ws.core.jaxws.spi.ServiceDelegateImpl.(ServiceDelegateImpl.java:137) at org.jboss.ws.core.jaxws.spi.ProviderImpl.createServiceDelegate(ProviderImpl.java:65) ... 10 more Caused by: javax.wsdl.WSDLException: WSDLException (at /definitions/types/s:schema): faultCode=PARSER_ERROR: Problem parsing 'http://schemas.xmlsoap.org/soap/envelope/'.: org.xml.sax.SAXParseException: Content is not allowed in prolog. at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.getDocument(JBossWSDLReaderImpl.java:2138) at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.parseSchema(JBossWSDLReaderImpl.java:833) at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.parseSchema(JBossWSDLReaderImpl.java:657) at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.parseTypes(JBossWSDLReaderImpl.java:618) at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.parseDefinitions(JBossWSDLReaderImpl.java:330) at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.readWSDL(JBossWSDLReaderImpl.java:2292) at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.readWSDL(JBossWSDLReaderImpl.java:2256) at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.readWSDL(JBossWSDLReaderImpl.java:2309) at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.readWSDL(JBossWSDLReaderImpl.java:2330) at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.readWSDL(JBossWSDLReaderImpl.java:2362) at org.jboss.ws.tools.wsdl.WSDLDefinitionsFactory.parse(WSDLDefinitionsFactory.java:128) ... 14 more Caused by: org.xml.sax.SAXParseException: Content is not allowed in prolog. at org.apache.xerces.parsers.DOMParser.parse(Unknown Source) at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source) at org.jboss.ws.tools.wsdl.JBossWSDLReaderImpl.getDocument(JBossWSDLReaderImpl.java:2128) ... 24 more
-
5. Re: WS-BA Coordinator Crash Recovery
Hi Masao,
I found out what is causing your problem loading the wsdl. The soapfault wsdl imports the wsdl for soap envelope in order to obtain a definition for the soap fault message. In 4.6.1 it is using the network URL to locate this wsdl file. Here are the relevant lines from file <JBOSSTS_4_6_1_GA_CP04>/XTS/WS-C/dev/dd/soapfault/wsdl/soapfault.wsdl
<?xml version="1.0" encoding="utf-8"?> <!-- a specification for a service which can be used to dispatch a SOAP 1.1 SoapFault to an arbitrary client --> <definitions xmlns:s="http://www.w3.org/2001/XMLSchema" xmlns:tns="http://jbossts.jboss.org/xts/soapfault" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:wsaw="http://www.w3.org/2006/02/addressing/wsdl" targetNamespace="http://jbossts.jboss.org/xts/soapfault" xmlns="http://schemas.xmlsoap.org/wsdl/"> <types> <s:schema> <s:import namespace="http://schemas.xmlsoap.org/soap/envelope/" schemaLocation="http://schemas.xmlsoap.org/soap/envelope"/> </s:schema> </types> <message name="SoapFault"> <part name="fault" element="soapenv:Fault" /> </message> . . .
The problme is that it is not always possible to pick up the wsdl file using the network URL "http://schemas.xmlsoap.org/soap/envelope". It appears that this URL has been redirected to another URL and the library code used by JBossWS can fail to load it. Also, using a network URL is not appropriate when the JBoss AS instances are ruinnign on private networks. This was fixed a few months ago in trunk but the patch has not yet been backported to 4.6.1. I will raise a JIRA for this.
When I run the demo on my machine ypur problem does not happen since I am able to read the wsdl form the URL. The web service is eventually told to compensate. After resending completed several times it sends a getstatus message and the coordinator sends it an invalid soap fault.
The only thing I am unhappy about is that it takes a very long time before the web service switches form sending completed to sending getStatus (about 1o minutes). I think it needs a small fix to make thsi work better. At the moment the web service keeps resending Completed at increasingly longer intervals (the resend time doubles every two resends( until eventually the resend delay reaches a maximum (by default this is configured at 300 seconds). At that point it switches to sending getStatus messages. The coordinator is just dropping the Completed messages even though ti knows that the participant shoudl eb compensated because this is what the BA spec says it is supposed to do .I think the coordinator should actually short circuit the wait by sending an invalid state fault when the completed message is received after it restarts. I will raise a JIRA for htis too and then check the specs to be sure that this change is ok.
-
6. Re: WS-BA Coordinator Crash Recovery
Hi Masao,
Here are the URLS for the two JIRAs
https://jira.jboss.org/browse/JBTM-801
https://jira.jboss.org/browse/JBTM-802
regards,
Andrew Dinn
-
7. Re: WS-BA Coordinator Crash Recovery
Hi Andrew,
Thank you very much for your help.
Now I am clear about this problem. My develop enviorment is on private network and our production enviorment too. I need to think about this.
regards,
Masao
-
8. Re: WS-BA Coordinator Crash Recovery
Hi Masao,
masao kunii wrote:
Thank you very much for your help.
Thank you very much for reporting this problem.
masao kunii wrote:
Now I am clear about this problem. My develop enviorment is on private network and our production enviorment too. I need to think about this.It is absolutely right to expect XTS to work successfully on a private network. The problem here lies with the XTS code and I will make sure that the EAP release is patched to correct it. Please let me know if you find any other problems which you think might be because your network is not connectedto public sites.
regards,
Andrew Dinn