1 2 Previous Next 24 Replies Latest reply on Feb 13, 2009 5:10 PM by beve

    Web service sometimes causes

      Hi,

      We have a problem with the ESB, which occurs then we call a web service through the ESB. The problem DOES NOT OCCUR ON EVERY CALL of the web service, thus it is a bit difficult to debug. What we are seeing is one of the following in the logs:

      -- snip --
      12:28:33,068 INFO [ServiceInvoker] Badly formed EPR [EPR: PortReference < <wsa:Address http://localhost:8765/> >] for Service [PROBLEM:ProblemWS] and Message [header: [ ]].Courier for EPR not supported: ESB-unaware EPR used!
      12:28:33,069 INFO [ServiceInvoker] Invalid EPR for service (probably ESB-unaware): ignoring for message: header: [ ]
      -- snip --

      and/or:

      -- snip --
      12:28:53,653 INFO [ServiceInvoker] Unresponsive EPR: JMSEpr [ PortReference < <wsa:Address jms://127.0.0.1:1099/queue/problem-ws/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.initial : org.jnp.interfaces.NamingContextFactory/>, <wsa:ReferenceProperties jbossesb:java.naming.provider.url : 127.0.0.1:1099/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.url.pkgs : org.jnp.interfaces/>, <wsa:ReferenceProperties jbossesb:destination-type : queue/>, <wsa:ReferenceProperties jbossesb:specification-version : 1.1/>, <wsa:ReferenceProperties jbossesb:connection-factory : ClusteredConnectionFactory/>, <wsa:ReferenceProperties jbossesb:persistent : true/>, <wsa:ReferenceProperties jbossesb:acknowledge-mode : AUTO_ACKNOWLEDGE/>, <wsa:ReferenceProperties jbossesb:transacted : false/>, <wsa:ReferenceProperties jbossesb:type : urn:jboss/esb/epr/type/jms/> > ] for message: header: [ To: JMSEpr [ PortReference < <wsa:Address jms://127.0.0.1:1099/queue/problem-ws/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.initial : org.jnp.interfaces.NamingContextFactory/>, <wsa:ReferenceProperties jbossesb:java.naming.provider.url : 127.0.0.1:1099/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.url.pkgs : org.jnp.interfaces/>, <wsa:ReferenceProperties jbossesb:destination-type : queue/>, <wsa:ReferenceProperties jbossesb:specification-version : 1.1/>, <wsa:ReferenceProperties jbossesb:connection-factory : ClusteredConnectionFactory/>, <wsa:ReferenceProperties jbossesb:persistent : true/>, <wsa:ReferenceProperties jbossesb:acknowledge-mode : AUTO_ACKNOWLEDGE/>, <wsa:ReferenceProperties jbossesb:transacted : false/>, <wsa:ReferenceProperties jbossesb:type : urn:jboss/esb/epr/type/jms/> > ] ReplyTo: JMSEpr [ PortReference < <wsa:Address jms://127.0.0.1:1099/queue/problem-ws_reply/>, <wsa:ReferenceProperties jbossesb:java.naming.provider.url : 127.0.0.1:1099/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.initial : org.jnp.interfaces.NamingContextFactory/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.url.pkgs : org.jnp.interfaces/>, <wsa:ReferenceProperties jbossesb:destination-type : queue/>, <wsa:ReferenceProperties jbossesb:specification-version : 1.1/>, <wsa:ReferenceProperties jbossesb:connection-factory : ClusteredConnectionFactory/>, <wsa:ReferenceProperties jbossesb:message-selector : jbossESBresponseUUID='c66de02c-f8a2-4fc1-8672-ffaa75c00b8c'/>, <wsa:ReferenceProperties jbossesb:persistent : true/>, <wsa:ReferenceProperties jbossesb:acknowledge-mode : AUTO_ACKNOWLEDGE/>, <wsa:ReferenceProperties jbossesb:transacted : false/>, <wsa:ReferenceProperties jbossesb:type : urn:jboss/esb/epr/type/jms/> > ] ]
      12:28:53,655 INFO [ServiceInvoker] Badly formed EPR [EPR: PortReference < <wsa:Address http://localhost:8765/> >] for Service [PROBLEM:ProblemWS] and Message [header: [ To: JMSEpr [ PortReference < <wsa:Address jms://127.0.0.1:1099/queue/problem-ws/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.initial : org.jnp.interfaces.NamingContextFactory/>, <wsa:ReferenceProperties jbossesb:java.naming.provider.url : 127.0.0.1:1099/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.url.pkgs : org.jnp.interfaces/>, <wsa:ReferenceProperties jbossesb:destination-type : queue/>, <wsa:ReferenceProperties jbossesb:specification-version : 1.1/>, <wsa:ReferenceProperties jbossesb:connection-factory : ClusteredConnectionFactory/>, <wsa:ReferenceProperties jbossesb:persistent : true/>, <wsa:ReferenceProperties jbossesb:acknowledge-mode : AUTO_ACKNOWLEDGE/>, <wsa:ReferenceProperties jbossesb:transacted : false/>, <wsa:ReferenceProperties jbossesb:type : urn:jboss/esb/epr/type/jms/> > ] ReplyTo: JMSEpr [ PortReference < <wsa:Address jms://127.0.0.1:1099/queue/problem-ws_reply/>, <wsa:ReferenceProperties jbossesb:java.naming.provider.url : 127.0.0.1:1099/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.initial : org.jnp.interfaces.NamingContextFactory/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.url.pkgs : org.jnp.interfaces/>, <wsa:ReferenceProperties jbossesb:destination-type : queue/>, <wsa:ReferenceProperties jbossesb:specification-version : 1.1/>, <wsa:ReferenceProperties jbossesb:connection-factory : ClusteredConnectionFactory/>, <wsa:ReferenceProperties jbossesb:message-selector : jbossESBresponseUUID='c66de02c-f8a2-4fc1-8672-ffaa75c00b8c'/>, <wsa:ReferenceProperties jbossesb:persistent : true/>, <wsa:ReferenceProperties jbossesb:acknowledge-mode : AUTO_ACKNOWLEDGE/>, <wsa:ReferenceProperties jbossesb:transacted : false/>, <wsa:ReferenceProperties jbossesb:type : urn:jboss/esb/epr/type/jms/> > ] ]].Courier for EPR not supported: ESB-unaware EPR used!
      12:28:53,656 INFO [ServiceInvoker] Invalid EPR for service (probably ESB-unaware): ignoring for message: header: [ To: JMSEpr [ PortReference < <wsa:Address jms://127.0.0.1:1099/queue/problem-ws/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.initial : org.jnp.interfaces.NamingContextFactory/>, <wsa:ReferenceProperties jbossesb:java.naming.provider.url : 127.0.0.1:1099/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.url.pkgs : org.jnp.interfaces/>, <wsa:ReferenceProperties jbossesb:destination-type : queue/>, <wsa:ReferenceProperties jbossesb:specification-version : 1.1/>, <wsa:ReferenceProperties jbossesb:connection-factory : ClusteredConnectionFactory/>, <wsa:ReferenceProperties jbossesb:persistent : true/>, <wsa:ReferenceProperties jbossesb:acknowledge-mode : AUTO_ACKNOWLEDGE/>, <wsa:ReferenceProperties jbossesb:transacted : false/>, <wsa:ReferenceProperties jbossesb:type : urn:jboss/esb/epr/type/jms/> > ] ReplyTo: JMSEpr [ PortReference < <wsa:Address jms://127.0.0.1:1099/queue/problem-ws_reply/>, <wsa:ReferenceProperties jbossesb:java.naming.provider.url : 127.0.0.1:1099/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.initial : org.jnp.interfaces.NamingContextFactory/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.url.pkgs : org.jnp.interfaces/>, <wsa:ReferenceProperties jbossesb:destination-type : queue/>, <wsa:ReferenceProperties jbossesb:specification-version : 1.1/>, <wsa:ReferenceProperties jbossesb:connection-factory : ClusteredConnectionFactory/>, <wsa:ReferenceProperties jbossesb:message-selector : jbossESBresponseUUID='c66de02c-f8a2-4fc1-8672-ffaa75c00b8c'/>, <wsa:ReferenceProperties jbossesb:persistent : true/>, <wsa:ReferenceProperties jbossesb:acknowledge-mode : AUTO_ACKNOWLEDGE/>, <wsa:ReferenceProperties jbossesb:transacted : false/>, <wsa:ReferenceProperties jbossesb:type : urn:jboss/esb/epr/type/jms/> > ] ]
      -- snip --

      The errors typically occurs after the request has hang for some seconds and because we need our web services to be quite responsive we need to solve this issue (even though the log is only on INFO level). Does anybody have any clue as to what is going on?

      We have created a small test-ESB that provokes this error and have collected the logs. I would like to send you our test code + .esb + logs if you provide an email or FTP.

      Thanks in advance!


      Venlig hilsen / Kind regards

      Anders Andersen

      IT Consultant - IT afdelingen / IT department
      Københavns Lufthavne A/S
      Lufthavnsboulevarden 6, DK-2770 Kastrup
      Tel./Phone no.: +45 60660306
      e-mail: anders.andersen@cph.dk

        • 1. Re: Web service sometimes causes
          beve

          Hi,

          We have created a small test-ESB that provokes this error and have collected the logs. I would like to send you our test code + .esb + logs if you provide an email or FTP.

          If you send me the test I'll take a look.

          Thanks,

          /Daniel

          • 2. Re: Web service sometimes causes
            mimra

            The quickstart webservice_producer does the same. Output from a single 'ant runtest':

            11:21:36,150 INFO [ServiceInvoker] Badly formed EPR [EPR: PortReference < <wsa:Address socket://localhost:8888/> >] for Service [MyServiceCategory:MyWSProducerService] and Message [header: [ ]].Courier for EPR not supported: ESB-unaware EPR used!
            11:21:36,150 INFO [ServiceInvoker] Invalid EPR for service (probably ESB-unaware): ignoring for message: header: [ ]
            11:21:36,150 INFO [ServiceInvoker] Badly formed EPR [EPR: PortReference < <wsa:Address http://localhost:8765/> >] for Service [MyServiceCategory:MyWSProducerService] and Message [header: [ ]].Courier for EPR not supported: ESB-unaware EPR used!
            11:21:36,150 INFO [ServiceInvoker] Invalid EPR for service (probably ESB-unaware): ignoring for message: header: [ ]
            11:21:36,169 INFO [STDOUT] [Quickstart_webservice_producer] BEFORE invoking jbossws endpoint:
            11:21:36,169 INFO [STDOUT] [<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:good="http://webservice_producer/goodbyeworld">
             <soapenv:Header/>
             <soapenv:Body>
             <good:sayGoodbye>
             <message>Goodbye!!</message>
             </good:sayGoodbye>
             </soapenv:Body>
            </soapenv:Envelope>].
            11:21:36,173 INFO [STDOUT] **** SOAPRequest perhaps mediated by ESB:
            <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:good="http://webservice_producer/goodbyeworld">
             <soapenv:Header/>
             <soapenv:Body>
             <good:sayGoodbye>
             <message>Goodbye!!</message>
             </good:sayGoodbye>
             </soapenv:Body>
            </soapenv:Envelope>
            11:21:36,173 INFO [STDOUT] Web Service Parameter - message=Goodbye!!
            11:21:36,174 INFO [STDOUT] [Quickstart_webservice_producer] AFTER invoking jbossws endpoint:
            11:21:36,174 INFO [STDOUT] [<env:Envelope xmlns:env='http://schemas.xmlsoap.org/soap/envelope/'><env:Header></env:Header><env:Body><ns2:sayGoodbyeResponse xmlns:ns2="http://webservice_producer/goodbyeworld"><return>... Ah Goodbye then!!!! - Goodbye!!</return></ns2:sayGoodbyeResponse></env:Body></env:Envelope>].
            11:21:36,174 WARN [ActionProcessingPipeline] No reply to address defined for reply message! To: JMSEpr [ PortReference < <wsa:Address jms://127.0.0.1:1099/queue/quickstart_webservice_producer_esb/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.initial : org.jnp.interfaces.NamingContextFactory/>, <wsa:ReferenceProperties jbossesb:java.naming.provider.url : 127.0.0.1:1099/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.url.pkgs : org.jnp.interfaces/>, <wsa:ReferenceProperties jbossesb:destination-type : queue/>, <wsa:ReferenceProperties jbossesb:specification-version : 1.1/>, <wsa:ReferenceProperties jbossesb:connection-factory : ConnectionFactory/>, <wsa:ReferenceProperties jbossesb:persistent : true/>, <wsa:ReferenceProperties jbossesb:acknowledge-mode : AUTO_ACKNOWLEDGE/>, <wsa:ReferenceProperties jbossesb:transacted : false/>, <wsa:ReferenceProperties jbossesb:type : urn:jboss/esb/epr/type/jms/> > ] MessageID: 1240b3c5-df42-4c00-816d-a65e56f955f3 RelatesTo: jms:correlationID#1240b3c5-df42-4c00-816d-a65e56f955f3
            11:21:37,109 INFO [ServiceInvoker] Badly formed EPR [EPR: PortReference < <wsa:Address http://localhost:8765/> >] for Service [MyServiceCategory:MyWSProducerService] and Message [header: [ ]].Courier for EPR not supported: ESB-unaware EPR used!
            11:21:37,109 INFO [ServiceInvoker] Invalid EPR for service (probably ESB-unaware): ignoring for message: header: [ ]
            11:21:37,118 INFO [STDOUT] [Quickstart_webservice_producer] BEFORE invoking jbossws endpoint:
            11:21:37,118 INFO [STDOUT] [<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:good="http://webservice_producer/goodbyeworld">
             <soapenv:Header/>
             <soapenv:Body>
             <good:sayGoodbye>
             <message>Goodbye!!</message>
             </good:sayGoodbye>
             </soapenv:Body>
            </soapenv:Envelope>].
            11:21:37,123 INFO [STDOUT] **** SOAPRequest perhaps mediated by ESB:
            <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:good="http://webservice_producer/goodbyeworld">
             <soapenv:Header/>
             <soapenv:Body>
             <good:sayGoodbye>
             <message>Goodbye!!</message>
             </good:sayGoodbye>
             </soapenv:Body>
            </soapenv:Envelope>
            11:21:37,123 INFO [STDOUT] Web Service Parameter - message=Goodbye!!
            11:21:37,124 INFO [STDOUT] [Quickstart_webservice_producer] AFTER invoking jbossws endpoint:
            11:21:37,124 INFO [STDOUT] [<env:Envelope xmlns:env='http://schemas.xmlsoap.org/soap/envelope/'><env:Header></env:Header><env:Body><ns2:sayGoodbyeResponse xmlns:ns2="http://webservice_producer/goodbyeworld"><return>... Ah Goodbye then!!!! - Goodbye!!</return></ns2:sayGoodbyeResponse></env:Body></env:Envelope>].
            11:21:38,067 INFO [ServiceInvoker] Badly formed EPR [EPR: PortReference < <wsa:Address socket://localhost:8888/> >] for Service [MyServiceCategory:MyWSProducerService] and Message [header: [ ]].Courier for EPR not supported: ESB-unaware EPR used!
            11:21:38,067 INFO [ServiceInvoker] Invalid EPR for service (probably ESB-unaware): ignoring for message: header: [ ]
            11:21:38,067 INFO [ServiceInvoker] Badly formed EPR [EPR: PortReference < <wsa:Address http://localhost:8765/> >] for Service [MyServiceCategory:MyWSProducerService] and Message [header: [ ]].Courier for EPR not supported: ESB-unaware EPR used!
            11:21:38,068 INFO [ServiceInvoker] Invalid EPR for service (probably ESB-unaware): ignoring for message: header: [ ]
            11:21:38,076 INFO [STDOUT] [Quickstart_webservice_producer] BEFORE invoking jbossws endpoint:
            11:21:38,076 INFO [STDOUT] [<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:good="http://webservice_producer/goodbyeworld">
             <soapenv:Header/>
             <soapenv:Body>
             <good:sayGoodbye>
             <message>Goodbye!!</message>
             </good:sayGoodbye>
             </soapenv:Body>
            </soapenv:Envelope>].
            11:21:38,081 INFO [STDOUT] **** SOAPRequest perhaps mediated by ESB:
            <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:good="http://webservice_producer/goodbyeworld">
             <soapenv:Header/>
             <soapenv:Body>
             <good:sayGoodbye>
             <message>Goodbye!!</message>
             </good:sayGoodbye>
             </soapenv:Body>
            </soapenv:Envelope>
            11:21:38,081 INFO [STDOUT] Web Service Parameter - message=Goodbye!!
            11:21:38,082 INFO [STDOUT] [Quickstart_webservice_producer] AFTER invoking jbossws endpoint:
            11:21:38,082 INFO [STDOUT] [<env:Envelope xmlns:env='http://schemas.xmlsoap.org/soap/envelope/'><env:Header></env:Header><env:Body><ns2:sayGoodbyeResponse xmlns:ns2="http://webservice_producer/goodbyeworld"><return>... Ah Goodbye then!!!! - Goodbye!!</return></ns2:sayGoodbyeResponse></env:Body></env:Envelope>].
            


            The ServiceInvoker gets 3 EPR when loading the service cluster info:

            jms://127.0.0.1:1099/queue/quickstart_webservice_producer_esb
            http://localhost:8765
            socket://localhost:8888
            


            When the ServiceInvoker picks the first one everything is fine, but when it gets one of the others things are not.

            Daniel: Any input on this?

            /Michael


            • 3. Re: Web service sometimes causes
              beve

               

              Daniel: Any input on this?

              Not at the moment but Anders has sent me his example and I'm looking into this matter. I'll post back here as soon as I know more.

              regards,

              /Daniel

              • 4. Re: Web service sometimes causes

                Hi,

                I have now spend a day more testing the Web Services and have come to a conclusion:

                If I deploy the sample "webservice_producer" and hit using some concurrent threads, via SoapUI (20 threads, 1 sec interval), over the ESB, then the log will be filled with:

                -- snip --
                2008-12-29 15:47:49,135 INFO [org.jboss.soa.esb.client.ServiceInvoker] Badly formed EPR [EPR: PortReference < <wsa:Address socket://localhost:8888/> >] for Service [MyServiceCategory:MyWSProducerService] and Message [header: [ To: JMSEpr [ PortReference < <wsa:Address jms://127.0.0.1:1099/queue/quickstart_webservice_producer_esb/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.initial : org.jnp.interfaces.NamingContextFactory/>, <wsa:ReferenceProperties jbossesb:java.naming.provider.url : 127.0.0.1:1099/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.url.pkgs : org.jnp.interfaces/>, <wsa:ReferenceProperties jbossesb:destination-type : queue/>, <wsa:ReferenceProperties jbossesb:specification-version : 1.1/>, <wsa:ReferenceProperties jbossesb:connection-factory : ConnectionFactory/>, <wsa:ReferenceProperties jbossesb:persistent : true/>, <wsa:ReferenceProperties jbossesb:acknowledge-mode : AUTO_ACKNOWLEDGE/>, <wsa:ReferenceProperties jbossesb:transacted : false/>, <wsa:ReferenceProperties jbossesb:type : urn:jboss/esb/epr/type/jms/> > ] ReplyTo: JMSEpr [ PortReference < <wsa:Address jms://127.0.0.1:1099/queue/quickstart_webservice_producer_esb_reply/>, <wsa:ReferenceProperties jbossesb:java.naming.provider.url : 127.0.0.1:1099/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.initial : org.jnp.interfaces.NamingContextFactory/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.url.pkgs : org.jnp.interfaces/>, <wsa:ReferenceProperties jbossesb:destination-type : queue/>, <wsa:ReferenceProperties jbossesb:specification-version : 1.1/>, <wsa:ReferenceProperties jbossesb:connection-factory : ConnectionFactory/>, <wsa:ReferenceProperties jbossesb:message-selector : jbossESBresponseUUID='c60160b2-5c17-4143-a6da-29500fcccf40'/>, <wsa:ReferenceProperties jbossesb:persistent : true/>, <wsa:ReferenceProperties jbossesb:acknowledge-mode : AUTO_ACKNOWLEDGE/>, <wsa:ReferenceProperties jbossesb:transacted : false/>, <wsa:ReferenceProperties jbossesb:type : urn:jboss/esb/epr/type/jms/> > ] ]].Courier for EPR not supported: ESB-unaware EPR used!
                -- snip --

                and others like this (might all relate to the same problem) - as stated in my first post.

                I have also tried to deploy a web service that calls ServiceInvoker directly (not via the SOAPProcessor) on org.jboss.soa.esb.actions.ContentBasedRouter and here the errors also arise.

                I.e. my first conclusions is that there is some serious concurrency problems with both the org.jboss.soa.esb.actions.soap.SOAPProcessor and the org.jboss.soa.esb.actions.ContentBasedRouter.

                Are you able to recreate the problem?


                Kind regards Anders.

                • 5. Re: Web service sometimes causes
                  sanjoa

                  I'm also getting Unresponsive EPR: JMSEpr, while doing soapui loadrunner tests on our jboss esb application.

                  • 6. Re: Web service sometimes causes
                    mimra

                    I looked a bit more at this, and his is the way I see things:

                    The quickstart webservice_producer outputs the following (not on every execution though):

                    D21:56:04,636 INFO [ServiceInvoker] Badly formed EPR [EPR: PortReference < <wsa:Address socket://localhost:8888/> >] for Service [MyServiceCategory:MyWSProducerService] and Message [header: [ ]].Courier for EPR not supported: ESB-unaware EPR used!
                    21:56:04,636 INFO [ServiceInvoker] Invalid EPR for service (probably ESB-unaware): ignoring for message: header: [ ]
                    21:56:04,636 INFO [ServiceInvoker] Badly formed EPR [EPR: PortReference < <wsa:Address http://localhost:8765/> >] for Service [MyServiceCategory:MyWSProducerService] and Message [header: [ ]].Courier for EPR not supported: ESB-unaware EPR used!
                    21:56:04,636 INFO [ServiceInvoker] Invalid EPR for service (probably ESB-unaware): ignoring for message: header: [ ]
                    


                    The webservice_producer quickstart has three providers:

                    JMS
                    JBoss Remoting for HTTP
                    JBoss Remoting for raw socket

                    When running the 'ant runtest' of the webservice_producer quickstart you hit each of these providers. The test invokes all providers: JMS, HTTP and socket.
                    If you edit the build.xml and remove the test for JMS and socket it leaves you with HTTP which - as far as I can see - still exposes the problem.

                    I don't think this is a concurrency problem.

                    The JBossRemotingGatewayListener ultimately invokes the ServiceInvoker.deliverSync and when the ServiceInvoker loads the cluster service info it gets three endpoints:

                    jms://127.0.0.1:1099/queue/quickstart_webservice_producer_esb
                    http://localhost:8765
                    socket://localhost:8888
                    


                    The ServiceInvoker uses a load balancing mechanism to choose the endpoint (I think it defaults to round robin) and if it doesn't get the JMS - it fails with the message shown in the console.
                    This is why this might look as a concurrency problem.

                    The JBossRemotingGatewayListener adds the endpoints:

                    http://localhost:8765
                    socket://localhost:8888
                    


                    This is what seems to cause the problem, but I'm not sure if this is wrong or not - we need Daniels input on this.

                    /Michael


                    • 7. Re: Web service sometimes causes

                      Yeah, you might be onto something, and this might very well explain the

                      D21:56:04,636 INFO [ServiceInvoker] Badly formed EPR [EPR: PortReference < <wsa:Address socket://localhost:8888/> >] for Service [MyServiceCategory:MyWSProducerService] and Message [header: [ ]].Courier for EPR not
                       supported: ESB-unaware EPR used!


                      log messages, but I think there might be more that one problem. As sanjoa also reports we ALSO get the "Unresponsive EPR: JMSEpr" log message sometimes. I do not get the "Unresponsive EPR: JMSEpr" when I run the standard "ant runtest", but when I try to run several threads concurrently hitting the http://localhost:8765 endpoint (in my test using SoapUI, see www.soapui.org) I get timeouts and the "Unresponsive EPR: JMSEpr" in the log. That is why I think there might be a problem with concurrency.


                      Kind regards,

                      Anders Andersen

                      • 8. Re: Web service sometimes causes
                        mimra

                        I haven't tried with multiple clients with the webservice_producer; I've only used 'ant runtest' and with that I have only seen the problem described above with the badly formed EPR and not the unresponsive EPR.

                        Maybe there is more than one problem here...

                        /Michael

                        • 9. Re: Web service sometimes causes
                          beve

                          Hi Guys,

                          Anders: could you try cleaning you juddi database?
                          I suspect that you might have stale epr in the database which might account for the reporting that the JMSEpr being invalid.
                          Could you tell me if you still get the stall that you reported in you application if you use the jbossesb-server from the main trunk (http://anonsvn.jboss.org/repos/labs/labs/jbossesb/trunk/).
                          I noticed an performance issue with the jbr gateway when investigating this (https://jira.jboss.org/jira/browse/JBESB-2252) and would be interested to see if you notice any difference here.

                          I'm looking into the rest of the issues here and I'll post back and try to explain this.

                          Thanks,

                          /Daniel

                          • 10. Re: Web service sometimes causes
                            beve

                            Hi Michael,

                            The ServiceInvoker gets 3 EPR when loading the service cluster info:
                            jms://127.0.0.1:1099/queue/quickstart_webservice_producer_esb
                            http://localhost:8765
                            socket://localhost:8888
                            When the ServiceInvoker picks the first one everything is fine, but when it gets one of the others things are not.
                            Daniel: Any input on this?

                            Yep, you are correct here. The service invoker will get a list of EPRs that are the ones above. ServiceInvoker will iterate through this list trying to use the eprs to deliver the message:
                            replyMessage = eprInvoker.attemptDelivery(message, epr);

                            attemptDelivery will end up in TwoWayCourierImpl.courierFromERP:
                            private Object courierFromEpr(EPR epr, boolean pickUpOnly) throws CourierException, MalformedEPRException
                            {
                             if (null == epr)
                             return null;
                             if (epr instanceof InVMEpr)
                             return CourierFactory.getInstance().getInVMCourier((InVMEpr) epr);
                             if (epr instanceof JMSEpr)
                             return new JmsCourier((JMSEpr) epr, pickUpOnly);
                             if (epr instanceof FileEpr)
                             return new FileCourier((FileEpr) epr, pickUpOnly);
                             if (epr instanceof JDBCEpr)
                             return new SqlTableCourier((JDBCEpr) epr, pickUpOnly);
                            
                             return courierFromGenericEPR(epr, pickUpOnly);
                            }
                            

                            As the socket epr is not in any of the if statements the courierFromGeneric method will be invoked:
                            private Object courierFromGenericEPR(EPR epr, boolean pickUpOnly) throws CourierException, MalformedEPRException
                            {
                             String addr = null;
                            
                             addr = epr.getAddr().getAddress();
                             if (addr.startsWith(InVMEpr.INVM_PROTOCOL))
                             return CourierFactory.getInstance().getInVMCourier(new InVMEpr(epr));
                             if (addr.startsWith(JMSEpr.JMS_PROTOCOL))
                             return new JmsCourier(new JMSEpr(epr), pickUpOnly);
                             if (addr.startsWith(JDBCEpr.JDBC_PROTOCOL))
                             return new SqlTableCourier(new JDBCEpr(epr), pickUpOnly);
                             // TODO magic strings
                             if (addr.startsWith("file://") || addr.startsWith("ftp://") || addr.startsWith("sftp://") || addr.startsWith("ftps://"))
                             return new FileCourier(new FileEpr(epr), pickUpOnly);
                            
                             throw new MalformedEPRException("Courier for " + epr.getClass().getSimpleName() + " not supported: ESB-unaware EPR used!");
                            }

                            What is happening is that we are looking for a bus transport protocol(Courier) that can deliver the message to a service. The supported bus transports are the ones above: InVM, JMS, JDBC, File.
                            These are the protocols that can deliver between servies. As we can see there is no if clause for a socket courier or http courier so a MalformedEPRException will be thrown.
                            I'm not sure why the gateways are added to the juddi database as I though they would not be, and the jms gateway is not being added. I will investigate further and post back.

                            The MalformedEPRException will be throw up to ServiceInvokers.attemptDelivery:
                            We will now see this in the servers console:
                            09:01:38,645 INFO [ServiceInvoker] Badly formed EPR [EPR: PortReference < <wsa:Address socket://localhost:8888/> >] for Service [MyServiceCategory:MyWSProducerService] and Message [header: [ ]].Courier for EPR not supported: ESB-unaware EPR used!

                            Back in ServiceInvokers.post method, it has a catch clause with MalformedERPException:
                            09:04:01,910 INFO [ServiceInvoker] Invalid EPR for service (probably ESB-unaware): ignoring for message: header: [ ]


                            ServiceInvoker will now try the next epr in the list and when a supported transport is found it will be delivered.

                            As mentioned I'll find out why the socket and http gateways are being added and post back.

                            regards,

                            /Daniel




                            • 11. Re: Web service sometimes causes
                              mimra

                              Daniel,

                              At deployment the JBossRemotingGatewayListener registers both EPR's:

                              localhost:8765 (HTTP)
                              localhost:8888 (Socket)
                              


                              See

                               private void registerEndpoint() throws ConfigurationException, RegistryException {
                               String serviceDescription = getConfig().getAttribute(ListenerTagNames.SERVICE_DESCRIPTION_TAG);
                               RegistryFactory.getRegistry().registerEPR(serviceCategory, serviceName, serviceDescription,
                               endpointReference, endpointReference.getAddr().getAddress());
                               }
                              


                              I don't see why this should be in the registry - I think this is wrong...

                              /Michael

                              • 12. Re: Web service sometimes causes
                                beve

                                Hi Michael,

                                thanks, I was just looking at that :)
                                Let me see that I can dig out form this and I'll post back.

                                Thanks for the feeback on this issue.

                                /Daniel

                                • 13. Re: Web service sometimes causes
                                  mimra

                                  Regarding the unresponsive EPR messages; I have tried with a clean ESB instance (deleted server/default/data) to make sure there is a clean juddi DB.

                                  When using SOAPui with 50 threads and a total run of 500 invocations, I get the same problem. The log contains:

                                  2008-12-30 10:47:05,565 INFO [org.jboss.soa.esb.client.ServiceInvoker] Unresponsive EPR: JMSEpr [ PortReference < <wsa:Address jms://127.0.0.1:1099/queue/quickstart_webservice_producer_esb/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.initial : org.jnp.interfaces.NamingContextFactory/>, <wsa:ReferenceProperties jbossesb:java.naming.provider.url : 127.0.0.1:1099/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.url.pkgs : org.jnp.interfaces/>, <wsa:ReferenceProperties jbossesb:destination-type : queue/>, <wsa:ReferenceProperties jbossesb:specification-version : 1.1/>, <wsa:ReferenceProperties jbossesb:connection-factory : ConnectionFactory/>, <wsa:ReferenceProperties jbossesb:persistent : true/>, <wsa:ReferenceProperties jbossesb:acknowledge-mode : AUTO_ACKNOWLEDGE/>, <wsa:ReferenceProperties jbossesb:transacted : false/>, <wsa:ReferenceProperties jbossesb:type : urn:jboss/esb/epr/type/jms/> > ] for message: header: [ To: JMSEpr [ PortReference < <wsa:Address jms://127.0.0.1:1099/queue/quickstart_webservice_producer_esb/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.initial : org.jnp.interfaces.NamingContextFactory/>, <wsa:ReferenceProperties jbossesb:java.naming.provider.url : 127.0.0.1:1099/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.url.pkgs : org.jnp.interfaces/>, <wsa:ReferenceProperties jbossesb:destination-type : queue/>, <wsa:ReferenceProperties jbossesb:specification-version : 1.1/>, <wsa:ReferenceProperties jbossesb:connection-factory : ConnectionFactory/>, <wsa:ReferenceProperties jbossesb:persistent : true/>, <wsa:ReferenceProperties jbossesb:acknowledge-mode : AUTO_ACKNOWLEDGE/>, <wsa:ReferenceProperties jbossesb:transacted : false/>, <wsa:ReferenceProperties jbossesb:type : urn:jboss/esb/epr/type/jms/> > ] ReplyTo: JMSEpr [ PortReference < <wsa:Address jms://127.0.0.1:1099/queue/quickstart_webservice_producer_esb_reply/>, <wsa:ReferenceProperties jbossesb:java.naming.provider.url : 127.0.0.1:1099/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.initial : org.jnp.interfaces.NamingContextFactory/>, <wsa:ReferenceProperties jbossesb:java.naming.factory.url.pkgs : org.jnp.interfaces/>, <wsa:ReferenceProperties jbossesb:destination-type : queue/>, <wsa:ReferenceProperties jbossesb:specification-version : 1.1/>, <wsa:ReferenceProperties jbossesb:connection-factory : ConnectionFactory/>, <wsa:ReferenceProperties jbossesb:message-selector : jbossESBresponseUUID='fe1a0a82-dfb5-42f0-85db-8cb1fbad30aa'/>, <wsa:ReferenceProperties jbossesb:persistent : true/>, <wsa:ReferenceProperties jbossesb:acknowledge-mode : AUTO_ACKNOWLEDGE/>, <wsa:ReferenceProperties jbossesb:transacted : false/>, <wsa:ReferenceProperties jbossesb:type : urn:jboss/esb/epr/type/jms/> > ] ]
                                  


                                  I don't think this is related to the other badly EPR... thing. Maybe we should split this thread?

                                  /Michael

                                  • 14. Re: Web service sometimes causes

                                    Yeah, we are probably looking at two different issues here... and yes, we should probably split the thread, one for the "Badly formed EPR" and one for the "Unresponsive EPR: JMSEpr". Michael, did you recreate the "Unresponsive EPR: JMSEpr" with the latest binary or did you build the ESB from trunk? If not, I will try to build from trunk and see if the trunk version also has the problem.

                                    Also, I recreated the "Unresponsive EPR: JMSEpr" using 20 threads and 100 invocations in SoapUI, and it can probably be done with only 5 threads...


                                    Kind regards,

                                    Anders Andersen.

                                    1 2 Previous Next