10 Replies Latest reply on Sep 18, 2008 4:43 AM by kconner

    Juddi problems

    viniciuscarvalho

      Hello there! I'm really happy and proud to say that our system finally made into production with JBoss ESB :-).

      We are not using juddi directly, I do not know if it is a core required component of ESB. After a few runs, we are having performance problems that seems to be related to juddi. It happens only on shutdow/startup operations. The time spent to create the auth tokens (sorry, what are they used for anyway??) are taking longer and longer. JBoss takes over 2 minutes (in a 8 processor machine??) just to delete the template bindings for juddi on the shutdown process.

      Is it possible to remove juddi from my esb deploy? Is it safe to do so? Any side effects? Should I just remove the sar from the deploy dir?

      Best regards

        • 1. Re: Juddi problems
          kurtstam

          jUDDI is the registry (like JNDI in JEE) nothing runs without it. What database are you running? Can you try to figure out where the time is spend (DB or ESB). How many services and service bindings do you have?

          -Kurt

          • 2. Re: Juddi problems
            marklittle

            Also, what version of JBossESB are you using?

            • 3. Re: Juddi problems
              viniciuscarvalho

              Hi there! Thanks for the quick reply :)

              We are using the 4.2.1GA version. On mysql. We have few services (8 only). I'll check to see if the time spent is on the database or esb. It's not big deal since it only happens on start/shutdown, and after the test phase, it shall really a rare thing to happen on production :) (at least we hope so).

              Regards

              • 4. Re: Juddi problems
                kconner

                Can you tell us why you think this is an issue with jUDDI? You mentioned the time taken to create the auth tokens but I don't think there is enough information in the logs to reach this conclusion. Have you profiled this?

                • 5. Re: Juddi problems
                  viniciuscarvalho

                  Kevin, it is an issue with juddi. I've enabled trace level for juddi and figured out that juddi performs 105.327 selects on my database on shutdown and 10.000 on startup. And that number is just growing, every time I start/stop the server. I have no idea what's going on, why does juddi needs to create so many selects on shutdown?

                  If I clean the entire database (drop the schema) the number drops to:

                  80 statements on startup
                  114 statements on shutdown

                  Really, I can't believe it. Now, how can I avoid this? Droping the database every time? Also, why does ESB has a juddi-service.sar + inside jbossesb-sar it has another juddi configuration file?

                  Best regards


                  Below is a really small amount of those selects:

                  2008-04-16 10:05:06,867 DEBUG [org.apache.juddi.datastore.jdbc.BindingTemplateTable] select from BINDING_TEMPLATE table:
                  
                   SELECT SERVICE_KEY,ACCESS_POINT_TYPE,ACCESS_POINT_URL,HOSTING_REDIRECTOR FROM BINDING_TEMPLATE WHERE BINDING_KEY=?
                   BINDING_KEY=45F54B10-C430-11DC-8B10-9229B70FBA2B
                  
                  2008-04-16 10:05:06,871 DEBUG [org.apache.juddi.datastore.jdbc.BindingDescTable] select from BINDING_DESCR table:
                  
                   SELECT LANG_CODE,DESCR, BINDING_DESCR_ID FROM BINDING_DESCR WHERE BINDING_KEY=? ORDER BY BINDING_DESCR_ID
                   BINDING_KEY=45F54B10-C430-11DC-8B10-9229B70FBA2B
                  
                  2008-04-16 10:05:06,872 DEBUG [org.apache.juddi.datastore.jdbc.BindingCategoryTable] select from BINDING_CATEGORY table:
                  
                   SELECT TMODEL_KEY_REF,KEY_NAME,KEY_VALUE, CATEGORY_ID FROM BINDING_CATEGORY WHERE BINDING_KEY=? ORDER BY CATEGORY_ID
                   BINDING_KEY=45F54B10-C430-11DC-8B10-9229B70FBA2B
                  
                  2008-04-16 10:05:06,872 DEBUG [org.apache.juddi.datastore.jdbc.TModelInstanceInfoTable] select from TMODEL_INSTANCE_INFO table:
                  
                   SELECT TMODEL_KEY,OVERVIEW_URL,INSTANCE_PARMS, TMODEL_INSTANCE_INFO_ID FROM TMODEL_INSTANCE_INFO WHERE BINDING_KEY=? ORDER BY TMODEL_INSTANCE_INFO_ID
                   BINDING_KEY=45F54B10-C430-11DC-8B10-9229B70FBA2B
                  
                  2008-04-16 10:05:06,873 DEBUG [org.apache.juddi.datastore.jdbc.BindingTemplateTable] select from BINDING_TEMPLATE table:
                  
                   SELECT SERVICE_KEY,ACCESS_POINT_TYPE,ACCESS_POINT_URL,HOSTING_REDIRECTOR FROM BINDING_TEMPLATE WHERE BINDING_KEY=?
                   BINDING_KEY=479C46C0-C611-11DC-86C0-E2AD654DE24F
                  
                  2008-04-16 10:05:06,874 DEBUG [org.apache.juddi.datastore.jdbc.BindingDescTable] select from BINDING_DESCR table:
                  
                   SELECT LANG_CODE,DESCR, BINDING_DESCR_ID FROM BINDING_DESCR WHERE BINDING_KEY=? ORDER BY BINDING_DESCR_ID
                   BINDING_KEY=479C46C0-C611-11DC-86C0-E2AD654DE24F


                  Is there an way to avoid this?

                  • 6. Re: Juddi problems
                    kconner

                     

                    "viniciuscarvalho" wrote:
                    Kevin, it is an issue with juddi. I've enabled trace level for juddi and figured out that juddi performs 105.327 selects on my database on shutdown and 10.000 on startup.

                    That is either an impressive registry or there is something very wrong within jUDDI.

                    "viniciuscarvalho" wrote:
                    And that number is just growing, every time I start/stop the server. I have no idea what's going on, why does juddi needs to create so many selects on shutdown?

                    I am not sure but we certainly need to find out what is going on.

                    "viniciuscarvalho" wrote:
                    Really, I can't believe it. Now, how can I avoid this? Droping the database every time?


                    Give me some time to investigate further, see if I can find out anything.

                    "viniciuscarvalho" wrote:
                    Also, why does ESB has a juddi-service.sar + inside jbossesb-sar it has another juddi configuration file?

                    This is a mistake, the juddi-service.sar should have been removed as the version in jbossesb.sar is more recent.

                    Can you try removing that sar and seeing if you still have the issue?

                    Thanks for all your work so far.

                    Kev

                    • 7. Re: Juddi problems
                      viniciuscarvalho

                      Hi Kevin, thanks for your support. Here's few other stuff I've been checking:

                      1st - I have removed the juddi.sar, yet, it seems that the number of selects is increasing within each start/stop

                      2nd - I have only 12 registred endpoints on my ESB deployment. Every time I get a request on my SOAP/JMS listener, juddi generates 14 selects (Don't know why not 12 :( ) the log presents this:

                      DEBUG [org.apache.juddi.datastore.jdbc.BindingCategoryTable] select from BINDING_CATEGORY table:

                      Ok, I don't understand UDDI internals (tried once, got bored, and too stupid to figure the relations T_MODEL, BINDING etc... sorry :P ), but it seems to me that juddi is looking for some services it should provide, ok thats fair (a cache really would help here, since I don't think my model_binding changes without a redploy does it?). Now, what's most problematic, is that when I dispatch from one service EPR to other (and I do this a few times during the business flow) each time it runs the N selects again. So for instance, I have this action chain:

                      InboundService -> CBRService -> CustomerService -> {ReplyService OR FaultService}

                      So, 4 services are invoked. I have a total of 56 selects for one business flow invocation. We have a 8 processor machine running jboss, and we do have quite a few interactions with database, SLSB , and SNMP protocols under the hood. The average response time is below 2 seconds, at the present time I'm tuning the server in order to get better response time, and I believe those queries do have an impact on it.

                      I'll keep checking this and any progress I made I'll post here.

                      Best regards, and thanks for the support

                      • 8. Re: Juddi problems
                        tcunning

                        I've created http://jira.jboss.com/jira/browse/JBESB-1675 to track this. Vinicius, if there's any other details you can add, or if you can provide a small sample that can help reproduce the problems you are seeing, that'd probably be pretty helpful.

                        • 9. Re: Juddi problems
                          teraryan

                          My company is having performance problems too, and I suspect that the JUDDI registry is the cause. But why doesn't everyone have this problem? The only thing non-standard about our approach is that we manually created the tables in the database. Is that your approach too?

                          • 10. Re: Juddi problems
                            kconner

                            The problem lies with scout and we have a workaround for it. This was committed into trunk for the 4.4GA release.

                            The issue is not related to the tables and their creation but is a consequence of scout pulling down a full graph of every EPR registered in the registry.

                            The JIRA issue mentioned earlier by Tom contains much more information.