10 Replies Latest reply on Oct 6, 2010 9:49 AM by Fernando Salceda

    getting empty responses

    Fernando Salceda Novice

      Using a "cxf-consumer" -> "cxf-provider" sequence on  "FUSE ESB 4.2.0-fuse-02-00". It's just a pass through (bridge) call used to test a problem I'm having in my real scenario.

       

      I'm doing performance test (with jmetter) and I usually get 0 bytes responses after a random amount of calls although the final service is responding correctly and consumer is receiving response from provider ( setted debug level ). I didn't detect any memory problem with jconsole.

       

      -


      SoapUI log messages before and after first empty response:

       

      Fri Sep 10 12:49:21 CEST 2010:INFO:Got response for SessioWSSoapBinding.altaRegistro:Request 1 in 124ms (498 bytes)

      Fri Sep 10 13:22:28 CEST 2010:DEBUG:HttpConnectionManager.getConnection:  config = HostConfiguration[host=http://localhost:9797], timeout = 0

      Fri Sep 10 13:22:28 CEST 2010:DEBUG:Getting free connection, hostConfig=HostConfiguration[host=http://localhost:9797]

      Fri Sep 10 13:22:28 CEST 2010:DEBUG:Freeing connection, hostConfig=HostConfiguration[host=http://localhost:9797]

      Fri Sep 10 13:22:28 CEST 2010:DEBUG:Notifying no-one, there are no waiting threads

      Fri Sep 10 13:22:28 CEST 2010:INFO:Got response for

       

      SessioWSSoapBinding.altaRegistro:Request 1 in 53ms (0 bytes)

      Fri Sep 10 13:22:35 CEST 2010:DEBUG:HttpConnectionManager.getConnection:  config = HostConfiguration[host=http://localhost:9797], timeout = 0

      Fri Sep 10 13:22:35 CEST 2010:DEBUG:Getting free connection, hostConfig=HostConfiguration[host=http://localhost:9797]

      Fri Sep 10 13:22:35 CEST 2010:DEBUG:Freeing connection, hostConfig=HostConfiguration[host=http://localhost:9797]

      Fri Sep 10 13:22:35 CEST 2010:DEBUG:Notifying no-one, there are no waiting threads

      -


       

      I would appreciate any help!

       

       

      Thanks.

       

      Fernando.

        • 1. Re: getting empty responses
          Fernando Salceda Novice

          I'm quite dissapointed because performance test are not as good as I expected. I'm changing some thread parameters (core and max threads) in config files but I think smx thread behaviour is not that one I have set. Per example, If I set core thread on 32, and max thread on 64, pool never create more than 32 thread per component (camel or cxf-bc), even If I lunch more than 32 concurrent request ( It will cause smx colapse threads waiting for exchange and no more request will be processed ).

           

          Setted example configuration in "/org.apache.servicemix.jbi.cfg":

          -


          shutdownTimeout=120000

          corePoolSize=32

          maximumPoolSize=64

          queueSize=1024

          allowCoreThreadTimeout=true

          keepAliveTime=60000

          -


           

          and "org.apache.servicemix.components.cxfbc.cfg":

          -


          threadPoolCorePoolSize=32

          threadPoolMaximumPoolSize=64

          threadPoolQueueSize=512

          -


           

           

          I would appreciate an answer.

           

          Thanks.

           

           

          If I don't find out a way to increase performance up to 100 user (threads) I will think in another possible solution to my scenario because now I'm using very simple tests (consumer -> provider) that should run better and faster than a real test.

          • 2. Re: getting empty responses
            Freeman(Yue) Fang Master

            Hi,

             

            Seems your threadpool configuration doesn't get picked up. Where you put org.apache.servicemix.components.cxfbc.cfg in? It should be $SMX_HOME/etc/ folder.

             

            Also, could you change

            threadPoolCorePoolSize=64

            threadPoolMaximumPoolSize=128

             

            to see if you get better performance?

             

            Freeman

            • 3. Re: getting empty responses
              Freeman(Yue) Fang Master

              Btw, you should use org.apache.servicemix.nmr.cfg but not org.apache.servicemix.jbi.cfg to configure the NMR threadpool.

               

              Freeman

              • 4. Re: getting empty responses
                Fernando Salceda Novice

                Created a new config file $SMX_HOME/etc/org.apache.servicemix.nmr.cfg and setted:

                 

                corePoolSize=64

                maximumPoolSize=128

                 

                But behaviour of smx didn't change. ThreadPool never creates more threads than the value of "corePoolSize", in this case 64.

                 

                camel threads => 64

                cxf-bc threads =>64

                 

                More than aprox. 60 users (client threads) will bring to a standstill to smx. Threads appear to be "waiting for exchange XXXXXXXXXXXXXXX".

                 

                I can increase corePoolSize, of course, but that's not the case. I would know if maximumPoolSize works and if it really set up the top amount of threads that will be created if there are more requests than value of corePoolSize threads...

                • 5. Re: getting empty responses
                  Freeman(Yue) Fang Master

                  Hi,

                   

                  Did you ever put queueSize(default is 1024 iirc) in org.apache.servicemix.nmr.cfg,

                   

                  Here is the rule about how *PoolSize works

                   

                       

                  • if the number of threads is less than corePoolSize, the executor will create a new thread to handle the job

                       

                  • if the number of queued jobs is less than queueSize, the job is queued

                       

                  • if the queue is full and the number of threads is less than maximumPoolSize, a new thread is created to handle the job

                       

                  • else, the current thread will handle the job

                   

                  so the job is queued until queue is full and you can't see a new thread created until queue is full. If you want to see create more thread than corePoolSize immediately, you need set queueSize=0

                   

                  You can get more details from .

                   

                  Btw, could you add synchronous="false" to your cxf bc consumer endpoint configuration, this will get better performance.

                   

                  http://servicemix.apache.org/thread-pools.html

                   

                  Freeman

                  • 6. Re: getting empty responses
                    Fernando Salceda Novice

                    Thx Freeman...

                     

                    I hadn't put queueSize parametger in org.apache.servicemix.nmr.cfg but I did it and changed also queueSize = 0 in every configuration file (org.apache.servicemix.nmr.cfg, org.apache.servicemix.jbi.cfg and org.apache.servicemix.components.cxfbc.cfg). And since them I can see how there are some variations in the amount of threads that before I couldn't see. But even setting queueSize = 0 and maximumPoolSize = 1024, smx never reaches the maximum value or exceeds the corePoolSize.

                     

                    Performance tests with more than corePoolSize value users usually brings to a standstill Smx and I have to restart it. Perhaps I am not understanding very well how thread pool works, but I think that It should queue a request (or 1024 request by default) and process them when there is a free thread left, don't it?

                     

                    I also tried to increase jvm heap and permSize memory, just in case, but there weren't any memory problems.

                     

                    The next I will check out is to add synchronous="false" to my cxf bc consumer endpoint configuration...

                    • 7. Re: getting empty responses
                      Yusuf Jakoet Newbie

                      Hi

                       

                      I don't know what the solution is, but I'd like to say that I'm experiencing the exact same problem.

                       

                      I usually resolve it by stopping/starting the CXF - BC/SE via the smx console..

                      • 8. Re: getting empty responses
                        Freeman(Yue) Fang Master

                        Hi,

                         

                        Add synchronous="false" attribute for your cxf bc consumer endpoint should resolve it. More details please see my blog "Why ESB server hang after runing for a while?" from http://freemanfang.blogspot.com

                         

                         

                        Freeman

                        • 9. Re: getting empty responses
                          Fernando Salceda Novice

                          Should I also add synchronous="false" attribute for my cxf bc provider endpoint? From time to time server hang, not as frequently as before but It still happens!

                           

                          After reading your blog post I think I did not configure my working flow correctly because smx do not create more than "corePoolSize value" threads even though I setted queueSize = 0.  I'm using:

                           

                          cxfbc-consumer ( synchronous="false" )

                          camel-enricher (bean implementing enrich method)

                          camel-recipientList (bean implementing getTargets())

                          camel-transformer (xslt file)

                          cxfbc-provider

                          • 10. Re: getting empty responses
                            Fernando Salceda Novice

                            First test (call 1):       200 users

                            Second test (call 2): 300 users

                            Third test (call 3):     400 users  -> smx hang (threads waiting for exchange).

                             

                            I atach a jconsole screenshot.

                             

                            - thread configuraction

                             

                            in "org.apache.servicemix.jbi.cfg":

                             

                            shutdownTimeout=120000

                            corePoolSize=512

                            maximumPoolSize=1024

                            queueSize=1024

                            allowCoreThreadTimeout=true

                            keepAliveTime=60000

                             

                            in "org.apache.servicemix.components.cxfbc.cfg":

                             

                            threadPoolCorePoolSize=512

                            threadPoolMaximumPoolSize=512

                            threadPoolQueueSize=1024

                             

                             

                            creating and setting "org.apache.servicemix.nmr.cfg" has no effect in my case.

                             

                            Edited by: fsalceda on Oct 6, 2010 1:48 PM