getting empty responses

[fsalceda]

Using a "cxf-consumer" -> "cxf-provider" sequence on  "FUSE ESB 4.2.0-fuse-02-00". It's just a pass through (bridge) call used to test a problem I'm having in my real scenario.

I'm doing performance test (with jmetter) and I usually get 0 bytes responses after a random amount of calls although the final service is responding correctly and consumer is receiving response from provider ( setted debug level ). I didn't detect any memory problem with jconsole.

-

---

SoapUI log messages before and after first empty response:

Fri Sep 10 12:49:21 CEST 2010:INFO:Got response for SessioWSSoapBinding.altaRegistro:Request 1 in 124ms (498 bytes)

Fri Sep 10 13:22:28 CEST 2010:DEBUG:HttpConnectionManager.getConnection:  config = HostConfiguration[host=http://localhost:9797], timeout = 0

Fri Sep 10 13:22:28 CEST 2010:DEBUG:Getting free connection, hostConfig=HostConfiguration[host=http://localhost:9797]

Fri Sep 10 13:22:28 CEST 2010:DEBUG:Freeing connection, hostConfig=HostConfiguration[host=http://localhost:9797]

Fri Sep 10 13:22:28 CEST 2010:DEBUG:Notifying no-one, there are no waiting threads

Fri Sep 10 13:22:28 CEST 2010:INFO:Got response for

SessioWSSoapBinding.altaRegistro:Request 1 in 53ms (0 bytes)

Fri Sep 10 13:22:35 CEST 2010:DEBUG:HttpConnectionManager.getConnection:  config = HostConfiguration[host=http://localhost:9797], timeout = 0

Fri Sep 10 13:22:35 CEST 2010:DEBUG:Getting free connection, hostConfig=HostConfiguration[host=http://localhost:9797]

Fri Sep 10 13:22:35 CEST 2010:DEBUG:Freeing connection, hostConfig=HostConfiguration[host=http://localhost:9797]

Fri Sep 10 13:22:35 CEST 2010:DEBUG:Notifying no-one, there are no waiting threads

-

---

I would appreciate any help!

Thanks.

Fernando.

[fsalceda]

I'm quite dissapointed because performance test are not as good as I expected. I'm changing some thread parameters (core and max threads) in config files but I think smx thread behaviour is not that one I have set. Per example, If I set core thread on 32, and max thread on 64, pool never create more than 32 thread per component (camel or cxf-bc), even If I lunch more than 32 concurrent request ( It will cause smx colapse threads waiting for exchange and no more request will be processed ).

Setted example configuration in "/org.apache.servicemix.jbi.cfg":

-

---

shutdownTimeout=120000

corePoolSize=32

maximumPoolSize=64

queueSize=1024

allowCoreThreadTimeout=true

keepAliveTime=60000

-

---

and "org.apache.servicemix.components.cxfbc.cfg":

-

---

threadPoolCorePoolSize=32

threadPoolMaximumPoolSize=64

threadPoolQueueSize=512

-

---

I would appreciate an answer.

Thanks.

If I don't find out a way to increase performance up to 100 user (threads) I will think in another possible solution to my scenario because now I'm using very simple tests (consumer -> provider) that should run better and faster than a real test.

[ffang]

Hi,

Seems your threadpool configuration doesn't get picked up. Where you put org.apache.servicemix.components.cxfbc.cfg in? It should be $SMX_HOME/etc/ folder.

Also, could you change

threadPoolCorePoolSize=64

threadPoolMaximumPoolSize=128

to see if you get better performance?

Freeman

[ffang]

Btw, you should use org.apache.servicemix.nmr.cfg but not org.apache.servicemix.jbi.cfg to configure the NMR threadpool.

Freeman

[fsalceda]

Created a new config file $SMX_HOME/etc/org.apache.servicemix.nmr.cfg and setted:

corePoolSize=64

maximumPoolSize=128

But behaviour of smx didn't change. ThreadPool never creates more threads than the value of "corePoolSize", in this case 64.

camel threads => 64

cxf-bc threads =>64

More than aprox. 60 users (client threads) will bring to a standstill to smx. Threads appear to be "waiting for exchange XXXXXXXXXXXXXXX".

I can increase corePoolSize, of course, but that's not the case. I would know if maximumPoolSize works and if it really set up the top amount of threads that will be created if there are more requests than value of corePoolSize threads...

[ffang]

Hi,

Did you ever put queueSize(default is 1024 iirc) in org.apache.servicemix.nmr.cfg,

Here is the rule about how *PoolSize works

• if the number of threads is less than corePoolSize, the executor will create a new thread to handle the job

• if the number of queued jobs is less than queueSize, the job is queued

• if the queue is full and the number of threads is less than maximumPoolSize, a new thread is created to handle the job

• else, the current thread will handle the job

so the job is queued until queue is full and you can't see a new thread created until queue is full. If you want to see create more thread than corePoolSize immediately, you need set queueSize=0

You can get more details from .

Btw, could you add synchronous="false" to your cxf bc consumer endpoint configuration, this will get better performance.

http://servicemix.apache.org/thread-pools.html

Freeman

[fsalceda]

Thx Freeman...

I hadn't put queueSize parametger in org.apache.servicemix.nmr.cfg but I did it and changed also queueSize = 0 in every configuration file (org.apache.servicemix.nmr.cfg, org.apache.servicemix.jbi.cfg and org.apache.servicemix.components.cxfbc.cfg). And since them I can see how there are some variations in the amount of threads that before I couldn't see. But even setting queueSize = 0 and maximumPoolSize = 1024, smx never reaches the maximum value or exceeds the corePoolSize.

Performance tests with more than corePoolSize value users usually brings to a standstill Smx and I have to restart it. Perhaps I am not understanding very well how thread pool works, but I think that It should queue a request (or 1024 request by default) and process them when there is a free thread left, don't it?

I also tried to increase jvm heap and permSize memory, just in case, but there weren't any memory problems.

The next I will check out is to add synchronous="false" to my cxf bc consumer endpoint configuration...

[yusuf.jakoet]

Hi

I don't know what the solution is, but I'd like to say that I'm experiencing the exact same problem.

I usually resolve it by stopping/starting the CXF - BC/SE via the smx console..

[ffang]

Hi,

Add synchronous="false" attribute for your cxf bc consumer endpoint should resolve it. More details please see my blog "Why ESB server hang after runing for a while?" from http://freemanfang.blogspot.com

Freeman

[fsalceda]

Should I also add synchronous="false" attribute for my cxf bc provider endpoint? From time to time server hang, not as frequently as before but It still happens!

After reading your blog post I think I did not configure my working flow correctly because smx do not create more than "corePoolSize value" threads even though I setted queueSize = 0.  I'm using:

cxfbc-consumer ( synchronous="false" )

camel-enricher (bean implementing enrich method)

camel-recipientList (bean implementing getTargets())

camel-transformer (xslt file)

cxfbc-provider

[fsalceda]

First test (call 1):       200 users

Second test (call 2): 300 users

Third test (call 3):     400 users  -> smx hang (threads waiting for exchange).

I atach a jconsole screenshot.

- thread configuraction

in "org.apache.servicemix.jbi.cfg":

shutdownTimeout=120000

corePoolSize=512

maximumPoolSize=1024

queueSize=1024

allowCoreThreadTimeout=true

keepAliveTime=60000

in "org.apache.servicemix.components.cxfbc.cfg":

threadPoolCorePoolSize=512

threadPoolMaximumPoolSize=512

threadPoolQueueSize=1024

creating and setting "org.apache.servicemix.nmr.cfg" has no effect in my case.

Edited by: fsalceda on Oct 6, 2010 1:48 PM