Performance Issue - CXF - CLOSE_WAIT

[jcs_jean-claude.souvignet]

hello,

I use :

ServiceMix Kernel (1.1.0.1-fuse)

ServiceMix (4.1.0.2-fuse)

I have a simple webservice hosted by fuse ESB. It is defined with CXF binding component consumer and a provider mapped on webservice hosted by Tomcat.

I use Soap UI to test the webservice on the bus (strategie simple).

With one thread everything works fine

With five thread process is running without stop

Everything is blocked : I can't call my webservice from any other client

when I list connexion on the server (netstat -an |grep 8090) :

every connection are in state CLOSE_WAIT.

Thanks in advance for help to solve this issue

[ubhole]

Can you specify what platform you are running on? I have doubt that it is something to do with the underlying TCP/IP settings. if it's linux you can try playing with net.ipv4.tcp_fin_timeout in /etc/sysctl.conf.

Ulhas

[jcs_jean-claude.souvignet]

Hello,

I have the same comportment with this configuration :

-> Injection under Windows - Service Mix under Linux - Tomcat (hosted web service)  under Linux

-> Injection under Windows - Service Mix under windows - Tomcat (hosted web service)  under Linux

I think as you it is not related to TCP/IP settings.

After my test, service mix is not available. I need to kill process.

One Idea ?

[ubhole]

Hi jcs,

Do you have some narrowed down testcase that shows the problem? I can take a look at it if you have one.

Regards,

Ulhas

[jcs_jean-claude.souvignet]

Issue test case in performanceIssue.zip :

--> Running WebService

unzip EndService/EndService.zip

mvn install

mvn jetty:run

--> Deploying Bundle OSGI

unzip OsgiBundle/ProtoPartDeuxFuseSv.zip

mvn install

in service mix (4.1.0.2) : osgi/install -s mvn:fr.sv.esb.protofuse/ProtoPartDeuxFuseSv-sa/0.0.1-SNAPSHOT/zip

--> Running testSoapUI

Test with TestSayHello

Run Load TestSayHello -> Threads 10, stratery simple

After 5 responses test failed. Service mix can't be stop. Test under SoapUi run without stop.

[ubhole]

Hi JCS,

I will try running the test but observed EndService folder is empty. Can you attach the service impl that is missing?

Regards,

Ulhas Bhole

[jcs_jean-claude.souvignet]

Sorry, good zip version...

[ubhole]

Managed to reproduce the issue after adding version attribute to jbi-maven-plugin. I will look into it more and get back to you as soon as I can.

/Ulhas

[ubhole]

Can you try using latest FUSE ESB 4.1 version 4.1.0-psc-01-00RC1? You will need to update you camel-context to camel 2.x namespace and camel version with latest 4.1 release is 2.1. For exact dependecy versions you can check release notes.

I have modified the testcase you sent to run with latest version and don't see any issues.

/Ulhas

[jcs_jean-claude.souvignet]

I'm not currently on this version. Changes between 4.1 and 4.1.0-psc-01-00RC1 are not simple. I need a little time to understand the action to make.

I have just done another test by enable trace in camelContext. I don't know the effective consequence but it seems to work well (with 4.1).

Do you have comprehensive explication for this comportment ?

-- change in camel context --

<camelContext id="dynamicrouteurCamel" trace="true" xmlns="http://activemq.apache.org/camel/schema/spring">

[ubhole]

4.1.0-psc-01-00RC1 is the latest 4.1 release and will soon be followed by a new GA release for FUSE ESB 4 and all the fixes will be available on new version which won't be done on 4.1.0.2.

Main difference is camel namespace will need to change for new namespace that Camel 2.x uses. you will need to change namespace to http://camel.apache.org/schema/spring

/Ulhas

[jcs_jean-claude.souvignet]

Hello,

After change for RC version I have this error :

Do your understande the problème ?

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dynamicrouteurCamel': Invocation of init method failed; nested exception is java.lang.NoSuchMethodError: fr.sv.esb.protofuse.partdeux.routing.PartDeuxRouteBuilder.from(Ljava/lang/String;)Lorg/apache/camel/model/RouteType;

     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1338)

     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:473)

     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)

     at java.security.AccessController.doPrivileged(Native Method)

     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)

     at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)

     at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)

     at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)

     at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)

     at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)

     at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:308)

     at org.springframework.context.support.AbstractApplicationContext.getBeansOfType(AbstractApplicationContext.java:947)

     at org.springframework.context.support.AbstractApplicationContext.registerListeners(AbstractApplicationContext.java:701)

     at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:377)

     at org.apache.servicemix.common.xbean.AbstractXBeanDeployer.deploy(AbstractXBeanDeployer.java:87)

     at org.apache.servicemix.camel.CamelSpringDeployer.deploy(CamelSpringDeployer.java:77)

     at org.apache.servicemix.common.BaseServiceUnitManager.doDeploy(BaseServiceUnitManager.java:88)

     at org.apache.servicemix.common.BaseServiceUnitManager.deploy(BaseServiceUnitManager.java:69)

     at org.apache.servicemix.jbi.deployer.artifacts.ServiceUnitImpl.deploy(ServiceUnitImpl.java:104)

     at org.apache.servicemix.jbi.deployer.impl.ServiceAssemblyInstaller.deploySUs(ServiceAssemblyInstaller.java:207)

     at org.apache.servicemix.jbi.deployer.impl.ServiceAssemblyInstaller.install(ServiceAssemblyInstaller.java:85)

     at org.apache.servicemix.jbi.deployer.impl.Deployer.onBundleStarted(Deployer.java:352)

     at org.apache.servicemix.jbi.deployer.impl.Deployer.bundleChanged(Deployer.java:282)

     at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:919)

     at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:227)

     at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:149)

     at org.eclipse.osgi.framework.internal.core.Framework.publishBundleEventPrivileged(Framework.java:1350)

     at org.eclipse.osgi.framework.internal.core.Framework.publishBundleEvent(Framework.java:1301)

     at org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(BundleHost.java:362)

     at org.eclipse.osgi.framework.internal.core.AbstractBundle.updateWorker(AbstractBundle.java:625)

     at org.eclipse.osgi.framework.internal.core.AbstractBundle.update(AbstractBundle.java:572)

     at org.eclipse.osgi.framework.internal.core.AbstractBundle.update(AbstractBundle.java:555)

     at org.apache.felix.karaf.shell.osgi.UpdateBundle.doExecute(UpdateBundle.java:37)

     at org.apache.felix.karaf.shell.osgi.BundleCommand.doExecute(BundleCommand.java:42)

     at org.apache.felix.karaf.shell.console.OsgiCommandSupport.execute(OsgiCommandSupport.java:41)

     at org.apache.felix.gogo.commands.basic.AbstractCommand.execute(AbstractCommand.java:34)

     at org.apache.felix.gogo.runtime.shell.CommandProxy.execute(CommandProxy.java:45)

     at org.apache.felix.gogo.runtime.shell.Closure.execute(Closure.java:211)

     at org.apache.felix.gogo.runtime.shell.Closure.executeStatement(Closure.java:146)

     at org.apache.felix.gogo.runtime.shell.Pipe.run(Pipe.java:91)

     at org.apache.felix.gogo.runtime.shell.Closure.execute(Closure.java:75)

     at org.apache.felix.gogo.runtime.shell.CommandSessionImpl.execute(CommandSessionImpl.java:71)

     at org.apache.felix.karaf.shell.console.jline.Console.run(Console.java:174)

     at java.lang.Thread.run(Thread.java:619)

Caused by: java.lang.NoSuchMethodError: fr.sv.esb.protofuse.partdeux.routing.PartDeuxRouteBuilder.from(Ljava/lang/String;)Lorg/apache/camel/model/RouteType;

     at fr.sv.esb.protofuse.partdeux.routing.PartDeuxRouteBuilder.configure(PartDeuxRouteBuilder.java:46)

     at org.apache.camel.builder.RouteBuilder.checkInitialized(RouteBuilder.java:295)

     at org.apache.camel.builder.RouteBuilder.configureRoutes(RouteBuilder.java:250)

     at org.apache.camel.builder.RouteBuilder.addRoutesToCamelContext(RouteBuilder.java:236)

     at org.apache.camel.impl.DefaultCamelContext.addRoutes(DefaultCamelContext.java:540)

     at org.apache.camel.spring.CamelContextFactoryBean.installRoutes(CamelContextFactoryBean.java:842)

     at org.apache.camel.spring.CamelContextFactoryBean.afterPropertiesSet(CamelContextFactoryBean.java:320)

     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1369)

     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1335)

     ... 43 more

[davsclaus]

You gotta recompile your Camel routes as 2.x is different than 1.x.

Eg this class RouteType has been renamed to RouteDefinition in 2.x. So you need to adjust your code accordingly.

[jcs_jean-claude.souvignet]

Hello,

I have not explicite reference to RouteType. So adjusting code can't be done.

By adjusting POM from my camel project with jaxb dependency :

Deployment work perfectly and services are available. I need time to make test but it seems that performance issue is not effective.

Have you explication to understand with this dependency solve this perturbations ?

[davsclaus]

I see this exception

method failed; nested exception is java.lang.NoSuchMethodError: fr.sv.esb.protofuse.partdeux.routing.PartDeuxRouteBuilder.from(Ljava/lang/String;)Lorg/apache/camel/model/RouteType;

Which tells me you are using your own Camel route builder in the class PartDeuxRouteBuilder. This class may use imports of Camel 1.x classes or somehow require use a Camel 1.x type which is RouteType.

But I have no clue why JAXB resolved it. Its a pain with JAXB being or not being in the JDK core and which version to use etc.