SCAInvoker contention on classloading in SwitchYard 1.1 and 1.0
synclpz Jun 10, 2014 6:13 AMHello, guys.
We are trying to use SCA references to integrate standalone services (independent SY jars) on one local server, so only invokeLocal() is userd. One of the task is high-concurrency, and there we see some performance lag in SCA (maybe). SwitchYard version is 1.1 and 1.0 tested. Stacktrace below is for 1.1.
Example:
Service 1 has REST Service binding with one operation "start"
Service 1 has SCA Reference binding to Service 2
Service 2 is a bean with Java interface and SCA service binding
Service 2 has a simple logic to output a log message only
Once "started" Service 1 starts a thread pool of 100 threads and starts calling Service 2 through SCA. The performance is analyzed via IBM WAIT tool, which just takes thread dumps and point us to a lock contentions. And there is one! 88% of time is spent in classloader, which is called from:
org/switchyard/common/type/Classes.forName() line 64
org/switchyard/common/type/Classes.forName() line 41
org/switchyard/common/xml/QNameUtil.toJavaMessageType() line 77
Is it a bug in SCA invocation subsystem?
Full stacktrace below:
java/lang/ClassLoader.loadClass() line 404 sun/misc/Launcher$AppClassLoader.loadClass() line 308 java/lang/ClassLoader.loadClass() line 357 java/lang/Class.forName0()native method java/lang/Class.forName() line 270 org/switchyard/common/type/Classes.forName() line 64 org/switchyard/common/type/Classes.forName() line 41 org/switchyard/common/xml/QNameUtil.toJavaMessageType() line 77 org/switchyard/internal/validate/BaseValidatorRegistry.getJavaFallbackValidator() line 140 org/switchyard/internal/validate/BaseValidatorRegistry.getValidator() line 93 org/switchyard/handlers/ValidateHandler.get() line 101 org/switchyard/handlers/ValidateHandler.handleMessage() line 66 org/switchyard/bus/camel/processors/HandlerProcessor.process() line 62 org/apache/camel/util/AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process() line 61 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/management/InstrumentationProcessor.process() line 73 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/processor/interceptor/TraceInterceptor.process() line 91 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/switchyard/bus/camel/audit/FaultProcessor.process() line 46 org/apache/camel/processor/RouteContextProcessor.processNext() line 45 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/processor/interceptor/DefaultChannel.process() line 303 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/Pipeline.process() line 117 org/apache/camel/processor/Pipeline.process() line 80 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/processor/FilterProcessor.process() line 58 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/management/InstrumentationProcessor.process() line 73 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/processor/interceptor/TraceInterceptor.process() line 91 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/switchyard/bus/camel/audit/FaultProcessor.process() line 46 org/apache/camel/processor/RouteContextProcessor.processNext() line 45 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/processor/interceptor/DefaultChannel.process() line 303 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/Pipeline.process() line 117 org/apache/camel/processor/Pipeline.process() line 80 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/TryProcessor.process() line 104 org/apache/camel/processor/TryProcessor.process() line 78 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/management/InstrumentationProcessor.process() line 73 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/processor/interceptor/TraceInterceptor.process() line 91 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/switchyard/bus/camel/audit/FaultProcessor.process() line 46 org/apache/camel/processor/RouteContextProcessor.processNext() line 45 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/processor/interceptor/DefaultChannel.process() line 303 org/apache/camel/processor/RouteContextProcessor.processNext() line 45 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/processor/UnitOfWorkProcessor.process() line 122 org/apache/camel/processor/RouteInflightRepositoryProcessor.processNext() line 48 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/management/InstrumentationProcessor.process() line 73 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/component/direct/DirectProducer.process() line 61 org/apache/camel/processor/UnitOfWorkProcessor.processAsync() line 150 org/apache/camel/processor/UnitOfWorkProcessor.process() line 117 org/apache/camel/util/AsyncProcessorHelper.process() line 99 org/apache/camel/processor/DelegateAsyncProcessor.process() line 86 org/apache/camel/processor/UnitOfWorkProducer.process() line 63 org/apache/camel/impl/ProducerCache$2.doInProducer() line 360 org/apache/camel/impl/ProducerCache$2.doInProducer() line 331 org/apache/camel/impl/ProducerCache.doInProducer() line 227 org/apache/camel/impl/ProducerCache.sendExchange() line 331 org/apache/camel/impl/ProducerCache.send() line 153 org/apache/camel/impl/DefaultProducerTemplate.send() line 106 org/apache/camel/impl/DefaultProducerTemplate.send() line 92 org/switchyard/bus/camel/ExchangeDispatcher.dispatch() line 87 org/switchyard/bus/camel/CamelExchange.sendInternal() line 234 org/switchyard/bus/camel/CamelExchange.send() line 171 org/switchyard/component/sca/SCAInvoker.invokeLocal() line 123 org/switchyard/component/sca/SCAInvoker.handleMessage() line 101 org/switchyard/bus/camel/processors/ProviderProcessor.process() line 29 org/apache/camel/util/AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process() line 61 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/management/InstrumentationProcessor.process() line 73 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/processor/interceptor/TraceInterceptor.process() line 91 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/switchyard/bus/camel/audit/FaultProcessor.process() line 46 org/apache/camel/processor/RouteContextProcessor.processNext() line 45 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/processor/interceptor/DefaultChannel.process() line 303 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/Pipeline.process() line 117 org/apache/camel/processor/Pipeline.process() line 80 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/TryProcessor.process() line 104 org/apache/camel/processor/TryProcessor.process() line 78 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/management/InstrumentationProcessor.process() line 73 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/processor/interceptor/TraceInterceptor.process() line 91 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/switchyard/bus/camel/audit/FaultProcessor.process() line 46 org/apache/camel/processor/RouteContextProcessor.processNext() line 45 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/processor/interceptor/DefaultChannel.process() line 303 org/apache/camel/processor/RouteContextProcessor.processNext() line 45 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/processor/UnitOfWorkProcessor.process() line 122 org/apache/camel/processor/RouteInflightRepositoryProcessor.processNext() line 48 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99 org/apache/camel/processor/DelegateAsyncProcessor.process() line 90 org/apache/camel/management/InstrumentationProcessor.process() line 73 org/apache/camel/util/AsyncProcessorHelper.process() line 73 org/apache/camel/component/direct/DirectProducer.process() line 61 org/apache/camel/processor/UnitOfWorkProcessor.processAsync() line 150 org/apache/camel/processor/UnitOfWorkProcessor.process() line 117 org/apache/camel/util/AsyncProcessorHelper.process() line 99 org/apache/camel/processor/DelegateAsyncProcessor.process() line 86 org/apache/camel/processor/UnitOfWorkProducer.process() line 63 org/apache/camel/impl/ProducerCache$2.doInProducer() line 360 org/apache/camel/impl/ProducerCache$2.doInProducer() line 331 org/apache/camel/impl/ProducerCache.doInProducer() line 227 org/apache/camel/impl/ProducerCache.sendExchange() line 331 org/apache/camel/impl/ProducerCache.send() line 153 org/apache/camel/impl/DefaultProducerTemplate.send() line 106 org/apache/camel/impl/DefaultProducerTemplate.send() line 92 org/switchyard/bus/camel/ExchangeDispatcher.dispatch() line 87 org/switchyard/bus/camel/CamelExchange.sendInternal() line 234 org/switchyard/bus/camel/CamelExchange.send() line 171 org/switchyard/component/bean/ClientProxyBean$ClientProxyInvocationHandler.invoke() line 292 com.sun.proxy com/sun/proxy/$Proxy69.tariff() Dispatching Overhead, JVM Reflection sun/reflect/GeneratedMethodAccessor33.invoke() sun/reflect/DelegatingMethodAccessorImpl.invoke() line 43 java/lang/reflect/Method.invoke() line 606 org/jboss/weld/bean/proxy/AbstractBeanInstance.invoke() line 45 org/jboss/weld/bean/proxy/ProxyMethodHandler.invoke() line 105 org/jboss/weld/proxies/TariffService$1366014918$Proxy$_$$_WeldClientProxy.tariff() com/myapp/switchyard/-----/---/observer_tariff/ObserverTariffService$TarifficationJob.run() line 98