5 Replies Latest reply on Jul 1, 2014 5:12 AM by synclpz

    SCAInvoker contention on classloading in SwitchYard 1.1 and 1.0

    synclpz

      Hello, guys.

       

      We are trying to use SCA references to integrate standalone services (independent SY jars) on one local server, so only invokeLocal() is userd. One of the task is high-concurrency, and there we see some performance lag in SCA (maybe). SwitchYard version is 1.1 and 1.0 tested. Stacktrace below is for 1.1.

       

      Example:

       

      Service 1 has REST Service binding with one operation "start"

      Service 1 has SCA Reference binding to Service 2

      Service 2 is a bean with Java interface and SCA service binding

      Service 2 has a simple logic to output a log message only

       

      Once "started" Service 1 starts a thread pool of 100 threads and starts calling Service 2 through SCA. The performance is analyzed via IBM WAIT tool, which just takes thread dumps and point us to a lock contentions. And there is one! 88% of time is spent in classloader, which is called from:

       

      org/switchyard/common/type/Classes.forName() line 64

      org/switchyard/common/type/Classes.forName() line 41

      org/switchyard/common/xml/QNameUtil.toJavaMessageType() line 77

       

      Is it a bug in SCA invocation subsystem?

       

      Full stacktrace below:

       

      java/lang/ClassLoader.loadClass() line 404
      sun/misc/Launcher$AppClassLoader.loadClass() line 308
      java/lang/ClassLoader.loadClass() line 357
      java/lang/Class.forName0()native method
      java/lang/Class.forName() line 270
      org/switchyard/common/type/Classes.forName() line 64
      org/switchyard/common/type/Classes.forName() line 41
      org/switchyard/common/xml/QNameUtil.toJavaMessageType() line 77
      org/switchyard/internal/validate/BaseValidatorRegistry.getJavaFallbackValidator() line 140
      org/switchyard/internal/validate/BaseValidatorRegistry.getValidator() line 93
      org/switchyard/handlers/ValidateHandler.get() line 101
      org/switchyard/handlers/ValidateHandler.handleMessage() line 66
      org/switchyard/bus/camel/processors/HandlerProcessor.process() line 62
      org/apache/camel/util/AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process() line 61
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/management/InstrumentationProcessor.process() line 73
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/processor/interceptor/TraceInterceptor.process() line 91
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/switchyard/bus/camel/audit/FaultProcessor.process() line 46
      org/apache/camel/processor/RouteContextProcessor.processNext() line 45
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/processor/interceptor/DefaultChannel.process() line 303
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/Pipeline.process() line 117
      org/apache/camel/processor/Pipeline.process() line 80
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/processor/FilterProcessor.process() line 58
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/management/InstrumentationProcessor.process() line 73
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/processor/interceptor/TraceInterceptor.process() line 91
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/switchyard/bus/camel/audit/FaultProcessor.process() line 46
      org/apache/camel/processor/RouteContextProcessor.processNext() line 45
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/processor/interceptor/DefaultChannel.process() line 303
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/Pipeline.process() line 117
      org/apache/camel/processor/Pipeline.process() line 80
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/TryProcessor.process() line 104
      org/apache/camel/processor/TryProcessor.process() line 78
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/management/InstrumentationProcessor.process() line 73
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/processor/interceptor/TraceInterceptor.process() line 91
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/switchyard/bus/camel/audit/FaultProcessor.process() line 46
      org/apache/camel/processor/RouteContextProcessor.processNext() line 45
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/processor/interceptor/DefaultChannel.process() line 303
      org/apache/camel/processor/RouteContextProcessor.processNext() line 45
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/processor/UnitOfWorkProcessor.process() line 122
      org/apache/camel/processor/RouteInflightRepositoryProcessor.processNext() line 48
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/management/InstrumentationProcessor.process() line 73
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/component/direct/DirectProducer.process() line 61
      org/apache/camel/processor/UnitOfWorkProcessor.processAsync() line 150
      org/apache/camel/processor/UnitOfWorkProcessor.process() line 117
      org/apache/camel/util/AsyncProcessorHelper.process() line 99
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 86
      org/apache/camel/processor/UnitOfWorkProducer.process() line 63
      org/apache/camel/impl/ProducerCache$2.doInProducer() line 360
      org/apache/camel/impl/ProducerCache$2.doInProducer() line 331
      org/apache/camel/impl/ProducerCache.doInProducer() line 227
      org/apache/camel/impl/ProducerCache.sendExchange() line 331
      org/apache/camel/impl/ProducerCache.send() line 153
      org/apache/camel/impl/DefaultProducerTemplate.send() line 106
      org/apache/camel/impl/DefaultProducerTemplate.send() line 92
      org/switchyard/bus/camel/ExchangeDispatcher.dispatch() line 87
      org/switchyard/bus/camel/CamelExchange.sendInternal() line 234
      org/switchyard/bus/camel/CamelExchange.send() line 171
      org/switchyard/component/sca/SCAInvoker.invokeLocal() line 123
      org/switchyard/component/sca/SCAInvoker.handleMessage() line 101
      org/switchyard/bus/camel/processors/ProviderProcessor.process() line 29
      org/apache/camel/util/AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process() line 61
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/management/InstrumentationProcessor.process() line 73
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/processor/interceptor/TraceInterceptor.process() line 91
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/switchyard/bus/camel/audit/FaultProcessor.process() line 46
      org/apache/camel/processor/RouteContextProcessor.processNext() line 45
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/processor/interceptor/DefaultChannel.process() line 303
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/Pipeline.process() line 117
      org/apache/camel/processor/Pipeline.process() line 80
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/TryProcessor.process() line 104
      org/apache/camel/processor/TryProcessor.process() line 78
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/management/InstrumentationProcessor.process() line 73
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/processor/interceptor/TraceInterceptor.process() line 91
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/switchyard/bus/camel/audit/FaultProcessor.process() line 46
      org/apache/camel/processor/RouteContextProcessor.processNext() line 45
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/processor/interceptor/DefaultChannel.process() line 303
      org/apache/camel/processor/RouteContextProcessor.processNext() line 45
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/processor/UnitOfWorkProcessor.process() line 122
      org/apache/camel/processor/RouteInflightRepositoryProcessor.processNext() line 48
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/processor/DelegateAsyncProcessor.processNext() line 99
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 90
      org/apache/camel/management/InstrumentationProcessor.process() line 73
      org/apache/camel/util/AsyncProcessorHelper.process() line 73
      org/apache/camel/component/direct/DirectProducer.process() line 61
      org/apache/camel/processor/UnitOfWorkProcessor.processAsync() line 150
      org/apache/camel/processor/UnitOfWorkProcessor.process() line 117
      org/apache/camel/util/AsyncProcessorHelper.process() line 99
      org/apache/camel/processor/DelegateAsyncProcessor.process() line 86
      org/apache/camel/processor/UnitOfWorkProducer.process() line 63
      org/apache/camel/impl/ProducerCache$2.doInProducer() line 360
      org/apache/camel/impl/ProducerCache$2.doInProducer() line 331
      org/apache/camel/impl/ProducerCache.doInProducer() line 227
      org/apache/camel/impl/ProducerCache.sendExchange() line 331
      org/apache/camel/impl/ProducerCache.send() line 153
      org/apache/camel/impl/DefaultProducerTemplate.send() line 106
      org/apache/camel/impl/DefaultProducerTemplate.send() line 92
      org/switchyard/bus/camel/ExchangeDispatcher.dispatch() line 87
      org/switchyard/bus/camel/CamelExchange.sendInternal() line 234
      org/switchyard/bus/camel/CamelExchange.send() line 171
      org/switchyard/component/bean/ClientProxyBean$ClientProxyInvocationHandler.invoke() line 292
      com.sun.proxy
      com/sun/proxy/$Proxy69.tariff()
      Dispatching Overhead, JVM Reflection
      sun/reflect/GeneratedMethodAccessor33.invoke()
      sun/reflect/DelegatingMethodAccessorImpl.invoke() line 43
      java/lang/reflect/Method.invoke() line 606
      org/jboss/weld/bean/proxy/AbstractBeanInstance.invoke() line 45
      org/jboss/weld/bean/proxy/ProxyMethodHandler.invoke() line 105
      org/jboss/weld/proxies/TariffService$1366014918$Proxy$_$$_WeldClientProxy.tariff()
      com/myapp/switchyard/-----/---/observer_tariff/ObserverTariffService$TarifficationJob.run() line 98
      
        • 1. Re: SCAInvoker contention on classloading in SwitchYard 1.1 and 1.0
          synclpz

          It's a kind of magic, but the cause was that service contract was

           

          int tariff(Request req);

           

          when we changed it to

           

          String tariff(Request req);

           

          the problem is gone and we have 8000 invocations per second (instead of 2000) on the same HW.

           

          May be the topic should be moved to devs forum for issue to be analysed, because the root cause is still unknown. Recommendation is NOT to use primitives in service contracts.

          • 2. Re: Re: SCAInvoker contention on classloading in SwitchYard 1.1 and 1.0
            synclpz

            Further investigation of this issue shows that there's a bug in SY validation mechanisms that slow down performance very much due to lock contention in java.lang.ClassLoader.loadClass(), because the method is synchronized.

             

            The issue is:

             

            1. While calling a service SY performs validation, this is mandatory and hardcoded in SY (why?)

            2. Upon validation, BaseValidatorRegistry is called to determine acceptable validator for the message, method getValidator(QName) is called

            3. getValidator scans it's config: Validator<?> validator = _validators.get(name) and gets null, obviously, because I don't use validators (and did not plan to use them, especially for SCA Java2Java invocations)

            4. After some other checks it decides to getJavaFallbackValidator(QName), which in turn calls QNameUtil.toJavaMessageType(QName) to get the Class<?> of the message.

            5. toJavaMessageType(QName) splits QName to get fully qualified class name and actually invokes Classes.forName(className)

            6. Classes.forName(className) makes the whole magic :-) It gets all available class loaders and tries to Class.forName() the requested className with every classloader until the first match. And, eventually, runs into system classloader, which has synchronized method! Furthermore, the logic relies on Exceptions - if class is not found with selected classloader (ClassNotFoundException), continue to the next one. This also add some performance penalty.

             

             

            I can't actually decide by myself what should be fixed in this scenario, but my suggestions are:

             

             

            1. Add ability to disable validation at all

            2. Do not use Classes.forName() while searching for java fallback validator, just use TCCL.loadClass()

            3. Make the Classes.forName not to use system classloader

            4. Fix JDK

            5. <Your option here>

             

            Mentioning here github committers

             

            igarashit

            tcunningham

            kcbabo

             

            Cheers.

            • 3. Re: SCAInvoker contention on classloading in SwitchYard 1.1 and 1.0
              kcbabo

              I think the answer here is to allow fallback validator lookup to be disabled via domain property.  The fallback logic has to load the class to check the type hierarchy.  It's unclear to me how common the fallback scenario actually is, but it's been the default behavior so far and changing that could potentially break things (hence the domain property override).  I will file a JIRA to track that work.  If you really need a fix in the meantime, a quick hack that might work would be using an ExchangeInterceptor to add a property that marks the message content as validated before it hits the ValidateHandler.  That would short-circuit the following block which cascades into the fallback lookup logic:

               

              core/runtime/src/main/java/org/switchyard/handlers/ValidateHandler.java at master · jboss-switchyard/core · GitHub

               

              Thanks for reporting this issue and for your detailed analysis.

              • 5. Re: SCAInvoker contention on classloading in SwitchYard 1.1 and 1.0
                synclpz

                We resolved this issue via implementing concrete but empty validators for all types used in SCA exchange on the both sides (service and reference sides).