10 Replies Latest reply on Mar 16, 2011 10:48 AM by shawkins

    How to improve performance of teiid

    tawalisa

      hi All,

       

                 I have tested teiid server.

      When I used 30 threads and every thread execute 100 SQL, teiid is ok.

      When I used 50 threads and every thread execute 100 SQl, teiid client throw exception which is below.

       

      Mar 8, 2011 3:39:39 PM org.teiid.jdbc.SocketProfile connect

      SEVERE: Could not create connection

      [SingleInstanceCommunicationException]Error establishing socket to host and port: mwm-18.gre.hp.com:31100. Reason: Read timed out

      1 [SocketTimeoutException]Read timed out

          at org.teiid.net.socket.SocketServerConnection.selectServerInstance(SocketServerConnection.java:162)

          at org.teiid.net.socket.SocketServerConnection.<init>(SocketServerConnection.java:94)

          at org.teiid.net.socket.SocketServerConnectionFactory.getConnection(SocketServerConnectionFactory.java:310)

          at org.teiid.jdbc.SocketProfile.createConnection(SocketProfile.java:104)

          at org.teiid.jdbc.SocketProfile.connect(SocketProfile.java:88)

          at org.teiid.jdbc.TeiidDriver.connect(TeiidDriver.java:87)

          at java.sql.DriverManager.getConnection(Unknown Source)

          at java.sql.DriverManager.getConnection(Unknown Source)

          at test.TeiidClient.getConnection(TeiidClient.java:46)

          at test.TestPerformance.dotest(TestPerformance.java:39)

          at test.TestPerformance.run(TestPerformance.java:26)

      Caused by: java.net.SocketTimeoutException: Read timed out

          at java.net.SocketInputStream.socketRead0(Native Method)

          at java.net.SocketInputStream.read(Unknown Source)

          at java.io.BufferedInputStream.fill(Unknown Source)

          at java.io.BufferedInputStream.read1(Unknown Source)

          at java.io.BufferedInputStream.read(Unknown Source)

          at java.io.DataInputStream.read(Unknown Source)

          at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.fillBuffer(ObjectDecoderInputStream.java:164)

          at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.findLength(ObjectDecoderInputStream.java:147)

          at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.readObjectOverride(ObjectDecoderInputStream.java:81)

          at java.io.ObjectInputStream.readObject(Unknown Source)

          at org.teiid.net.socket.OioOjbectChannelFactory$OioObjectChannel.read(OioOjbectChannelFactory.java:114)

          at org.teiid.net.socket.SocketServerInstanceImpl.doHandshake(SocketServerInstanceImpl.java:112)

          at org.teiid.net.socket.SocketServerInstanceImpl.connect(SocketServerInstanceImpl.java:93)

          at org.teiid.net.socket.SocketServerConnectionFactory.getServerInstance(SocketServerConnectionFactory.java:274)

          at org.teiid.net.socket.SocketServerConnection.connect(SocketServerConnection.java:172)

          at org.teiid.net.socket.SocketServerConnection.selectServerInstance(SocketServerConnection.java:124)

          ... 10 more

       

       

      When client throw this exception , Teiid Server look fine. It don't output any log. After a few time, client will continue running.  I want to know what is reason. And how to improve performance of teiid.

       

      Thanks.

      Best Regards

        • 1. How to improve performance of teiid
          rareddy

          Allen,

           

          What is the 100 SQL mean? You mean it executes a SQL that returns 100 rows in a resultset? I understand the threads part. What I suspect in your case the thread usage is maxed out, and you connection request is put on the queue but not serviced in the allowed timeout value for the connection, thus the exception above.

           

          Read the following for the tuning tips.

           

          http://docs.jboss.org/teiid/7.3.0.Final/admin-guide/en-US/html_single/#performance

           

          Especially you need to increase the maxThreads, maxActivePlans along with increased JVM memory settings based on capabilities of the Server configuration.  Note that increasing threads with a low end server to a very high number will not serve any good other than exhausting the resources. IMO, 32 threads per core may be an sensible number depending upon the work load.

           

          Also if you are working with multiple sources in the back end (in -ds.xml files) you need to create  correct number of connections required for your concurrency in each source. By default the pool only contains max of 20 connections. So, if you involved the source in 20 simultaneous queries the 21 st will wait. However, note in non-xa situations these connections are released as soon as the work is done. In XA situations these connections are held until transaction is either committed or rollback.

           

          If you have enough RAM on this server, you can allocate more heap for the Buffer Services, increase "maxReserveBatchColumns" property per suggestions in the file.

           

          All the above properties can be modified in the <jboss-as>/server/<default>/deploy/teiid/teiid-jboss-beans.xml file.

           

          Hope this helps.

           

           

          Ramesh..

          • 2. How to improve performance of teiid
            tawalisa

            Mega thanks for Ramesh help.

             

            I found another question,  when I am running my test case,  Teiid server will have not any response sometimes. I have waited 12 hours. It still has not any response.

            I just restart teiid server to fix it.

            When I running 15 threads, server will return result.

            When I running 20 threads, Teiid server has not any response sometimes.

            When I running 30 threads, Teiid server has not any response usually.

            The server like corpse. If I use teiid client tool to connect Teiid server, it has not any response.

            I have updated many configuration file with your help. This issue still is happend.

            I updated Datasoucre max pool is 100.

            <jboss-as>/server/<default>/deploy/teiid/teiid-jboss-beans.xml

            maxThreads 128

            maxActivePlans 60

            maxReserveBatchColumns 16384

             

            If you have any suggestion, welcome tell me.

             

            Thanks

            Allen

            • 3. How to improve performance of teiid
              rareddy

              Allen,

               

              Without logs I can not advise. If you have logs please attach them, not post in the message. Also when the server is hung like that you can take thread dump (kiil -3 on *unix, CTRL-BREAK on Windows) and attach them too.

               

              Ramesh..

              • 4. Re: How to improve performance of teiid
                tawalisa

                Hi Ramesh,

                 

                I have got dump information as below. Do you have any advice? Thanks

                 

                2011-03-14 15:38:55

                Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.1-b03 mixed mode):

                 

                "WorkManager TimeoutMonitor" daemon prio=10 tid=0x000000005b77c000 nid=0x7c7d waiting on condition [0x0000000049a1c000]

                   java.lang.Thread.State: TIMED_WAITING (sleeping)

                    at java.lang.Thread.sleep(Native Method)

                    at org.jboss.util.threadpool.BasicThreadPool$TimeoutMonitor.run(BasicThreadPool.java:686)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Worker26_QueryProcessorQueue109" daemon prio=10 tid=0x00002aabbe4f6000 nid=0x7c7c runnable [0x000000004991a000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.SocketInputStream.socketRead0(Native Method)

                    at java.net.SocketInputStream.read(SocketInputStream.java:129)

                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

                    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)

                    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)

                    - locked <0x00002aab09efb880> (a java.io.BufferedInputStream)

                    at java.io.DataInputStream.read(DataInputStream.java:132)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.fillBuffer(ObjectDecoderInputStream.java:164)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.findLength(ObjectDecoderInputStream.java:147)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.readObjectOverride(ObjectDecoderInputStream.java:81)

                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:344)

                    at org.teiid.net.socket.OioOjbectChannelFactory$OioObjectChannel.read(OioOjbectChannelFactory.java:114)

                    - locked <0x00002aab09efb8b0> (a java.lang.Object)

                    at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:253)

                    at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                    at java.lang.reflect.Method.invoke(Method.java:597)

                    at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:107)

                    at $Proxy275.read(Unknown Source)

                    at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:347)

                    at org.teiid.jdbc.StatementImpl.sendRequestMessageAndWait(StatementImpl.java:915)

                    at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:479)

                    at org.teiid.jdbc.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:214)

                    at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:342)

                    at org.teiid.translator.jdbc.JDBCQueryExecution.execute(JDBCQueryExecution.java:84)

                    at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:263)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:281)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:211)

                    at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:154)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.SelectNode.nextBatchDirect(SelectNode.java:96)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

                    at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

                    at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

                    at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

                    at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:214)

                    at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:168)

                    at org.teiid.query.processor.relational.SourceState.sort(SourceState.java:193)

                    at org.teiid.query.processor.relational.MergeJoinStrategy.loadLeft(MergeJoinStrategy.java:328)

                    at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:182)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:162)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:161)

                    at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:150)

                    at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:105)

                    at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:115)

                    at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:250)

                    at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:184)

                    at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)

                    at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:188)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:116)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:290)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Worker24_QueryProcessorQueue108" daemon prio=10 tid=0x0000000068ea2800 nid=0x7c7a runnable [0x0000000049718000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.SocketInputStream.socketRead0(Native Method)

                    at java.net.SocketInputStream.read(SocketInputStream.java:129)

                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

                    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)

                    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)

                    - locked <0x00002aab09daa328> (a java.io.BufferedInputStream)

                    at java.io.DataInputStream.read(DataInputStream.java:132)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.fillBuffer(ObjectDecoderInputStream.java:164)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.findLength(ObjectDecoderInputStream.java:147)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.readObjectOverride(ObjectDecoderInputStream.java:81)

                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:344)

                    at org.teiid.net.socket.OioOjbectChannelFactory$OioObjectChannel.read(OioOjbectChannelFactory.java:114)

                    - locked <0x00002aab09daa498> (a java.lang.Object)

                    at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:253)

                    at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                    at java.lang.reflect.Method.invoke(Method.java:597)

                    at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:107)

                    at $Proxy275.read(Unknown Source)

                    at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:347)

                    at org.teiid.jdbc.StatementImpl.sendRequestMessageAndWait(StatementImpl.java:915)

                    at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:479)

                    at org.teiid.jdbc.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:214)

                    at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:342)

                    at org.teiid.translator.jdbc.JDBCQueryExecution.execute(JDBCQueryExecution.java:84)

                    at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:263)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:281)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:211)

                    at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:154)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.SelectNode.nextBatchDirect(SelectNode.java:96)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

                    at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

                    at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

                    at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

                    at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:214)

                    at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:168)

                    at org.teiid.query.processor.relational.SourceState.sort(SourceState.java:193)

                    at org.teiid.query.processor.relational.MergeJoinStrategy.loadLeft(MergeJoinStrategy.java:328)

                    at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:182)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:162)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:161)

                    at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:150)

                    at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:105)

                    at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:115)

                    at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:250)

                    at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:184)

                    at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)

                    at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:188)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:116)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:290)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Worker22_QueryProcessorQueue69" daemon prio=10 tid=0x000000006a35d000 nid=0x7c78 runnable [0x0000000049516000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.SocketInputStream.socketRead0(Native Method)

                    at java.net.SocketInputStream.read(SocketInputStream.java:129)

                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

                    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)

                    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)

                    - locked <0x00002aab160f9968> (a java.io.BufferedInputStream)

                    at java.io.DataInputStream.read(DataInputStream.java:132)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.fillBuffer(ObjectDecoderInputStream.java:164)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.findLength(ObjectDecoderInputStream.java:147)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.readObjectOverride(ObjectDecoderInputStream.java:81)

                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:344)

                    at org.teiid.net.socket.OioOjbectChannelFactory$OioObjectChannel.read(OioOjbectChannelFactory.java:114)

                    - locked <0x00002aab160f9ad8> (a java.lang.Object)

                    at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:253)

                    at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                    at java.lang.reflect.Method.invoke(Method.java:597)

                    at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:107)

                    at $Proxy275.read(Unknown Source)

                    at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:347)

                    at org.teiid.jdbc.StatementImpl.sendRequestMessageAndWait(StatementImpl.java:915)

                    at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:479)

                    at org.teiid.jdbc.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:214)

                    at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:342)

                    at org.teiid.translator.jdbc.JDBCQueryExecution.execute(JDBCQueryExecution.java:84)

                    at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:263)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:281)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:211)

                    at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:154)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.SelectNode.nextBatchDirect(SelectNode.java:96)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

                    at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

                    at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

                    at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

                    at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:214)

                    at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:168)

                    at org.teiid.query.processor.relational.SourceState.sort(SourceState.java:193)

                    at org.teiid.query.processor.relational.MergeJoinStrategy.loadLeft(MergeJoinStrategy.java:328)

                    at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:182)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:162)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:161)

                    at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:150)

                    at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:105)

                    at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:115)

                    at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:250)

                    at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:184)

                    at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)

                    at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:188)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:116)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:290)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Worker21_QueryProcessorQueue138" daemon prio=10 tid=0x000000005ec06800 nid=0x7c77 runnable [0x0000000049415000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.SocketInputStream.socketRead0(Native Method)

                    at java.net.SocketInputStream.read(SocketInputStream.java:129)

                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

                    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)

                    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)

                    - locked <0x00002aab09e63e28> (a java.io.BufferedInputStream)

                    at java.io.DataInputStream.read(DataInputStream.java:132)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.fillBuffer(ObjectDecoderInputStream.java:164)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.findLength(ObjectDecoderInputStream.java:147)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.readObjectOverride(ObjectDecoderInputStream.java:81)

                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:344)

                    at org.teiid.net.socket.OioOjbectChannelFactory$OioObjectChannel.read(OioOjbectChannelFactory.java:114)

                    - locked <0x00002aab09e63f98> (a java.lang.Object)

                    at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:253)

                    at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                    at java.lang.reflect.Method.invoke(Method.java:597)

                    at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:107)

                    at $Proxy275.read(Unknown Source)

                    at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:347)

                    at org.teiid.jdbc.StatementImpl.sendRequestMessageAndWait(StatementImpl.java:915)

                    at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:479)

                    at org.teiid.jdbc.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:214)

                    at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:342)

                    at org.teiid.translator.jdbc.JDBCQueryExecution.execute(JDBCQueryExecution.java:84)

                    at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:263)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:281)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:211)

                    at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:154)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.SelectNode.nextBatchDirect(SelectNode.java:96)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

                    at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

                    at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

                    at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

                    at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:214)

                    at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:168)

                    at org.teiid.query.processor.relational.SourceState.sort(SourceState.java:193)

                    at org.teiid.query.processor.relational.MergeJoinStrategy.loadLeft(MergeJoinStrategy.java:328)

                    at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:182)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:162)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:161)

                    at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:150)

                    at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:105)

                    at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:115)

                    at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:250)

                    at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:184)

                    at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)

                    at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:188)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:116)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:290)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Worker20_QueryProcessorQueue209" daemon prio=10 tid=0x0000000069051000 nid=0x7c76 runnable [0x0000000049314000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.SocketInputStream.socketRead0(Native Method)

                    at java.net.SocketInputStream.read(SocketInputStream.java:129)

                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

                    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)

                    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)

                    - locked <0x00002aab725488b8> (a java.io.BufferedInputStream)

                    at java.io.DataInputStream.read(DataInputStream.java:132)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.fillBuffer(ObjectDecoderInputStream.java:164)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.findLength(ObjectDecoderInputStream.java:147)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.readObjectOverride(ObjectDecoderInputStream.java:81)

                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:344)

                    at org.teiid.net.socket.OioOjbectChannelFactory$OioObjectChannel.read(OioOjbectChannelFactory.java:114)

                    - locked <0x00002aab72548a48> (a java.lang.Object)

                    at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:253)

                    at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                    at java.lang.reflect.Method.invoke(Method.java:597)

                    at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:107)

                    at $Proxy275.read(Unknown Source)

                    at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:347)

                    at org.teiid.jdbc.StatementImpl.sendRequestMessageAndWait(StatementImpl.java:915)

                    at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:479)

                    at org.teiid.jdbc.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:214)

                    at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:342)

                    at org.teiid.translator.jdbc.JDBCQueryExecution.execute(JDBCQueryExecution.java:84)

                    at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:263)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:281)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:211)

                    at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:154)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.SelectNode.nextBatchDirect(SelectNode.java:96)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

                    at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

                    at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

                    at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

                    at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:214)

                    at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:168)

                    at org.teiid.query.processor.relational.SourceState.sort(SourceState.java:193)

                    at org.teiid.query.processor.relational.MergeJoinStrategy.loadLeft(MergeJoinStrategy.java:328)

                    at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:182)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:162)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:161)

                    at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:150)

                    at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:105)

                    at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:115)

                    at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:250)

                    at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:184)

                    at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)

                    at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:188)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:116)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:290)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Worker17_QueryProcessorQueue197" daemon prio=10 tid=0x000000006aa5f800 nid=0x7c73 runnable [0x0000000049010000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.SocketInputStream.socketRead0(Native Method)

                    at java.net.SocketInputStream.read(SocketInputStream.java:129)

                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

                    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)

                    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)

                    - locked <0x00002aab09e85798> (a java.io.BufferedInputStream)

                    at java.io.DataInputStream.read(DataInputStream.java:132)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.fillBuffer(ObjectDecoderInputStream.java:164)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.findLength(ObjectDecoderInputStream.java:147)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.readObjectOverride(ObjectDecoderInputStream.java:81)

                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:344)

                    at org.teiid.net.socket.OioOjbectChannelFactory$OioObjectChannel.read(OioOjbectChannelFactory.java:114)

                    - locked <0x00002aab09e85908> (a java.lang.Object)

                    at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:253)

                    at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                    at java.lang.reflect.Method.invoke(Method.java:597)

                    at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:107)

                    at $Proxy275.read(Unknown Source)

                    at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:347)

                    at org.teiid.jdbc.StatementImpl.sendRequestMessageAndWait(StatementImpl.java:915)

                    at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:479)

                    at org.teiid.jdbc.StatementImpl.execute(StatementImpl.java:308)

                    at org.jboss.resource.adapter.jdbc.CheckValidConnectionSQL.isValidConnection(CheckValidConnectionSQL.java:58)

                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnectionFactory.isValidConnection(BaseWrapperManagedConnectionFactory.java:559)

                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.checkValid(BaseWrapperManagedConnection.java:321)

                    at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.matchManagedConnections(XAManagedConnectionFactory.java:492)

                    at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:219)

                    at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:716)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:404)

                    at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:381)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)

                    at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89)

                    at org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(JDBCExecutionFactory.java:245)

                    at org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(JDBCExecutionFactory.java:80)

                    at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:198)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:281)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:211)

                    at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:154)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.SelectNode.nextBatchDirect(SelectNode.java:96)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

                    at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

                    at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

                    at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

                    at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:214)

                    at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:168)

                    at org.teiid.query.processor.relational.SourceState.sort(SourceState.java:193)

                    at org.teiid.query.processor.relational.MergeJoinStrategy.loadLeft(MergeJoinStrategy.java:328)

                    at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:182)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:162)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:161)

                    at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:150)

                    at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:105)

                    at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:115)

                    at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:250)

                    at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:184)

                    at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)

                    at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:188)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:116)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:290)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Worker16_QueryProcessorQueue170" daemon prio=10 tid=0x00002aaba4f84000 nid=0x7c72 runnable [0x0000000048f0f000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.SocketInputStream.socketRead0(Native Method)

                    at java.net.SocketInputStream.read(SocketInputStream.java:129)

                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

                    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)

                    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)

                    - locked <0x00002aab16549f70> (a java.io.BufferedInputStream)

                    at java.io.DataInputStream.read(DataInputStream.java:132)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.fillBuffer(ObjectDecoderInputStream.java:164)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.findLength(ObjectDecoderInputStream.java:147)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.readObjectOverride(ObjectDecoderInputStream.java:81)

                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:344)

                    at org.teiid.net.socket.OioOjbectChannelFactory$OioObjectChannel.read(OioOjbectChannelFactory.java:114)

                    - locked <0x00002aab1654a100> (a java.lang.Object)

                    at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:253)

                    at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                    at java.lang.reflect.Method.invoke(Method.java:597)

                    at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:107)

                    at $Proxy275.read(Unknown Source)

                    at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:347)

                    at org.teiid.jdbc.StatementImpl.sendRequestMessageAndWait(StatementImpl.java:915)

                    at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:479)

                    at org.teiid.jdbc.StatementImpl.execute(StatementImpl.java:308)

                    at org.jboss.resource.adapter.jdbc.CheckValidConnectionSQL.isValidConnection(CheckValidConnectionSQL.java:58)

                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnectionFactory.isValidConnection(BaseWrapperManagedConnectionFactory.java:559)

                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.checkValid(BaseWrapperManagedConnection.java:321)

                    at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.matchManagedConnections(XAManagedConnectionFactory.java:492)

                    at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:219)

                    at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:716)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:404)

                    at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:381)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)

                    at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89)

                    at org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(JDBCExecutionFactory.java:245)

                    at org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(JDBCExecutionFactory.java:80)

                    at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:198)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:281)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:211)

                    at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:154)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.SelectNode.nextBatchDirect(SelectNode.java:96)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

                    at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

                    at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

                    at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

                    at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:214)

                    at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:168)

                    at org.teiid.query.processor.relational.SourceState.sort(SourceState.java:193)

                    at org.teiid.query.processor.relational.MergeJoinStrategy.loadLeft(MergeJoinStrategy.java:328)

                    at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:182)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:162)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:161)

                    at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:150)

                    at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:105)

                    at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:115)

                    at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:250)

                    at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:184)

                    at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)

                    at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:188)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:116)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:290)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Worker15_QueryProcessorQueue142" daemon prio=10 tid=0x00002aabb5b0d000 nid=0x7c71 runnable [0x0000000048e0e000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.SocketInputStream.socketRead0(Native Method)

                    at java.net.SocketInputStream.read(SocketInputStream.java:129)

                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

                    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)

                    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)

                    - locked <0x00002aab09df2418> (a java.io.BufferedInputStream)

                    at java.io.DataInputStream.read(DataInputStream.java:132)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.fillBuffer(ObjectDecoderInputStream.java:164)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.findLength(ObjectDecoderInputStream.java:147)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.readObjectOverride(ObjectDecoderInputStream.java:81)

                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:344)

                    at org.teiid.net.socket.OioOjbectChannelFactory$OioObjectChannel.read(OioOjbectChannelFactory.java:114)

                    - locked <0x00002aab09df2588> (a java.lang.Object)

                    at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:253)

                    at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                    at java.lang.reflect.Method.invoke(Method.java:597)

                    at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:107)

                    at $Proxy275.read(Unknown Source)

                    at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:347)

                    at org.teiid.jdbc.StatementImpl.sendRequestMessageAndWait(StatementImpl.java:915)

                    at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:479)

                    at org.teiid.jdbc.StatementImpl.execute(StatementImpl.java:308)

                    at org.jboss.resource.adapter.jdbc.CheckValidConnectionSQL.isValidConnection(CheckValidConnectionSQL.java:58)

                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnectionFactory.isValidConnection(BaseWrapperManagedConnectionFactory.java:559)

                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.checkValid(BaseWrapperManagedConnection.java:321)

                    at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.matchManagedConnections(XAManagedConnectionFactory.java:492)

                    at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:219)

                    at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:716)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:404)

                    at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:381)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)

                    at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89)

                    at org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(JDBCExecutionFactory.java:245)

                    at org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(JDBCExecutionFactory.java:80)

                    at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:198)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:281)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:211)

                    at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:154)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.SelectNode.nextBatchDirect(SelectNode.java:96)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

                    at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

                    at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

                    at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

                    at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:214)

                    at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:168)

                    at org.teiid.query.processor.relational.SourceState.sort(SourceState.java:193)

                    at org.teiid.query.processor.relational.MergeJoinStrategy.loadLeft(MergeJoinStrategy.java:328)

                    at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:182)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:162)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:161)

                    at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:150)

                    at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:105)

                    at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:115)

                    at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:250)

                    at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:184)

                    at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)

                    at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:188)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:116)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:290)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Worker14_QueryProcessorQueue155" daemon prio=10 tid=0x00002aaba61d1000 nid=0x7c70 runnable [0x0000000048d0d000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.SocketInputStream.socketRead0(Native Method)

                    at java.net.SocketInputStream.read(SocketInputStream.java:129)

                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

                    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)

                    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)

                    - locked <0x00002aab09e13f40> (a java.io.BufferedInputStream)

                    at java.io.DataInputStream.read(DataInputStream.java:132)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.fillBuffer(ObjectDecoderInputStream.java:164)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.findLength(ObjectDecoderInputStream.java:147)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.readObjectOverride(ObjectDecoderInputStream.java:81)

                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:344)

                    at org.teiid.net.socket.OioOjbectChannelFactory$OioObjectChannel.read(OioOjbectChannelFactory.java:114)

                    - locked <0x00002aab09e140b0> (a java.lang.Object)

                    at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:253)

                    at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                    at java.lang.reflect.Method.invoke(Method.java:597)

                    at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:107)

                    at $Proxy275.read(Unknown Source)

                    at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:347)

                    at org.teiid.jdbc.StatementImpl.sendRequestMessageAndWait(StatementImpl.java:915)

                    at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:479)

                    at org.teiid.jdbc.StatementImpl.execute(StatementImpl.java:308)

                    at org.jboss.resource.adapter.jdbc.CheckValidConnectionSQL.isValidConnection(CheckValidConnectionSQL.java:58)

                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnectionFactory.isValidConnection(BaseWrapperManagedConnectionFactory.java:559)

                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.checkValid(BaseWrapperManagedConnection.java:321)

                    at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.matchManagedConnections(XAManagedConnectionFactory.java:492)

                    at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:219)

                    at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:716)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:404)

                    at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:381)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)

                    at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89)

                    at org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(JDBCExecutionFactory.java:245)

                    at org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(JDBCExecutionFactory.java:80)

                    at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:198)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:281)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:211)

                    at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:154)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.SelectNode.nextBatchDirect(SelectNode.java:96)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

                    at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

                    at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

                    at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

                    at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:214)

                    at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:168)

                    at org.teiid.query.processor.relational.SourceState.sort(SourceState.java:193)

                    at org.teiid.query.processor.relational.MergeJoinStrategy.loadLeft(MergeJoinStrategy.java:328)

                    at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:182)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:162)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:161)

                    at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:150)

                    at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:105)

                    at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:115)

                    at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:250)

                    at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:184)

                    at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)

                    at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:188)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:116)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:290)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Worker13_QueryProcessorQueue133" daemon prio=10 tid=0x000000006a361800 nid=0x7c6f runnable [0x0000000048c0d000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.SocketInputStream.socketRead0(Native Method)

                    at java.net.SocketInputStream.read(SocketInputStream.java:129)

                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

                    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)

                    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)

                    - locked <0x00002aab09f04458> (a java.io.BufferedInputStream)

                    at java.io.DataInputStream.read(DataInputStream.java:132)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.fillBuffer(ObjectDecoderInputStream.java:164)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.findLength(ObjectDecoderInputStream.java:147)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.readObjectOverride(ObjectDecoderInputStream.java:81)

                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:344)

                    at org.teiid.net.socket.OioOjbectChannelFactory$OioObjectChannel.read(OioOjbectChannelFactory.java:114)

                    - locked <0x00002aab09f04488> (a java.lang.Object)

                    at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:253)

                    at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                    at java.lang.reflect.Method.invoke(Method.java:597)

                    at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:107)

                    at $Proxy275.read(Unknown Source)

                    at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:347)

                    at org.teiid.jdbc.StatementImpl.sendRequestMessageAndWait(StatementImpl.java:915)

                    at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:479)

                    at org.teiid.jdbc.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:214)

                    at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:342)

                    at org.teiid.translator.jdbc.JDBCQueryExecution.execute(JDBCQueryExecution.java:84)

                    at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:263)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:281)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:211)

                    at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:154)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.SelectNode.nextBatchDirect(SelectNode.java:96)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

                    at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

                    at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

                    at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

                    at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:214)

                    at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:168)

                    at org.teiid.query.processor.relational.SourceState.sort(SourceState.java:193)

                    at org.teiid.query.processor.relational.MergeJoinStrategy.loadLeft(MergeJoinStrategy.java:328)

                    at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:182)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:162)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:161)

                    at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:150)

                    at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:105)

                    at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:115)

                    at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:250)

                    at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:184)

                    at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)

                    at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:188)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:116)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:290)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Worker12_QueryProcessorQueue217" daemon prio=10 tid=0x00002aabbf0c2800 nid=0x7c6e runnable [0x0000000048b0b000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.SocketInputStream.socketRead0(Native Method)

                    at java.net.SocketInputStream.read(SocketInputStream.java:129)

                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

                    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)

                    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)

                    - locked <0x00002aab32ca6df8> (a java.io.BufferedInputStream)

                    at java.io.DataInputStream.read(DataInputStream.java:132)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.fillBuffer(ObjectDecoderInputStream.java:164)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.findLength(ObjectDecoderInputStream.java:147)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.readObjectOverride(ObjectDecoderInputStream.java:81)

                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:344)

                    at org.teiid.net.socket.OioOjbectChannelFactory$OioObjectChannel.read(OioOjbectChannelFactory.java:114)

                    - locked <0x00002aab32ca6f88> (a java.lang.Object)

                    at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:253)

                    at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                    at java.lang.reflect.Method.invoke(Method.java:597)

                    at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:107)

                    at $Proxy275.read(Unknown Source)

                    at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:347)

                    at org.teiid.jdbc.StatementImpl.sendRequestMessageAndWait(StatementImpl.java:915)

                    at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:479)

                    at org.teiid.jdbc.StatementImpl.execute(StatementImpl.java:308)

                    at org.jboss.resource.adapter.jdbc.CheckValidConnectionSQL.isValidConnection(CheckValidConnectionSQL.java:58)

                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnectionFactory.isValidConnection(BaseWrapperManagedConnectionFactory.java:559)

                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.checkValid(BaseWrapperManagedConnection.java:321)

                    at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.matchManagedConnections(XAManagedConnectionFactory.java:492)

                    at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:219)

                    at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:716)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:404)

                    at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:381)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)

                    at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89)

                    at org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(JDBCExecutionFactory.java:245)

                    at org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(JDBCExecutionFactory.java:80)

                    at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:198)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:281)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:211)

                    at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:154)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.SelectNode.nextBatchDirect(SelectNode.java:96)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

                    at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

                    at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

                    at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

                    at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:214)

                    at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:168)

                    at org.teiid.query.processor.relational.SourceState.sort(SourceState.java:193)

                    at org.teiid.query.processor.relational.MergeJoinStrategy.loadLeft(MergeJoinStrategy.java:328)

                    at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:182)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:162)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:161)

                    at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:150)

                    at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:105)

                    at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:115)

                    at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:250)

                    at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:184)

                    at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)

                    at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:188)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:116)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:290)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "New I/O server worker #1-4" daemon prio=10 tid=0x000000005caa7000 nid=0x7c6c runnable [0x000000004890b000]

                   java.lang.Thread.State: RUNNABLE

                    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

                    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)

                    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

                    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

                    - locked <0x00002aab05c8f458> (a sun.nio.ch.Util$1)

                    - locked <0x00002aab05c8f440> (a java.util.Collections$UnmodifiableSet)

                    - locked <0x00002aab05c84ae0> (a sun.nio.ch.EPollSelectorImpl)

                    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

                    at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:38)

                    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:164)

                    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

                    at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "New I/O server worker #1-3" daemon prio=10 tid=0x000000005c2ad800 nid=0x7c6b runnable [0x000000004880a000]

                   java.lang.Thread.State: RUNNABLE

                    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

                    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)

                    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

                    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

                    - locked <0x00002aab05caea10> (a sun.nio.ch.Util$1)

                    - locked <0x00002aab05cae9f8> (a java.util.Collections$UnmodifiableSet)

                    - locked <0x00002aab05ca4738> (a sun.nio.ch.EPollSelectorImpl)

                    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

                    at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:38)

                    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:164)

                    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

                    at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "New I/O server worker #1-2" daemon prio=10 tid=0x000000005eb5d800 nid=0x7c6a runnable [0x0000000048709000]

                   java.lang.Thread.State: RUNNABLE

                    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

                    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)

                    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

                    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

                    - locked <0x00002aab05c728f8> (a sun.nio.ch.Util$1)

                    - locked <0x00002aab05c728e0> (a java.util.Collections$UnmodifiableSet)

                    - locked <0x00002aab05c66c18> (a sun.nio.ch.EPollSelectorImpl)

                    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

                    at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:38)

                    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:164)

                    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

                    at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "New I/O server worker #1-1" daemon prio=10 tid=0x000000006f919000 nid=0x7c69 runnable [0x0000000048608000]

                   java.lang.Thread.State: RUNNABLE

                    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

                    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)

                    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

                    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

                    - locked <0x00002aab05c81878> (a sun.nio.ch.Util$1)

                    - locked <0x00002aab05c81860> (a java.util.Collections$UnmodifiableSet)

                    - locked <0x00002aab05c75b60> (a sun.nio.ch.EPollSelectorImpl)

                    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

                    at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:38)

                    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:164)

                    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

                    at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "SocketPing" daemon prio=10 tid=0x00002aabbdc30000 nid=0x7c68 in Object.wait() [0x0000000048507000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aab05cd7440> (a java.util.TaskQueue)

                    at java.util.TimerThread.mainLoop(Timer.java:509)

                    - locked <0x00002aab05cd7440> (a java.util.TaskQueue)

                    at java.util.TimerThread.run(Timer.java:462)

                 

                "Worker10_QueryProcessorQueue164" daemon prio=10 tid=0x00002aabb7a1f800 nid=0x7c67 runnable [0x0000000048404000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.SocketInputStream.socketRead0(Native Method)

                    at java.net.SocketInputStream.read(SocketInputStream.java:129)

                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

                    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)

                    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)

                    - locked <0x00002aab05ced150> (a java.io.BufferedInputStream)

                    at java.io.DataInputStream.read(DataInputStream.java:132)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.fillBuffer(ObjectDecoderInputStream.java:164)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.findLength(ObjectDecoderInputStream.java:147)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.readObjectOverride(ObjectDecoderInputStream.java:81)

                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:344)

                    at org.teiid.net.socket.OioOjbectChannelFactory$OioObjectChannel.read(OioOjbectChannelFactory.java:114)

                    - locked <0x00002aab05ced2c0> (a java.lang.Object)

                    at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:253)

                    at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                    at java.lang.reflect.Method.invoke(Method.java:597)

                    at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:107)

                    at $Proxy275.read(Unknown Source)

                    at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:347)

                    at org.teiid.jdbc.StatementImpl.sendRequestMessageAndWait(StatementImpl.java:915)

                    at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:479)

                    at org.teiid.jdbc.StatementImpl.execute(StatementImpl.java:308)

                    at org.jboss.resource.adapter.jdbc.CheckValidConnectionSQL.isValidConnection(CheckValidConnectionSQL.java:58)

                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnectionFactory.isValidConnection(BaseWrapperManagedConnectionFactory.java:559)

                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.checkValid(BaseWrapperManagedConnection.java:321)

                    at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.matchManagedConnections(XAManagedConnectionFactory.java:492)

                    at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:219)

                    at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:716)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:404)

                    at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:381)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)

                    at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89)

                    at org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(JDBCExecutionFactory.java:245)

                    at org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(JDBCExecutionFactory.java:80)

                    at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:198)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:281)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:211)

                    at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:154)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.SelectNode.nextBatchDirect(SelectNode.java:96)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

                    at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

                    at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

                    at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

                    at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:214)

                    at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:168)

                    at org.teiid.query.processor.relational.SourceState.sort(SourceState.java:193)

                    at org.teiid.query.processor.relational.MergeJoinStrategy.loadLeft(MergeJoinStrategy.java:328)

                    at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:182)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:162)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:161)

                    at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:150)

                    at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:105)

                    at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:115)

                    at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:250)

                    at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:184)

                    at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)

                    at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:188)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:116)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:290)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "IdleRemover" daemon prio=10 tid=0x00002aabbe9b2000 nid=0x7c66 in Object.wait() [0x0000000048305000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aab1605cdf0> (a java.util.concurrent.CopyOnWriteArrayList)

                    at org.jboss.resource.connectionmanager.IdleRemover$IdleRemoverRunnable.run(IdleRemover.java:167)

                    - locked <0x00002aab1605cdf0> (a java.util.concurrent.CopyOnWriteArrayList)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Worker8_QueryProcessorQueue203" daemon prio=10 tid=0x00002aabbd7bd800 nid=0x7c64 runnable [0x0000000048101000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.SocketInputStream.socketRead0(Native Method)

                    at java.net.SocketInputStream.read(SocketInputStream.java:129)

                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

                    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)

                    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)

                    - locked <0x00002aab09dd0ad8> (a java.io.BufferedInputStream)

                    at java.io.DataInputStream.read(DataInputStream.java:132)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.fillBuffer(ObjectDecoderInputStream.java:164)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.findLength(ObjectDecoderInputStream.java:147)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.readObjectOverride(ObjectDecoderInputStream.java:81)

                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:344)

                    at org.teiid.net.socket.OioOjbectChannelFactory$OioObjectChannel.read(OioOjbectChannelFactory.java:114)

                    - locked <0x00002aab09dd0c48> (a java.lang.Object)

                    at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:253)

                    at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                    at java.lang.reflect.Method.invoke(Method.java:597)

                    at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:107)

                    at $Proxy275.read(Unknown Source)

                    at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:347)

                    at org.teiid.jdbc.StatementImpl.sendRequestMessageAndWait(StatementImpl.java:915)

                    at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:479)

                    at org.teiid.jdbc.StatementImpl.execute(StatementImpl.java:308)

                    at org.jboss.resource.adapter.jdbc.CheckValidConnectionSQL.isValidConnection(CheckValidConnectionSQL.java:58)

                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnectionFactory.isValidConnection(BaseWrapperManagedConnectionFactory.java:559)

                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.checkValid(BaseWrapperManagedConnection.java:321)

                    at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.matchManagedConnections(XAManagedConnectionFactory.java:492)

                    at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:219)

                    at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:716)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:404)

                    at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:381)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)

                    at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89)

                    at org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(JDBCExecutionFactory.java:245)

                    at org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(JDBCExecutionFactory.java:80)

                    at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:198)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:281)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:211)

                    at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:154)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.SelectNode.nextBatchDirect(SelectNode.java:96)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

                    at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

                    at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

                    at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

                    at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:214)

                    at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:168)

                    at org.teiid.query.processor.relational.SourceState.sort(SourceState.java:193)

                    at org.teiid.query.processor.relational.MergeJoinStrategy.loadLeft(MergeJoinStrategy.java:328)

                    at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:182)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:162)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:161)

                    at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:150)

                    at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:105)

                    at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:115)

                    at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:250)

                    at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:184)

                    at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)

                    at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:188)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:116)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:290)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Worker6_QueryProcessorQueue147" daemon prio=10 tid=0x00002aaba5ba8000 nid=0x7c62 runnable [0x0000000047eff000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.SocketInputStream.socketRead0(Native Method)

                    at java.net.SocketInputStream.read(SocketInputStream.java:129)

                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

                    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)

                    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)

                    - locked <0x00002aab09f02ff0> (a java.io.BufferedInputStream)

                    at java.io.DataInputStream.read(DataInputStream.java:132)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.fillBuffer(ObjectDecoderInputStream.java:164)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.findLength(ObjectDecoderInputStream.java:147)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.readObjectOverride(ObjectDecoderInputStream.java:81)

                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:344)

                    at org.teiid.net.socket.OioOjbectChannelFactory$OioObjectChannel.read(OioOjbectChannelFactory.java:114)

                    - locked <0x00002aab09f03020> (a java.lang.Object)

                    at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:253)

                    at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                    at java.lang.reflect.Method.invoke(Method.java:597)

                    at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:107)

                    at $Proxy275.read(Unknown Source)

                    at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:347)

                    at org.teiid.jdbc.StatementImpl.sendRequestMessageAndWait(StatementImpl.java:915)

                    at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:479)

                    at org.teiid.jdbc.StatementImpl.execute(StatementImpl.java:308)

                    at org.jboss.resource.adapter.jdbc.CheckValidConnectionSQL.isValidConnection(CheckValidConnectionSQL.java:58)

                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnectionFactory.isValidConnection(BaseWrapperManagedConnectionFactory.java:559)

                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.checkValid(BaseWrapperManagedConnection.java:321)

                    at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.matchManagedConnections(XAManagedConnectionFactory.java:492)

                    at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:219)

                    at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:716)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:404)

                    at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:381)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)

                    at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89)

                    at org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(JDBCExecutionFactory.java:245)

                    at org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(JDBCExecutionFactory.java:80)

                    at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:198)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:281)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:211)

                    at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:154)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.SelectNode.nextBatchDirect(SelectNode.java:96)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

                    at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

                    at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

                    at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

                    at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:214)

                    at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:168)

                    at org.teiid.query.processor.relational.SourceState.sort(SourceState.java:193)

                    at org.teiid.query.processor.relational.MergeJoinStrategy.loadLeft(MergeJoinStrategy.java:328)

                    at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:182)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:162)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:161)

                    at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:150)

                    at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:105)

                    at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:115)

                    at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:250)

                    at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:184)

                    at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)

                    at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:188)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:116)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:290)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Worker5_QueryProcessorQueue211" daemon prio=10 tid=0x00002aabbf292800 nid=0x7c61 runnable [0x0000000047dff000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.SocketInputStream.socketRead0(Native Method)

                    at java.net.SocketInputStream.read(SocketInputStream.java:129)

                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

                    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)

                    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)

                    - locked <0x00002aab7254c5b8> (a java.io.BufferedInputStream)

                    at java.io.DataInputStream.read(DataInputStream.java:132)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.fillBuffer(ObjectDecoderInputStream.java:164)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.findLength(ObjectDecoderInputStream.java:147)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.readObjectOverride(ObjectDecoderInputStream.java:81)

                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:344)

                    at org.teiid.net.socket.OioOjbectChannelFactory$OioObjectChannel.read(OioOjbectChannelFactory.java:114)

                    - locked <0x00002aab7254c748> (a java.lang.Object)

                    at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:253)

                    at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                    at java.lang.reflect.Method.invoke(Method.java:597)

                    at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:107)

                    at $Proxy275.read(Unknown Source)

                    at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:347)

                    at org.teiid.jdbc.StatementImpl.sendRequestMessageAndWait(StatementImpl.java:915)

                    at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:479)

                    at org.teiid.jdbc.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:214)

                    at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:342)

                    at org.teiid.translator.jdbc.JDBCQueryExecution.execute(JDBCQueryExecution.java:84)

                    at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:263)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:281)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:211)

                    at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:154)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.SelectNode.nextBatchDirect(SelectNode.java:96)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

                    at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

                    at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

                    at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

                    at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:214)

                    at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:168)

                    at org.teiid.query.processor.relational.SourceState.sort(SourceState.java:193)

                    at org.teiid.query.processor.relational.MergeJoinStrategy.loadLeft(MergeJoinStrategy.java:328)

                    at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:182)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:162)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:161)

                    at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:150)

                    at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:105)

                    at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:115)

                    at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:250)

                    at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:184)

                    at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)

                    at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:188)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:116)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:290)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Worker4_QueryProcessorQueue196" daemon prio=10 tid=0x00002aabbea19800 nid=0x7c60 runnable [0x0000000047cfd000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.SocketInputStream.socketRead0(Native Method)

                    at java.net.SocketInputStream.read(SocketInputStream.java:129)

                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

                    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)

                    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)

                    - locked <0x00002aab09efc1c0> (a java.io.BufferedInputStream)

                    at java.io.DataInputStream.read(DataInputStream.java:132)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.fillBuffer(ObjectDecoderInputStream.java:164)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.findLength(ObjectDecoderInputStream.java:147)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.readObjectOverride(ObjectDecoderInputStream.java:81)

                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:344)

                    at org.teiid.net.socket.OioOjbectChannelFactory$OioObjectChannel.read(OioOjbectChannelFactory.java:114)

                    - locked <0x00002aab09efc1f0> (a java.lang.Object)

                    at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:253)

                    at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                    at java.lang.reflect.Method.invoke(Method.java:597)

                    at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:107)

                    at $Proxy275.read(Unknown Source)

                    at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:347)

                    at org.teiid.jdbc.StatementImpl.sendRequestMessageAndWait(StatementImpl.java:915)

                    at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:479)

                    at org.teiid.jdbc.StatementImpl.execute(StatementImpl.java:308)

                    at org.jboss.resource.adapter.jdbc.CheckValidConnectionSQL.isValidConnection(CheckValidConnectionSQL.java:58)

                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnectionFactory.isValidConnection(BaseWrapperManagedConnectionFactory.java:559)

                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.checkValid(BaseWrapperManagedConnection.java:321)

                    at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.matchManagedConnections(XAManagedConnectionFactory.java:492)

                    at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:219)

                    at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:716)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:404)

                    at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:381)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)

                    at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89)

                    at org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(JDBCExecutionFactory.java:245)

                    at org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(JDBCExecutionFactory.java:80)

                    at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:198)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:281)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:211)

                    at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:154)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.SelectNode.nextBatchDirect(SelectNode.java:96)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

                    at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

                    at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

                    at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

                    at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:214)

                    at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:168)

                    at org.teiid.query.processor.relational.SourceState.sort(SourceState.java:193)

                    at org.teiid.query.processor.relational.MergeJoinStrategy.loadLeft(MergeJoinStrategy.java:328)

                    at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:182)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:162)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:161)

                    at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:150)

                    at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:105)

                    at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:115)

                    at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:250)

                    at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:184)

                    at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)

                    at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:188)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:116)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:290)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Worker3_QueryProcessorQueue80" daemon prio=10 tid=0x000000005ecbf800 nid=0x7c5f runnable [0x0000000047bfd000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.SocketInputStream.socketRead0(Native Method)

                    at java.net.SocketInputStream.read(SocketInputStream.java:129)

                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

                    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)

                    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)

                    - locked <0x00002aab1610c7d0> (a java.io.BufferedInputStream)

                    at java.io.DataInputStream.read(DataInputStream.java:132)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.fillBuffer(ObjectDecoderInputStream.java:164)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.findLength(ObjectDecoderInputStream.java:147)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.readObjectOverride(ObjectDecoderInputStream.java:81)

                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:344)

                    at org.teiid.net.socket.OioOjbectChannelFactory$OioObjectChannel.read(OioOjbectChannelFactory.java:114)

                    - locked <0x00002aab1610c960> (a java.lang.Object)

                    at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:253)

                    at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                    at java.lang.reflect.Method.invoke(Method.java:597)

                    at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:107)

                    at $Proxy275.read(Unknown Source)

                    at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:347)

                    at org.teiid.jdbc.StatementImpl.sendRequestMessageAndWait(StatementImpl.java:915)

                    at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:479)

                    at org.teiid.jdbc.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:214)

                    at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:342)

                    at org.teiid.translator.jdbc.JDBCQueryExecution.execute(JDBCQueryExecution.java:84)

                    at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:263)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:281)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:211)

                    at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:154)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.SelectNode.nextBatchDirect(SelectNode.java:96)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

                    at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

                    at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

                    at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

                    at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:214)

                    at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:168)

                    at org.teiid.query.processor.relational.SourceState.sort(SourceState.java:193)

                    at org.teiid.query.processor.relational.MergeJoinStrategy.loadLeft(MergeJoinStrategy.java:328)

                    at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:182)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:162)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:161)

                    at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:150)

                    at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:105)

                    at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:115)

                    at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:250)

                    at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:184)

                    at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)

                    at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:188)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:116)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:290)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Worker2_QueryProcessorQueue235" daemon prio=10 tid=0x00002aabbe152000 nid=0x7c5e runnable [0x0000000047afb000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.SocketInputStream.socketRead0(Native Method)

                    at java.net.SocketInputStream.read(SocketInputStream.java:129)

                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

                    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)

                    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)

                    - locked <0x00002aab32c5bdf8> (a java.io.BufferedInputStream)

                    at java.io.DataInputStream.read(DataInputStream.java:132)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.fillBuffer(ObjectDecoderInputStream.java:164)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.findLength(ObjectDecoderInputStream.java:147)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.readObjectOverride(ObjectDecoderInputStream.java:81)

                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:344)

                    at org.teiid.net.socket.OioOjbectChannelFactory$OioObjectChannel.read(OioOjbectChannelFactory.java:114)

                    - locked <0x00002aab32c5bf88> (a java.lang.Object)

                    at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:253)

                    at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                    at java.lang.reflect.Method.invoke(Method.java:597)

                    at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:107)

                    at $Proxy275.read(Unknown Source)

                    at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:347)

                    at org.teiid.jdbc.StatementImpl.sendRequestMessageAndWait(StatementImpl.java:915)

                    at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:479)

                    at org.teiid.jdbc.StatementImpl.execute(StatementImpl.java:308)

                    at org.jboss.resource.adapter.jdbc.CheckValidConnectionSQL.isValidConnection(CheckValidConnectionSQL.java:58)

                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnectionFactory.isValidConnection(BaseWrapperManagedConnectionFactory.java:559)

                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.checkValid(BaseWrapperManagedConnection.java:321)

                    at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.matchManagedConnections(XAManagedConnectionFactory.java:492)

                    at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:219)

                    at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:716)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:404)

                    at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:381)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)

                    at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89)

                    at org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(JDBCExecutionFactory.java:245)

                    at org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(JDBCExecutionFactory.java:80)

                    at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:198)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:281)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:211)

                    at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:154)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.SelectNode.nextBatchDirect(SelectNode.java:96)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

                    at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

                    at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

                    at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

                    at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:214)

                    at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:168)

                    at org.teiid.query.processor.relational.SourceState.sort(SourceState.java:193)

                    at org.teiid.query.processor.relational.MergeJoinStrategy.loadLeft(MergeJoinStrategy.java:328)

                    at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:182)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:162)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:161)

                    at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:150)

                    at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:105)

                    at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:115)

                    at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:250)

                    at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:184)

                    at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)

                    at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:188)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:116)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:290)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Worker1_QueryProcessorQueue219" daemon prio=10 tid=0x00002aabbd999800 nid=0x7c5d runnable [0x00000000479fa000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.SocketInputStream.socketRead0(Native Method)

                    at java.net.SocketInputStream.read(SocketInputStream.java:129)

                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

                    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)

                    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)

                    - locked <0x00002aab32caae28> (a java.io.BufferedInputStream)

                    at java.io.DataInputStream.read(DataInputStream.java:132)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.fillBuffer(ObjectDecoderInputStream.java:164)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.findLength(ObjectDecoderInputStream.java:147)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.readObjectOverride(ObjectDecoderInputStream.java:81)

                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:344)

                    at org.teiid.net.socket.OioOjbectChannelFactory$OioObjectChannel.read(OioOjbectChannelFactory.java:114)

                    - locked <0x00002aab32caafb8> (a java.lang.Object)

                    at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:253)

                    at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                    at java.lang.reflect.Method.invoke(Method.java:597)

                    at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:107)

                    at $Proxy275.read(Unknown Source)

                    at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:347)

                    at org.teiid.jdbc.StatementImpl.sendRequestMessageAndWait(StatementImpl.java:915)

                    at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:479)

                    at org.teiid.jdbc.StatementImpl.execute(StatementImpl.java:308)

                    at org.jboss.resource.adapter.jdbc.CheckValidConnectionSQL.isValidConnection(CheckValidConnectionSQL.java:58)

                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnectionFactory.isValidConnection(BaseWrapperManagedConnectionFactory.java:559)

                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.checkValid(BaseWrapperManagedConnection.java:321)

                    at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.matchManagedConnections(XAManagedConnectionFactory.java:492)

                    at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:219)

                    at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:716)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:404)

                    at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:381)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)

                    at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)

                    at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89)

                    at org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(JDBCExecutionFactory.java:245)

                    at org.teiid.translator.jdbc.JDBCExecutionFactory.getConnection(JDBCExecutionFactory.java:80)

                    at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:198)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:281)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:211)

                    at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:154)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.SelectNode.nextBatchDirect(SelectNode.java:96)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

                    at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

                    at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

                    at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

                    at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:214)

                    at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:168)

                    at org.teiid.query.processor.relational.SourceState.sort(SourceState.java:193)

                    at org.teiid.query.processor.relational.MergeJoinStrategy.loadLeft(MergeJoinStrategy.java:328)

                    at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:182)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:162)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:161)

                    at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:150)

                    at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:105)

                    at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:115)

                    at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:250)

                    at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:184)

                    at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)

                    at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:188)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:116)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:290)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Worker0_QueryProcessorQueue81" daemon prio=10 tid=0x00002aaba41dd800 nid=0x7c5c runnable [0x00000000478fa000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.SocketInputStream.socketRead0(Native Method)

                    at java.net.SocketInputStream.read(SocketInputStream.java:129)

                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

                    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)

                    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)

                    - locked <0x00002aab05d15600> (a java.io.BufferedInputStream)

                    at java.io.DataInputStream.read(DataInputStream.java:132)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.fillBuffer(ObjectDecoderInputStream.java:164)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.findLength(ObjectDecoderInputStream.java:147)

                    at org.teiid.netty.handler.codec.serialization.ObjectDecoderInputStream.readObjectOverride(ObjectDecoderInputStream.java:81)

                    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:344)

                    at org.teiid.net.socket.OioOjbectChannelFactory$OioObjectChannel.read(OioOjbectChannelFactory.java:114)

                    - locked <0x00002aab05d15770> (a java.lang.Object)

                    at org.teiid.net.socket.SocketServerInstanceImpl.read(SocketServerInstanceImpl.java:253)

                    at sun.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                    at java.lang.reflect.Method.invoke(Method.java:597)

                    at org.teiid.net.socket.SocketServerConnectionFactory$ShutdownHandler.invoke(SocketServerConnectionFactory.java:107)

                    at $Proxy275.read(Unknown Source)

                    at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:347)

                    at org.teiid.jdbc.StatementImpl.sendRequestMessageAndWait(StatementImpl.java:915)

                    at org.teiid.jdbc.StatementImpl.executeSql(StatementImpl.java:479)

                    at org.teiid.jdbc.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:214)

                    at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:342)

                    at org.teiid.translator.jdbc.JDBCQueryExecution.execute(JDBCQueryExecution.java:84)

                    at org.teiid.dqp.internal.datamgr.ConnectorWorkItem.execute(ConnectorWorkItem.java:263)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.getResults(DataTierTupleSource.java:281)

                    at org.teiid.dqp.internal.process.DataTierTupleSource.nextTuple(DataTierTupleSource.java:211)

                    at org.teiid.query.processor.relational.AccessNode.nextBatchDirect(AccessNode.java:154)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.UnionAllNode.nextBatchDirect(UnionAllNode.java:90)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.SelectNode.nextBatchDirect(SelectNode.java:96)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.BatchIterator.finalRow(BatchIterator.java:70)

                    at org.teiid.common.buffer.AbstractTupleSource.getCurrentTuple(AbstractTupleSource.java:69)

                    at org.teiid.query.processor.BatchIterator.getCurrentTuple(BatchIterator.java:82)

                    at org.teiid.common.buffer.AbstractTupleSource.nextTuple(AbstractTupleSource.java:48)

                    at org.teiid.query.processor.relational.SortUtility.initialSort(SortUtility.java:214)

                    at org.teiid.query.processor.relational.SortUtility.sort(SortUtility.java:168)

                    at org.teiid.query.processor.relational.SourceState.sort(SourceState.java:193)

                    at org.teiid.query.processor.relational.MergeJoinStrategy.loadLeft(MergeJoinStrategy.java:328)

                    at org.teiid.query.processor.relational.JoinNode.nextBatchDirect(JoinNode.java:182)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.ProjectNode.nextBatchDirect(ProjectNode.java:162)

                    at org.teiid.query.processor.relational.RelationalNode.nextBatch(RelationalNode.java:274)

                    at org.teiid.query.processor.relational.RelationalPlan.nextBatch(RelationalPlan.java:161)

                    at org.teiid.query.processor.QueryProcessor.nextBatchDirect(QueryProcessor.java:150)

                    at org.teiid.query.processor.QueryProcessor.nextBatch(QueryProcessor.java:105)

                    at org.teiid.query.processor.BatchCollector.collectTuples(BatchCollector.java:115)

                    at org.teiid.dqp.internal.process.RequestWorkItem.processMore(RequestWorkItem.java:250)

                    at org.teiid.dqp.internal.process.RequestWorkItem.process(RequestWorkItem.java:184)

                    at org.teiid.dqp.internal.process.AbstractWorkItem.run(AbstractWorkItem.java:49)

                    at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:188)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$RunnableWrapper.run(ThreadReuseExecutor.java:116)

                    at org.teiid.dqp.internal.process.ThreadReuseExecutor$3.run(ThreadReuseExecutor.java:290)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "New I/O server worker #2-4" daemon prio=10 tid=0x00002aabbdc1a800 nid=0x7c5b runnable [0x00000000477fa000]

                   java.lang.Thread.State: RUNNABLE

                    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

                    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)

                    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

                    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

                    - locked <0x00002aab05ca0ae0> (a sun.nio.ch.Util$1)

                    - locked <0x00002aab05ca0ac8> (a java.util.Collections$UnmodifiableSet)

                    - locked <0x00002aab05c9bda0> (a sun.nio.ch.EPollSelectorImpl)

                    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

                    at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:38)

                    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:164)

                    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

                    at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "New I/O server worker #2-3" daemon prio=10 tid=0x00002aabbe764000 nid=0x7c5a runnable [0x00000000476f9000]

                   java.lang.Thread.State: RUNNABLE

                    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

                    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)

                    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

                    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

                    - locked <0x00002aab05cb68f8> (a sun.nio.ch.Util$1)

                    - locked <0x00002aab05cb68e0> (a java.util.Collections$UnmodifiableSet)

                    - locked <0x00002aab05caa770> (a sun.nio.ch.EPollSelectorImpl)

                    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

                    at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:38)

                    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:164)

                    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

                    at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "New I/O server worker #2-2" daemon prio=10 tid=0x00002aabbd696800 nid=0x7c59 runnable [0x00000000471f4000]

                   java.lang.Thread.State: RUNNABLE

                    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

                    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)

                    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

                    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

                    - locked <0x00002aab05cdaec0> (a sun.nio.ch.Util$1)

                    - locked <0x00002aab05cdaea8> (a java.util.Collections$UnmodifiableSet)

                    - locked <0x00002aab05cc6388> (a sun.nio.ch.EPollSelectorImpl)

                    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

                    at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:38)

                    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:164)

                    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

                    at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "New I/O server worker #2-1" daemon prio=10 tid=0x00002aabbe7f6800 nid=0x7c58 runnable [0x00000000473f6000]

                   java.lang.Thread.State: RUNNABLE

                    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

                    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)

                    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

                    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

                    - locked <0x00002aab05cbff80> (a sun.nio.ch.Util$1)

                    - locked <0x00002aab05cbff68> (a java.util.Collections$UnmodifiableSet)

                    - locked <0x00002aab05cb26f0> (a sun.nio.ch.EPollSelectorImpl)

                    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

                    at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:38)

                    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:164)

                    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

                    at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "ajp-15.154.146.168-8109-Acceptor-0" daemon prio=10 tid=0x000000006841b800 nid=0x7c4d runnable [0x00000000475f8000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.PlainSocketImpl.socketAccept(Native Method)

                    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                    - locked <0x00002aaaeb24d4c0> (a java.net.SocksSocketImpl)

                    at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                    at java.net.ServerSocket.accept(ServerSocket.java:421)

                    at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)

                    at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "http-15.154.146.168-8180-Acceptor-0" daemon prio=10 tid=0x0000000063543000 nid=0x7c4c runnable [0x00000000470f3000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.PlainSocketImpl.socketAccept(Native Method)

                    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                    - locked <0x00002aaaeb24e208> (a java.net.SocksSocketImpl)

                    at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                    at java.net.ServerSocket.accept(ServerSocket.java:421)

                    at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)

                    at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:310)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "JBossLifeThread" prio=10 tid=0x000000005b6b7000 nid=0x7c4b in Object.wait() [0x00000000472f5000]

                   java.lang.Thread.State: WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aab017a89c0> (a java.lang.Object)

                    at java.lang.Object.wait(Object.java:485)

                    at org.jboss.bootstrap.AbstractServerImpl$LifeThread.run(AbstractServerImpl.java:866)

                    - locked <0x00002aab017a89c0> (a java.lang.Object)

                 

                "Connection Consumer for dest JBossQueue[CallbackQueue] id=4" daemon prio=10 tid=0x00002aabb5342000 nid=0x7c2d in Object.wait() [0x00000000474f7000]

                   java.lang.Thread.State: WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaf3dc1888> (a java.lang.Object)

                    at java.lang.Object.wait(Object.java:485)

                    at org.jboss.jms.client.container.ClientConsumer.getMessage(ClientConsumer.java:866)

                    at org.jboss.jms.client.container.ClientConsumer.receive(ClientConsumer.java:506)

                    - locked <0x00002aaaf3dc1888> (a java.lang.Object)

                    at org.jboss.jms.client.container.ConsumerAspect.handleReceive(ConsumerAspect.java:194)

                    at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect_z_handleReceive_587063509.invoke(ConsumerAspect_z_handleReceive_587063509.java)

                    at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)

                    at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)

                    at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)

                    at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)

                    at org.jboss.jms.client.delegate.ClientConsumerDelegate.receive(ClientConsumerDelegate.java)

                    at org.jboss.jms.client.JBossConnectionConsumer.run(JBossConnectionConsumer.java:248)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "WorkManager(2)-5" daemon prio=10 tid=0x0000000070a03800 nid=0x7c28 waiting on condition [0x0000000046ff2000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaeab1fc88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "pool-8-thread-20" prio=10 tid=0x000000005ba33800 nid=0x7c27 waiting on condition [0x0000000046ef1000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaef4f0f48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "pool-8-thread-19" prio=10 tid=0x000000006ba73800 nid=0x7c26 waiting on condition [0x0000000046df0000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaef4f0f48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "pool-8-thread-18" prio=10 tid=0x000000006b0a7000 nid=0x7c25 waiting on condition [0x0000000046cef000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaef4f0f48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "pool-8-thread-17" prio=10 tid=0x000000005c33f000 nid=0x7c24 waiting on condition [0x0000000046bee000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaef4f0f48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "pool-8-thread-16" prio=10 tid=0x0000000063819000 nid=0x7c23 waiting on condition [0x0000000046aed000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaef4f0f48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "pool-8-thread-15" prio=10 tid=0x000000005ecc0800 nid=0x7c22 waiting on condition [0x00000000469ec000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaef4f0f48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "pool-8-thread-14" prio=10 tid=0x00000000698bf000 nid=0x7c21 waiting on condition [0x00000000468eb000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaef4f0f48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "pool-8-thread-13" prio=10 tid=0x000000005ec78000 nid=0x7c20 waiting on condition [0x00000000467ea000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaef4f0f48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Thread-31" prio=10 tid=0x00002aabb5e1c000 nid=0x7c1f waiting on condition [0x00000000466e9000]

                   java.lang.Thread.State: TIMED_WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaf3f78af8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2116)

                    at org.jboss.soa.esb.services.jbpm.integration.msg.RetryExecutor.run(RetryExecutor.java:166)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "H2 Log Writer JBPMDB" daemon prio=10 tid=0x00002aaba5d8d000 nid=0x7c1e waiting on condition [0x00000000465e8000]

                   java.lang.Thread.State: TIMED_WAITING (sleeping)

                    at java.lang.Thread.sleep(Native Method)

                    at org.h2.store.WriterThread.run(WriterThread.java:149)

                 

                "H2 File Lock Watchdog /opt/jboss-teiid-5.0.1/jboss-as/server/var/mwm/jboss/server/teiid-server/data/h2/jbpmDB.lock.db" daemon prio=10 tid=0x00002aaba435c800 nid=0x7c1d waiting on condition [0x00000000464e7000]

                   java.lang.Thread.State: TIMED_WAITING (sleeping)

                    at java.lang.Thread.sleep(Native Method)

                    at org.h2.store.FileLock$1.run(FileLock.java:200)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Connection Consumer for dest JBossQueue[JbpmJobQueue] id=3" daemon prio=10 tid=0x00002aabb78f9800 nid=0x7c1c in Object.wait() [0x00000000463e6000]

                   java.lang.Thread.State: WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaf4015e40> (a java.lang.Object)

                    at java.lang.Object.wait(Object.java:485)

                    at org.jboss.jms.client.container.ClientConsumer.getMessage(ClientConsumer.java:866)

                    at org.jboss.jms.client.container.ClientConsumer.receive(ClientConsumer.java:506)

                    - locked <0x00002aaaf4015e40> (a java.lang.Object)

                    at org.jboss.jms.client.container.ConsumerAspect.handleReceive(ConsumerAspect.java:194)

                    at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect_z_handleReceive_587063509.invoke(ConsumerAspect_z_handleReceive_587063509.java)

                    at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)

                    at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)

                    at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)

                    at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)

                    at org.jboss.jms.client.delegate.ClientConsumerDelegate.receive(ClientConsumerDelegate.java)

                    at org.jboss.jms.client.JBossConnectionConsumer.run(JBossConnectionConsumer.java:248)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Connection Consumer for dest JBossQueue[JbpmTimerQueue] id=2" daemon prio=10 tid=0x0000000069093000 nid=0x7c1a in Object.wait() [0x00000000461e4000]

                   java.lang.Thread.State: WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaf4067488> (a java.lang.Object)

                    at java.lang.Object.wait(Object.java:485)

                    at org.jboss.jms.client.container.ClientConsumer.getMessage(ClientConsumer.java:866)

                    at org.jboss.jms.client.container.ClientConsumer.receive(ClientConsumer.java:506)

                    - locked <0x00002aaaf4067488> (a java.lang.Object)

                    at org.jboss.jms.client.container.ConsumerAspect.handleReceive(ConsumerAspect.java:194)

                    at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect_z_handleReceive_587063509.invoke(ConsumerAspect_z_handleReceive_587063509.java)

                    at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)

                    at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)

                    at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)

                    at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)

                    at org.jboss.jms.client.delegate.ClientConsumerDelegate.receive(ClientConsumerDelegate.java)

                    at org.jboss.jms.client.JBossConnectionConsumer.run(JBossConnectionConsumer.java:248)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Connection Consumer for dest JBossQueue[JbpmCommandQueue] id=1" daemon prio=10 tid=0x00002aabbeab3000 nid=0x7c19 in Object.wait() [0x00000000460e3000]

                   java.lang.Thread.State: WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaf40f1210> (a java.lang.Object)

                    at java.lang.Object.wait(Object.java:485)

                    at org.jboss.jms.client.container.ClientConsumer.getMessage(ClientConsumer.java:866)

                    at org.jboss.jms.client.container.ClientConsumer.receive(ClientConsumer.java:506)

                    - locked <0x00002aaaf40f1210> (a java.lang.Object)

                    at org.jboss.jms.client.container.ConsumerAspect.handleReceive(ConsumerAspect.java:194)

                    at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect_z_handleReceive_587063509.invoke(ConsumerAspect_z_handleReceive_587063509.java)

                    at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)

                    at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)

                    at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)

                    at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)

                    at org.jboss.jms.client.delegate.ClientConsumerDelegate.receive(ClientConsumerDelegate.java)

                    at org.jboss.jms.client.JBossConnectionConsumer.run(JBossConnectionConsumer.java:248)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "pool-19-thread-1" daemon prio=10 tid=0x00002aabbdf0e800 nid=0x7c14 waiting on condition [0x0000000045cdf000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaf2add568> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "WorkManager(2)-4" daemon prio=10 tid=0x000000006a98d800 nid=0x7c13 waiting on condition [0x0000000045bde000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaeab1fc88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "WorkManager(2)-3" daemon prio=10 tid=0x000000006a7f4000 nid=0x7c12 waiting on condition [0x0000000045add000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaeab1fc88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "WorkManager(2)-2" daemon prio=10 tid=0x000000006abcb800 nid=0x7c11 waiting on condition [0x00000000459dc000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaeab1fc88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Timer-2" daemon prio=10 tid=0x00002aabbfed4000 nid=0x7c10 in Object.wait() [0x00000000458db000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaf2bfb918> (a java.util.TaskQueue)

                    at java.util.TimerThread.mainLoop(Timer.java:509)

                    - locked <0x00002aaaf2bfb918> (a java.util.TaskQueue)

                    at java.util.TimerThread.run(Timer.java:462)

                 

                "RMI Scheduler(0)" daemon prio=10 tid=0x000000005bda0000 nid=0x7c0f waiting on condition [0x00000000457da000]

                   java.lang.Thread.State: TIMED_WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaad58a24c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)

                    at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)

                    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)

                    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "RMI TCP Connection(17)-15.154.146.168" daemon prio=10 tid=0x00002aabbf789800 nid=0x7c0e runnable [0x00000000456d9000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.SocketInputStream.socketRead0(Native Method)

                    at java.net.SocketInputStream.read(SocketInputStream.java:129)

                    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

                    at java.io.BufferedInputStream.read(BufferedInputStream.java:237)

                    - locked <0x00002aab5bf8f260> (a java.io.BufferedInputStream)

                    at java.io.FilterInputStream.read(FilterInputStream.java:66)

                    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)

                    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)

                    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "RMI RenewClean-[15.154.146.168:1198]" daemon prio=10 tid=0x000000005bc3f000 nid=0x7c0d in Object.wait() [0x00000000455d8000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaf32482f0> (a java.lang.ref.ReferenceQueue$Lock)

                    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

                    - locked <0x00002aaaf32482f0> (a java.lang.ref.ReferenceQueue$Lock)

                    at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "JBoss System Threads(1)-3" daemon prio=10 tid=0x00000000709e3800 nid=0x7c0c waiting on condition [0x00000000454d7000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaad2bc09f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "ESBScheduler:jbossesb.esb" prio=10 tid=0x00002aabbf1f1800 nid=0x7c0b sleeping[0x00000000453d6000]

                   java.lang.Thread.State: TIMED_WAITING (sleeping)

                    at java.lang.Thread.sleep(Native Method)

                    at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:394)

                 

                "ESBScheduler:jbossesb.esb_Worker-0" prio=10 tid=0x00002aabc47bb000 nid=0x7c0a in Object.wait() [0x00000000452d5000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaf2b60a78> (a java.lang.Object)

                    at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)

                    - locked <0x00002aaaf2b60a78> (a java.lang.Object)

                    at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)

                    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)

                 

                "Thread-22" prio=10 tid=0x00002aabbf918800 nid=0x7c09 in Object.wait() [0x00000000451d4000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaf40e9250> (a java.lang.Object)

                    at org.jboss.jms.client.container.ClientConsumer.waitOnLock(ClientConsumer.java:833)

                    at org.jboss.jms.client.container.ClientConsumer.getMessage(ClientConsumer.java:880)

                    at org.jboss.jms.client.container.ClientConsumer.receive(ClientConsumer.java:530)

                    - locked <0x00002aaaf40e9250> (a java.lang.Object)

                    at org.jboss.jms.client.container.ConsumerAspect.handleReceive(ConsumerAspect.java:194)

                    at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect_z_handleReceive_587063509.invoke(ConsumerAspect_z_handleReceive_587063509.java)

                    at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)

                    at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)

                    at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)

                    at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)

                    at org.jboss.jms.client.delegate.ClientConsumerDelegate.receive(ClientConsumerDelegate.java)

                    at org.jboss.jms.client.JBossMessageConsumer.receive(JBossMessageConsumer.java:86)

                    at sun.reflect.GeneratedMethodAccessor430.invoke(Unknown Source)

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                    at java.lang.reflect.Method.invoke(Method.java:597)

                    at org.jboss.internal.soa.esb.rosetta.pooling.JmsSession$ExceptionHandler.invoke(JmsSession.java:479)

                    at $Proxy271.receive(Unknown Source)

                    at org.jboss.internal.soa.esb.couriers.JmsCourier.internalPickupPayload(JmsCourier.java:504)

                    - locked <0x00002aaaf1eb9ff0> (a org.jboss.internal.soa.esb.couriers.JmsCourier)

                    at org.jboss.internal.soa.esb.couriers.JmsCourier.pickupPayload(JmsCourier.java:465)

                    at org.jboss.internal.soa.esb.couriers.JmsCourier.pickup(JmsCourier.java:458)

                    at org.jboss.internal.soa.esb.couriers.TwoWayCourierImpl.pickup(TwoWayCourierImpl.java:228)

                    at org.jboss.internal.soa.esb.couriers.TwoWayCourierImpl.pickup(TwoWayCourierImpl.java:204)

                    at org.jboss.soa.esb.listeners.message.MessageAwareListener.waitForEventAndProcess(MessageAwareListener.java:297)

                    at org.jboss.soa.esb.listeners.message.MessageAwareListener.doRun(MessageAwareListener.java:253)

                    at org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle.run(AbstractThreadedManagedLifecycle.java:115)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "pool-8-thread-12" prio=10 tid=0x00002aabbeb8d800 nid=0x7c08 waiting on condition [0x00000000450d3000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaef4f0f48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "pool-8-thread-11" prio=10 tid=0x00002aabbedef000 nid=0x7c07 waiting on condition [0x0000000044fd2000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaef4f0f48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "pool-8-thread-10" prio=10 tid=0x00002aabbdebb800 nid=0x7c06 waiting on condition [0x0000000044ed1000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaef4f0f48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "pool-8-thread-9" prio=10 tid=0x00002aabbd556800 nid=0x7c05 waiting on condition [0x0000000044dd0000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaef4f0f48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "pool-8-thread-8" prio=10 tid=0x00002aabbd42e800 nid=0x7c04 waiting on condition [0x0000000044ccf000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaef4f0f48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "pool-8-thread-7" prio=10 tid=0x00002aabbd8b4000 nid=0x7c03 waiting on condition [0x0000000044bce000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaef4f0f48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "pool-8-thread-6" prio=10 tid=0x00002aabbdf24000 nid=0x7c02 waiting on condition [0x0000000044acd000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaef4f0f48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "pool-8-thread-5" prio=10 tid=0x00002aabc9836800 nid=0x7c01 waiting on condition [0x00000000447ca000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaef4f0f48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "pool-8-thread-4" prio=10 tid=0x00002aabbdc19800 nid=0x7c00 waiting on condition [0x00000000448cb000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaef4f0f48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "pool-8-thread-3" prio=10 tid=0x00002aabbe95a800 nid=0x7bff waiting on condition [0x00000000445c8000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaef4f0f48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "pool-8-thread-2" prio=10 tid=0x00002aabbd4a6800 nid=0x7bfe waiting on condition [0x00000000446c9000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaef4f0f48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "pool-8-thread-1" prio=10 tid=0x000000005cad9000 nid=0x7bfd waiting on condition [0x00000000449cc000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaef4f0f48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Thread-18" daemon prio=10 tid=0x00002aabc8c14800 nid=0x7bf5 waiting on condition [0x00000000443c6000]

                   java.lang.Thread.State: TIMED_WAITING (sleeping)

                    at java.lang.Thread.sleep(Native Method)

                    at org.jboss.messaging.core.impl.memory.SimpleMemoryManager$MemoryRunnable.run(SimpleMemoryManager.java:126)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "DefaultQuartzScheduler_QuartzSchedulerThread" prio=10 tid=0x00002aabc8620800 nid=0x7bf4 sleeping[0x00000000442c5000]

                   java.lang.Thread.State: TIMED_WAITING (sleeping)

                    at java.lang.Thread.sleep(Native Method)

                    at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:394)

                 

                "DefaultQuartzScheduler_Worker-9" prio=10 tid=0x00002aabbecea000 nid=0x7bf3 in Object.wait() [0x00000000441c4000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaedf16450> (a java.lang.Object)

                    at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)

                    - locked <0x00002aaaedf16450> (a java.lang.Object)

                    at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)

                    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)

                 

                "DefaultQuartzScheduler_Worker-8" prio=10 tid=0x00002aabbedcc800 nid=0x7bf2 in Object.wait() [0x00000000440c3000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaedf16450> (a java.lang.Object)

                    at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)

                    - locked <0x00002aaaedf16450> (a java.lang.Object)

                    at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)

                    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)

                 

                "DefaultQuartzScheduler_Worker-7" prio=10 tid=0x00002aabbd7bf000 nid=0x7bf1 in Object.wait() [0x0000000043fc2000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaedf16450> (a java.lang.Object)

                    at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)

                    - locked <0x00002aaaedf16450> (a java.lang.Object)

                    at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)

                    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)

                 

                "DefaultQuartzScheduler_Worker-6" prio=10 tid=0x00002aabbd67d000 nid=0x7bf0 in Object.wait() [0x0000000043ec1000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaedf16450> (a java.lang.Object)

                    at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)

                    - locked <0x00002aaaedf16450> (a java.lang.Object)

                    at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)

                    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)

                 

                "DefaultQuartzScheduler_Worker-5" prio=10 tid=0x00002aabbd67b800 nid=0x7bef in Object.wait() [0x0000000043dc0000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaedf16450> (a java.lang.Object)

                    at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)

                    - locked <0x00002aaaedf16450> (a java.lang.Object)

                    at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)

                    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)

                 

                "DefaultQuartzScheduler_Worker-4" prio=10 tid=0x00002aabbdf5b000 nid=0x7bee in Object.wait() [0x0000000043cbf000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaedf16450> (a java.lang.Object)

                    at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)

                    - locked <0x00002aaaedf16450> (a java.lang.Object)

                    at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)

                    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)

                 

                "DefaultQuartzScheduler_Worker-3" prio=10 tid=0x00002aabbdf67800 nid=0x7bed in Object.wait() [0x0000000043bbe000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaedf16450> (a java.lang.Object)

                    at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)

                    - locked <0x00002aaaedf16450> (a java.lang.Object)

                    at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)

                    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)

                 

                "DefaultQuartzScheduler_Worker-2" prio=10 tid=0x00002aabbdf66000 nid=0x7bec in Object.wait() [0x0000000043abd000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaedf16450> (a java.lang.Object)

                    at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)

                    - locked <0x00002aaaedf16450> (a java.lang.Object)

                    at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)

                    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)

                 

                "DefaultQuartzScheduler_Worker-1" prio=10 tid=0x00002aabbe042800 nid=0x7beb in Object.wait() [0x00000000439bc000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaedf16450> (a java.lang.Object)

                    at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)

                    - locked <0x00002aaaedf16450> (a java.lang.Object)

                    at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)

                    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)

                 

                "DefaultQuartzScheduler_Worker-0" prio=10 tid=0x00002aabc85b8800 nid=0x7bea in Object.wait() [0x00000000438bb000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaedf16450> (a java.lang.Object)

                    at org.quartz.simpl.SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)

                    - locked <0x00002aaaedf16450> (a java.lang.Object)

                    at org.quartz.simpl.SimpleThreadPool.access$000(SimpleThreadPool.java:47)

                    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)

                 

                "WorkManager(2)-1" daemon prio=10 tid=0x00002aabbdf25800 nid=0x7be9 waiting on condition [0x00000000437ba000]

                   java.lang.Thread.State: WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaee177588> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                    at java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:220)

                    at org.jboss.resource.adapter.mail.inflow.NewMsgsWorker.run(NewMsgsWorker.java:76)

                    at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:205)

                    at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "RMI TCP Accept-0" daemon prio=10 tid=0x00002aabc7f28800 nid=0x7be8 runnable [0x00000000436b9000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.PlainSocketImpl.socketAccept(Native Method)

                    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                    - locked <0x00002aaaee1e6e18> (a java.net.SocksSocketImpl)

                    at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                    at java.net.ServerSocket.accept(ServerSocket.java:421)

                    at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)

                    at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Timer-1" daemon prio=10 tid=0x00002aabbdfcd800 nid=0x7be7 in Object.wait() [0x00000000435b8000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaef4ccd30> (a java.util.TaskQueue)

                    at java.util.TimerThread.mainLoop(Timer.java:509)

                    - locked <0x00002aaaef4ccd30> (a java.util.TaskQueue)

                    at java.util.TimerThread.run(Timer.java:462)

                 

                "JCA PoolFiller" prio=10 tid=0x00002aabbf6b0800 nid=0x7be6 in Object.wait() [0x000000004094c000]

                   java.lang.Thread.State: WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaed07fad0> (a java.util.LinkedList)

                    at java.lang.Object.wait(Object.java:485)

                    at org.jboss.resource.connectionmanager.PoolFiller.run(PoolFiller.java:87)

                    - locked <0x00002aaaed07fad0> (a java.util.LinkedList)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "New I/O server boss #3 (channelId: 1059514998, /15.154.146.168:35532)" daemon prio=10 tid=0x00002aabbe053800 nid=0x7be5 runnable [0x000000004084b000]

                   java.lang.Thread.State: RUNNABLE

                    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

                    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)

                    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

                    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

                    - locked <0x00002aaaed4ecc48> (a sun.nio.ch.Util$1)

                    - locked <0x00002aaaed4ecc30> (a java.util.Collections$UnmodifiableSet)

                    - locked <0x00002aaaed4ec590> (a sun.nio.ch.EPollSelectorImpl)

                    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

                    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:241)

                    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

                    at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "New I/O server boss #2 (channelId: 394187016, /15.154.146.168:31543)" daemon prio=10 tid=0x000000005c3db000 nid=0x7be4 runnable [0x00000000434b7000]

                   java.lang.Thread.State: RUNNABLE

                    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

                    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)

                    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

                    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

                    - locked <0x00002aaaed4e3648> (a sun.nio.ch.Util$1)

                    - locked <0x00002aaaed4e3630> (a java.util.Collections$UnmodifiableSet)

                    - locked <0x00002aaaed4e2f90> (a sun.nio.ch.EPollSelectorImpl)

                    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

                    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:241)

                    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

                    at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "New I/O server boss #1 (channelId: 1573409133, /15.154.146.168:31100)" daemon prio=10 tid=0x0000000069783800 nid=0x7be3 runnable [0x00000000433b6000]

                   java.lang.Thread.State: RUNNABLE

                    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

                    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)

                    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

                    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

                    - locked <0x00002aaaed4de300> (a sun.nio.ch.Util$1)

                    - locked <0x00002aaaed4de2e8> (a java.util.Collections$UnmodifiableSet)

                    - locked <0x00002aaaed4ddc38> (a sun.nio.ch.EPollSelectorImpl)

                    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

                    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:241)

                    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

                    at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "AOPListner" daemon prio=10 tid=0x00002aabbecd5800 nid=0x7be2 waiting on condition [0x000000004074a000]

                   java.lang.Thread.State: TIMED_WAITING (sleeping)

                    at java.lang.Thread.sleep(Native Method)

                    at org.jboss.console.plugins.AOPLister$RefreshPoller.run(AOPLister.java:898)

                 

                "ContainerBackgroundProcessor[StandardEngine[jboss.web]]" daemon prio=10 tid=0x00002aabbcfc0000 nid=0x7be1 waiting on condition [0x00000000432b5000]

                   java.lang.Thread.State: TIMED_WAITING (sleeping)

                    at java.lang.Thread.sleep(Native Method)

                    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1590)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "RMI TCP Accept-4544" daemon prio=10 tid=0x00002aabbe7cf000 nid=0x7be0 runnable [0x00000000431b4000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.PlainSocketImpl.socketAccept(Native Method)

                    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                    - locked <0x00002aaaeb1896e0> (a java.net.SocksSocketImpl)

                    at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                    at java.net.ServerSocket.accept(ServerSocket.java:421)

                    at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)

                    at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "PooledInvokerAcceptor#0-4545" prio=10 tid=0x00002aabbe614000 nid=0x7bdf runnable [0x00000000430b3000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.PlainSocketImpl.socketAccept(Native Method)

                    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                    - locked <0x00002aaaebd2a3b8> (a java.net.SocksSocketImpl)

                    at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                    at java.net.ServerSocket.accept(ServerSocket.java:421)

                    at org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:262)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Thread-15" prio=10 tid=0x00002aabbe016000 nid=0x7bde in Object.wait() [0x0000000042fb2000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaebf7dc58> (a java.lang.Object)

                    at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doPeriodicWait(PeriodicRecovery.java:675)

                    at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:434)

                    - locked <0x00002aaaebf7dc58> (a java.lang.Object)

                 

                "Listener:4812" daemon prio=10 tid=0x00002aabbd393000 nid=0x7bdd runnable [0x0000000041553000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.PlainSocketImpl.socketAccept(Native Method)

                    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                    - locked <0x00002aaaeb25a640> (a java.net.SocksSocketImpl)

                    at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                    at java.net.ServerSocket.accept(ServerSocket.java:421)

                    at com.arjuna.ats.internal.arjuna.recovery.Listener.run(Listener.java:118)

                 

                "Thread-14" daemon prio=10 tid=0x0000000067ec9000 nid=0x7bdb waiting on condition [0x0000000041452000]

                   java.lang.Thread.State: TIMED_WAITING (sleeping)

                    at java.lang.Thread.sleep(Native Method)

                    at com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor.run(ExpiredEntryMonitor.java:237)

                 

                "Thread-13" daemon prio=10 tid=0x000000005cc6a000 nid=0x7bda in Object.wait() [0x0000000041249000]

                   java.lang.Thread.State: WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaeb25d920> (a java.util.LinkedList)

                    at java.lang.Object.wait(Object.java:485)

                    at com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellations(TransactionReaper.java:521)

                    - locked <0x00002aaaeb25d920> (a java.util.LinkedList)

                    at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:74)

                 

                "Thread-12" daemon prio=10 tid=0x000000005c99a000 nid=0x7bd9 in Object.wait() [0x0000000040649000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaeb25d068> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)

                    at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:104)

                    - locked <0x00002aaaeb25d068> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)

                 

                "Listener:4813" daemon prio=10 tid=0x000000006a31f800 nid=0x7bd8 runnable [0x0000000042eb1000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.PlainSocketImpl.socketAccept(Native Method)

                    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                    - locked <0x00002aaaeb262070> (a java.net.SocksSocketImpl)

                    at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                    at java.net.ServerSocket.accept(ServerSocket.java:421)

                    at com.arjuna.ats.internal.arjuna.recovery.Listener.run(Listener.java:118)

                 

                "SessionMonitor" daemon prio=10 tid=0x000000005eb04000 nid=0x7bd7 in Object.wait() [0x0000000042db0000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaeabbeb40> (a java.util.TaskQueue)

                    at java.util.TimerThread.mainLoop(Timer.java:509)

                    - locked <0x00002aaaeabbeb40> (a java.util.TaskQueue)

                    at java.util.TimerThread.run(Timer.java:462)

                 

                "AcceptorThread[ServerSocket[addr=/15.154.146.168,port=0,localport=4546]]" prio=10 tid=0x000000005eb1c000 nid=0x7bd6 runnable [0x0000000041148000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.PlainSocketImpl.socketAccept(Native Method)

                    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                    - locked <0x00002aaaeaadad28> (a java.net.SocksSocketImpl)

                    at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                    at java.net.ServerSocket.accept(ServerSocket.java:421)

                    at org.jboss.remoting.transport.socket.SocketServerInvoker.accept(SocketServerInvoker.java:1145)

                    at org.jboss.remoting.transport.socket.SocketServerInvoker.access$200(SocketServerInvoker.java:69)

                    at org.jboss.remoting.transport.socket.SocketServerInvoker$AcceptThread.run(SocketServerInvoker.java:1056)

                 

                "ServerSocketRefresh" daemon prio=10 tid=0x000000005b648000 nid=0x7bd5 in Object.wait() [0x0000000041047000]

                   java.lang.Thread.State: WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaeaadb640> (a java.lang.Object)

                    at java.lang.Object.wait(Object.java:485)

                    at org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:880)

                    - locked <0x00002aaaeaadb640> (a java.lang.Object)

                 

                "HDScanner" prio=10 tid=0x000000005cf42800 nid=0x7bd4 waiting on condition [0x0000000040f46000]

                   java.lang.Thread.State: TIMED_WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaaead788a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)

                    at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)

                    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)

                    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)

                    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "AcceptorThread[ServerSocket[addr=/15.154.146.168,port=0,localport=3973]]" prio=10 tid=0x000000006b8bb800 nid=0x7bd3 runnable [0x0000000042caf000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.PlainSocketImpl.socketAccept(Native Method)

                    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                    - locked <0x00002aaaead8cd38> (a java.net.SocksSocketImpl)

                    at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                    at java.net.ServerSocket.accept(ServerSocket.java:421)

                    at org.jboss.remoting.transport.socket.SocketServerInvoker.accept(SocketServerInvoker.java:1145)

                    at org.jboss.remoting.transport.socket.SocketServerInvoker.access$200(SocketServerInvoker.java:69)

                    at org.jboss.remoting.transport.socket.SocketServerInvoker$AcceptThread.run(SocketServerInvoker.java:1056)

                 

                "ServerSocketRefresh" daemon prio=10 tid=0x000000005d2b9000 nid=0x7bd2 in Object.wait() [0x0000000042bae000]

                   java.lang.Thread.State: WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaaead8d650> (a java.lang.Object)

                    at java.lang.Object.wait(Object.java:485)

                    at org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:880)

                    - locked <0x00002aaaead8d650> (a java.lang.Object)

                 

                "secondaryServerSocketThread[0]" daemon prio=10 tid=0x00002aabb5ae9000 nid=0x7bc6 runnable [0x0000000042aad000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.PlainSocketImpl.socketAccept(Native Method)

                    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                    - locked <0x00002aaae848ffc0> (a java.net.SocksSocketImpl)

                    at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                    at java.net.ServerSocket.accept(ServerSocket.java:421)

                    at org.jboss.remoting.transport.bisocket.BisocketServerInvoker$SecondaryServerSocketThread$1.run(BisocketServerInvoker.java:1120)

                    at java.security.AccessController.doPrivileged(Native Method)

                    at org.jboss.remoting.transport.bisocket.BisocketServerInvoker$SecondaryServerSocketThread.run(BisocketServerInvoker.java:1116)

                 

                "AcceptorThread[ServerSocket[addr=/15.154.146.168,port=0,localport=4557]]" prio=10 tid=0x00002aabc02e5800 nid=0x7bc4 runnable [0x00000000429ac000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.PlainSocketImpl.socketAccept(Native Method)

                    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                    - locked <0x00002aaae84944b8> (a java.net.SocksSocketImpl)

                    at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                    at java.net.ServerSocket.accept(ServerSocket.java:421)

                    at org.jboss.remoting.transport.socket.SocketServerInvoker.accept(SocketServerInvoker.java:1145)

                    at org.jboss.remoting.transport.socket.SocketServerInvoker.access$200(SocketServerInvoker.java:69)

                    at org.jboss.remoting.transport.socket.SocketServerInvoker$AcceptThread.run(SocketServerInvoker.java:1056)

                 

                "ServerSocketRefresh" daemon prio=10 tid=0x00002aabc0aef000 nid=0x7bc3 in Object.wait() [0x0000000040548000]

                   java.lang.Thread.State: WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaae8494e18> (a java.lang.Object)

                    at java.lang.Object.wait(Object.java:485)

                    at org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:880)

                    - locked <0x00002aaae8494e18> (a java.lang.Object)

                 

                "Thread-5" prio=10 tid=0x0000000063c54000 nid=0x7ba6 waiting on condition [0x0000000040e45000]

                   java.lang.Thread.State: TIMED_WAITING (parking)

                    at sun.misc.Unsafe.park(Native Method)

                    - parking to wait for  <0x00002aaae8497b00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

                    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2116)

                    at org.jboss.internal.soa.esb.couriers.transport.InVMTemporaryTransport$ReaperThread.run(InVMTemporaryTransport.java:902)

                 

                "HSQLDB Timer @7d083fe3" daemon prio=10 tid=0x0000000063a13000 nid=0x7ba5 in Object.wait() [0x0000000040209000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaae8498030> (a org.hsqldb.lib.HsqlTimer$TaskQueue)

                    at org.hsqldb.lib.HsqlTimer$TaskQueue.park(Unknown Source)

                    - locked <0x00002aaae8498030> (a org.hsqldb.lib.HsqlTimer$TaskQueue)

                    at org.hsqldb.lib.HsqlTimer.nextTask(Unknown Source)

                    - locked <0x00002aaae8498030> (a org.hsqldb.lib.HsqlTimer$TaskQueue)

                    at org.hsqldb.lib.HsqlTimer$TaskRunner.run(Unknown Source)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "JBoss System Threads(1)-2" daemon prio=10 tid=0x000000005c24d000 nid=0x7b7c runnable [0x00000000428ab000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.PlainSocketImpl.socketAccept(Native Method)

                    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                    - locked <0x00002aaad54a3e40> (a java.net.SocksSocketImpl)

                    at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                    at java.net.ServerSocket.accept(ServerSocket.java:421)

                    at org.jnp.server.Main$AcceptHandler.run(Main.java:597)

                    at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:147)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "GC Daemon" daemon prio=10 tid=0x000000005cbad000 nid=0x7b7b in Object.wait() [0x00000000427aa000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaad2c0a828> (a sun.misc.GC$LatencyLock)

                    at sun.misc.GC$Daemon.run(GC.java:100)

                    - locked <0x00002aaad2c0a828> (a sun.misc.GC$LatencyLock)

                 

                "RMI Reaper" prio=10 tid=0x000000005cbab800 nid=0x7b7a in Object.wait() [0x00000000426a9000]

                   java.lang.Thread.State: WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaad2c0a838> (a java.lang.ref.ReferenceQueue$Lock)

                    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

                    - locked <0x00002aaad2c0a838> (a java.lang.ref.ReferenceQueue$Lock)

                    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

                    at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "RMI TCP Accept-1198" daemon prio=10 tid=0x000000005c914800 nid=0x7b79 runnable [0x0000000041661000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.PlainSocketImpl.socketAccept(Native Method)

                    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                    - locked <0x00002aaad2c0a870> (a java.net.SocksSocketImpl)

                    at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                    at java.net.ServerSocket.accept(ServerSocket.java:421)

                    at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)

                    at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "JBoss System Threads(1)-1" daemon prio=10 tid=0x000000005c83b000 nid=0x7b78 runnable [0x0000000041dcd000]

                   java.lang.Thread.State: RUNNABLE

                    at java.net.PlainSocketImpl.socketAccept(Native Method)

                    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                    - locked <0x00002aaad2c0aa78> (a java.net.SocksSocketImpl)

                    at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                    at java.net.ServerSocket.accept(ServerSocket.java:421)

                    at org.jboss.web.WebServer.run(WebServer.java:320)

                    at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:147)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                    at java.lang.Thread.run(Thread.java:662)

                 

                "Timer-Log4jService" daemon prio=10 tid=0x000000005c9e3000 nid=0x7b77 in Object.wait() [0x0000000041ccc000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaad2c0acb0> (a java.util.TaskQueue)

                    at java.util.TimerThread.mainLoop(Timer.java:509)

                    - locked <0x00002aaad2c0acb0> (a java.util.TaskQueue)

                    at java.util.TimerThread.run(Timer.java:462)

                 

                "Timer-0" daemon prio=10 tid=0x000000005bb0b800 nid=0x7b6f in Object.wait() [0x00000000424a7000]

                   java.lang.Thread.State: TIMED_WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaad2c0ad90> (a java.util.TaskQueue)

                    at java.util.TimerThread.mainLoop(Timer.java:509)

                    - locked <0x00002aaad2c0ad90> (a java.util.TaskQueue)

                    at java.util.TimerThread.run(Timer.java:462)

                 

                "DestroyJavaVM" prio=10 tid=0x000000005b52f800 nid=0x7b5c waiting on condition [0x0000000000000000]

                   java.lang.Thread.State: RUNNABLE

                 

                "Low Memory Detector" daemon prio=10 tid=0x000000005b5eb800 nid=0x7b69 runnable [0x0000000000000000]

                   java.lang.Thread.State: RUNNABLE

                 

                "CompilerThread1" daemon prio=10 tid=0x000000005b5e9800 nid=0x7b68 waiting on condition [0x0000000000000000]

                   java.lang.Thread.State: RUNNABLE

                 

                "CompilerThread0" daemon prio=10 tid=0x000000005b5e4000 nid=0x7b67 waiting on condition [0x0000000000000000]

                   java.lang.Thread.State: RUNNABLE

                 

                "JDWP Event Helper Thread" daemon prio=10 tid=0x000000005b5ce800 nid=0x7b66 runnable [0x0000000000000000]

                   java.lang.Thread.State: RUNNABLE

                 

                "JDWP Transport Listener: dt_socket" daemon prio=10 tid=0x000000005b5ca800 nid=0x7b65 runnable [0x0000000000000000]

                   java.lang.Thread.State: RUNNABLE

                 

                "Signal Dispatcher" daemon prio=10 tid=0x000000005b5bf000 nid=0x7b64 waiting on condition [0x0000000000000000]

                   java.lang.Thread.State: RUNNABLE

                 

                "Finalizer" daemon prio=10 tid=0x000000005b59c000 nid=0x7b63 in Object.wait() [0x0000000041bcb000]

                   java.lang.Thread.State: WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaad542c4d8> (a java.lang.ref.ReferenceQueue$Lock)

                    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

                    - locked <0x00002aaad542c4d8> (a java.lang.ref.ReferenceQueue$Lock)

                    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

                    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

                 

                "Reference Handler" daemon prio=10 tid=0x000000005b59a000 nid=0x7b62 in Object.wait() [0x0000000041aca000]

                   java.lang.Thread.State: WAITING (on object monitor)

                    at java.lang.Object.wait(Native Method)

                    - waiting on <0x00002aaad2c0a818> (a java.lang.ref.Reference$Lock)

                    at java.lang.Object.wait(Object.java:485)

                    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

                    - locked <0x00002aaad2c0a818> (a java.lang.ref.Reference$Lock)

                 

                "VM Thread" prio=10 tid=0x000000005b595800 nid=0x7b61 runnable

                 

                "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000005b542800 nid=0x7b5d runnable

                 

                "GC task thread#1 (ParallelGC)" prio=10 tid=0x000000005b544800 nid=0x7b5e runnable

                 

                "GC task thread#2 (ParallelGC)" prio=10 tid=0x000000005b546800 nid=0x7b5f runnable

                 

                "GC task thread#3 (ParallelGC)" prio=10 tid=0x000000005b548000 nid=0x7b60 runnable

                 

                "VM Periodic Task Thread" prio=10 tid=0x000000005b5f6800 nid=0x7b6a waiting on condition

                 

                JNI global references: 15512

                 

                Heap

                PSYoungGen      total 751872K, used 448446K [0x00002aab5b4f0000, 0x00002aaba0240000, 0x00002aaba0240000)

                  eden space 376000K, 39% used [0x00002aab5b4f0000,0x00002aab6459ca60,0x00002aab72420000)

                  from space 375872K, 79% used [0x00002aab72420000,0x00002aab849630c8,0x00002aab89330000)

                  to   space 375872K, 0% used [0x00002aab89330000,0x00002aab89330000,0x00002aaba0240000)

                PSOldGen        total 2255552K, used 1726142K [0x00002aaad1a40000, 0x00002aab5b4f0000, 0x00002aab5b4f0000)

                  object space 2255552K, 76% used [0x00002aaad1a40000,0x00002aab3afef9c0,0x00002aab5b4f0000)

                PSPermGen       total 262144K, used 115567K [0x00002aaab1a40000, 0x00002aaac1a40000, 0x00002aaad1a40000)

                  object space 262144K, 44% used [0x00002aaab1a40000,0x00002aaab8b1bc28,0x00002aaac1a40000)

                • 5. Re: How to improve performance of teiid
                  rareddy

                  Allen,

                   

                  Looking through your log, looks like you using Teiid to call another Teiid instance or same Teiid instance as source, and that is currently running but not hung. Depending upon query you are running each may take variable amount time, mean while system seems to hung but it is not dead locked. I also see there are over 20 concurrent requests to the same.

                   

                  Again, without knowing more on what you are doing I can not help.  I will still refer to my orignal post on this email.

                   

                  Ramesh..

                  • 6. Re: How to improve performance of teiid
                    tawalisa

                    Hello Ramesh,

                     

                              You look right. Yes, Our VDB has 2 Datasource, one is oracle, another is VDB. This VDB is deployed by this jboss teiid server ,too. This VDB has 2 Datasouce, one is sqlserver, another is oracle. This oracle is different with the oracle of its parent VDB. This architecture looks so large.

                     

                              My test case use 30 threads concurrent requests VDB. Each thread just execute "select * from table" to VDB.  At this time, Teiid will hang as you looked log.

                              If I use 20 threads, Teiid will hang sometimes.

                              If I use 10 threads, Teiil will be ok usually.

                     

                              So it is strange why this hang is caused by our concurrent requests.

                     

                    Thanks

                    Best Regards

                    Allen

                    • 7. Re: How to improve performance of teiid
                      rareddy

                      Allen,

                       

                      Why not create single VDB with all the three sources? Also, we have a way to merge 2 VDBs into one, look at the "mergeVDB" Admin API call. However, word of caution is "mergeVDB" is not persistent, so when server restarts it forgets that the VDB has been merged. If you use merge call, you can get benefit of what you are doing with two VDBs but access them as one.

                       

                      Also, Teiild is not hung, it is waiting for threads to become free. if you are running multiple clients you need to size it accordingly.  By default, there are only 20 source connections in the connection pool created by the -ds.xml file, if you are running out of connections you need to increase max connections allowed.

                       

                      Currently due to a bug https://issues.jboss.org/browse/TEIID-1505

                       

                      Teiid only supports upto 20 concurrent source calls, so 21 st request will be queued, and wait until at least one the previously submitted job is finished. If you are running a long running queries, you could see this behavior immediately.

                       

                      Ramesh..

                      • 8. Re: How to improve performance of teiid
                        tawalisa

                        Dear Ramesh,

                         

                             Is this maxactiveplans for teiid server or for each VDB?

                             We are using 7.2 final vesion Teiid. Do you give us a patch to fix this bug? Or shall we wait next final version to fix it?

                         

                        Thanks

                        Best Regards

                        Li Jia

                        • 9. Re: How to improve performance of teiid
                          rareddy

                          No, this will be in 7.4 release. You can get alpha build of it now.

                          • 10. Re: How to improve performance of teiid
                            shawkins

                            Allen,

                             

                            The current 7.4 alpha does not have the fix.  We will have a 7.4 beta out by the beginning of next week containing the fix

                             

                            If you want to patch an older branch I've attached a specific patch to https://issues.jboss.org/browse/TEIID-1505 - don't use the fisheye change set as the issue was worked as part of a larger set of changes.

                             

                            Steve