Errors running simultaneous queries

[snadji]

Hi,

We see the exception listed below when triggering lots of queries (50+) at the same time. Most of the queries succeed. Just a few throw this exception.

Any ideas why these errors are thrown? I tried this with 8.2 and 8.3-beta2.

Thanks

Feb 12, 2013 9:41:16 AM org.teiid.jdbc.TeiidDriver connect

SEVERE: Could not create connection

org.teiid.jdbc.TeiidSQLException: TEIID20018 Unable to find a component used authenticate on to Teiid

        at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:135)

        at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:71)

        at org.teiid.jdbc.SocketProfile.connect(SocketProfile.java:56)

        at org.teiid.jdbc.TeiidDriver.connect(TeiidDriver.java:107)

        at org.teiid.jdbc.TeiidDriver.connect(TeiidDriver.java:55)

        at java.sql.DriverManager.getConnection(DriverManager.java:582)

        at java.sql.DriverManager.getConnection(DriverManager.java:185)

        at com.ca.dvs.testutils.ScalabilityQueryThread.run(ScalabilityQueryThread.java:42)

        at java.lang.Thread.run(Thread.java:662)

Caused by: org.teiid.net.CommunicationException: TEIID20018 Unable to find a component used authenticate on to Teiid

        at org.teiid.net.socket.SocketServerConnection.selectServerInstance(SocketServerConnection.java:144)

        at org.teiid.net.socket.SocketServerConnection.<init>(SocketServerConnection.java:92)

        at org.teiid.net.socket.SocketServerConnectionFactory.getConnection(SocketServerConnectionFactory.java:312)

        at org.teiid.jdbc.SocketProfile.connect(SocketProfile.java:54)

        ... 6 more

Caused by: org.teiid.core.TeiidComponentException: TEIID10006 Decryption failed: javax.crypto.BadPaddingException   Given final block not pr

operly padded

        at org.teiid.client.util.ExceptionUtil.convertException(ExceptionUtil.java:59)

        at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler.invoke(SocketServerInstanceImpl.java:374)

        at $Proxy0.logon(Unknown Source)

        at org.teiid.net.socket.SocketServerConnection.logon(SocketServerConnection.java:173)

        at org.teiid.net.socket.SocketServerConnection.selectServerInstance(SocketServerConnection.java:125)

        ... 9 more

Caused by: org.teiid.core.crypto.CryptoException: TEIID10006 Decryption failed: javax.crypto.BadPaddingException   Given final block not pro

perly padded

        at org.teiid.core.crypto.BasicCryptor.unsealObject(BasicCryptor.java:147)

        at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.convertResult(SocketServerInstanceImpl.java:326)

        at org.teiid.client.util.ResultsFuture.get(ResultsFuture.java:116)

        at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:357)

        at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler.invoke(SocketServerInstanceImpl.java:366)

        ... 12 more

[rareddy]

hmm, any errors in the server log?

[snadji]

Ramesh,

The following error is in server.log. Would enabling TRACE level and re-running the tests help?

09:17:18,402 ERROR [org.teiid.TRANSPORT] (New I/O server worker #2-2) TEIID40017 Unexpected exception for session null: org.teiid.core.crypto.CryptoException: TEIID10006 Decryption failed: javax.crypto.BadPaddingException   Given final block not properly padded

          at org.teiid.core.crypto.BasicCryptor.unsealObject(BasicCryptor.java:167) [teiid-common-core-8.3.0.Beta2.jar:8.3.0.Beta2]

          at org.teiid.transport.ServerWorkItem.run(ServerWorkItem.java:77) [teiid-runtime-8.3.0.Beta2.jar:8.3.0.Beta2]

          at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:253) [teiid-engine-8.3.0.Beta2.jar:8.3.0.Beta2]

          at org.teiid.transport.SocketClientInstance.processMessagePacket(SocketClientInstance.java:166) [teiid-runtime-8.3.0.Beta2.jar:8.3.0.Beta2]

          at org.teiid.transport.SocketClientInstance.receivedMessage(SocketClientInstance.java:152) [teiid-runtime-8.3.0.Beta2.jar:8.3.0.Beta2]

          at org.teiid.transport.SSLAwareChannelHandler.messageReceived(SSLAwareChannelHandler.java:210) [teiid-runtime-8.3.0.Beta2.jar:8.3.0.Beta2]

          at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100) [netty-3.2.6.Final.jar:]

          at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.2.6.Final.jar:]

          at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783) [netty-3.2.6.Final.jar:]

          at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:149) [netty-3.2.6.Final.jar:]

          at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.2.6.Final.jar:]

          at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783) [netty-3.2.6.Final.jar:]

          at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302) [netty-3.2.6.Final.jar:]

          at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317) [netty-3.2.6.Final.jar:]

          at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299) [netty-3.2.6.Final.jar:]

          at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:214) [netty-3.2.6.Final.jar:]

          at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80) [netty-3.2.6.Final.jar:]

          at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.2.6.Final.jar:]

          at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.2.6.Final.jar:]

          at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) [netty-3.2.6.Final.jar:]

          at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) [netty-3.2.6.Final.jar:]

          at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351) [netty-3.2.6.Final.jar:]

          at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282) [netty-3.2.6.Final.jar:]

          at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202) [netty-3.2.6.Final.jar:]

          at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.2.6.Final.jar:]

          at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44) [netty-3.2.6.Final.jar:]

          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [rt.jar:1.7.0_10]

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [rt.jar:1.7.0_10]

          at java.lang.Thread.run(Unknown Source) [rt.jar:1.7.0_10]

[rareddy]

Can you check to see you are using the correct client jdbc jar? Teiid driver does support backwards compatible versions, just in case.

[snadji]

I was using teiid-client-8.2.0.Final.jar. I will retest with the 8.3 version and let you know. But as I mentioned earlier, we saw the same error going against 8.2 as well.

[snadji]

Same outcome using teiid-client-8.3.0.Beta2.jar

[rareddy]

Samier,

We have not seen this kind error before. I believe this may be something to do with network itself. Not really sure what is going on here. What is your test case like? you using JMeter to pump simultaneous queries? May be run in debug and see of there any other error occurring.

Ramesh..

[shawkins]

It looks like this is the exception that can happen if a mismatched symmetric encryption pair is used.  However I don't see how we can cause this to happen as the relevant server/client code explicitly holds a single connection instance that will only negatiate a single key.

[snadji]

We are using a tool that we wrote that launches a large number for threads that trigger slow-running queries on Teiid (these queries make teiid get data from multiple data sources then ORDER BY several columns)

We haven't tried jmeter to run these queries. Could our tool be doing something that is not thread-safe that messes up teiid connections? I will test with jmeter and see if problem happens there.

Thanks

[rareddy]

As long as you are not using the same teiid connection in multiple threads simultaneously you should be fine which tool you use. If there is cross over then you will see issues.

[shawkins]

Using the same connection across multiple threads would not affect something as low level as this.  However warning still applies as higher level Connection/ResultSet etc. constructs are not entirely thread safe.

Steve

[snadji]

Each thread in our tools has it's own Teiid connection.

I was not able to reproduce this error using jmeter. Actually, for a while, I didn't see this error using our tool either. I'll keep looking and will post any new findings.

Thanks again.

[kchen007]

we saw similar issue with our JUnit test program. It happened randomly, hard to reproduce consistently.

[shawkins]

So to understand where this is at, Samier saw this behavior initially with an 8.2 client and then with an 8.3 (probably against an 8.2 server), but no longer sees this exception. 

kchen what version are you on?

If you are on an 8.3 server/client the above code should no longer be hit as the sealed object construct had issues relying on the context classloader.

Steve

[kchen007]

At the time we are using Teiid 8.3 client against Teidd 8.4 server, then we upgrade client to Teiid 8.4 too. and the problem still there.

Right now, we are just creating one connection as workaround. If Teiid can fix this problem, that will be great.

thanks

Kevin