[kchen007]

Ramesh:

I tried the servlet, it is about the same time on both windows and linux, around 10s for 100 top level entities, inlinie with above windows timing when using socket and much better than linux's data(>60s).

[rareddy]

Kevin,

This proves that it definitely something do with the network configuration in your Linux environment. Now switching gears, if your client accesses the DBMS directly do you still see the lag or the lag is only with Teiid in the mix?

Ramesh..

[kchen007]

Before we only profiling server side, today when I profile the client side, looks like the socket read take most of the time, see picture below:

[kchen007]

We got the bottom of it, if we disable the Nagle's alorithm, then the problem went away.

BTW, what is the best way to disable the Nagle's alorithm without changing the Teiid client code or change the machine wide setting.

thanks.

[shawkins]

> We got the bottom of it, if we disable the Nagle's alorithm, then the problem went away.

We'll look into that as well and see if local/remote socket connections in our scenarios have a performance degradation.

> BTW, what is the best way to disable the Nagle's alorithm without changing the Teiid client code or change the machine wide setting.

There is the system/Teiid client property org.teiid.sockets.conserveBandwidth, but it already defaults to false (which maps to Socket.setTCPNodelay(true)) which should disable Nagle's algorithm.

[kchen007]

That is strange, If I started my client program with -Dorg.teiid.sockets.conserveBandwidth=true, instead of false,  it will make the program performance better. maybe the flag got flipped again somewhere?

[rareddy]

I believe the issue is with how it was confusingly described in the javadoc, which led to a bug in Teiid interpretation

{code}

     * Enable/disable TCP_NODELAY (disable/enable Nagle's algorithm).

     *

     * @param on <code>true</code> to enable TCP_NODELAY,

     * <code>false</code> to disable.

{code}

true = enable tcp_delay = disable Nagle

Either we can fix doc, or can log a JIRA. I do remember that it was conscious decision not to disable Nagle's algorithm while back, but do remember the reasons behind it now.

Ramesh..

[shawkins]

> I believe the issue is with how it was confusingly described in the javadoc, which led to a bug in Teiid interpretation

I'm not sure I follow you.  Our property, org.teiid.sockets.conserveBandwidth is doc'ed as true=Nagle's algorithm.  Then when we setTCPNoDelay we flip the value, so conserveBandwidth=true which means TCP_NODELAY=false which means enable Nagle's algorithm.  Do you see something else?

[rareddy]

You are correct, I read wrong.

org.teiid.sockets.conserveBandwidth =  false is default setting,  so Teiid is setting TCP_DELAY = true, that disable Nagle's algorithm by default. Flipping it to true, should enable Nagle's algorithm.

Kevin: how did you disable Nagle's algorithm first time when you found out about it? I guess my confusion started there thinking Teiid by default enabled Nagle's algorithm.

Ramesh..

[kchen007]

I recompile Teiid client code after changing the boolean to true. then it started to perform well.

[kchen007]

When I suggested one of our user to set the flag, he told me from time to time, the client cannot connect to server with the following error msgs.

thanks.

on Server side:

09:22:24,257 ERROR [org.teiid.TRANSPORT] (New I/O worker #5) TEIID40017 Unexpected exception for session null: org.teiid.core.crypto.CryptoException:

TEIID10006 Decryption failed: javax.crypto.BadPaddingException   Given final block not properly padded

        at org.teiid.core.crypto.BasicCryptor.decrypt(BasicCryptor.java:94) [teiid-common-core-8.4.0.Final.jar:8.4.0.Final]

        at org.teiid.core.crypto.BasicCryptor.unsealObject(BasicCryptor.java:176) [teiid-common-core-8.4.0.Final.jar:8.4.0.Final]

        at org.teiid.transport.ServerWorkItem.run(ServerWorkItem.java:77) [teiid-runtime-8.4.1.jar:8.4.1]

        at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:269) [teiid-engine-8.4.1.jar:8.4.1]

        at org.teiid.transport.SocketClientInstance.processMessagePacket(SocketClientInstance.java:192) [teiid-runtime-8.4.1.jar:8.4.1]

        at org.teiid.transport.SocketClientInstance.receivedMessage(SocketClientInstance.java:178) [teiid-runtime-8.4.1.jar:8.4.1]

        at org.teiid.transport.SSLAwareChannelHandler.messageReceived(SSLAwareChannelHandler.java:201) [teiid-runtime-8.4.1.jar:8.4.1]

        at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.6.2.Final.jar:]

        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.6.2.Final.jar:]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45]

        at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]

and here is from the client side:

SEVERE: Could not create connection

org.teiid.jdbc.TeiidSQLException: TEIID20018 Unable to find a component used authenticate on to Teiid

        at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:135)

        at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:71)

        at org.teiid.jdbc.SocketProfile.connect(SocketProfile.java:56)

        at org.teiid.jdbc.TeiidDriver.connect(TeiidDriver.java:107)

        at org.teiid.jdbc.TeiidDriver.connect(TeiidDriver.java:55)

        at java.sql.DriverManager.getConnection(DriverManager.java:582)

        at java.sql.DriverManager.getConnection(DriverManager.java:185)

        at com.lgc.dsds.performance.TestPrepStatementClose.runTest(TestPrepStatementClose.java:29)

        at com.lgc.dsds.performance.TestPrepStatementClose.main(TestPrepStatementClose.java:208)

Caused by: org.teiid.net.CommunicationException: TEIID20018 Unable to find a component used authenticate on to Teiid

        at org.teiid.net.socket.SocketServerConnection.selectServerInstance(SocketServerConnection.java:144)

        at org.teiid.net.socket.SocketServerConnection.<init>(SocketServerConnection.java:92)

        at org.teiid.net.socket.SocketServerConnectionFactory.getConnection(SocketServerConnectionFactory.java:312)

        at org.teiid.jdbc.SocketProfile.connect(SocketProfile.java:54)

        ... 6 more

Caused by: org.teiid.core.TeiidComponentException: TEIID10006 Decryption failed: javax.crypto.BadPaddingException   Given final block not properly padded

        at org.teiid.client.util.ExceptionUtil.convertException(ExceptionUtil.java:59)

        at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler.invoke(SocketServerInstanceImpl.java:388)

        at $Proxy0.logon(Unknown Source)

        at org.teiid.net.socket.SocketServerConnection.logon(SocketServerConnection.java:173)

        at org.teiid.net.socket.SocketServerConnection.selectServerInstance(SocketServerConnection.java:125)

        ... 9 more

Caused by: org.teiid.core.crypto.CryptoException: TEIID10006 Decryption failed: javax.crypto.BadPaddingException   Given final block not properly padded

        at org.teiid.core.crypto.BasicCryptor.decrypt(BasicCryptor.java:94)

        at org.teiid.core.crypto.BasicCryptor.unsealObject(BasicCryptor.java:176)

        at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.convertResult(SocketServerInstanceImpl.java:340)

        at org.teiid.client.util.ResultsFuture.get(ResultsFuture.java:116)

        at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:371)

        at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler.invoke(SocketServerInstanceImpl.java:380)

        ... 12 more

Exception in thread "main" org.teiid.jdbc.TeiidSQLException: TEIID20018 Unable to find a component used authenticate on to Teiid

        at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:135)

        at org.teiid.jdbc.TeiidSQLException.create(TeiidSQLException.java:71)

        at org.teiid.jdbc.SocketProfile.connect(SocketProfile.java:56)

        at org.teiid.jdbc.TeiidDriver.connect(TeiidDriver.java:107)

        at org.teiid.jdbc.TeiidDriver.connect(TeiidDriver.java:55)

        at java.sql.DriverManager.getConnection(DriverManager.java:582)

        at java.sql.DriverManager.getConnection(DriverManager.java:185)

        at com.lgc.dsds.performance.TestPrepStatementClose.runTest(TestPrepStatementClose.java:29)

        at com.lgc.dsds.performance.TestPrepStatementClose.main(TestPrepStatementClose.java:208)

Caused by: org.teiid.net.CommunicationException: TEIID20018 Unable to find a component used authenticate on to Teiid

        at org.teiid.net.socket.SocketServerConnection.selectServerInstance(SocketServerConnection.java:144)

        at org.teiid.net.socket.SocketServerConnection.<init>(SocketServerConnection.java:92)

        at org.teiid.net.socket.SocketServerConnectionFactory.getConnection(SocketServerConnectionFactory.java:312)

        at org.teiid.jdbc.SocketProfile.connect(SocketProfile.java:54)

        ... 6 more

Caused by: org.teiid.core.TeiidComponentException: TEIID10006 Decryption failed: javax.crypto.BadPaddingException   Given final block not properly padded

        at org.teiid.client.util.ExceptionUtil.convertException(ExceptionUtil.java:59)

        at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler.invoke(SocketServerInstanceImpl.java:388)

        at $Proxy0.logon(Unknown Source)

        at org.teiid.net.socket.SocketServerConnection.logon(SocketServerConnection.java:173)

        at org.teiid.net.socket.SocketServerConnection.selectServerInstance(SocketServerConnection.java:125)

        ... 9 more

Caused by: org.teiid.core.crypto.CryptoException: TEIID10006 Decryption failed: javax.crypto.BadPaddingException   Given final block not properly padded

        at org.teiid.core.crypto.BasicCryptor.decrypt(BasicCryptor.java:94)

        at org.teiid.core.crypto.BasicCryptor.unsealObject(BasicCryptor.java:176)

        at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.convertResult(SocketServerInstanceImpl.java:340)

        at org.teiid.client.util.ResultsFuture.get(ResultsFuture.java:116)

        at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler$1.get(SocketServerInstanceImpl.java:371)

        at org.teiid.net.socket.SocketServerInstanceImpl$RemoteInvocationHandler.invoke(SocketServerInstanceImpl.java:380)

        ... 12 more

[shawkins]

I'm not entirely sure how the two would be related.  You wouldn't expect to get these bad padding exceptions except if the wrong key is being used to decrypt, which shouldn't be possible.  How reproducible is this, and what if any other changes are in effect?

[kchen007]

I cannot reproduce it myself. After examine the call stack again, It looks like there is mismatch between runtime jar and common-core jar. We built the runtime jar to pickup some fix for 8.4.0 and it requuires corresponding common-core.jar. I have given him the new common-core module and hopefully that will fix the problem.

It only happened twice for him too.

thanks

Kevin