I suggest you run any java optimizer program and capture the method times to figure out. I suspect something to do with JVM. Do you consistently see the behavior on 2nd, 3rd, ...10th call.
Couple of questions:
- do you perform any warmup, because vm initialization could cause some difference. Try running 1 or 2 queries before starting your test.
- Can you provide the startup parameters for each server from the server log?
- Also, I assume both OS's are 64 bit, right?
to answer your question, yes I run the test a lot of times and they always give the comparable answer on both servers, it should not be issue of vm initializatin.
both machine are 64 bits, linux is redhat 5 and windows is 7.
on Linux, the startup parameter is:
17:03:26,025 DEBUG [org.jboss.as.config] (MSC service thread 1-7) VM Arguments: -D[Standalone] -XX:+UseCompressedOops -Xms1024m -Xmx4096m -XX:MaxPermSize=512m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=standalone.xml -agentlib:jdwp=transport=dt_socket,address=8787,server=y,suspend=n
on Windows, the startup parameter is:
15:14:02,962 DEBUG [org.jboss.as.config] (MSC service thread 1-6) VM Arguments: -XX:+TieredCompilation -XX:+UseCompressedOops -Dprogram.name=standalone.bat -Xms1024M -Xmx4096M -XX:MaxPermSize=512M -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djboss.server.default.config=standalone.xml -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n
Also, I forgot to mention, we are using Teiid 8.4.0 final.
a little bit background on what we are doing, we started from top level entity and then retrieve its related entities(parents/childeren).
the number I give is for 500 top level entities. there is a mistake on windows number, the client reported 54s to finish, and teiid processing time is 34s.
today I run with just 1 top level entities. and turned on the teiid TRANSPORT log for both windows and linux.
then I exact the time when a request received by server to long  recv, and when the request sent out from server long send.
The ∑(send[i] -recv[i]) represents the total teiid process time, the ∑(recv[i+1] -send[i]) represents the network time+client process time.
the number for linux are 197ms/1960ms, while for windows they are 112ms/777ms.
The number definitely shows that on linux it took more time to send/recv messages, 1960ms v.s 777ms.
The jboss server use XNIO to handle socket request, is it possible the problem is there and how can we configure it to improve its performance?
Turn on COMMAND logging on the server, it should give you timings for
* When user query submitted, and ended
* Source query submitted, start and end
From this, you can make sure both in Linux and Windows you are executing the same number and same queries and their respective times. The log you got is little verbose.
These logs will get you accurate timings, will not reveal anything other than that. I see that your one call to entity generating 36 different user queries Teiid, and Linux took 1 sec more than windows. But relatively it is twice.
user command = is command submitted by user, in this case may be your JPA or Hibernate framework
data src command = what Teiid sending it to physical data source.
Did you try profiling?
the teiid process the request does not differ much on both linux and windows, if you read my previos post carefully we are thinking the problem is on the network layer, the sending/receiving took much more time on linux machine. In our test case, the client is a separate jdbc program running on the same machine.
On you server side, is your send() call blocking call? or it will just hand the message over to network layer and return.
the profiling does not help, it did not show hotspot.
> The jboss server use XNIO to handle socket request, is it possible the problem is there and how can we configure it to improve its performance?
Teiid does not use XNIO. We use Netty to handle our NIO.
> On you server side, is your send() call blocking call? or it will just hand the message over to network layer and return.
It is handed off to Netty. The processing threads should typically not block unless they are doing source work.
Thanks Steven. Any advice on how to debug the problem?
I would think Netty would behave similarly in both environments unless it is some JVM optimization issue. If the both client and server both are same machine, or they going over loopback interface or ethernet address? may be Wireshark can show that traffic details?
We are using Sun JVM 1.6, what JVM do you recommend? Is there any particular parameter need to be set?
the client and server are running at the same machine, different process and use loopback interface, also use real IP and it behaves the same.
We also test the other cases, client/server on different machines, in those cases, the performance is also not good as client/server all on the same Windows machine.
the test is using 100 top level entities.
I typically use Oracle VM or Open JDK. We have no recommendations at community level.
If you believe that transport layer is culprit, there is one more test you could do to validate it. If you do a in-VM client query and measure the times. Typically you would write simple servlet and issue a query using Local Connection. If in this case Windows and Linux are nearly same then problem surely isolated to Netty related layer and/or JVM.