1 2 Previous Next 27 Replies Latest reply on Apr 14, 2014 3:24 PM by kchen007

    Teiid performance Linux v.s. Windows

    kchen007

      I have deployed our connector to 2 jboss server, one runs on Linux machine, the other runs on Windows machine. Load the same vdb with the same data source.

       

      the client is a standalone java jdbc program that will just query data from the above vdb, the data queried is exactly the same. but the time taken between the 2 platform shows big difference.

      The client is single threaded, it will query data one by one.

       

      On Windows it took 22 seconds for the client to finish while on Linux it took 561s.  On both server, no cpu usage spike, 4g allocated to each vm and it only used 2g.

       

      I modified the teiid runtime jar to capture the time between a request reached to Teiid and the time it was sent out to client from Teiid, it reported 14s for Windows and 58s for Linux. Though the linux number is a little bit higher but there are still 500s missing ....

       

      Any idea what happened?

       

      thanks

      Kevin

       

       

        • 1. Re: Teiid performance Linux v.s. Windows
          rareddy

          Kevin,

           

          I suggest you run any java optimizer program and capture the method times to figure out. I suspect something to do with JVM. Do you consistently see the behavior on 2nd, 3rd, ...10th call.

           

          Ramesh..

          • 2. Re: Teiid performance Linux v.s. Windows
            van.halbert

            Couple of questions:

             

            -  do you perform any warmup, because vm initialization could cause some difference.   Try running 1 or 2 queries before starting your test.

            -  Can you provide the startup parameters for each server from the server log?

            -  Also, I assume both OS's are 64 bit, right?

             

            Thanks.

             

            Van

            • 3. Re: Teiid performance Linux v.s. Windows
              kchen007

              Van:

              to answer your question, yes I run the test a lot of times and they always give the comparable answer on both servers, it should not be issue of vm initializatin.

               

              both machine are 64 bits, linux is redhat 5 and windows is 7.

               

               

              on Linux, the startup parameter is:

              17:03:26,025 DEBUG [org.jboss.as.config] (MSC service thread 1-7) VM Arguments: -D[Standalone] -XX:+UseCompressedOops -Xms1024m -Xmx4096m -XX:MaxPermSize=512m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=standalone.xml -agentlib:jdwp=transport=dt_socket,address=8787,server=y,suspend=n

               

              on Windows, the startup parameter is:

              15:14:02,962 DEBUG [org.jboss.as.config] (MSC service thread 1-6) VM Arguments: -XX:+TieredCompilation -XX:+UseCompressedOops -Dprogram.name=standalone.bat -Xms1024M -Xmx4096M -XX:MaxPermSize=512M -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djboss.server.default.config=standalone.xml -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n

               

              thanks

              Kevin

              • 4. Re: Teiid performance Linux v.s. Windows
                kchen007

                Also, I forgot to mention, we are using Teiid 8.4.0 final.

                • 5. Re: Re: Teiid performance Linux v.s. Windows
                  kchen007

                  a little bit background on what we are doing, we started from top level entity and then retrieve its related entities(parents/childeren).

                  the number I give is for 500 top level entities. there is a mistake on windows number, the client reported 54s to finish, and teiid processing time is 34s.

                   

                  today I run with just 1 top level entities. and turned on the teiid TRANSPORT log for both windows and linux.

                   

                  then I exact the time when a request received by server to long [] recv, and when the request sent out from server long[] send.

                   

                  The ∑(send[i] -recv[i]) represents the total teiid process time, the ∑(recv[i+1] -send[i]) represents the network time+client process time.

                  the number for linux are 197ms/1960ms, while for windows they are 112ms/777ms.

                   

                  The number definitely shows that on linux it took more time to send/recv messages, 1960ms v.s  777ms.

                   

                  The jboss server use XNIO to handle socket request, is it possible the problem is there and how can we configure it to improve its performance?

                   

                  thanks

                  Kevin

                  • 6. Re: Re: Teiid performance Linux v.s. Windows
                    rareddy

                    Kevin,

                     

                    Turn on COMMAND logging on the server, it should give you timings for

                    * When user query submitted, and ended

                    * Source query submitted, start and end

                     

                    From this, you can make sure both in Linux and Windows you are executing the same number and same queries and their respective times. The log you got is little verbose.

                     

                    Ramesh..

                    • 7. Re: Re: Re: Teiid performance Linux v.s. Windows
                      kchen007

                      Ramesh:

                       

                      thanks, I produce the command log, it is for just 1 top level entity. see attached, let me know what do you find. BTW, what is the difference between user command and data source command?

                       

                      Kevin

                      • 8. Re: Re: Re: Teiid performance Linux v.s. Windows
                        rareddy

                        Kevin,

                         

                        These logs will get you accurate timings, will not reveal anything other than that.  I see that your one call to entity generating 36 different user queries Teiid, and Linux took 1 sec more than windows. But relatively it is twice.

                         

                        user command = is command submitted by user, in this case may be your JPA or Hibernate framework

                        data src command = what Teiid sending it to physical data source.

                         

                        Did you try profiling?

                         

                        Ramesh..

                        • 9. Re: Re: Re: Teiid performance Linux v.s. Windows
                          kchen007

                          the teiid process the request does not differ much on both linux and windows, if you read my previos post carefully we are thinking the problem is on the network layer, the sending/receiving took much more time on linux machine.   In our test case, the client is a separate jdbc program running on the same machine.

                           

                          On you server side, is your send() call blocking call? or it will just hand the message over to network layer and return.

                           

                          the profiling does not help, it did not show hotspot.

                           

                          thanks,

                          Kevin

                          • 10. Re: Re: Teiid performance Linux v.s. Windows
                            shawkins

                            > The jboss server use XNIO to handle socket request, is it possible the problem is there and how can we configure it to improve its performance?

                             

                            Teiid does not use XNIO.  We use Netty to handle our NIO.

                             

                            > On you server side, is your send() call blocking call? or it will just hand the message over to network layer and return.

                             

                            It is handed off to Netty.  The processing threads should typically not block unless they are doing source work.

                            • 11. Re: Re: Teiid performance Linux v.s. Windows
                              kchen007


                              Thanks Steven. Any advice on how to debug the problem?

                              • 12. Re: Re: Teiid performance Linux v.s. Windows
                                rareddy

                                I would think Netty would behave similarly in both environments unless it is some JVM optimization issue. If the both client and server both are same machine, or they going over loopback interface or ethernet address? may be Wireshark can show that traffic details?

                                 

                                Ramesh..

                                • 13. Re: Re: Re: Teiid performance Linux v.s. Windows
                                  kchen007

                                  Thanks Ramesh.

                                   

                                  We are using Sun JVM 1.6,  what JVM do you recommend? Is there any particular parameter need to be set?

                                   

                                  the client and server are running at the same machine, different process and use loopback interface, also use real IP and it behaves the same.

                                   

                                  We also test the other cases, client/server on different machines, in those cases, the performance is also not good as client/server all on the same Windows machine.

                                   

                                  the test is using 100 top level entities.

                                   

                                   




                                     
                                     
                                     
                                     
                                     
                                     
                                     
                                     
                                     
                                     
                                     
                                   
                                   
                                   
                                   
                                   
                                   
                                               
                                    Client


                                   


                                  Server



                                  Windows 1



                                  Windows 2



                                  Linux 1


                                   



                                  Linux 2


                                   



                                  Windows 1



                                  9s



                                  278s



                                  72s



                                  N/A



                                  Windows 2



                                  202s



                                  12s



                                  53s



                                  N/A



                                  Linux 1



                                  227s



                                  213s



                                  61s



                                  N/A



                                  Linux 2



                                  151s



                                  169s



                                  32s



                                  N/A


                                  • 14. Re: Re: Re: Teiid performance Linux v.s. Windows
                                    rareddy

                                    Kevin,

                                     

                                    I typically use Oracle VM or Open JDK. We have no recommendations at community level.

                                     

                                    If you believe that transport layer is culprit, there is one more test you could do to validate it. If you do a in-VM client query and measure the times. Typically you would write simple servlet and issue a query using Local Connection. If in this case Windows and Linux are nearly same then problem surely isolated to Netty related layer and/or JVM.

                                     

                                    Ramesh..

                                    1 2 Previous Next