11 Replies Latest reply on Jun 5, 2013 10:08 AM by Stephen Coy

    I/O performance down in Jboss AS 7 for Enterprise application

    kishore routhu Newbie

      I am working with EnterpriseApplication on Jboss AS 7 platform.

      Functionality of application it that it takes XML data as request

      and process it and sends XML data as response to client. Here

      XML request (in kilobyts) is relatively less in size when compared to

      XML response ( ranges fro KB to MB). I enabled some loggings of

      request processing from this i conclude that 90% of request processing

      (i.e 1.parsingXML + 2.executeBusinessLogic + 3.constructsXMLResponse +

      4. SendXML response to client)

      is to write back the XML response to client(i.e step 4 from the above).

      And observed from live that Ram usage is always 50% - 70% and

      CPU load is 0.16 - 1.0

       

       

      But still i couldn't found that only write response back to client

      takes 90% of response time.

       

       

      following is Web configuration in Standalone.xml of Jboss As 7 and target client

      has same configuration which receives the response.

      <subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host">

          <configuration>

                <static-resources sendfile="524288"/>

          </configuration>

          <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http" enable-lookups="true" redirect-port="8443" executor="http-executor" max-connections="50"/>

          <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" redirect-port="8443" secure="true" max-connections="400">

                <ssl name="ssl" key-alias="chapter8" password="123456" certificate-key-file="../standalone/configuration/cultswitch.cultuzz.de.keystore" cipher-suite="SSL_RSA_WITH_RC4_128_MD5,SSL_RSA_WITH_RC4_128_SHA" protocol="TLSv1" verify-client="false" verify-depth="10" keystore-type="PKCS12" truststore-type="PKCS12"/>

          </connector>

          <virtual-server name="default-host" enable-welcome-root="true">

                <alias name="localhost"/>

                <alias name="vela"/>

          </virtual-server>

      </subsystem>

       

       

      Can anybody help with the way to find the reasons for this ?

       

       

      For more information :

      1. Using non-transactional datasource with following pool configuration

         <pool>

           <min-pool-size>20</min-pool-size>

           <max-pool-size>50</max-pool-size>

           <prefill>false</prefill>

           <use-strict-min>false</use-strict-min>

           <flush-strategy>FailingConnectionOnly</flush-strategy>

         </pool>

      2. Local EJB's with CONTAINER managed transactions.

      3. Configured for recovery-environment for transactions recovery for every 2 minutes.

      4. Threads subsystem configuration as follows

         <subsystem xmlns="urn:jboss:domain:threads:1.1">

              <bounded-queue-thread-pool name="http-executor">

                 <core-threads count="50"/>

                 <queue-length count="50"/>

                 <max-threads count="50"/>

                 <keepalive-time time="10" unit="seconds"/>

                 </bounded-queue-thread-pool>

         </subsystem>

       

       

      Hardware Specification : 8GB RAM

       

       

      Processor Information (2 processors):

       

       

      vendor_id          : GenuineIntel

      cpu family          : 6

      model                    : 44

      model name          : Intel(R) Xeon(R) CPU           E5645  @ 2.40GHz

      stepping          : 2

      cpu MHz                    : 2400.085

      cache size          : 12288 KB

      physical id          : 0

      siblings          : 2

      core id                    : 1

      cpu cores          : 2

      apicid                    : 1

      initial apicid          : 1

      fpu                    : yes

      fpu_exception          : yes

      cpuid level          : 11

      wp                    : yes

      flags                    : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perf

      mon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm arat epb dts

      bogomips          : 4800.17

      clflush size          : 64

      cache_alignment          : 64

      address sizes          : 40 bits physical, 48 bits virtual

       

       

       

       

      Softwares Specification:

      ApplicationType : EnterpriseApplication

      Java : 7.1

      Application Server : Jboss AS 7

      DataBase : MySQL

      EJB : 3.0

      Hibenate : 4.0.1

       

       

      Do i need any changes in Standalone configuration ?

      Do i need any changes in I/O of OpertingSystem ?

        • 1. Re: I/O performance down in Jboss AS 7 for Enterprise application
          Stephen Coy Master

          If you have potentially large responses then your process can be improved considerably by changing steps 3 & 4 of your logic.

           

          You can use the javax.xml.stream API to build your response and write it directly to the output stream as you go. This saves a lot of memory and the GC side effects from generating the entire XML structure in a growing buffer.

           

          It's possible to use this in conjunction with other DOM to XML strategies such as JAXB to simplify your XML generation. In the past I have used JAXB to generate small XML sub-elements and then stream them out using XML streaming. This trick makes your fat XML generation scalable.

           

          What are you comparing JBossAS 7.1 with when you say that your I/O performance is down?

          1 of 1 people found this helpful
          • 2. Re: I/O performance down in Jboss AS 7 for Enterprise application
            kishore routhu Newbie

            Thank you Stephen Coy for your reply

             

            Actually i build the XML response as a string and writing bytes from the string to outputStream.

            Is it cuase to slow down by writing bytes to outputstream ?

            Example :

            This is only end of processing request before it includes lots of business calcultions over xml data

             

                           1. byte[] rsBytes = resXML.getBytes();

                            2.response.setContentType("text/xml");

                            3.int contentLength = otaRSBytes.length;

                            4.response.setContentLength(contentLength);

                            5.response.setCharacterEncoding("UTF-8");

                            6.out = response.getOutputStream();

                            7.out.write(rsBytes);  

                            8.out.flush(); 

                            9.out.close();

            90% response time takes from line 2 to line 7 .

             

            What are you comparing JBossAS 7.1 with when you say that your I/O performance is down?

             

            I am expecting that because of any bad configuration on jboss standalone mode some backgroud

            processes are always accessing to I/O that may cause to slow down the actuall response write to outputstream.

             

            Can you please suggest me in this way ?

            • 3. Re: I/O performance down in Jboss AS 7 for Enterprise application
              Stephen Coy Master

              What is reading the response?

               

              If you're using a web browser you may be slowed down by the ability of the browser to read and render the response.

               

              Try using curl or wget on a unix platform in order to avoid this possibility.

              • 4. Re: I/O performance down in Jboss AS 7 for Enterprise application
                kishore routhu Newbie

                Thank you for your replay

                 

                Here i am not using the browser to render the response. But the response is send back to Client Program from other system

                in same network ?

                Is there any changes required from Client end to receive response fastly ?

                • 5. Re: I/O performance down in Jboss AS 7 for Enterprise application
                  Stephen Coy Master

                  Are you able to use curl or wget from a unix system to see what effect this has on the server?

                   

                  What is the nature of the client?

                   

                  You may need to profile both the client and server to discover where the bottlenect lies.

                  • 6. Re: I/O performance down in Jboss AS 7 for Enterprise application
                    kishore routhu Newbie

                    Thank Stephen for your replay

                     

                    i can able to use curl to findout bottleneck .

                    But i don't know how to work with curl to findout where the bottlenecks are

                    can you provide me that inforamtion ?

                     

                    What is the nature of the client?

                    The nature of client is also same as Server means that it has same configuration as Server with Jboss as 7

                    and Hardware.

                     

                    Can provide me how to monitor client and server to discover I/O bottlenecks ?

                    • 7. Re: I/O performance down in Jboss AS 7 for Enterprise application
                      Stephen Coy Master

                      Using curl is very easy.

                      {code:bash}[steve@steves-mbpw ~]$ curl http://www.jboss.org/overview/{code}

                       

                      You can find more information about curl using:

                      {code:bash}[steve@steves-mbpw ~]$ man curl{code}

                       

                      As curl is a native application, it is unlikely to become a read-side bottleneck. It will read the output as fast as it can be delivered.

                       

                      If the output from curl is slow, then the problem is likely to be server side. If it's fast, then it will be your java client that is slowing things down.

                      • 8. Re: I/O performance down in Jboss AS 7 for Enterprise application
                        kishore routhu Newbie

                        Thank u Stephen Coy for your continues replies on my posts

                         

                        All your previous answers were helpful for me.

                         

                        But here my concern is how do i improve the  performance of send data to client from Servlet.

                        Here the response data encapsulated in String and size varies from 200KB to 3MB.

                         

                        Following is current code snippet to send response to client

                         

                                       1. byte[] rsBytes = resXML.getBytes();

                                       2.response.setContentType("text/xml");

                                       3.int contentLength = otaRSBytes.length;

                                       4.response.setContentLength(contentLength);

                                       5.response.setCharacterEncoding("UTF-8");

                                       6.out = response.getOutputStream();

                                       7.out.write(rsBytes);  

                                       8.out.flush(); 

                                       9.out.close();

                         

                        What are the varies ways to send response from Server (through Servlet) to Client and Which is efficient among

                        them ?

                        How to tune Client and Server  at any level (i.e 1.ApplicationServerConfiguration, 2. OS level , 3.Java) 

                        in order to acheive over network latency of sending response between Server and Client. ?

                         

                        ***** And i already  provided all ApplicatoinServler configurations for Client and Server.

                         

                        Please suggest me for the above.

                         

                        Thanks in advance

                        • 9. Re: I/O performance down in Jboss AS 7 for Enterprise application
                          Stephen Coy Master

                          What happens if you do this:

                          {code:java}response.setContentType("text/xml");

                          response.setCharacterEncoding("UTF-8");

                          response.getWriter().print(resXML);

                          {code}

                           

                          What happened when you tested with curl?

                          1 of 1 people found this helpful
                          • 10. Re: I/O performance down in Jboss AS 7 for Enterprise application
                            kishore routhu Newbie

                            Now my code is working with ServletOutputStream

                             

                                           response.setContentType("text/xml");

                                           int contentLength = otaRSBytes.length;

                                           response.setContentLength(contentLength);

                                           response.setCharacterEncoding("UTF-8");

                                           out = response.getOutputStream();

                                           out.write(rsBytes); 

                                           out.flush();

                             

                            but didn't tried with response.getWriter().print(resXML);

                             

                            whether PrintWriter on String is efficient than the OutputStream ?

                             

                             

                            What happened when you tested with curl?

                            Here i observed that the problem is from server side .

                             

                            • 11. Re: I/O performance down in Jboss AS 7 for Enterprise application
                              Stephen Coy Master

                              Well, if you have 50MB of XML in a java.lang.String, this

                              {code:java}byte[] rsBytes = resXML.getBytes(){code}

                              creates a copy of the character data. Now you have consumed 100MB of memory,

                               

                              and then:

                              {code:java}out.write(rsBytes);{code}

                              buffers the bytes again, consuming a further 50MB of memory.

                               

                              You can mitigate the last one by adding:

                              {code:java}...

                              response.setCharacterEncoding("UTF-8");

                              response.setBufferSize(1024 * 1024); // 1MB

                              ...{code}

                               

                              Furthermore, resXLM.getBytes() returns the character bytes in the platform encoding. If that's not UTF-8 then all sorts of things could be going on.